Avoid SharePoint Throttling
The SharePoint connector retrieves content and metadata from an on-premises SharePoint repository.
You have a few options to avoid throttling:
Decrease the number of threads
If you see many 429/503 errors, you are probably hitting SharePoint Online with too many concurrent fetchers.
-
Set Crawl Performance > Fetch Threads to a lower value.
-
Set Crawl Performance > Prefetch Threads to a lower value.
Stagger the datasource jobs
If you have multiple SharePoint Online datasource jobs that run at the same time, use the job scheduler to stagger their schedules instead.
Increase the number of retries
By default, the connector is configured with retries. This provides a chance for the requests that were rate-limited to run again.
You can increase the number of retries and the interval between retries. The process is called exponential backoff, which gradually increases the delays between retries to increase the chances of a successful retry. This helps prevent missing documents due to rate limiting.
Retry configuration parameters include:
-
Retry attempts
-
Retry max wait
-
Retryer backoff delay (milliseconds)
-
Retryer backoff max delay (milliseconds)
-
Retryer backoff multiplier (decimal)
| When you are receiving too many rate limiting errors, it is likely too many requests are being sent too frequently. Retrying may not help. One option is to decrease your traffic instead. If you want to continue sending the maximum number of requests, configure the Retryer backoff multiplier so it gets larger after every retry. The crawler will slow significantly and allow SharePoint to relax the throttling. |