Avoid SharePoint Throttling
The SharePoint connector retrieves content and metadata from an on-premises SharePoint repository.
You have a few options to avoid throttling:
If you see many
503 errors, you are probably hitting SharePoint Online with too many concurrent fetchers.
Set Crawl Performance > Fetch Threads to a lower value.
Set Crawl Performance > Prefetch Threads to a lower value.
By default, the connector is configured with retries. This provides a chance for the requests that were rate-limited to run again.
You can increase the number of retries and the interval between retries. The process is called exponential backoff, which gradually increases the delays between retries to increase the chances of a successful retry. This helps prevent missing documents due to rate limiting.
Retry configuration parameters include:
Retry max wait
Retryer backoff delay (milliseconds)
Retryer backoff max delay (milliseconds)
Retryer backoff multiplier (decimal)
|When you are receiving too many rate limiting errors, it is likely too many requests are being sent too frequently. Retrying may not help. One option is to decrease your traffic instead. If you want to continue sending the maximum number of requests, configure the Retryer backoff multiplier so it gets larger after every retry. The crawler will slow significantly and allow SharePoint to relax the throttling.