Add Custom Headers to HTTP Requests
The Web connector V1 retrieves data from a Web site using HTTP and starting from a specified URL.
In Fusion 4.1 and later, you can optionally add custom headers to all http
get requests from the Web connector.
For example, you might want to add a header that includes
Connection: keep-alive to prevent the connector from timing out while crawling your Web site.
To add a custom header, use the configuration parameter
f.addedHeaders. To send multiple headers, use the following format:
Click Indexing > Datasources.
Click Add+, then Web.
Enter a datasource ID and a start link.
Click Link discovery.
Fill in the Headers to add to HTTP requests field.
Add each header in the format
To add multiple headers to all HTTP requests, put each header on a new line.
Fill in any remaining options to configure the datasource.