Legacy Product

Fusion 5.4

Add Custom Headers to HTTP Requests

The Web connector V1 retrieves data from a Web site using HTTP and starting from a specified URL.

In Fusion 4.1 and later, you can optionally add custom headers to all http get requests from the Web connector. For example, you might want to add a header that includes Connection: keep-alive to prevent the connector from timing out while crawling your Web site.

To add a custom header, use the configuration parameter f.addedHeaders. To send multiple headers, use the following format:

Header1: Value1
Header2: Value2
To add a custom header in the Fusion UI on any node:

  1. Click Indexing > Datasources.

  2. Click Add+, then Web.

  3. Enter a datasource ID and a start link.

  4. Click Link discovery.

  5. Fill in the Headers to add to HTTP requests field.

    • Add each header in the format HeaderName: HeaderValue.

    • To add multiple headers to all HTTP requests, put each header on a new line.

  6. Fill in any remaining options to configure the datasource.

  7. Click Save.