Azure V1 Connector Configuration Reference

Table of Contents

Configuration

The Azure connector is used to crawl an Azure instance. Its connector type is "lucid.azure" and its plugin type is "azure".

V1 deprecation and removal notice

Starting in Fusion 5.12.0, all V1 connectors are deprecated. This means they are no longer being actively developed and will be removed in Fusion 5.13.0.

The replacement for this connector is in active development at this time and will be released at a future date.

If you are using this connector, you must migrate to the replacement connector or a supported alternative before upgrading to Fusion 5.13.0. We recommend migrating to the replacement connector as soon as possible to avoid any disruption to your workflows.

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

Configuration

Crawl Azure objects.

id - stringrequired

Unique name for this datasource.

>= 1 characters

Match pattern: ^[a-zA-Z0-9_-]+$

pipeline - stringrequired

Name of an existing index pipeline for processing documents.

>= 1 characters

description - string

Optional description for this datasource.

parserId - string

Parser used when parsing raw content. For some connectors, a configuration to 'retry' parsing if an error occurs is available as an advanced setting

properties - Properties

Datasource configuration properties

db - Connector DB

Type and properties for a ConnectorDB implementation to use with this datasource.

type - string

Fully qualified class name of ConnectorDb implementation.

>= 1 characters

Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb

inlinks - boolean

Keep track of incoming links. This negatively impacts performance and size of DB.

Default: false

aliases - boolean

Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

Default: false

inv_aliases - boolean

Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

Default: false

service_type - string

The Azure service type to crawl, either Blobs or Tables.

Allowed values: Azure TableAzure Blob

storage_account - string

The Azure storage account name.

>= 1 characters

token_secret - string

A valid Azure Access Key for authentication.

>= 1 characters

max_threads - integer

The maximum number of threads to use for fetching data. Each thread will create a new connection to the repository, which may make overall throughput faster, but will also require more system resources including CPU and memory.

Default: 5

max_connections - integer

Maximum number of simultaneous connections to the repository. This value usually does not need to be changed.

Default: 5000

storage_container - string

The name of an Azure Blob container.

>= 1 characters

max_bytes - integer

The maximum size, in bytes, of a document to crawl.

Default: 10485760

tables - string

The Azure table to index.

>= 1 characters

table_filter_statement - string

A filter to apply to the crawl. Use Azure's syntax for filtering table queries.

commit_on_finish - boolean

Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

Default: true

initial_mapping - Initial field mapping

Provides mapping of fields before documents are sent to an index pipeline.

skip - boolean

Set to true to skip this stage.

Default: false

label - string

A unique label for this stage.

<= 255 characters

condition - string

Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

reservedFieldsMappingAllowed - boolean

Default: false

retentionMappings - array[object]

Define here which fields should be kept or deleted

object attributes:{field required : {
display name: Field
type: string
}operation : {
display name: Operation
type: string
}}

updateMappings - array[object]

Define here values that should be added to or set on a field. When a value is added, any values previously on the field will be retained. When a value is set, any values previously on the field will be overridden.

object attributes:{field required : {
display name: Field
type: string
}value required : {
display name: Value
type: string
}operation : {
display name: Operation
type: string
}}

translationMappings - array[object]

Define here fields that should be moved or copied to another field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.

object attributes:{source required : {
display name: Source Field
type: string
}target required : {
display name: Target Field
type: string
}operation : {
display name: Operation
type: string
}}

unmappedRule - Unmapped Fields

Define here how fields not mapped by the above rules should be handled. By default, any remaining fields will be kept on the document.

keep - boolean

Keep all unmapped fields

Default: true

delete - boolean

Delete all unmapped fields

Default: false

fieldToMoveValuesTo - string

Move all unmapped field values to this field

fieldToCopyValuesTo - string

Copy all unmapped field values to this field

valueToAddToUnmappedFields - string

Add this value to all unmapped fields

valueToSetOnUnmappedFields - string

Set this value on all unmapped fields