db - Connector DB
Type and properties for a ConnectorDB implementation to use with this datasource.
type - string
Fully qualified class name of ConnectorDb implementation.
>= 1 characters
Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb
inlinks - boolean
Keep track of incoming links. This negatively impacts performance and size of DB.
Default: false
aliases - boolean
Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.
Default: false
inv_aliases - boolean
Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.
Default: false
startLinks - array[string]
The URL of the Windows Share folders or files to crawl, e.g., smb://172.17.0.3:32778/MyShare/. UNC paths like \\172.17.0.3\MyShare are also allowed
dedupe - boolean
If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.
Default: false
dedupeField - string
Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.
dedupeScript - string
Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.
dedupeSaveSignature - boolean
If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.
Default: false
delete - boolean
Set to true to remove documents from the index when they can no longer be accessed as unique documents.
Default: true
deleteErrorsAfter - integer
Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.
Default: -1
fetchThreads - integer
The number of threads to use during fetching. The default is 5.
Default: 5
emitThreads - integer
The number of threads used to send documents from the connector to the index pipeline. The default is 5.
Default: 5
chunkSize - integer
The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.
Default: 1
fetchDelayMS - integer
Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.
Default: 0
refreshAll - boolean
Set to true to always recrawl all items found in the crawldb.
Default: true
refreshStartLinks - boolean
Set to true to recrawl items specified in the list of start links.
Default: false
refreshErrors - boolean
Set to true to recrawl items that failed during the last crawl.
Default: false
refreshOlderThan - integer
Number of seconds to recrawl items whose last fetched date is longer ago than this value.
Default: -1
refreshIDPrefixes - array[string]
A prefix to recrawl all items whose IDs begin with this value.
refreshIDRegexes - array[string]
A regular expression to recrawl all items whose IDs match this pattern.
refreshScript - string
A JavaScript function ('shouldRefresh()') to customize the items recrawled.
forceRefresh - boolean
Set to true to recrawl all items even if they have not changed since the last crawl.
Default: false
forceRefreshClearSignatures - boolean
If true, signatures will be cleared if force recrawl is enabled.
Default: true
retryEmit - boolean
Set to true for emit batch failures to be retried on a document-by-document basis.
Default: true
depth - integer
Number of levels in a directory or site tree to descend for documents.
Default: -1
maxItems - integer
Maximum number of documents to fetch. The default (-1) means no limit.
Default: -1
failFastOnStartLinkFailure - boolean
If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.
Default: true
crawlDBType - string
The type of crawl database to use, in-memory or on-disk.
Default: on-disk
Allowed values: in-memoryon-disk
commitAfterItems - integer
Commit the crawlDB to disk after this many items have been received. A smaller number here will result in a slower crawl because of commits to disk being more frequent; conversely, a larger number here will cause a resumed job after a crash to need to recrawl more records.
Default: 10000
initial_mapping - Initial field mapping
Provides mapping of fields before documents are sent to an index pipeline.
skip - boolean
Set to true to skip this stage.
Default: false
label - string
A unique label for this stage.
<= 255 characters
condition - string
Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.
reservedFieldsMappingAllowed - boolean
Default: false
mappings - array[object]
List of mapping rules
Default: {"source":"charSet","target":"charSet_s","operation":"move"}{"source":"fetchedDate","target":"fetchedDate_dt","operation":"move"}{"source":"lastModified","target":"lastModified_dt","operation":"move"}{"source":"signature","target":"dedupeSignature_s","operation":"move"}{"source":"length","target":"length_l","operation":"move"}{"source":"mimeType","target":"mimeType_s","operation":"move"}{"source":"parent","target":"parent_s","operation":"move"}{"source":"owner","target":"owner_s","operation":"move"}{"source":"group","target":"group_s","operation":"move"}
object attributes:{source required : {
display name: Source Field
type: string
}target : {
display name: Target Field
type: string
}operation : {
display name: Operation
type: string
}}
unmapped - Unmapped Fields
If fields do not match any of the field mapping rules, these rules will apply.
source - string
The name of the field to be mapped.
target - string
The name of the field to be mapped to.
operation - string
The type of mapping to perform: move, copy, delete, add, set, or keep.
Default: copy
Allowed values: copymovedeletesetaddkeep
f.maxSizeBytes - integer
Maximum size (in bytes) of documents to fetch or -1 for unlimited file size.
Default: 4194304
f.minSizeBytes - integer
Minimum size, in bytes, of documents to fetch.
Default: 0
f.addFileMetadata - boolean
Set to true to add information about documents found in the filesystem to the document, such as document owner, group, or ACL permissions.
Default: true
f.index_items_discarded - boolean
Enable to index discarded document metadata
Default: false
retainOutlinks - boolean
Set to true for links found during fetching to be stored in the crawldb. This increases precision in certain recrawl scenarios, but requires more memory and disk space.
Default: false
aliasExpiration - integer
The number of crawls after which an alias will expire. The default is 1 crawl.
Default: 1
excludeExtensions - array[string]
File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.
excludeRegexes - array[string]
Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.
includeExtensions - array[string]
File extensions to be fetched. This will limit this datasource to only these file extensions.
includeRegexes - array[string]
Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.
reevaluateCrawlDBOnStart - boolean
Reevaluate exisiting crawldb entries for legality on startup?
Default: false
f.fs.username - string
Windows File Share Username
f.fs.password - string
Windows File Share Password
f.fs.domain - string
Windows File Share Domain
f.fs.timeout - integer
Timeout sets Read, Write, and Transact timeout (default is 60 seconds)
exclusiveMinimum: false
Default: 60
f.fs.socket_timeout - integer
Socket Timeout (default is 0 seconds, blocks forever)
exclusiveMinimum: false
Default: 0
f.fs.verify_access - boolean
Set to true to require successful connection to the filesystem before saving this datasource.
Default: true
enable_security_trimming - Enable security trimming
Set to true to fetch and index access control information from files.
ad_url - string
Fully qualified URL of the LDAP or AD server where user information is stored, in the format: 'ldap:://hostname:389' or 'ldap://hostname:636'.
Match pattern: ldaps?://.+
ad_user_principal_name - string
A User Principal with permissions to access the Active Directory server in the format user@domain.
Match pattern: .+@.+
ad_credentials - string
A password for the User Principal.
ad_user_filter - string
Valid LDAP filter to find user objects in Active Directory, such as '(&(objectclass=user)(sAMAccountName={0}))'.
ad_group_filter - string
Valid LDAP filter to find group objects in Active Directory, such as '(&(objectclass=group))'.
ad_user_base_dn - string
Active Directory node or directory where user objects reside.
ad_group_base_dn - string
Active Directory node or directory where group objects reside.
ad_cache_groups - boolean
Set to true to enable caching of the Active Directory group hierarchy to speed up security trimming processes.
Default: false
enable_SIDs_cache - boolean
Set to true to cache user SIDs to reduce the number of Active Directory requests.
Default: true
max_cache_size - integer
Maximum number of items to store in the SIDS cache before refreshing. The default is 100 items.
exclusiveMinimum: false
Default: 1000
cache_element_expiration_time - integer
Time in seconds to store items in the caches. The default is 7200 seconds (2 hours).
exclusiveMinimum: false
Default: 7200
ad_read_token_groups - boolean
Set to true to read groups using TOKEN_GROUPS. Only applied if 'Cache AD groups' is disabled.
Default: true
ad_context_factory - string
The classname of the context factory to use to create the initial context.
Default: com.sun.jndi.ldap.LdapCtxFactory
ad_security_authentication - string
The type of security authentication to use.
Default: simple
ad_referral - string
The method for processing referrals encountered by the service provider.
Default: follow
ad_read_timeout - integer
Time in milliseconds for reading responses from Active Directory. Default is 5000 ms.
exclusiveMinimum: false
Default: 5000
ad_connect_timeout - integer
Time in milliseconds for connecting to Active Directory. Default is 3000 ms.
exclusiveMinimum: false
Default: 3000
rewriteLinkScript - string
A Javascript function 'rewriteLink(link) { }' to modify links to documents before they are fetched.
diagnosticMode - boolean
Enable to print more detailed information to the logs about each request.
Default: false