Twitter Stream V1 Connector Configuration Reference
The Twitter Stream connector uses Twitter’s streaming API to continually index Twitter. The datasource can be configured to limit tweets or it can be run indefinitely, until Twitter cuts off your access or you stop the datasource. This connector only retrieves tweets created after the datasource has been started.
|
|
This connector is no longer functional in Fusion 5.9 and later. This incompatibility arises due to changes implemented in the data source version or related APIs.
Although the V1 connector might still be visible within the Fusion UI, it cannot be used effectively. To ensure uninterrupted operation, we strongly recommend switching to a valid V2 alternate connector.
The replacement for this connector is in active development at this time and will be released at a future date.
|
|
|
When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.
|
Twitter Stream uses Twitter's stream API to continually index tweets. It includes the ability to set some properties for limiting the tweets, and it never ends unless Twitter stops your access or you stop the datasource.
description - string
Optional description for this datasource.
id - stringrequired
Unique name for this datasource.
>= 1 characters
Match pattern: ^[a-zA-Z0-9_-]+$
parserId - string
Parser used when parsing raw content. Retry parsing setting is available under crawl performance (advance setting)
pipeline - stringrequired
Name of an existing index pipeline for processing documents.
>= 1 characters
properties - Properties
Datasource configuration properties
access_token - string
The OAuth Access Token is provided by Twitter when registering the application.
>= 1 characters
commit_on_finish - boolean
Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.
Default: true
consumer_key - string
The OAuth Consumer Key is provided by Twitter when registering the application.
>= 1 characters
consumer_secret - string
The OAuth Consumer Secret is provided by Twitter when registering the application.
>= 1 characters
db - Connector DB
Type and properties for a ConnectorDB implementation to use with this datasource.
aliases - boolean
Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.
Default: false
inlinks - boolean
Keep track of incoming links. This negatively impacts performance and size of DB.
Default: false
inv_aliases - boolean
Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.
Default: false
type - string
Fully qualified class name of ConnectorDb implementation.
>= 1 characters
Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb
filter_follow - array[string]
Set of users (user ids) to track
filter_locations - array[string]
Set of bounding boxes (e.g. 'left,bottom,right,top' lat/long coordinates)
filter_track - array[string]
Keywords or phrases to track
initial_mapping - Initial field mapping
Provides mapping of fields before documents are sent to an index pipeline.
condition - string
Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.
label - string
A unique label for this stage.
<= 255 characters
mappings - array[object]
List of mapping rules
object attributes:{operation : {
display name: Operation
type: string
}source required : {
display name: Source Field
type: string
}target : {
display name: Target Field
type: string
}}
reservedFieldsMappingAllowed - boolean
Default: false
skip - boolean
Set to true to skip this stage.
Default: false
unmapped - Unmapped Fields
If fields do not match any of the field mapping rules, these rules will apply.
operation - string
The type of mapping to perform: move, copy, delete, add, set, or keep.
Default: copy
Allowed values: copymovedeletesetaddkeep
source - string
The name of the field to be mapped.
target - string
The name of the field to be mapped to.
max_docs - integer
The maximum number of documents to pull down, as a long. -1 for no limit
Default: -1
sleep - integer
The amount of time, in milliseconds, to sleep when listening so as to not get throttled
Default: 10000
token_secret - string
The OAuth Token Secret is provided by Twitter when registering the application.
>= 1 characters