Slack V2 Connector Configuration Reference

Table of Contents

Authentication
Required application scopes
Initial actions
Document filtering
Crawling process
Incremental Crawling processors
Security filtering

The Slack V2 connector allows you to index Slack channels and messages.

Due to rate limit restrictions in the Slack Web API, this connector has the following restrictions:

Only one connector-plugin instance should be started.
Only one fetching thread should be set when creating the datasource configuration.

Authentication

You must install a Slack application in the workspace that will be indexed before using the Slack V2 connector. See Slack’s Basic app setup for more information about installing a Slack application.

Once the app is correctly installed in the workspace, Slack will provide an access token, which you will then use in the Slack V2 connecter configuration.

Required application scopes

The Slack application must provide access to the following User Token Scopes in order to work with Slack V2 connector:

Slack User Token Scopes

Initial actions

The Slack V2 connector will perform the following actions at the beginning of each job:

Load all users in the users cache. This information will be used later to resolve user names when processing mentions in messages.

It will also be useful for avoiding requests to retrieve user’s metadata.
Load the channels to be processed using the channel filtering properties from the configuration.

This will be useful to establish a source of truth when determining whether a channel should be indexed or deleted on each job.

Document filtering

Profiles indexing

Profiles indexing
`Index profiles`	Determines whether users are indexed
`Index guest profiles`	Determines whether profiles from guest users are indexed
`Index inactive profiles`	Determines whether profiles of deactivated users are indexed
Channels indexing
`Index channels`	Determines whether a document per channel is indexed
`Index archived channels`	Determines whether archived channels are indexed to the content collection
`Index public channels`	Determines whether public channels are indexed
`Index private channels`	Determines whether private channels are indexed
`Index direct messages`	Determines whether direct and multi-party direct message only are indexed
`Channels to include`	A list of the channel names to be included in the crawling process
Messages indexing
`Index messages`	Determines whether messages from the channels are indexed
`Index bot messages`	Determines where messages from bots in channels are indexed
`Index messages replies (threads)`	Determines whether the replies (if applicable) to messages are indexed

Index profiles

Determines whether users are indexed

Index guest profiles

Determines whether profiles from guest users are indexed

Index inactive profiles

Determines whether profiles of deactivated users are indexed

Channels indexing

Index channels

Determines whether a document per channel is indexed

Index archived channels

Determines whether archived channels are indexed to the content collection

Index public channels

Determines whether public channels are indexed

Index private channels

Determines whether private channels are indexed

Index direct messages

Determines whether direct and multi-party direct message only are indexed

Channels to include

A list of the channel names to be included in the crawling process

Messages indexing

Index messages

Determines whether messages from the channels are indexed

Index bot messages

Determines where messages from bots in channels are indexed

Index messages replies (threads)

Determines whether the replies (if applicable) to messages are indexed

Due to limitations in the Slack Web API, edited and deleted messages cannot be retrieved. You must clear the datasource and start a new connectors job to index edited messages and/or remove deleted messages.

Crawl process diagram

Endpoints used in the crawling process

Endpoints Description

Endpoints	Description
`https://slack.com/api/users.list`	Lists all users in the workspace. This is not controlled by the properties in Document Filtering. All users are retrieved at the beginning of each job.
`https://slack.com/api/conversations.list`	Lists the channels that should be indexed. This fetching is guided by the properties listed in Document Filtering.
`https://slack.com/api/conversations.members`	Lists the members of specific channels being indexed.
`https://slack.com/api/conversations.history`	Retrieves a page of messages from a specific channel
`https://slack.com/api/users.info`	Retrieves the information of a single user, if it’s not present in the users cache (message document generation).
`https://slack.com/api/conversations.info`	Retrieves the information of a single channel, if it’s not present in the channel’s cache (message document generation).
`https://slack.com/api/team.info`	Retrieves the information of a team (message document generation).
`https://slack.com/api/conversations.replies`	Retrieves a page of replies from a specific message from a channel.

https://slack.com/api/users.list

Lists all users in the workspace. This is not controlled by the properties in Document Filtering. All users are retrieved at the beginning of each job.

https://slack.com/api/conversations.list

Lists the channels that should be indexed. This fetching is guided by the properties listed in Document Filtering.

https://slack.com/api/conversations.members

Lists the members of specific channels being indexed.

https://slack.com/api/conversations.history

Retrieves a page of messages from a specific channel

https://slack.com/api/users.info

Retrieves the information of a single user, if it’s not present in the users cache (message document generation).

https://slack.com/api/conversations.info

Retrieves the information of a single channel, if it’s not present in the channel’s cache (message document generation).

https://slack.com/api/team.info

Retrieves the information of a team (message document generation).

https://slack.com/api/conversations.replies

Retrieves a page of replies from a specific message from a channel.

Message mentions processing

The Slack V2 connector retrieves messages from the Slack API in the following format:

 "type": "message",
 "text": "this is a mention in a message: <@U01U580HSDN>",

In this example, @U01U580HSDN is the user Id. The connector will resolve the user Id with the user’s display name, if it exists. Otherwise, it will use the full name in the user’s profile.

Incremental Crawling processors

Each incremental job will re-index the User ACL to update all users in the Access Control Collection, and the Channel ACL to update the members of each channel.

Processor Description

Processor	Description
`ProfilesCheckpointProcessor`	Performs the following actions: Triggers the indexing of all user profiles and the profile’s ACL Verifies that profiles should still be indexed. If not, it deletes them Verifies guest and deleted profiles
`ChannelsCheckpointProcessor`	Detects new and deleted channels by comparison and the triggers the re-indexing of the channel ACL documents.
`ChannelCheckpointProcessor`	Verifies that the current channel should still be indexed. If the channel should still be indexed, then it triggers the indexing of new messages providing the timestamp (`ts`) of the last seen message from the previous job.

ProfilesCheckpointProcessor

Performs the following actions:

Triggers the indexing of all user profiles and the profile’s ACL
Verifies that profiles should still be indexed. If not, it deletes them
Verifies guest and deleted profiles

ChannelsCheckpointProcessor

Detects new and deleted channels by comparison and the triggers the re-indexing of the channel ACL documents.

ChannelCheckpointProcessor

Verifies that the current channel should still be indexed.

If the channel should still be indexed, then it triggers the indexing of new messages providing the timestamp (ts) of the last seen message from the previous job.

Security filtering

Security filtering processors

Processor Description

Processor	Description
`ProfileAclProcessor`	Builds a document representing a Slack user, which is then sent to the Access Control Collection. Regular users can access public channels, even if they are not members of the channel. Guest users can access only the channels they are members of.
`ChannelAclProcessor`	Retrieves the current members of the given channel, builds a document with those values, and sends the document to the Access Control Collection. In public channels, an extra document is sent to the Access Control Collection to provide access to regular users who are not members of public groups.

ProfileAclProcessor

Builds a document representing a Slack user, which is then sent to the Access Control Collection.

Regular users can access public channels, even if they are not members of the channel.

Guest users can access only the channels they are members of.

ChannelAclProcessor

Retrieves the current members of the given channel, builds a document with those values, and sends the document to the Access Control Collection.

In public channels, an extra document is sent to the Access Control Collection to provide access to regular users who are not members of public groups.

Access Control Document links

Security trimming rules

Security trimming uses the following rules:

All users (regular and guest) can see the profile of any user in the Slack workspace.
All regular users can access public channels and messages, even if they are not members of the channel.
Guest users can only access the public or private channels they are members of.
Only the members of private channels can access the channel and messages.
Only the users involved can access direct messages and multi-party direct messages.

Connector for Slack

description - string

Optional description

<= 125 characters

pipeline - stringrequired

Name of the IndexPipeline used for processing output.

>= 1 characters

Match pattern: ^[a-zA-Z0-9_-]+$

diagnosticLogging - boolean

Enable diagnostic logging; disabled by default

Default: false

parserId - stringrequired

The Parser to use in the associated IndexPipeline.

coreProperties - Core Properties

Common behavior and performance settings.

fetchSettings - Fetch Settings

System level settings for controlling fetch behavior and performance.

indexingThreads - number

Maximum number of indexing threads; defaults to 4.This setting controls the number of threads in the indexing service used for processing content documents emitted by this datasource.Higher values can sometimes help with overall fetch performance.

>= 1

<= 10

exclusiveMinimum: false

exclusiveMaximum: false

Default: 4

Multiple of: 1

pluginInstances - number

Maximum number of plugin instances for distributed fetching. Only specified number of plugin instanceswill do fetching. This is useful for distributing load between different instances.

<= 500

exclusiveMinimum: false

exclusiveMaximum: false

Default: 0

Multiple of: 1

fetchResponseScheduledTimeout - number

The maximum amount of time for a response to be scheduled. The task will be canceled if this setting is exceeded.

>= 1000

<= 500000

exclusiveMinimum: false

exclusiveMaximum: false

Default: 300000

Multiple of: 1

indexMetadata - boolean

When enabled the metadata of skipped items will be indexed to the content collection

Default: false

indexContentFields - boolean

When enabled, content fields will be indexed to the crawl-db collection

Default: false

id - stringrequired

A unique identifier for this Configuration.

>= 1 characters

Match pattern: ^[a-zA-Z0-9_-]+$

properties - Slack properties

Plugin specific properties.

slackAppToken - string

The Slack application access token, this token is provided after installing the application in the workspace

profilesConfig - Profiles settings

Settings to control the indexation of Slack profiles

indexProfiles - boolean

Should profiles be indexed

Default: true

indexGuestProfiles - boolean

If indexing profiles, include profiles of guest users

Default: true

indexInactiveProfiles - boolean

If indexing profiles, include profiles of inactive users

Default: false

channelsConfig - Channels and messages settings

Settings to control the indexation of Slack channels and messages

indexChannels - boolean

Generate a document in the index for each channel

Default: true

indexMessages - boolean

Should individual messages be indexed

Default: true

indexFromArchivedChannels - boolean

If indexing messages or channel data, include data from archived channels

Default: false

indexFromPublicChannels - boolean

If indexing messages or channel data, include data from public channels

Default: true

indexPrivateChannels - boolean

If indexing messages or channel data, include data from private channels

Default: false

indexDirectMessages - boolean

If indexing messages, include private messages

Default: false

indexBotMessages - boolean

If indexing messages, include messages that came from bots

Default: false

indexThreadReplies - boolean

If indexing messages, include messages that are replies to threads. Note: Each thread found will result in it's own API call, and is subject to rate limiting. Using this can significantly slow down indexing.

Default: false

channelFilter - array[string]

If indexing messages or channel data, include data only from channels with names matching the specified regular expressions

requests - Requests settings

pageSize - number

The number of objects requested per page from Slack

>= 1

<= 1000

exclusiveMinimum: false

exclusiveMaximum: false

Default: 200

Multiple of: 1

itemRetryProperties - Item retry settings

Options to configure the retry operation for items.

maxRetries - number

The maximum number of attempts for a failed item

<= 20

exclusiveMinimum: false

exclusiveMaximum: false

Default: 3

Multiple of: 1

retryDelayInSeconds - number

The amount of time, in seconds, before process again a failed item

>= 1

<= 600

exclusiveMinimum: false

exclusiveMaximum: false

Default: 30

Multiple of: 1

connectionsProperties - Connections Settings

A set of options for configuring the HTTP client.

maxConnections - number

The maximum number of connections

>= 1

<= 2147483647

exclusiveMinimum: false

exclusiveMaximum: false

Default: 20

Multiple of: 1

timeouts - Timeout Settings

A set of options for configuring the HTTP client.

readTimeoutMs - number

<= 600000

exclusiveMinimum: false

exclusiveMaximum: false

Default: 300000

Multiple of: 1

connectTimeoutMs - number

<= 300000

exclusiveMinimum: false

exclusiveMaximum: false

Default: 60000

Multiple of: 1

security - Security filtering configuration

enabled - boolean

Enable query-time security-trimming

Default: true

collectionId - string

Id of the collection to be used for storing ACL records. If not specified, ACL collection name will be generated automatically using pattern '<datasource_id>_access_control_hierarchy'.