com.lucidworks.fusion.connector.plugin.api.fetcher.type.content

Interface FetchGroup

All Known Implementing Classes:

Candidate, FetchInput

public interface FetchGroup

A fetch group represents a grouping of #Candidates. Items within a fetch group are consistently routed to the same plugin service instance, making data locality possible.

#CoreConfig.FetchSystemSettings.numFetchThreads() can be used to control the total number of threads per connector job, normally just 1 pool with n-threads. Inputs with fetch groups though, are consistently mapped across #CoreConfig.FetchSystemSettings.numFetchThreads() thread pools, where each thread pool has a dedicated set of threads.

For example, if a connector emits items that result in 20 distinct fetch groups, a numFetchThreads value of 10 would result in 10 thread pools, each with numThreadsPerGroup threads, with an even distribution of groups assigned to those thread pools; 2 groups per thread pool.

The two main benefits of fetch groups are:

• Fine grained control of processing for items within groups; items within a group are always processed on the same plugin node.
• Better control over group threading, by sharing a fixed number of thread pools across groups. If fetch calls for a particular group is inherently slow (i.e., blocking), groups assigned to other thread pools will not be impacted.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getFetchGroup()
default boolean	hasFetchGroup()

Method Detail

getFetchGroup

java.lang.String getFetchGroup()

hasFetchGroup

default boolean hasFetchGroup()