Legacy Product

Fusion 5.10
    Fusion 5.10

    Fusion Microservices

    Fusion is comprised of microservices that drive features and functionality within a deployment. The services running in your deployment depend on the version of Fusion and the features you have enabled.

    Get deployment details

    You can view your deployment details using kubectl.

    1. First, verify that you have access to your cluster, replacing the example values with your parameters. This example is for a Fusion instance deployed using GKE.

      gcloud container clusters get-credentials EXAMPLE-CLUSTER --region EXAMPLE-REGION --project EXAMPLE-PROJECT
    2. Get the ports and services:

      kubectl get svc -n EXAMPLE-NAME
    3. Get the StatefulSets:

      kubectl get statefulsets -n EXAMPLE-NAME
    4. Get the deployments:

      kubectl get deploy -n EXAMPLE-NAME

    Overview

    The table below lists the Fusion microservices deployed by the Helm chart. It also include Kubernetes services that manage traffic to microservices.

    Fusion is a complex distributed application composed of many stateful and stateless services designed to support demanding search-oriented workloads at high scale.

    For Docker image versions associated with microservices, see the list of Docker images and versions for each Fusion release.

    Microservice Required for Fusion Protocol Deployment or StatefulSet Node Pool Assignment Autoscaling Supported Description

    admin

    Yes

    REST/HTTP

    Deployment

    system

    Not required. Minimum of 1, but 2 pods are recommended for high availability.

    Exposes endpoints for admin tasks, such as creating applications and running jobs.

    admin-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves static Web assets for the admin UI.

    argo

    No

    HTTP

    Deployment

    system

    Yes (CPU or custom metric).

    Orchestrates parallel jobs on Kubernetes.

    argo-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Stores logs and prior Argo workflow runs.

    auth-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves static Web assets for the login form.

    connectors-classic

    No

    REST/HTTP

    StatefulSet

    analytics or system

    Not supported.

    REST service for supporting non-RPC connector plugins. This microservice was previously named classic-rest-service.

    connectors

    No

    REST/HTTP

    Deployment

    analytics or system

    Not required. One pod is enough for most clusters.

    Routes REST API requests to connectors-classic and connectors-rpc.

    connectors-backend

    No

    gRPC

    Deployment

    analytics or system

    Yes (CPU or custom metric).

    gRPC service for managing SDK-based connector plugins.

    connector-plugin-<connector_plugin>

    No

    HTTP/TCP

    Deployment

    analytics or system

    Yes (CPU or custom metric).

    Deployment for each connector plugin type. Note: There is a base deployment, connector-plugin with 0 replicas. This is used as a deployment template for each connector plugin type. It should not be deleted or scaled.

    devops-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves static Web assets for the DevOps UI. Available in Fusion 5.0 through 5.5.

    indexing

    Yes

    REST/HTTP

    Deployment

    search or analytics depending on write-volume

    Yes (CPU or custom metric).

    Processes indexing requests.

    insights

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves the App Insights UI

    job-launcher

    No

    REST/HTTP

    Deployment

    analytics

    Not required. One pod is enough for most clusters.

    Configures and launches the Spark driver pod for running Spark jobs

    job-rest-server

    No

    REST/HTTP

    Deployment

    analytics

    Not required. One pod is enough for most clusters.

    Performs admin tasks for creating and running Spark jobs.

    jupyter

    No

    HTTP

    Deployment

    analytics

    Not required. One pod is enough for most clusters.

    Jupyter notebook for ad hoc analytics and visualization.

    logstash

    No

    HTTP

    StatefulSet

    system

    Not required. Minimum of 1, but 2 pods are recommended for high availability.

    Collects logs from the other microservices and either indexes into system_logs or ships them to an external service like Elastic

    milvus

    No

    REST/HTTP

    Deployment

    analytics or system

    Not required. One pod is enough for most clusters.

    Dense Vector Search Engine for ML models active.

    milvus-mysql

    No

    REST/HTTP

    Deployment

    analytics or system

    Not required. One pod is enough for most clusters.

    Handles metadata for Milvus service active.

    ml-model-service

    No

    REST/HTTP and gRPC

    Deployment

    search

    Yes (CPU or custom metric).

    Exposes gRPC endpoints for generating predictions from ML models.

    pm-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves static Web assets for the Predictive Merchandiser app.

    proxy / api-gateway

    Yes

    HTTP

    Deployment

    search

    Not required. Minimum of 1, but 2 pods are recommended for high availability.

    Performs authentication, authorization, and traffic routing.

    pulsar-bookkeeper

    Yes

    HTTP

    StatefulSet

    search

    At least 3 nodes in high availability. You need to run 3 or 5 to keep quorum.

    Write Ahead Log (WAL) used for persistent message storage.

    pulsar-broker

    Yes

    HTTP and TCP

    Deployment

    search

    At least 3 nodes in high availability.

    Contains REST API for managing administration and dispatcher for handling all message transfers.

    query

    Yes

    REST/HTTP

    Deployment

    search

    Yes (CPU or custom metric).

    Processes query requests.

    rules-ui

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves static Web assets for the Rules UI.

    seldon-ambassador

    No

    Web

    Deployment

    system

    Not required. Minimum of 1, but 2 pods are recommended for high availability.

    Load balancing and proxy for Seldon Core deployments.

    seldon-core

    No

    REST/GRPC

    Deployment

    system

    Yes (CPU or custom metric).

    Serves models built in any model building framework.

    seldon-webhook-service

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Maintains Seldon Core deployments for ML model serving active.

    solr

    Yes

    HTTP

    StatefulSet

    At least 3 nodes in search, 2 in analytics, and 2 in system

    Yes (CPU or custom metric).

    Search engine.

    spark-driver

    No

    n/a

    single pod per job

    analytics or dedicated Node Pool for Spark jobs

    One per job.

    Launched by the job-launcher to run a Spark job

    spark-executor

    No

    n/a

    one or more pods launched by the Spark driver for executing job tasks

    analytics or dedicated Node Pool for Spark jobs

    Depends on job configuration. Controlled by the spark.executor.instances setting.

    Executes tasks for a Spark job

    sql-service

    No

    REST/HTTP and JDBC

    Deployment

    analytics

    Not required. One pod is enough for most clusters.

    Performs admin tasks for creating and managing SQL catalog assets. Exposes a JDBC endpoint for the SQL service.

    templating

    No

    Web

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Retrieves and renders Predictive Merchandiser templates.

    tikaserver

    No

    HTTP

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Is responsible for text extraction using Tika Server Parser

    webapps

    No

    REST/HTTP

    Deployment

    system

    Not required. One pod is enough for most clusters.

    Serves App Studio-based Web apps.

    zookeeper

    Yes

    TCP

    StatefulSet

    system

    No. You need to run 1, 3, or 5 ZooKeeper pods to keep quorum. Do not use HPA for scaling ZooKeeper.

    Stores centralized configuration and performs distributed coordination tasks.

    Ports used by Fusion

    Below you will find the list of pod ports for intra-cluster communications.

    Service Port

    admin

    8765

    admin-ui

    8080

    auth-ui

    8080

    connector-plugin

    9020, 5701

    connectors

    9010

    connectors-backend

    8771

    connectors-classic

    9000

    devops-ui

    8080

    fusion-argo-argo-ui

    2746

    fusion-jupyter

    8888

    fusion-kafka

    9092, 9093

    fusion-headless

    9092, 9093

    fusion-ml-model-service-ambassador

    80, 443

    fusion-ml-model-service-mysql

    3306

    fusion-pulsar-bookkeeper

    3181, 8000

    fusion-pulsar-broker

    8080, 6650

    fusion-solr-exporter

    8989

    fusion-solr-headless

    8983

    fusion-solr-svc

    8983

    fusion-zookeeper

    2181, 2281

    fusion-zookeeper-headless

    2181, 3888, 2888, 2281

    indexing

    8765

    insights

    8080

    job-launcher

    8083

    job-rest-server

    8081

    milvus

    19530, 19121

    ml-model-grpc

    6565

    ml-model-service

    8086

    moos-ui

    8080

    pm-ui

    8080

    prometheus-pushgateway

    9091

    proxy

    6764

    pulsar-broker

    8080, 6650

    query

    8787

    rules-ui

    8080

    seldon-webhook-service

    443

    sql-jdbc

    8088

    sql-service

    8085

    superset

    8088

    superset-proxy

    80

    templating

    5250

    tikaserver

    9998

    webapps

    8780