Legacy Product

Fusion 5.4

Fusion 5 Upgrade from 5.0.x

This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the General upgrade process.

Remember, upgrade instructions may vary between deployment types too. Review Upgrades by deployment type for details.

You must upgrade to 5.1.x before upgrading to 5.2.x.

Upgrades from 5.0.0

to 5.0.1

Upgrade from 5.0.0 to 5.0.1 using the General upgrade process. You may need to upgrade to Helm v3 before using the general upgrade process.

to 5.0.2

Before upgrading from 5.0.0/5.0.1 to 5.0.2 using the General upgrade process, you must first:

  1. Upgrade to Helm v3

  2. Migrate your cluster to Helm v3

  3. Upgrade Solr from 8.2.0 to 8.3.1 and ZooKeeper from 3.4.14 to 3.5.6

  4. Install Prometheus and Grafana to the exisiting cluster

1. Upgrade to Helm v3

  1. List our your releases for Helm v2:

    helm ls --all-namespaces

  2. When you’re ready to upgrade, run the following:

    brew upgrade kubernetes-helm
    This step is for Mac users. For other operating systems, download directly from Helm.

  3. Verify the Helm version

    helm version --short

    Expected output:

    v3.0.0+ge29ce2a

2. Migrate your cluster to Helm v3

If you installed your Fusion 5 cluster using Helm v2, you need to migrate it to Helm v3 with the helm-2to3 plugin using the process described in the Helm blog. These instructions teach you how to migrate the release metadata in Tiller to your local system.

If you installed your cluster with Helm v3, you don’t need to do this step. Verify your release with helm ls.

3. Upgrade Solr and ZooKeeper

Fusion 5.0.0 and 5.0.1 run different versions of Solr and ZooKeeper than 5.0.2. You need to upgrade Solr in your existing cluster and make minor changes to the custom values YAML.

ZooKeeper upgrades to 3.5.6 during the Fusion upgrade, but due to a bug with Kubernetes, you must delete the exisiting StatefulSet to switch charts during the upgrade.

During testing, we found upgrading Solr before ZooKeeper is more stable.

Solr

To upgrade Solr:

  1. Edit your custom values YAML file and change the Solr version to 8.3.1.:

    solr:
  image:
    tag: 8.3.1
  updateStrategy:
    type: "RollingUpdate"

  2. Determine the version of the Fusion chart you are running:

    helm ls -n <namespace>`

  3. Use the Fusion chart version in place of <CHART_VERSIOM> in the setup script. For instance, your chart version may be fusion-5.0.1. In this case, pass --version 5.0.1.

    ./setup_f5_gke.sh -c <existing_cluster> -p <gcp_project_id> -r <release> -n <namespace> \
  --version <CHART_VERSION> \
  --values gke_<cluster>_<release>_fusion_values.yaml --upgrade

  4. Wait until Solr is running and heatlhy again.

ZooKeeper

Before beginning, edit your custom values file, and move the ZooKeeper settings out from under the solr: section to the main level. For example change this:

solr:
  ...
  zookeeper:
    ...

... to this:

solr:
  ...

zookeeper:
  ...

At this point you’re ready to upgrade to ZooKeeper 3.5.6.

You cannot upgrade ZooKeeper wit zero downtime. Your cluster will lose quorum momentarily, resulting in a minute or so of downtime in the cluster. To avoid as much downtime as possible, be ready to upgrade to Fusion 5.0.2 immediately after deleting the existing Statefulset.

When you’re ready to upgrade:

  1. Run the following command:

    kubectl delete statefulset ${RELEASE}-solr
kubectl delete statefulset ${RELEASE}-zookeeper
    Deleting the StatefulSet does not remove the persistent volumes backing Zookeeper and Solr. No data is lost.

  2. After editing your custom values YAML file, run:

    cd fusion-cloud-native

./setup_f5_gke.sh -c <CLUSTER> -p <PROJECT> -z <ZONE> \
  -n <NAMESPACE> -r <RELEASE> \
    --values <MY_VALUES> --version 5.0.2 --upgrade --force

  3. Wait several minutes, and then verify ZooKeeper establishes quorum:

    kubectl get pods

    Kubernetes pulls new Docker images and performs the rolling upgrade for each Fusion service. The upgrade rollout may take some time to complete.

  4. After upgrading, verify the versions of each pod:

    kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq

4. Install Prometheus and Grafana to Existing Cluster

As of Fusion 5.0.2, the setup scripts provide the option to install Prometheus and Grafana using the --prometheus option. If you installed Fusion 5.0.0 or 5.0.1, the upgrade does not install Prometheus and Grafana.

Once you complete the upgrade to Fusion 5.0.2, you can run the install_prom.sh script to install these additional services into your namespace. Pass the --help option to see script usage details.

  1. To install into a GKE cluster and schedule the new pods in the default Node Pool:

    ./install_prom.sh -c <cluster> -r <release> -n <namespace> \
  --node-pool "cloud.google.com/gke-nodepool: default-pool" --provider gke

  2. Once Prometheus and Grafana are deployed, edit your custom values YAML file for Fusion to enable the Solr exporter:

    solr:
  ...
  exporter:
    enabled: true
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9983"
      prometheus.io/path: "/metrics"
    nodeSelector:
      cloud.google.com/gke-nodepool: default-pool

  3. Add pod annotations to the query-pipeline, fusion-indexing, api-gateway services, as needed, to allow Prometheus to scrape metrics:

    fusion-indexing:
  ...
  pod:
    annotations:
      prometheus.io/port: "8765"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"
    query-pipeline:
  ...
  pod:
    annotations:
      prometheus.io/port: "8787"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"
    api-gateway:
  ...
  pod:
    annotations:
      prometheus.io/port: "6764"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"

  4. Complete the process by running and upgrade on the Fusion Helm chart.

to 5.1.0

If you’re running Fusion 5.0.0 or 5.0.1, upgrade to 5.0.2 before proceeding with this section. If you’re running Fusion 5.0.2, perform these steps before upgrading to 5.1.0:

  1. Prepare to upgrade to Solr 8.4.1

  2. Backup the query_rewrite and query_rewrite_staging collections

  3. Upgrade to Solr 8.4.1

  4. Install Kubernetes Custom Resource Definition (CRDs) for Seldon Core

  5. Delete logstash StatefulSet

  6. Update the Prometheus Scrape Path for query, index, and gateway services

1. Prepare to upgrade to Solr 8.4.1

Lucene 8.4.1 introduced an incompatible change to the underlying postingsFormat for the tagger field type in the schema for query_rewrite collections. For more information, see the Solr guide.

Before you upgrade to Solr 8.4.1, re-index the query_rewrite documents to remove the use of the postingsFormat. Otherwise, when Solr 8.4.1 initializes, it will not be able to load the query_rewrite collections. After upgrading, you’ll re-index again to restore the postingsFormat using the new implementation. The custom postingsFormat is essential for achieving optimal text tagging performance.

2. Back up the query_rewrite and query_rewrite_staging collections

DO NOT make changes to the query_rewrite collections with the Rules Editor during the upgrade process. For production systems, perform this upgrade process during a maintenance window.

Create a backup of your query_rewrite and query_rewrite_staging collections to ensure your data is not lost during the upgrade, especially in production environments. Depending on your Ingress config, the export may take too long and timeout. We recommend using kubectl port-forward to the Fusion Gateway pod:

kubectl port-forward <POD> 6764

Export the collection(s) to a local JSON file using the /query/query-rewrite/export/<COLL> endpoint. For example:

PROXY="http://localhost:6764"
APP="APP_NAME"
curl -u $CREDS "$PROXY/query/query-rewrite/export/${APP}_query_rewrite_staging" > ${APP}_query_rewrite_staging.json
curl -u $CREDS "$PROXY/query/query-rewrite/export/${APP}_query_rewrite" > ${APP}_query_rewrite.json

Replace $CREDS with your Fusion admin username and password, for example -u admin:somepassword

Repeat this command for every Fusion application with data indexed in the query_rewrite_staging and query_rewrite collections.

3. Upgrade to Solr 8.4.1

To upgrade from Solr 8.3.1 to 8.4.1, re-index all query_rewrite and query_rewrite_staging collections that have indexed data. Fusion provides a utility Docker image to drive the re-index process.

If your installation does not have indexed documents in any of the query_rewrite collections, you can upgrade to Solr 8.4.1 using a Helm upgrade:

  1. Run the prepare step:

    kubectl run --generator=run-pod/v1 \
  --image="lucidworks/fm-upgrade-query-rewrite:1.x" \
  --restart=Never \
  --env="HELM_RELEASE=<CHANGEME>" \
  --env="ACTION=prepare" prepare-upgrade-solr841

    The prepare step re-indexes the query_rewrite collections into a temp collection after removing the postingsFormat from the tagger field type in the Solr schema. This ensures the temp collections can be restored when Solr 8.4.1 initializes.

    Change the HELM_RELEASE value to the release name (NOT the version) of your Fusion 5 installation. You can find this using helm list against your Fusion 5 namespace. Locate the release that’s using the "fusion" chart and check the name column. Typically, the release name is the same as your namespace name.

  2. Wait until the prepare-upgrade-solr841 pod shows status Completed.

  3. Set the Solr tag version in the custom values YAML files to 8.4.1.

  4. Verify all *_temp_fix collections are online and healthy.

  5. Run the restore step:

    kubectl run --generator=run-pod/v1 \
  --image="lucidworks/fm-upgrade-query-rewrite:1.x" \
  --restart=Never \
  --env="HELM_RELEASE=<CHANGEME>" \
  --env="ACTION=restore" restore-upgrade-solr841

    The restore step re-indexes the temp collections back into the original query_rewrite collections after restoring the postingsFormat on the tagger field with the new implementation added in Lucene 8.4.1.

    Change the HELM_RELEASE value to the release name of your Fusion 5 installation.

  6. Wait until the restore-upgrade-solr841 pod shows status Completed.

  7. Verify all query rewrite collections are online and healthy.

  8. Delete the prepare and restore pods:

    kubectl delete po prepare-upgrade-solr841
kubectl delete po restore-upgrade-solr841

4. Install Seldon Core CRDs

Fusion 5.1.0 introduces Seldon Core for machine learning (ML) model serving. Seldon Core installs Kuberentes Custom Resource Definitions (CRDs). Due to a limitation in how Helm handles CRDs during upgrades to an existing cluster, you may need to install the CRDs into a temporary namespace before attempting an upgrade to your existing namespace.

  1. Check if the Seldon Core CRDs are present in your cluster:

    kubectl api-versions | grep machinelearning.seldon.io/v1

  2. If this returns no results, create a temporary namespace and install the Seldon Core CRDs into the K8s cluster:

    kubectl create namespace tmp-crd-install
helm install --namespace tmp-crd-install tmp-crd lucidworks/fusion --version 5.1.0 --debug \
  --set "solr.enabled=false" --set "fusion-admin.enabled=false" \
  --set "fusion-indexing.enabled=false" --set "query-pipeline.enabled=false" \
  --set "api-gateway.enabled=false" --set "classic-rest-service.enabled=false" \
  --set "sql-service.enabled=false" --set "zookeeper.enabled=false" \
  --set "job-launcher.enabled=false" --set "job-rest-service.enabled=false" \
  --set "rest-service.enabled=false" --set "rpc-service.enabled=false" \
  --set "logstash.enabled=false" --set "webapps.enabled=false"
helm delete --namespace tmp-crd-install tmp-crd
kubectl delete namespace tmp-crd-install

  3. Verify the Seldon Core CRDs were installed successfully:

    k api-versions | grep machinelearning.seldon.io/v1;

    Your output should resemble the following:

    machinelearning.seldon.io/v1
machinelearning.seldon.io/v1alpha2
machinelearning.seldon.io/v1alpha3

5. Delete Logstash StatefulSet

If you’re running Fusion 5.0.2 or earlier, delete the Logstash StatefulSet. The data remains intact and Logstash restores correctly during the Fusion upgrade. Run the following:

kubectl delete sts <RELEASE>-logstash

Proceed with the upgrade to Fusion 5.1.0. Update the CHART_VERSION to 5.1.0 in your upgrade script.

6. Update Prometheus Scrape Path

Add the prometheus.io/path: "/actuator/prometheus" annotation to the api-gateway, query-pipeline, and fusion-indexing sections of your custom values YAML:

query-pipeline:
  ... existing settings
  pod:
    annotations:
      prometheus.io/port: "8787"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"

api-gateway:
  ... existing settings
  pod:
    annotations:
      prometheus.io/port: "6764"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"

fusion-indexing:
  ... existing settings
  pod:
    annotations:
      prometheus.io/port: "8765"
      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"

General upgrade process

Fusion natively supports deployments on various Kubernetes platforms, including AKS, EKS, and GKE. You can also deploy on a different Kubernetes platform of your choice.

Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments. In this case, see Other Kubernetes deployment upgrades.

Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.

Natively supported deployment upgrades

Deployment type Platform

Azure Kubernetes Service (AKS)

aks

Amazon Elastic Kubernetes Service (EKS)

eks

Google Kubernetes Engine (GKE)

gke

Fusion includes upgrade scripts for natively supported deployment types. To upgrade:

  1. Open the <platform>_<cluster>_<release>_upgrade_fusion.sh upgrade script file for editing.

  2. Update the CHART_VERSION to your target Fusion version, and save your changes.

  3. Run the <platform>_<cluster>_<release>_upgrade_fusion.sh script. The <release> value is the same as your namespace, unless you overrode the default value using the -r option.

After running the upgrade, use kubectl get pods to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:

kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq