Legacy Product

Fusion 5.10
    Fusion 5.10

    Configure Grafana, Prometheus, Promtail, and Loki in Fusion

    Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.

    Clone the fusion-cloud-native repository

    Open a terminal window and run following command:

    git clone https://github.com/lucidworks/fusion-cloud-native.git

    Install Grafana

    1. In your local fusion-cloud-native repository, run the following command for your <cluster> and <namespace>:

      ./install-prom.sh -c <cluster> -n <namespace>

      The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.

      Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping

Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded


Successfully installed Prometheus and Grafana into the fe-foundry namespace.

NAME                 	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART                  	APP VERSION
fe-foundry           	fe-foundry	11      	2023-08-07 15:10:55.373825 -0700 PDT	deployed	fusion-5.8.0           	5.8.0
fe-foundry-jupyter   	fe-foundry	2       	2023-07-20 11:29:38.481329 -0700 PDT	deployed	fusion-jupyter-0.2.5   	1.0
fe-foundry-monitoring	fe-foundry	1       	2023-08-10 11:41:06.113257 -0700 PDT	failed  	fusion-monitoring-1.0.1	1.0.1

    2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:

      kubectl get services

      The following is a sample output.

      NAME          TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)          AGE
grafana       LoadBalancer   <IP Address>   <IP Address>  3000:32589/TCP    87m

      If the output does not display, run the following command to expose Grafana, including an EXTERNAL_IP for your Grafana LoadBalancer service:

      kubectl expose deployment <grafana-deployment-name> --type=LoadBalancer --name=grafana --port=3000 --target-port=3000

    Install Loki

    To obtain Loki from the helm chart repository, run the following command for the unique <loki-release-name> for your cluster:

    helm upgrade --install <loki-release-name> --namespace=<namespace> grafana/loki-stack

    If you do no enter the <loki-release-name> correctly, an error similar to the following displays:

    Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".

    If the helm upgrade is successful, the following is a sample output.

    Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370   39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246   39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See http://docs.grafana.org/features/datasources/loki/ for more detail.

    Obtain Admin credentials for Grafana

    1. After you validate Grafana is running by accessing <EXTERNAL-IP>:3000, run the following command to obtain an <admin_password> for your Grafana instance:

      kubectl get secret --namespace <namespace> <release_name>-monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

    2. Sign in to Grafana and change the password for security purposes.

    3. Run the following command to display the promtail pods that are running:

      kubectl get pods | grep -i promtail | nl

      Promtail pods must match the number of Kubernetes nodes since an instance of promptail runs on each node.

    Add the Loki datsource

    1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.

    2. In the Configuration section, click Data sources.

    3. Click Add new data source.

    4. In the search bar for the data source, enter Loki.

    5. In the URL field on the Settings screen, enter your unique <loki-release-name:port>. The default port for Loki is 3100.

      If you encounter issues with the <loki-release-name:port> information, open a terminal and run kubectl get services | grep loki to display a list of every service with a name that contains loki along with its associated IP address and port.

    6. Complete the other fields and click Save & test.

    Additional information

    For more information, see View logs in Self-Hosted Fusion 5.6 and later.