Troubleshooting Apache Pulsar Issues

Execute the following command to obtain the number of existing replicas of statefulsets (STS) for both Pulsar Bookkeeper and Broker:

BOOKKEEPER_REPLICAS=$(kubectl get sts -l "app=pulsar,component=bookkeeper" -o jsonpath="{.items[0].spec.replicas}" ; echo)
BROKER_REPLICAS=$(kubectl get sts -l "app=pulsar,component=broker" -o jsonpath="{.items[0].spec.replicas}" ; echo)

Execute the following command to scale down the STS:

kubectl get sts -l "app=pulsar" --no-headers | awk '{print $1}'  | xargs kubectl scale --replicas=0 sts

Execute the following command to delete the Pulsar PVCs:

kubectl get pvc -l "app=pulsar" --no-headers | awk '{print $1}' | xargs kubectl delete pvc

Sign in to Zookeeper and execute the following command:

ZOOKEEPER_NODE=$(kubectl get pods -l "app=zookeeper" --no-headers | awk '{print $1}' | head -n1)
kubectl exec -it $ZOOKEEPER_NODE /bin/bash

Execute the following commands in Zookeeper to run the zzkCli.sh script and delete Pulsar information:

cd bin
zkCli.sh
deleteall /pulsar
quit
exit

Execute the following commands to scale the pulsar-bookkeeper replica count to its previous numbers:

kubectl get sts -l "app=pulsar,component=bookkeeper" --no-headers | awk '{print $1}'  | xargs kubectl scale --replicas=$BOOKKEEPER_REPLICAS sts

When at least one pulsar-bookkeeper created from the previous command is successfully running, execute the following command to scale the pulsar-broker replica count to its previous numbers:

kubectl get sts -l "app=pulsar,component=broker" --no-headers | awk '{print $1}'  | xargs kubectl scale --replicas=$BROKER_REPLICAS sts

When at least one pulsar-broker created from the previous command is successfully running, sign in to that pulsar-broker instance and execute the following commands:

BROKER_NODE=$(kubectl get pods -l "app=pulsar,component=broker" --no-headers | grep '1/1' | grep Running | head -n1 | awk '{print $1}')
kubectl exec -it $BROKER_NODE /bin/bash
cd bin

While still signed into the pulsar-broker pod, execute the following commands to create a:

Execute the following command to restart all applicable deployments:

kubectl get deployments --no-headers | awk '{print $1}' | kubectl rollout restart deployment

Verify the managedLedgerDefaultWriteQuorum field contains an adequate number of bookkeepers for a write quorum. The default value is 2. If necessary, increase the value.

Execute the following command to add more bookkeeper nodes to scale up the pulsar-bookkeeper STS:

kubectl scale -n example example-pulsar-bookkeeper --replicas 4

When a sufficient number of healthy bookkeeper nodes exist for a quorum, execute the following command to obtain the failing bookkeeper cookie IDs from Zookeeper:

kubectl exec -n example example-zookeeper-0 -- bin/zkCli.sh ls /pulsar/ledgers/cookies

$ kubectl exec -n example example-zookeeper-0 -- bin/zkCli.sh ls /pulsar/ledgers/cookies

[example-pulsar-bookkeeper-0.example-pulsar-bookkeeper.example.svc.cluster.local:3181, example-pulsar-bookkeeper-1.example-pulsar-bookkeeper.example.svc.cluster.local:3181, example-pulsar-bookkeeper-2.example-pulsar-bookkeeper.example.svc.cluster.local:3181]

Execute the following command to decommission each of the corrupted bookkeeper nodes:

decommissionbookie

kubectl exec -n example example-pulsar-bookkeeper-0 -- /pulsar/bin/bookkeeper shell decommissionbookie -bookieid example-pulsar-bookkeeper-1.example-pulsar-bookkeeper.example.svc.cluster.local:3181

Restore the readiness probe and liveness probe to their original state if they were removed or modified earlier in this procedure.

Execute the following command to delete the each of the failing pods that were decommissioned. For example:

kubectl delete pod -n example example-pulsar-bookkeeper-1

If necessary, set the value of the pulsar-bookkeeper STS to its original size.

Execute the following commands to scale down the pulsar-broker and pulsar-bookkeeper STS to zero replicas:

kubectl scale -n example sts/example-pulsar-bookkeeper --replicas=0
kubectl scale -n example sts/example-pulsar-broker --replicas=0

Execute the following command to list all of the PVCs for pulsar-bookkeeper pods:

kubectl get pvc -n example -o name | grep pulsar-bookkeeper

Verify the correct bookkeeper PVCs are listed for your cluster/namespace.

Execute the following command to delete each of the PVCs owned by previously-existing pulsar-bookkeeper pods:

kubectl get pvc -n example -o name | grep pulsar-bookkeeper | xargs -I {} kubectl delete -n example {}

Execute the following command to delete all nodes under the /pulsar path in Zookeeper:

kubectl exec -n example example-zookeeper-0 -- bin/zkCli.sh deleteall /pulsar

Execute the following commands to scale the pulsar-broker and pulsar-bookkeeper STS to the appropriate number of replicas:

kubectl scale -n example sts/example-pulsar-bookkeeper --replicas=3
kubectl scale -n example sts/example-pulsar-broker --replicas=2

Restart all remaining Fusion services pods except zookeeper, solr, pulsar-broker, and pulsar-bookkeeper. Based on the Fusion deployment customization for your system, there may be other pods that should not be restarted.

Access the bookkeeper.configData.PULSAR_MEM helm parameter.

Add -XX:MaxDirectMemorySize with the appropriate value.