rogulski.it

My name is Piotr, a passionate pythonista and this is my blog!

    Troubleshooting and debugging knative with kafka

    Posted at — Jan 24, 2023

    Useful commands

    1. Display list of knative services

      kn services list

    2. Display config for specific service

      kn service describe --verbose <SERVICE_NAME>

    3. List all revisions

      kn revisions list

    4. Display triggers

      kn triggers list
kubectl get triggers

    5. Display kafka sources

      kubectl get kafkasource

    6. Run Kafka Producer container

      kubectl -n kafka run kafka-producer -ti --image=strimzi/kafka:0.14.0-kafka-2.3.0 --rm=true --restart=Never -- bash

    7. List kafka topics

      bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVER_HOST> --list

    8. List messages produced to a specific topic

      bin/kafka-console-consumer.sh --bootstrap-server <BOOTSTRAP_SERVER_HOST> --topic <TOPIC_NAME> --from-beginning

    9. List messages from knative broker topic

      bin/kafka-console-consumer.sh --bootstrap-server <BOOTSTRAP_SERVER_HOST> --topic knative-broker-default-default --from-beginning

    10. List all consumer groups in Kafka

      bin/kafka-consumer-groups.sh  --list --bootstrap-server <BOOTSTRAP_SERVER_HOST>

    11. Display all consumer groups with details

      bin/kafka-consumer-groups.sh --describe --all-groups --bootstrap-server <BOOTSTRAP_SERVER_HOST>

    12. List offset with lag for each knative topic

      bin/kafka-consumer-groups.sh --describe --group knative-group --bootstrap-server <BOOTSTRAP_SERVER_HOST>

    13. Count the lag of the offset from previous group command

      bin/kafka-consumer-groups.sh --describe --group knative-group --bootstrap-server <BOOTSTRAP_SERVER_HOST> | awk 'NR>1 {sum += $6} END {print sum}'

    Debugging flow

    1. Check if your source topic is working:

      • Run kafka-producer (6)
      • Produce kafka message to the source topic
      • Check if message arrived to the topic (8)

    2. Check if the Trigger is properly connected/filtered

      • Check status of triggers (4)
      • Check filter defined in yaml file and check headers in the consumer service if it is matching the query

    3. Check if your events are copied into broker topic

      • Run kafka-producer (6)
      • Produce kafka message to the source topic
      • Check if message arrived to the topic (8) - if default topic name is used, the name is: knative-broker-default-default

    4. Check if pods in knative-eventing namespace do not raise errors e.g. kafka-broker-dispatcher

    Useful resources/tools