Logging Kafka topics and writing to Elastic Search using Logstash

To install and setup ElasticSearch in local. Click Here

Download LogStash

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.15.2-linux-x86_64.tar.gz

tar -xvf logstash-8.15.2-linux-x86_64.tar.gz

Configure and run it

Create a file called logstash.conf

input { 
  kafka {
    topics => ["test_microservice"]
    decorate_events => 'basic'
  }
}
filter {
        json {
                source => "message"
        }
        mutate {
            add_field => {
                "kafka-topic" => "%{[@metadata][kafka][topic]}"
            }
        }
}
output {
  stdout{}
  elasticsearch {
        hosts => "http://127.0.0.1:9200"
        user => 'elastic'
        password => 'HkieavcIJjSa9EZanrtr'
        index => 'es_index'
  }
}

In the above conf,

Logstash reads messages from kafka topic test_microservice and writes into elasticsearch.

Run Logstash

bin/logstash -f logstash.conf