Distributed/Cluster nodes performance monitoring Live using metricbeat

-
Live performance monitoring is the critical part of the big data ecosystem. There are many ways to monitor around it. However, i feel the metricbeat is one of the best choices.
Metricbeat always runs in conjunction with EK(Elastic search and Kibana) stack. It's really special for the platform components (Kubernetes, Docker, fluentd, hyperkube, etcd, HA proxy etc  ), where we have inbuilt monitoring support in Metricbeat. 
Metricbeat collects other system statistics such as CPU, Memory, IO, network stats etc and pushed them into elasticsearch. From Elasticsearch we can visualise them using Kibana.
In my experiment, i set up the Elastic-search and Kibana as isolated system( on 192.168.1.2) and Metricbeat on all the cluster nodes which require performance statistics monitoring.  

Usage

  1. Download the tar from the link here to all the cluster nodes: https://github.com/Indu-sharma/Utilities/blob/master/metricbeat.tar.gz
  2. Extract the tar in a folder:
    mkdir /data/metricbeat
    cd  /data/metricbeat
    tar -zxf /data/metricbeat/metricbeat.tar.gz

  3. Point metricbeat to Elasticsearch/Kibana and change the ES indices:
    Change the following parameters in
    /data/metricbeat/metricbeat-6.1.3-darwin-x86_64/metricbeat.yml
    index: "sec-%{+yyyy.MM.dd}" # Its good to give index name 
    prefix as the project name.
    setup.template.name: "sec" # Same as  Index name prefix
    setup.template.pattern: "sec-" # Same as  Index name prefix  & '-'
    In the Kibana section on the yml file, update the kibana server:
    host: "kibana:5601" ## Note its the hostname of Kibana server 
    not the IP, so all the cluster nodes should have the mapping of kibana hostname to IP mapping in /etc/hosts
    In the 'Elasticsearch output' section update the hostname as
    ELasticserver hostname : hosts: ["192.168.1.2:6775"]
  4. Start the metricbeat service on all the cluster nodes:
    $ /data/metricbeat/metricbeat-6.1.3-darwin-x86_64/metricbeat 
    -v -e &
  5. Open the Kibana UI and configure Visulisations and dashboards:
    http://kibana:5601
    Go to Management tab & create the index. Provide 
    Index name as that of configured in step-3 & make as default index.
  6.  Now,  From Visualisation button create the timeserise -> Visual builder and publish the graphs such as CPU, Memory by process etc to the dashboards. 
  7. Kibana UI Samples: 
A. Top Hosts with CPU usage on cluster and Top Processes  with CPU usage:
B. Top Hosts with RSS memory usage on cluster and Top Processes  with RSS memory usage:
Above statistics are Live, and can be monitored continuously.  Along with default, customisations can be done to automatically monitor/detect the statistics of all spark/MR jobs by dynamically updating the PID value of them in elastic-search with the actual strings as provided in the file 

Comments

Post a Comment