ELK stack consists of a set of applications for retrieving and managing log files. In the software development industry, log files play a vital role to identify the problem and troubleshoot the issue. ELK stack is a collection of different open-source application tools such as Elasticsearch, Kibana, and Logstash. ELK can be used to collect, search and visualize the logs generated from any source in any pattern using a query. In this article, we will learn how to install and configure the ELK stack on Ubuntu and Debian.
Prerequisites:
- Fresh Ubuntu 20.04 or Debian 10 Server
- Root Privileged Account
- Proper Internet connection
Install Java
Installation of ELK stack requires a Java environment. Run the following command to install java on Ubuntu/Debian
$ sudo apt install openjdk-8-jdk
Verify the installation by checking the java version
$ java -version
Output:
Install and configure Elasticsearch
Once java is installed, now it's time to install and configure Elasticsearch. Since Elasticsearch packages are not available by default on Ubuntu/Debian we need to add the elasticsearch apt repository. Run the following command to add the GPG repository key.
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Now create the repository file using the command.
$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Once the repository file is created, elasticsearch can be installed using the command.
$ sudo apt update
$ sudo apt install elasticsearch
The default configuration file of elasticsearch is located at /etc/elasticsearch/elasticsearch.yml . Use any text editor and uncomment the lines:
network.host: localhost
http.port: 9200
Start and enable elasticsearch
$ sudo systemctl start elasticsearch
$ sudo systemctl enable elasticsearch
Run the following command to view Elasticsearch status and details
$ curl -X GET "localhost:9200"
Output:
Install and configure Logstash
Logstash package is available by default in Ubuntu/Debian systems. Run the following command to install.
$ sudo apt install logstash
Start and enable the service
$ sudo systemctl start logstash
$ sudo systemctl enable logstash
Check the service using the command
$ systemctl status logstash
Default configuration directory of logstash is /etc/logstash/conf.d/ . Once the installation is completed INPUT, FILTER and OUTPUT pipelines can be configured based on required use cases.
Install and configure Kibana
Kibana is a web based GUI tool used for analyzing and parsing the collected logs. Kibana is available in the default repository of Ubuntu/Debian. Run the following command to install the package.
$ sudo apt install kibana
To configure kibana, go to the default configuration directory and uncomment the following lines
$ sudo vim /etc/kibana/kibana.yml
server.port: 5601 server.host: "localhost" elasticsearch.hosts: ["http://localhost:9200"]
Start and enable the service
$ sudo systemctl start kibana
$ sudo systemctl enable kibana
Allow the kibana port in the firewall
$ sudo ufw allow 5601/tcp
Now access the Kibana dashboard using the url http://localhost:5601
Install and configure filebeat
Filebeat is used for sending logs to elasticsearch and logstash for parsing. Filebeat is available by default in Ubuntu/Debian repository. Run the following command to install.
$ sudo apt install filebeat -y
To configure the filebeat, go to the default configuration directory and comment out the following.
$ sudo vim /etc/filebeat/filebeat.yml
# output.elasticsearch: # Array of hosts to connect to . # hosts: ["localhost:9200"]
Uncomment the following line and save the file
output.logstash: hosts: [“localhost:5044”]
In the next step, enable filebeat system module
$ sudo filebeat modules enable system
Now run the following command to load the index template
$ sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
Start and enable filebeat service
$ sudo systemctl start filebeat
$ sudo systemctl enable filebeat
Check the status
$ sudo systemctl status filebeat
Conclusion
In this article, I have covered how to install and configure the ELK stack on Debian/Ubuntu in the correct way. Also, we have learned how to use different components such as Kibana, Logstash, and Kibana to analyze and visualize the logs from any source.