greggm263
Posts: 1
Joined: Fri Mar 20, 2020 11:11 am

Elasticsearch (ELK)

Fri Mar 20, 2020 11:14 am

Has anyone installed Elasticsearch (ELK) on a Raspberry pi 4 successfully? If so, would you happen to have the steps/procedure?

Thanks

GW

DirkS
Posts: 10345
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: Elasticsearch (ELK)

Fri Mar 20, 2020 3:09 pm

Their website says it's available for Win/Mac and Linux x86-64
No mention of Arm and/or Pi...

IanS
Posts: 248
Joined: Wed Jun 20, 2012 2:51 pm
Location: Southampton, England

Re: Elasticsearch (ELK)

Sat Mar 21, 2020 8:27 pm

I got it running on a 3B+, albeit in a very limited fashion. Moving it to a 4GB Pi4 made a big difference to performance. You really need the extra memory. Google for installation methods - that's how I found working instructions.
You cannot run the latest version of the stack. I believe it went to 64bit only at some recent point. I am running version 5.6.15 of the stack.

freeridetheworld
Posts: 5
Joined: Mon Apr 20, 2020 1:50 am

Re: Elasticsearch (ELK)

Mon Apr 20, 2020 1:54 am

I can help you setup Elasticsearch on a Raspberry Pi 4 if needed. Feel free to reach out to me. I have it running but I'm in the process of fine-tuning my notes so the step by step process might need a little work. Overall I have a pretty good working setup.

Vini_
Posts: 1
Joined: Wed Apr 29, 2020 6:08 pm

Re: Elasticsearch (ELK)

Wed Apr 29, 2020 6:19 pm

I came across your post about running elk on a raspberry pi. Is it worth the effort?

I need to do a little exercise with logs from my home firewall. Do you think it can cope with a little load?

IanS
Posts: 248
Joined: Wed Jun 20, 2012 2:51 pm
Location: Southampton, England

Re: Elasticsearch (ELK)

Mon May 04, 2020 5:09 pm

Home firewall logs is what I use my ELK stack to process. Having the 4GB of RAM makes a big difference if you want to run queries that return more than a day or two of data.

freeridetheworld
Posts: 5
Joined: Mon Apr 20, 2020 1:50 am

Re: Elasticsearch (ELK)

Mon May 04, 2020 6:45 pm

Vini_ wrote:
Wed Apr 29, 2020 6:19 pm
I came across your post about running elk on a raspberry pi. Is it worth the effort?

I need to do a little exercise with logs from my home firewall. Do you think it can cope with a little load?
I just got 7.6.2 running on a Rpi 4B this weekend. I need a bit of time to go through my command history and fine-tune the process but I think I could have this done by Thursday night. Does that work for everyone here? I just don't want to put out directions that don't work because I rushed through them. I for one can't stand when steps get left out or glossed over so I'll try to have them perfected by Thursday and post on here.

geektechstuff.com
Posts: 37
Joined: Sat Mar 02, 2019 8:08 pm
Contact: Website

Re: Elasticsearch (ELK)

Tue May 05, 2020 12:09 pm

greggm263 wrote:
Fri Mar 20, 2020 11:14 am
Has anyone installed Elasticsearch (ELK) on a Raspberry pi 4 successfully? If so, would you happen to have the steps/procedure?

Thanks

GW
The whole ELK stack or just ElasticSearch and Kibana? An older version of ElasticSearch can be installed using:

Code: Select all

sudo apt-get update

sudo apt-get upgrade

sudo apt-get install default-jre

curl https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.3.deb -o elastic.deb

sudo apt install ./elastic.deb

sudo service elastic start
www.geektechstuff.com

freeridetheworld
Posts: 5
Joined: Mon Apr 20, 2020 1:50 am

Re: Elasticsearch (ELK)

Tue May 05, 2020 2:41 pm

I have Elasticsearch 7.2.0 and 7.6.2 up and running. I'll take your word that those instructions work for 5.5.x as I havent tried it yet but I don't have much of a desire to go that far back in the versions. I have not setup Kibana on the Rpi 4B yet as I purchased a LattePanda which is Windows based and I'm running it on that. The idea being that you learn how to install beats etc on both major platforms. Everything I have is running in a C4 Labs case. I'd be interested in seeing the instructions for installing Kibana but while I was working through this project Kibana seemed, keyword seemed, a lot more difficult to install on Rpi than Elastic so I just went with the LattePanda to save me time and provide a better teaching platform for the whole product.

freeridetheworld
Posts: 5
Joined: Mon Apr 20, 2020 1:50 am

Re: Elasticsearch (ELK)

Fri May 08, 2020 3:44 am

Below is my complete installation setup guide for Elasticsearch on RPi for versions 7.0.0 to 7.3.2. Scroll down to the bottom for the elasticsearch installation. I had 7.6.2 working but I've run into java/JNA errors when I double-checked my steps and now I can't get it to work. Hopefully, I will be able to post the installation steps for 7.4.0-7.6.2 soon. I am running a three node RPi 4b+ elasticsearch cluster with a single node Kibana cluster running on a Windows based LattePanda. All of this is housed in a C4 Labs 8 bay case.

My Linux skills are not the greatest and I am really summarizing some of the links that I've come across on the discussion forums. Big thank you to Jason_Baumgartner, glassman (David Glass) and anyone else who posted in the threads below.
https://discuss.elastic.co/t/elasticsea ... 4-b/187976
https://discuss.elastic.co/t/elasticsea ... n/200535/7
https://discuss.elastic.co/t/installing ... r/202599/9

***Note: If you do not need these step feel free to skip them.
microSD and Raspbian Buster setup:
Download the first version in the upper left called "Raspbian Buster with desktop and recommended software" zip file from https://www.raspberrypi.org/downloads/raspbian/
------
Download balenaEtcher from https://www.balena.io/etcher/ and OPEN it after installation in order to format the microSD card for the Raspberry Pi's correctly
• ***Note: If the Windows cmd/admin access screen pops up just say Yes
------
Click "Select Image" in balenaEtcher and find the Raspbian Buster image that you downloaded (2019-09-26-raspbian-buster-full). ***Note: There is no need to unzip the Raspbian Buster file. Also, If you get a bunch of errors or "Format Disk" questions when you insert the microSD card close them all out and let balenaEtcher run through its steps. If you get a "Failed device" warning in belnaEtcher proceed with the installation. Error seems to be a false positive.
------
Click Flash, wait until complete
------
Remove the microSD card adapter from your computer
------
Repeat these steps as needed as you add nodes to the cluster.

Insert micro SD card into the Raspberry Pi slot located on the back side of the single board computer
------
Attach HDMI cable to the Pi and monitor/TV. ***Note: You may need an adapter for your HDMI cable as the Raspberry Pi has microHDMI ports as its output.
------
Connect the power supply or ethernet cable to your Raspberry Pi if you have a PoE switch & PoE hat. Power is supplied to the Pi via the switch thanks to the Power over Ethernet (PoE) hat that we put on the Pi.
------
Insert the Raspberry Pi into the appropriate bay in the C4 Labs case
------
***Note: Write down the IP address of your pi here. This can be found right above the Next button on the welcome screen.
Your Raspberry Pi should boot to the "Welcome to Raspberry Pi" welcome screen. Take note of the IP address in the bottom right hand corner above the Next button. Click, Next.
(Bay 1 LattePanda LattePanda 169.254.102.95)(Bay4: 169.254.70.17) (Bay5: 169.254.44.165) (Bay6: 169.254.98.9)(Logstash 169.254.114.75)
------
Set your country, language and timezone. If you are in the US click on "Use US keyboard" so that you don’t end up with the English keyboard which has a few characters in different places. This can cause a lot of confusion with passwords, trust me I know. :)
------
Set the password, if you'd like to use the default password its raspberry (username is pi). This of course is not recommended but it’s the easiest way to get things started. Click, Next.
------
If you would like to remove the black border check the box and click next. This step isnt necessary but visually its more appealing if you want to use this GUI at some point.
------
Ideally you have a WiFi network to connect to. If you do select the WiFi network that you'd like to use and click next.
Enter the password for your Wifi network and click next.
------
***Note: The update might take a while so if you would like to take a break here I would click Next and let the RPi update or you can click Skip and continue with the Elasticsearch/Beats installation. Once the "System has been updated", restart the machine.
------
Setup Complete, click Later if you would like to continue or wait until the RPi comes back from its restart.
------
Click on the Raspberry icon in the top left hand corner, click on Preferences-->Raspberry Pi Configuration-->Interfaces (tab)-->Enable SSH-->OK. Restart/Reboot?
• This will enable us to Secure Socket Shell (SSH) into the Pi from the LattePanda.
------
Repeat these steps as needed as you add nodes to the cluster.

In order to access my RPi I use the free edition of MobaXterm on my Windows based LattePanda. If you do not need these step feel free to skip to the Elasticsearch installation.
MobaXterm, Java & Elasticsearch Installation:
Download, install and open MobaXTerm from https://mobaxterm.mobatek.net/download.html
• Teaching Note: This should have be downloaded by the student prior to class or taken off of the thumb drive.
------
Configure MobaXTerm
• Click on Session in the upper left hand corner under Terminal
• Click on SSH in the upper left hand corner
• Remote host should be the IP address of your Rpi and the username should be pi, click OK.
○ If the correct information has been entered you should get a MoTTY Security Warning, click Yes here.
○ Once you enter the password have Moba save it
• Welcome to the command line of your Raspberry Pi!
------
java -version
• This will allow us to check that the correct java version has been installed which you should
------
Elasticsearch
Some of the following directions can be found online through the following link. https://www.elastic.co/guide/en/elastic ... t/deb.html
I believe that these steps will work for 7.0.0 through 7.3.2 but I have not tested each version. ***Note: I will add the 7.4 and higher installation instructions in this thread at a later time.
• wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - ***Note: This should respond with OK
• sudo apt-get install apt-transport-https
• echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
• cd ~
• wget https://artifacts.elastic.co/downloads/ ... -amd64.deb
• sudo dpkg -i --force-all --ignore-depends=lib6 elasticsearch-7.3.2-no-jdk-amd64.deb ***Note: You may see some download errors here
• Change the permissions for the /etc/elasticsearch folder for easier access
• sudo chmod g+w /etc/elasticsearch
• sudo chmod 755 -R /etc/elasticsearch
• sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch
Edit the elasticsearch yaml file
• sudo nano /etc/elasticsearch/elasticsearch.yml ***Note:This is where you edit the node settings for Elasticsearch.
  • Clustername: elasticpi
  • Node.name: <name your node, i.e. bay1>
  • Path.data: /var/lib/elasticsearch
  • Path.logs: /var/log/elasticsearch
  • Network.host: <computers ip address>
  • Http.port: 9200
  • discovery.seed_hosts: ["169.254.xx.xx","169.254.xx.xx","169.254.xx.xx"] ***Note: If you are creating a cluster you need to add all of the node IP addresses here.
  • xpack.ml.enabled: false // add to yaml at the bottom
  • node.master: true // add to yaml at the bottom
  • node.data: true // add to yaml at the bottom
  • node.ingest: true // add to yaml at the bottom
  • discovery.type: single-node // ***Note: Comment this out if you are building a cluster. This line is temporary and used if you just have one RPi.
  • bootstrap.system_call_filter: false ***Note: This is needed to bypass some of the checks that Elasticsearch does.
§ Ctrl + o, enter, ctrl + x
• sudo nano /etc/default/elasticsearch
  • Edit the JAVA_HOME section to have JAVA_HOME=/usr/lib/jvm/java-11-openjdk-armhf
  • Ctrl 0, enter, Ctrl + X
    • sudo nano /etc/elasticsearch/jvm.options
    • Change the -Xms1g and -Xmx1g to half of the memory that your system has. For a 4B its should be changed to -Xms2g and -Xmx2g
    • Ctrl + 0, enter, Ctrl + X
    • sudo systemctl start elasticsearch
    • sudo systemctl status elasticsearch

freeridetheworld
Posts: 5
Joined: Mon Apr 20, 2020 1:50 am

Re: Elasticsearch (ELK)

Sun May 17, 2020 12:50 pm

Seems like theres also some good news on this front from Elastic.
https://github.com/elastic/beats/pull/17301
https://github.com/elastic/beats/issues/18334

Return to “Advanced users”