Splunk in My Homelab: Part One

Over time my homelab has grown to over 40 virtual machines spread over three subnets. All the boxes are active and I want more insight into what is going on, so I’m adding a Splunk server. Over the next three blog posts, I’ll walk you through setting up and searching Splunk. This first post is going to cover registering for, downloading, and installing the Splunk software. The next post will cover installing the forwarders giving Splunk something to index. Lastly, the third post will cover adding apps and querying data.

Before we begin, lets quickly look at the parts of Splunk. The layer closest to us is the search head. This is the front end that we interact with via search and the Splunk web UI. The indexer is the heart of Splunk. The indexer receives data, compresses it, and then indexes it. The last part is the forwarder. This is a Splunk instance on an end point that generates machine data. The forwarder forwards the data to the indexer. Typically you would separate the roles, but because of the small amount of data and for simplicity sake, I’m putting search head and indexer on one box.

 

First go to splunk.com and register an account. Its free, but be aware they may ask to validate your email, in case you like to use throw away ones. Once logged in, click the big green free Splunk button. I chose the Splunk enterprise server free download.

 

I’m using Ubuntu 16.04 for my Splunk server, so I will chose Linux. At this point, you can download the .deb file, but as the second screen shot shows, you can also use wget to download the bits right to your box. That is what I will be doing. You’ll also notice that I pointed out the data limitation. This is not an issue for me since I’m just curious about my traffic and don’t actually need more than that. But if you want more and are willing to follow up, Google “Splunk Developer License.”

After running wget from the screen shot above run the following:

#Install splunk enterprise server
dpkg -i splunk-6.6.2-4b804538c686-linux-2.6-amd64.deb
#If you receive any errors looking for dependencies run this
sudo apt-get install -f
#Now start the splunk server
sudo /opt/splunk/bin/splunk --accept-license
#Finally set splunk to start on boot
sudo /opt/splunk/bin/splunk enable boot-start

Now navigate to http://splunkserver:8000.  Success!!!

In the next post, I will be installing a universal forwarder which will provide some data that I will later query in part three of this series.

PuTTYgen and an Ubuntu Server

Recently I used PuTTYgen to create a key pair which I intended to use to connect to one of the Ubuntu servers I have in my home lab. Not paying attention to detail, I uploaded the public key, ran

cat key.pub >> .ssh/authorized_keys

and was surprised to find that I was not able to connect. It turns out the formatting of the key is different in subtle ways. I’m going to cover how I manually modified the key so it would work, and then later found an easier way via ssh-keygen.

Manual Process

First I opened the key in Atom, an alternative to sublime text which I favor (when not using Vim that is 😉 ).

Turns out *Nix systems expect no other formatting or characters apart from the key itself. So I had to remove white spaces, comments and control characters. I manually removed all the words and comments. Then I hit find and searched via regex for carriage return and line feed. \r\n in the picture. I left the replace blank as I wanted to remove everything except for the key and replace it with nothing. You can see where Atom highlighted the space we would be removing.

Then I added ssh-rsa followed by a space at the start of the file. This signals the file encoding being used. Check RFC 4253 secion 6.6 (link) for a detailed, albeit boring, read about why we use ssh-rsa. If I want a comment, I can add a space followed by a comment at the end of the line.

Finally, just to be sure there were no issues with EOL between Windows and Linux, I installed an Atom package called line ending converter. To do this in Atom, I went to settings, chose install and searched community packages for the package. Once it is installed, it can be found under packages at Convert Line Endings to.

So now I have an acceptable key. I ssh to the box via username/password, open .ssh/authorized_keys, paste the key, and I’m good to go.

Or, if you’re the more streamlined type, this whole blog post can be done in one step. But, you wouldn’t learn what’s going on under the hood, and that’s the fun part.

ssh-keygen -i -f putty_key > new_key

Proxy Variables

While setting up squid proxy on my pfSense home lab gateway, I had trouble getting apt-get update to work on my Ubuntu/snort box which was behind the proxy. After some quick Googling, I tried the first response (because the first response is always the best right 😉 ), and the first response failed. After reading a few more blogs, I noticed there were many different ways to setup a proxy properly. Here’s what I found.

Setting up a proxy on the command line starts by declaring the proxy environment variable. Applicable variable options are:

http_proxy
https_proxy
ftp_proxy
socks_proxy
all_proxy

Next, check if you currently have a proxy already set.

$ env | grep -i proxy 

If you get nothing from the command above, you know you don’t have a proxy yet. If you only want http and ftp to go through your proxy, export those variables instead of some of the other options above.

$ export {http,ftp}_proxy="http://proxy_name_or_ip:port_number"

This will only export the variable for the current session. If you log out or reset the computer, you will lose the proxy setting. If you need to make it permanent, use /etc/environment which is Ubuntu’s system-wide location for environment variables. You could also put it in /etc/profile.d as a file since this directory is ultimately read by /etc/profile. It’s not best practice to set it in /etc/bash.bashrc because variables in this file are specific to shells. Finally, if you want only a specific user to receive the proxy, you should set it at ~/.bashrc

If you want to read more about proper placement of environment variables, read this Ubuntu Environment Variables.

Now add your settings to /etc/environment

 echo "http_proxy=http://proxy_name_or_ip:port_number" >> /etc/environment;\
 echo "ftp_proxy=http://proxy_name_or_ip:port_number" >> /etc/environment 

If your proxy requires a username and password, the following format is often used:

echo "http_proxy=http://username:password@proxy_name_or_ip:port_number" >> /etc/environment

Most of the time you would be done at this point. But I had an issue with APT where I had to set the proxy in the APT configuration file.

I had to edit /etc/apt/apt.conf and add

Acquire::http::proxy "http://proxy_name_or_ip:port_number";

If you are curious about how to configure YUM similar to APT, you need to edit /etc/yum.conf

Once the file is open, add these lines to the section [main]

proxy=http://proxy_name_or_ip:port_numaer
proxy_username=jamey
proxy_password=PW

While we’re talking about proxies, CNTLM is another proxy that you install locally and point your proxy variable to localhost. It is a middle-man proxy that sits between you and a proxy that requires NTLM authentication. I have found this incredibly helpful when using Linux in a Windows environment. It’s a really cool piece of software and really easy to setup.

It’s in the Ubuntu repositories.

apt-get install cntlm

Its configuration file is found at /etc/cntlm.conf

Add the following:

Username        jamey
Domain          jamey.info
Proxy           proxy_ip:proxy_port
Listen          127.0.0.1:proxy_port

Next we create password hashes.

cntlm -H

The output should look like this

PassLM          ACF337F47B2E22ED552C4BCA4AEBFB11
PassNT          2A22CC95E275BE3150326D0C1E86A58E
PassNTLMv2      F001B46C503A3A01611D2859EBEA8762    # Only for user 'jamey', domain 'jamey.info'   

Copy/paste your output to /etc/cntlm.conf

Finally configure the local proxy variable as we did above to point to localhost instead of an external proxy.

export http_proxy=http://127.0.0.1:3128