Monitoring iptables in realtime

I just had the pleasurable experience of debugging iptables again.

Here is a short oneliner that lets you debug your iptables in realtime.

watch -n 1 "sudo iptables-save -t nat -c"

iptables-save has the convienient flag -c, which is showing it's counters as [packet:byte]

If you have a lot of web traffic on vm's like me you might want to filter out 443. Also rules that do not trigger at all [0:0] can be filtered out.

watch -n 1 "sudo iptables-save -t nat -c | grep -v '0:0' |grep -v '443'"

This gives you a good indication as to wich rules get applied in realtime.
A useful tool for debugging.

Happy hacking :)


Mounting external LUKS encrypted drive on Ubuntu 20

Conveniently in Ubuntu Desktop OS's if you connect an external whole disk encrypted drive, it will ask you to input a password and mount it.

So far so convenient.
However it does not mount all partitions, but ony the boot partition.

This is why you can only see the the kernel, the init ramdisk, grub etc.
To mount the other partitions first use lvscan to scan for logical volumes on all connected block devices.

You can see that one LG (logical group) is currently mounted, the other is not. Using vgchange -ay you can list the active volumegroups to get thier label.

The label is a softlink for a device. You need the devicename to mount the partition. Then check the content of /dev/mapper to see the device name.
Then create a mountpoint and mount the partition.

And now you can access the content of that partition :)

Cheers,
Ori


Integration of collabora online on a Nextcloud behind NAT on a KVM hypervisor

This one had me struggle quite a bit.
I felt like this is something that cannot possibly be super hard to setup.
Before running Nextcloud in a VM I remember the setup to be quite trivial.

And once again it has proven to be true that there are only three networking issues. DNS DNS and DNS. But lets take a step back.

I am running a Nextcloud for my family and wanted to integrate collabora.

This is the How-To I used: https://www.linuxbabe.com/cloud-storage/integrate-collabora-online-server-nextcloud-ubuntu-16-04

Collabora, much like Google Docs or Office365, enables you to edit documents online in your browser. Collabora is a child of Libreoffice Online. The Idea is to still perform well even if 20 people or more are working on a document simultaneously.

The average collabora installation guide is using the collabora\code docker container. It expect you to run it in a "classical" nextcloud setup in wich the Apache or Nginx Server is run on the Aplication Layer of the Host Operating System of the Server.

In such a setup the docker container is running in the kernelspace of the host machine that is hooked up to the internet.

If we now add a layer of virtualisation things become a little more tricky.

The docker container is still running in the same kernelspace as the nextcloud instance. However the NAT is having some effects on the DNS resolution.

The docker container gets run using the hostname for the subdomain that will be used by the webserver for the rewrite rules interacting with the docker container.

sudo docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=nextcloud\\.your-domain\\.com' --restart always --cap-add MKNOD collabora/code

If we run this setup without nat, the docker container can resolve nextcloud.your-domain.com to 127.0.0.1.

Then the container can speak to the nextcloud using the loopback interface.
Behind a nat, this is not working.

A simple change does the trick.
We will have to edit the hosts file inside the docker conainer.

EDIT: Docker has a parameter to perform this task (since version 17) called --add-host. Using this option your changes wont be lost if the container restarts (e.g. if the host reboots). Lets say the IP of your Nextcloud behind the NAT is 192.168.122.10. Then you will have to start the container like this:

sudo docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=nextcloud\\.your-domain\\.com' --restart always --cap-add MKNOD --add-host=nextcloud.your-domain.com:192.168.122.10 --add-host=office.your-domain.com:192.168.122.10 collabora/code

General container debugging

EDIT: I will leave this bit in for general purpose docker container debugging.

Check docker ps -a to find the name the collabora docker container has been assigned. Then start a shell inside the container using docker exec -i -t <ContainerName> /bin/bash.

There we edit the /etc/hosts file and add two lines.
One for office.your-domain.com and one for nextcloud.your-domain.com. It worked best for me using the IP address of the virtual machine.

Beeing lazy I did run apt update and apt install vi to have an editor inside of the container. You could also just echo it into the hosts file.

Afterwards you also need to edit the /etc/hosts file of the VM.
To be able to properly communicate with the container I had to have office.your-domain.com resolve to 127.0.0.1 rather then my public IP.

Cheers,
Ori


Setup Grafana on Ubuntu 18.04 with LetsEncrypt

In this article I will show you how to get the data visualisation solution Grafana to work with clean HTTPS on Ubuntu 18.04.
As alwaysI recommend not running the service natively on your server but rather to run it in a VM.

See: virtualization with KVM

Installation

Simply follow along the instructions of the  official guide on the Grafana website.

LetsEncrypt

To secure our webserver with valid SSL certificates we generate an certificate using LetsEncrypt
Ubuntu comes with certbot installed nativley.

sudo certbot certonly -d your.website

Write down the fullchain.pem and privkey.pem path.
You will later put that into the grafana.ini configuration file.

Before we do that, we have to make sure grafana can access these certificates.
To do that we create a new group.

sudo groupadd sslcerts

/etc/letsencrypt is owned by the user root and the group root.
We will change the group ownership recursivley to sslcerts.

user chown -R root:sslcerts /etc/letsencrypt/

Now we will add the user grafana (added when installing grafana) to this grop.

sudo usermod -G sslcerts -a grafana

Now we will have to adjust the permissions of /etc/letsencrypt/live and /etc/letsencrypt/archive

sudo chmod 755 /etc/letsencrypt/live
sudo chmod 755 /etc/letsencrypt/archive

Editing the configfile /etc/grafana/grafana.ini

You will have to change the following lines:

30 [server]
31 # Protocol (http, https, socket)
32 protocol = https

37 # The http port to use
38 http_port = 443

40 # The public facing domain name used to access grafana from a browser
41 domain = your.grafana.url

47 # The full public facing url you use in browser, used for redirects and emails
48 # If you use reverse proxy and sub path specify full url (with sub path)
49 root_url = https://your.grafana.url

60 # https certs & key file
61 cert_file = /etc/letsencrypt/live/your.grafana.url/fullchain.pem
62 cert_key = /etc/letsencrypt/live/your.grafana.url/privkey.pem

Empowering Grafana to bind 443

The grafana service is not running as root.
This is why in the default configuration a ein highport is beeing used for the webserver.

But we want to use 443...

To do this without granting grafana super user, we explicitly allow it to bind ports below 1024 using setcap.
sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/grafana-server

Further read:
https://wiki.apache.org/httpd/NonRootPortBinding
https://wiki.archlinux.org/index.php/Capabilities

Now, finally, restart the grafana service.

sudo systemctl restart grafana-server.service

If you have done everything right, a clean HTTPS should be greeting you.
If it does not work, a look into the logfile can be quite helpful.

sudo tail -f /var/log/grafana/grafana.log

At this webinterface you can now login using admin admin.
You will be asked to change that password on the first login.

Now you can carry on using this guid: https://grafana.com/docs/guides/getting_started/

Cheers,
Ori


MaSSHandra Installation on Ubuntu 18.04 with LetsEncrypt

In this article I will describe how to get MaSSHandra running on an Ubuntu 18.04.
As alwaysI recommend not running the service natively on your server but rather to run it in a VM.

See: virtualization with KVM

Preperation

First things first.
Update your sources and install pending updates.

sudo apt update -y && sudo apt upgrade -y

Now we install the following packages:

  • Sendmail - will be used to send emails to the MaSSHandra users
  • NodeJS - will run the services
  • npm - Node packet manager, will be used to update NodeJS
  • mysql - an SQL Server, that is going to hold MaSSHandras data

sudo apt install -y sendmail nodejs npm apache2 mysql

Now we empty the npm cache and install the current version of node.

sudo npm cache clean -f
sudo npm install -g n

The command node -v should now be showing a Version above 10.0.

Clone the git repo to your home diretory.
git clone https://github.com/pablomarle/networkmaps

Now we create a few directories MaSSHandra is going to use.

sudo mkdir /etc/networkmaps/
sudo mkdir /sendmail/
sudo mkdir /sendmail/queue/
sudo mkdir /sendmail/sent/
sudo mkdir /diagrams/

SQL Database

First we are going to harden the SQL Database a bit.
Mysql comes with a script that is going to interactivley ask you a few settings to make it a bit more secure.

sudo mysql_secure_installation

Now login to mysql.

sql -u root -p

Create a database that MaSSHandra will later use to handle users.
create database users;

You can ofcourse use another database name then users if you want.
Just make sure that you use this altered name on the database import and later when configurung the config.json.

Logoff by typing

exit;

Import the sql database included in the git repo.
There are no users in there, however a bunch of tables that will handle users, passwords (binary64 with salt), diagrams and permissions.

mysql -u root -p users < ~/networkmaps/database_schema/users.sql

Now log back into mysql.
sql -u root -p

We will now create a SQL user that MaSSHandra can use to access the database.
Please change "YourMasshandraSqlPassword".
Here you can use a username of your choice that later will be put in the config.json.

CREATE USER 'masshandra'@'localhost' IDENTIFIED BY 'YourMasshandraSqlPassword';
GRANT ALL ON Users.* TO 'masshandra'@'localhost' IDENTIFIED BY 'YourMasshandraSqlPassword' WITH GRANT OPTION;

Then reload the sql permissions and exit mysql.
FLUSH PRIVILEGES;
EXIT;

LetsEncrypt

To secure our webserver with valid SSL certificates we generate an certificate using LetsEncrypt
Ubuntu comes with certbot installed nativley.

sudo certbot certonly -d your.website

MaSSHandra configuration

So far so prepearing.
Lets now head over to tweaking masshandras settings.

MaSSHandra is expecting a configuration file at /etc/networkmaps/config.json
So we copy the example config included in the git repo to that location.

sudo cp ~/networkmaps/docs/sample_config.json /etc/networkmaps/config.json

In it, change the settings marked in red:

{
"comment": "This file is expected to be in /etc/networkmaps",
"timers": {
"usertimeout": 3600,
"savediagram": 300
},
"use_ssl_socket": true,
"use_ssl": true,
"socket": {
"address": "IP OF YOUR SERVER",
"port": "3000",
"cert": "/etc/letsencrypt/live/your.website/fullchain.pem",
"key": "/etc/letsencrypt/live/your.website/privkey.pem"

},
"server":
{
"hostname": "your.website",
"port": 3000
},
"staticserver":
{
"hostname": "your.website",
"port": 443
},
"db":
{
"users":
{
"database": "users",
"host": "localhost",
"user": "masshandra",
"password": "YourMasshandraSqlPassword"
}
},
"diagrams":
{
"path": "/diagrams/"
},
"sendmail":
{
"queue": "/sendmail/queue/",
"sent": "/sendmail/sent/",
"server": "your.mailserver",
"port": 465,
"is_secured": true,
"user": "mailuser@your.mailserver",
"password": "YourMailPassword",
"from": "your.website.url <noreply@your.website.url>"
}
}

Starting the server

Now we start the services that will open a websocket on :3000 and handle the emails.

sudo node ~/networkmaps/server.js
sudo node ~/networkmaps/smtp_daemon.js

Remember that you can send the processes to the background by appending & to the command.
Leave them as they are if you want to debug.

When you now head over to your MaSSHandra instance and register a user, you sould see some logs.

Once you confirmed the Email you should be able to login and use MaSSHandra.

Cheers,
Ori


MaSSHandra

In this article I want to tell you about the 3D network diagram editor MaSSHandra.
Its origins are a software project that was cancelled in 2016 by the same name.

Version 2.3 of MaSSHandra was a standalone application for Windows, Linux and Mac.
A commercial license was at 20€, the software was free for private users.

Amongst other features it came with SNMP autodiscovery, access to devices via SSH, RDP and was packed with 3D models.

The new project is also called MaSSHandra and it is web based.
It relies on NodeJS und ThreeJS, which means it will natively run in modern browsers.
ThreeJS is using WebGL, a way to let your browser access parts of your GPU directly.
Very impressive tech.

The MaSSHandra source code is on github under MIT license.

The project itself is still in its early days but is showing a lot of potential if you ask me.
If you think so as well, please donate a coffee to Pablo, the developer of MaSSHandra here.
Or help further developing the software.

In upcomming articles I will show you how to get your own MaSSHandra instnace up and running.

Cheers,
Ori


KVM - resize qcow2

To resize the qcow2 image of a KVM Virtual Machine there is an easy command line tool.
First shut down the VM and check where the qcow image is located.

Using the virt-manager go to view > details > highlight the hard drive > source path

On the comand line you can check virsh edit <VM NAME> to open the XML descriptor file where the qcow is located.

Now you can resize it using the following command.

qemu-img resize path_to_.qcow +??G

In my case this would be:

sudo qemu-img resize /var/lib/uvtool/libvirt/images/vm_haproxy.qcow +20G

If you now start the vm, you can take a look at df -h to find that it worked.
Also it should be quite clear in the Monitoring, in case you monitor the vm.

Cheers,
Ori


VxLAN

I want to present to you a nice YouTube series about VxLAN.

As you can find in my Juniper artikle V-Lan configuration the IEEE 802.1X standard enlarges the frame on Layer-2.
Even if at a first glance adding another 32Bit to every frame seems qute oversized, you will eventually run into its limits.
VxLAN is a way to wrap Layer-2 Traffic inside Layer-4 UDP traffic to surpass V-Lan's limitations.

https://www.youtube.com/watch?v=YNqKDI_bnPM

Ceers,
Ori


Process management from the CLI

#sudo kill $(ps aux|sed -e 's/[ ]* / /g'|cut -f2 -d' ')

 


Mapping users to devices using SMB sessions

If you cannot find out what user is working on a specific device using PSexec or PSloggedon, this trickt might help.

Go to the fileserver, start an administrative powershell and run get-smbsession.
This will list all currently open smb sessions.
This way you can map user accounts to IP addresses.

To filter for the IP address you are looking for you can pipe to findstr.
If I am looking for an IP that has 130 in the fourth octett like 10.20.30.130 it would look like this.

get-smbsession | findstr .130.

Cheers,
Ori