Not all Uplinks are made equal

In this article I want to remind you of an effect that we sometimes tend to forget.
I am talking about the limited publicly available IPv4 addresses.

If you want to privately host a service like VPN, or have a customer that only has a consumer line, you should be aware of this fact.
To handle the problem of having more end-users then public IPs the Internet Service Providers are using a technique called NAT.
When an ISP is "natting" their users, it is called Carrier Grade NAT.
This means that the large pool of internal IP (v4 or v6) addresses is being translated to a smaller pool of public addresses before being transmitted.

This results in multiple end-users using the same public IP at the same time.
The ISP only has to remember what open port belongs to what session e.g. internal IP.
NAT in a nutshell. (pun not intended)

In addition to that most of the consumer lines today only use IPv6.
In order to make IPv4 services available to hier customers the ISPs are using what is called a DS (Dual Stack) Lite Tunnel.
This is resulting in the private IPv6 address facing one of the public IPv4 addresses of the ISP.
The IPv4 packets get encapsulated in IPv6 and then forwarded to the ISP sending it out as IPv4.
The other way around the IPv4 response is then again encapsulated by the ISP and forwarded to the customer.

So much for the background of whats going on.
Lets say I am checking what IPv4 address i am currently surfing on and try to connect to it using port 8080.
My ISP is not expecting anyone to contact him on that IP on port 8080 (since he is most likely not holding a session there) and cannot tell wich one of the end-users I want to contact.
The packets get dropped or rejected.

You will have to tell your ISP that you want to provide services over the internet and ask him to deactivate or rather change the Carrier Grade NAT for you.
Consumer hotlines usually know this as "changing / disabling the DS Lite Tunnel" or simply "enable port forwarding".

If the ISP is going to do that for you depends on the number of public IPs that the provider has available.
As far as I know you have no right to claim, that the your carrier deactivates the NAT for you.
Since hardly any of the end-users are asking for this however, ISPs normally have no problems enableing this 1:1 NAT for you.

Once this has been setup, the request to 8080 (or any other) can be directly associated to your internal IP and gets forwarded.

Please remember that will also have to enable port forwarding in your router behind the consumer line, as this one is also using NAT to masquerade your internal IP Addresses.
Also on some consumer devices there are settings like "filters" or "parenting controls" that limit fiddle with your portforwardings.
It's actually quite a s**t-show to properly configure some of these end-user routers.

Also your public IP most likely rotates, just saying.

Cheers,
Ori


Bad, bad VMware tools

Sometimes you come across things that have the potential of letting fecal matter hit the rotary impeller.
Let me warn you.

Imagine you have successfully completed the ESXi upgrades, systems are back alive, nothing broke and everything is fine and dandy.
After a while you are thinking "shit I forgot to update the VMware Tools on those Servers".
In this purely hypothetical scenario you connect to the Hypervisor select your VMs, right klick and select Guest > update VMware Tools.

A couple minutes later the VMs reboot. (!)
No warning, no check if that is fine with you, the VMs just reboot.

Conclusion: To make sure you do not melt some customers productive systems, plan your VMware Tools updates along with the ESXi upgrade downtimes.

Holy shit,
Ori


Export data from AzureAD using PowerShell

In this article I want to show you how Data can be exported from Azure AD using PowerShell
At first you have to connect your PowerShell with AzureAD.

To do this use the PowerShell commandlet connect-azuread.
This should spawn a pop-up asking you for authentication.
Use Azure Credentials that have the permissions needed to at least read all properties.

If the connection worked it should look something like this

Getting the data

Here is a good reference for PowerShell commandlets related to AzureAD.

We use get-azureaddevice to get a list of the devices in the Azure Domain.

As you can see the devices do have an ObjectID, a DeviceID and a DisplayName.
These are by far not the only Infos that are available regarding devices.

To get an overview of what values get stored for each device just take one at random (-objectid) and pipe to format list (fl).

In this case I would like to export data on the attributes DisplayName, ObjectId, ObjectType and ApproximateLastLogonTimeStamp.

These can be listed for all devices using the following command.
get-azureaddevice | select DisplayName, ObjectId, ObjectType, ApproximateLastLogonTimeStamp

Exporting the data

For better visibility you can export this info to a CSV file.
To do this pipe STDOUT to export-csv and use a local path to store the file.

In my case the command looks like this:
get-azureaddevice | select DisplayName, ObjectId, ObjectType, ApproximateLastLogonTimeStamp | export-csv C:\temp\foo.csv

Now we can open the file, for example with excel.
Just start excel and select Data > From Text / CSV.

In the import Dialoge select comma separated import and load the file.

Using this import method you can sort columns alphabeticaly.

Cheers,
Ori


NCurses Disk Usage

In this article I want to show you a neat little tool: ncdu.
Ncdu stands for NCurses Disk Usage and helps you get an fast overview of the disk usage.

Normal way

Normally I would check the disk using df (disk free) with the parameter -h for human readable.

df -h

Then take a look at the filesystems either having the highest usage (Use%) or that store the most data (Used).
This way you can start digging deep layer by layer du(wiki) to find the cause of unusual disk usage.

ncdu

The genious thing with ncdu is, that it searches directories and subdirectories to give you a neat overview, sorted by size.
This way you can crack down on issues way faster.

First you should be updating your sources, apply pending upgrades and then install ncdu.

apt update -y && apt upgrade -y && apt install ncdu

Once this is done, the program can be run from the shell.

ncdu

Cheers,

Ori


Install firmware updates

In this article I want to show you how to run a firmware update on a juniper router.
First you have to download the firwmare.
In this case EX3300.

You can run the firmware update using a USB stick that is FAT32 formated that is exactly 2GB sized.
Using a 2GB Fat23 partition will not do the trick and you can end up crunching the filesystem of the router...

Alternativley you can choose your a Web- or FTP server to be the source of the update files.
I choose to use the HFS Webserver, feel free to use what ever suites you best.

Firmwareupdate

After installing the webserver, load the .tgz firmware file into its root directory and copy the link to it.
As you can see on the switch I am checking the currently running firmware version (12.3R12-S7)

> show version

After that I start the installation of the new firmware

> request system software add <http link>

Depending on the device this process can take 15 minutes or more.
When the upgrade is complete you will be asked to reboot the device.
Please do not just pull the cable, give it a graceful reboot.
(Trust me)

> request system reboot

When your reboot is done, check that the new version (15.1R6-S3) has been applied.

Cheers,
Ori

 

 


Find connected devices

In this article I want to show you how you can find devices that are attached to a juniper router.

Search the MAC table

You can print out the entire mac table using this command

> show ethernet switching table

If you want to know what is connected to a specific interface use this

> show ethernet switching-table interface ge-0/0/2

If you know the MAC of the device you are looking for, this command will find it

> show ethernet switching-table | match <string>

If you only want to see what devices are connected to the Gigabit Ethernet interfaces do this
> show ehternet switching table | match ge-

Be careful... if you use aggregated interfaces (AE) or glas modules (SFP) this search pattern would not work.

Cheers,
Ori


Vlan configuration

In this article I will show you how to configure V-Lans on a Juniper Router.
VLANs are virtual networks defined by the IEEE Standard 802.1q.

The idea is to run separated networks on the same physical infrastructure.
This gets accomplished by adding another 32Bit to the frame on layer 2, containing the V-Lan related information.


It is important that all devices that are supposed to forward tagged frames do support 802.1q.
If a device is unable to handle frames of that size it will simply destroy them.

Creating a VLAN

First we check if V-Lans on that device already exist.

#show vlans

Now we define two V-Lans named first und second.
First
is going to be assigned the VLAN-ID 10.
Second
is going to be assigned the VLAN-ID 20.

# set vlans first vlan-id 10
# set vlans second vlan-id 20

Lets take a look at the V-Lan config.

root# show vlans
default {
    l3-interface vlan.0;
}
first {
    vlan-id 10;
}
second {
    vlan-id 20;
}

This output should be pretty self explanatory.
Before making an interface become a member in one of these V-Lans we should take a look at the difference between tagged and untagged interfaces.

Tagged and untagged interfaces

V-Lans can be configured in two modes on an interface.

Access (native / untagged):

  • Every interface can only be of type Access for one V-Lan
  • An Access Interface receives untagged traffic and attaches a V-Lan tag to that traffic before forwarding it
  • You will usually use this setting to connect clients or devices that are unable of speaking V-Lan

Example: Gigabit Ethernet Interface 10
This interface will receive untagged (normal) traffic and forward it with the V-Lan Tag "10".
The V-Lan Tag will removed before sending traffic outbound.
For example broadcast traffic for V-Lan 10 or traffic for a device located behind this interface (ARP table).
This way the device on the other end can understand the traffic and does not even know it ever had a V-Lan tag attached to it.

ge-0/0/10 {
    unit 0 {
      family ethernet-switching {
          vlan {
            members first;
          }
      }
   

Trunk (tagged):

  • If an Interface is defined as a trunk it receives and sends tagged frames for certain V-Lans
  • Usually interfaces that have other V-Lan capable devices on the other end get defined that way

Example: Gigabit Ethernet Interface 12
This interface receives tagged traffic for the V-Lans 10 and 20.
If traffic that is supposed to be sent to that network, the V-Lan Tag will be attached to the frame.

Important: A Trunk Interface does not need to be a member in all V-Lans!

ge-0/0/12 {
    unit 0 {
      family ethernet-switching {
         port-mode trunk;
            vlan {
               members [ first second ];
            }
         }
      }
   }

Setting up an IP address on a V-Lan Interface
Please make sure to make the Unit ID the same as th VLAN ID to minimize confusion!!
Your colleagues will hate you if you don't.

At first we will configure a virtual interface with an IPv4 address

#set interface irb unit <UNIT ID> family inet address <IP/subnetmask>

Then we make that interface a member of a V-Lan
# set vlans <VLAN NAME> l3interface irb.<Interface ID>

Then we define this interface to be of type access in this V-Lan

# set interface ge-0/0/2.20 family ethernet-swtiching port mode access

In this case the commands could look like this:

# set interface irb unit 10 family inet address 10.0.10.1
# set vlans 10 l3interface irb.10
# set interface ge-0/0/2.10 family ethernet-swtiching port mode access

If we now connect a cable to the interface ge-0/0/2 our traffic will be sent to V-Lan 10.
Since the we configured the router to have an IP address on the layer 3 interface irb.10, we can now connect to it via ssh on 10.0.10.1.

Cheers,
Ori


Basic Setup

In this article I want to show you how to configure a basic meraki setup.

Devices

I am going to use the following devices:

  • Firewall - MX65
  • Switch - MS120-8PL
  • AccessPoint - MR33

Registering the devices

At first you will have to log in to your Meraki Account.
After the first login to your account you will be facing a pop-up that is asking you to register your devices.

Select to Register Meraki devices and then click on Next.
Since on a new account there are no networks that you can add these devices to, you will now be presented the Create Network wizard.
Give the Network a name and go to Add devices to claim devices for this network.

There will be another pop-up, asking you to enter the serial numbers of the devices you wish to add.

You will find these, as shown in the dialoge, on the devices or on the boxes they arrive in.

After you have claimed them, you can review the added devices before creating the network.

Cabeling

You could configure the devices before performing the cabeling, but in this case I am not going to.
In a real world scenario this is a big advantage as you can already configure the devices before the hardware arrives at the customer.

At first we connect the uplink with the subtly labeled interface named "Internet".

Then we connect the firewall to the switch.
I like to use eth1 on both devices for that.

Then we connect the Access Point with one of the POE Interfaces.
(In this case they are all POE Interfaces)

The status LED is showing you the current state of the device.

When the devices are starting they are connecting to the meraki servers.
If they find a configuration on the servers that is more recent then the one that they are currently holding, they apply it.

After a while the devices should be online and be visible in the meraki portal.

 

Renaming devices

When you log in to the Meraki portal you should now see the left hand side navigation bar.
Your organisation and network are already selected.

Go to Network-wide > Topology.

If your devices successfully connected to the Meraki Cloud they should appear as green.
If not, the devices either have no route to the internet, did not complete booting yet or you have some other issue like a license problem or a hardware defect.

You will see that the devices are still named after thier MAC addresses.
We should change that.

Go to the device that you want to rename, perform a mouseover and click on the devices name.

Here you can click on the pen symbol, change the name and add the correct address.

Especially  when you are using dozens of sites with hundreds of devices it really pays out if you use this feature.
Also you should upload Floor Plans at Wireless > MonitorMap & floor plans and place the Access Points on them.
This really eases troubleshooting when you are trying to figure out why a specific client is roaming like crazy or why a certain corner seems to have bad wifi.

The Topology View should now look like this.

Configuring the Access-Point

Next up we will take a look at what SSIDs the Access Point will send out.
To do this we change to Wireless > Configure > SSIDs.

You will find that there already is an active SSID on your Access Point.
We will rename it and save the configuration.

This SSID is currently "Open" and we should give it a password.
To do this go to Access control > edit settings right under the name.

Most of these settings you do not have to touch on a first setup.
I might go into them in other articles.

When editing the settings go to Network access select Pre-shared Key with WPA-2 and choose a password for your wifi.

In the Addressing and Traffic section I would suggest to use Bridged mode.
This way all clients will be in the same Natwork. .
If you are using the default NAT mode verwendet, all clients connecting to the Access Point will be put to into a separate network.
In this separate network the devices are isolated from one another as well as the internal network and can only use the Internet.

This is perfect for a guest wifi and a good default for an unconfigured SSID.
(Even though I think it is unprofessional that the devices have an open SSID as their default setting...)

Do not forget to save your configuration changes!
This still is not enough for the clients connected to the wifi can communicated with clients in the LAN.

This first SSID has, in addition to the default of isolating the clients via NAT, another surprise for you.
Go to Wireless > Configuration > Firewall & traffic shaping

Hooray! A layer 3 firewall rule that denies traffic of wireless clients accessing the LAN.
Also this rule CAN NOT be deleted. You can change it from Deny to Allow however.


Why did I not just ask you to close this SSID and use one of the 14 others, that do not have those presets?
Because it is important to know about them!

 

Setting the timezone

The last thing we need to do is set the timezone.
Go to Network-wide > Configure > General

I choose Berlin.

This is important so the automatic firmware updates get applied to the devices on the time you would expect them to.
You can configure that at Network-wide > General > Firmware upgrades.


This should be enough for a basic Meraki setup.

Cheers,
Ori


Synology NAS: Reducing the rights of domain admins

In this article i want to show you how to reduce the rights of domain admins on domain integrated synology NAS.

Why should you domain join a NAS?

The basic idea is to centralise user management.
When users and groups from AD are imported and used you can revoke a users access rights simply by deactivating his AD account.

Also you can provide read / write premissions to AD groups, and in the future only have to add users to those groups in AD.

 

Why would you want to revoke the rights of admin users?

It is a likley scenario that a crypto trojan or other malicious software at some point somehow gains domain admin privileges.
I have seen this a couple times.

In a case like this you do not want you NAS to be rendered useless.
Especially if you are using it to store Backups.

 

Problem

A Syno NAS usually get configured using the webinterface.
Here you can join it to your domain using Control Panel > Domain / LDAP > Domain.

Now you can change share permissions for users and groups imported from your domain.
To do this go to Shared Folder > Edit > Permissions and select Domain Users or Domain Groups.
However the fields "Read only" and "Custom" are grayed out for Users in the DOM\Domain-Admins group.

 

Solution

Go to the domain configuration again and click on Domain Options.
Here you can go to Domain Administrators and kick out the default group.

Then you can change the permissions correctly.

Dont forget to create a user for your backup solution that has write access to the share.

Cheers,
Ori


Troubleshooting of windows server hard disk extensions / ESXi

This article desribes how to handle issues you can run into when extending windows server hard disks on ESXi.
In this perticular case the C:\ Partition should be extended from 139GB to 156GB.

After the extending the disk the C:\ Partition to 156GB the Explorer and Monitoring still show that the volume size is 139GB.

 

 

Troubleshooting

Always check that there is enaugh disk space remaining on the storage.
If a full storage is not the cause of this issue, keep looking.

A reboot always is a good idea, however in a production envoirement that is not always possible and does not help in this case.
A good trick is to extend the harddrive for one more GB.

After a refresh of the disk management utility using F5 there should be 1GB that is showing as "unallocated".

After extending the volume again, issues like this are normally solved.

Do not wonder that the harddrive is showing as 156GB angezeigt wird...

... this is Windows, and there is a 100MB reserved partition in front of C:\ and the explorer only counts full GB.

 

Cheers,
Ori