Setting up a simple Linux VM for internal routing within virtual environment

Essential network components are sometimes the hardest to configure without knowing the basics. When I first started working on a virtual environment, I didn't even know I had to setup networking! It took me weeks to understand fully how all the VMware "parts" speak to each other. The vKernels, vSwitches, vLANs versus VLANs... yeah, that's a story for another time. 

For now, let's talk about the basics. In this post I'll quickly walk you through the process of setting up a virtual router (vRouter) - a Linux VM responsible for most of your internal virtual networking. Once we have that up and going, we can continue our journey through the exciting world of virtual networking inside both VMware vSphere and Hyper-V.

Table of Contents

I. Intro - Why virtual router? (backstory)

II. Setting up virtual router - step-by-step tutorial

III. Alternative setup for quick rollout

IV. End notes

I. Intro - Why virtual router?

One the first challenges I had to face while setting up my first virtual infrastructure was figuring out how to do networking. The diagram below shows how the physical side of things looked like* back then.
*Networks, service names, types and other details about this setup were changes due to internal security policy within the company. 

Internal LAN & WAN (172.18.18.0/24 & 10.0.1.0/24) for the VI were supplied by the DC staff and separated from the parent networks with VLAN tagging and dedicated ports on their switches:

As you can see, the groundwork was quite good, however once you reached the ESXi hosts (and vSphere), you were stuck with one public and one private network provided by the parent DC. This setup quickly became too simple for the following reasons:

• 3 public IP addresses for 10+ public services - as soon as the 1:1 service to IP ratio was exceeded, routing (at least port forwarding) became essential; 
• Security - the default private & public networks supplied by the DC had no firewall configured at all, meaning anyone from anywhere could try to get access to our VMs. On top of that, some services we were running inside the virtual environment required multiple independent (but route-able) networks;
• Testing, VPNs etc. - as the infrastructure main purpose was backing up remote client networks, there was a need of a test site or test environment where recovered machines could be tested and troubleshoot before going back to production. This required additional supporting services such as DNS, VPN and so on to be included into our VI. 

Knowing an usual infrastructure admin, you're probably thinking now "Why the f*** didn't you just get a physical router, switches and so on?". I'm glad you asked, because that's something a sane person would do. Frankly, I hate my life too much to go for an easy option.

On a serious note, there was no time nor resources to upgrade the infrastructure with additional equipment. While funding wasn't an unsolvable problem, I was frankly running out of time. Getting approval for additional hardware purchase, renegotiating contract with the DC (additional rack space) & waiting for the new stuff to come in simply wasn't an option at this stage. I had to set this whole thing up fast.

That's where a virtual router comes in.

II. Setting up virtual router - step-by-step tutorial

1.1. Choosing the OS - Why Linux?

Where do I begin...

Firstly, the costs. As described above, the cost of getting dedicated devices wasn't a problem, so why would I worry about licensing costs? Surely I could get something more robust than a simple Linux distro for a fraction of full-fledged firewall/router price? The answer is...

First, setting up a virtual router on Windows doesn't make sense. There are obviously pre-made, ready for use programs which do the job for you (no need to worry about all the internal stuff), however if you think about it for a second, is Windows really something you'd like to set as the main point of security in your network? Again...

Second option would be getting something pre-made by a renowned company like Cisco or Juniper. At first it seems like a reasonable thing to do, but if you ever dealt with getting a license for Cisco/Juniper virtual appliance you probably know that it would be faster to order one of their physical devices instead... which, as mentioned earlier, I didn't have time for.

I hope you're not thinking at this stage about OSX or openBSD because I don't have any more "No/Nope" images handy. 

1.2. Choosing the OS - Distributions

Now as we got that off our minds, let's focus on choosing a distribution for our Linux box.

There are multiple options (and probably as many opinions) which distro to chose, however as I don't really want the comment section to eventually turn into a battlefield between RedHat, Debian and Arch fanboys, let's group potential candidate distros:

• Redhat/Fedora - obvious choice for most of Linux admins, however please do keep in mind Redhat is a licensed (non-free) distribution, while Fedora might a bit too "cutting edge" for a router (untested features, potential backdoors etc.);
• Debian/Ubuntu - personal favorite for its stability (one of best on the market) and price (free, obviously). Ubuntu is a good pick as well, however I don't see a point in choosing it over Debian for server/service role; 
• Arch - probably a better choice than Debian for experienced Linux admins, however setup time may be longer without a pre-made script or vanilla OVF template;  
• Other - please keep in mind the guide below might not be compatible with your distro of choice.

2.1. Initial setup - OS/VM requirements

Let's quickly run through what a VM acting as a router for our network should have:

• Disk: distribution minimum should suffice, however you might want to add something on top of that for /swap and extra features in the future. I recommend 16GB in total (4GB /swap);
• RAM: in the setup shown on the diagram at the begging of this post, my initial vRouter had about 2GB of RAM memory. This turned out to be enough for a couple of months. If your infrastructure can handle it, I recommend going for 4GB+ straight away. Remember to add extra RAM if your setup includes a desktop environment (Gnome, KDE etc.);
• CPU: one vCPU core should do the job if you don't restrict its share too much;
• NICs: at least two required; one connected to LAN-vDS, the other to WAN-vDS. In my setup, I decided to add three to the VM - the additional NIC was connected to a brand new VM-LAN network through LAN-vDS.

I won't be going through the process on installing a Linux operating system on a VM in this post. If I ever decide to write a post describing this process, it would be linked here*.

*As there is no such article yet, this link redirects to a third-party site with a good enough guide for Linux beginners.

2.2. Initial setup - Planning

Here's how I planned new network setup after vRouter rollout:

You might need to have your virtual network(s) setup differently, but I highly recommend to at least write everything down before continuing.

In the setup above, the vRouter is connected to three networks:

• 10.0.1.0/24 WAN - interfaces eth0, eth0:1, eth0:2 (three sub-interfaces; more in part II/2/c)
• DC Router WAN interface: 10.0.1.1/24
• eth0 10.0.1.10/24,
• eth0:1 10.0.1.20/24,
• eth0:2 10.0.1.30/24;
• 172.18.18.0/24 LAN - interface eth2
• DC Router LAN interface: 172.18.18.1/24
• eth2 172.18.18.2/24
• 172.18.100.0/24 VM-LAN - interface eth1:
• eth1: 172.18.100.1/24;

Interface setup on ESXi hosts:

• ESXi Host1
• Uplink11 & Uplink12: 
• LAN port group with an IP address: 172.18.18.101/24 (LAN-vDS)
• VM-LAN port group with an IP address: 172.18.100.101/24 (LAN-vDS)
• Uplink13 & Uplink14: WAN port group without IP address assigned (WAN-vDS)
• ESXi Host 2
• Uplink21 & Uplink22: 
• LAN port group with an IP address: 172.18.18.102/24 (LAN-vDS)
• VM-LAN port group with an IP address: 172.18.100.102/24 (LAN-vDS)
• Uplink23 & Uplink24: WAN port group without IP address assigned (WAN-vDS)

2.3. Initial setup - Linux interface setup (within OS)

Now with everything properly laid-out, we can start setting up our vRouter.

First thing that needs to be done is setting up the interfaces according to the planned networking. Let's get to it by opening /etc/network/interfaces (or appropriate file in your distro) with root privileges by running the following command:

> sudo nano /etc/network/interfaces

Let's configure the WAN facing interface eth0. Paste the following into the config file and adjust the IPs according to your setup

#The main external WAN interface (eth0)  
allow-hotplug eth0                       
iface eth0 inet static                   
         address 10.0.1.10               
         netmask 255.255.255.0           
         network 10.0.1.0                
         gateway 10.0.1.1                
         broadcast 10.0.1.255            

#eth0:1 sub-interface for public IP no. 2
allow-hotplug eth0:1                     
iface eth0:1 inet static                 
         address 10.0.1.20               
         netmask 255.255.255.0           
         network 10.0.1.0                
         gateway 10.0.1.1                
         broadcast 10.0.1.255            

#eth0:2 sub-interface for public IP no. 3
allow-hotplug eth0:2                     
iface eth0:2 inet static                 
         address 10.0.1.30               
         netmask 255.255.255.0           
         network 10.0.1.0                
         gateway 10.0.1.1                
         broadcast 10.0.1.255            

The sub-interfaces do not require additional NICs to be installed. You probably won't need to include them in your setup if your infrastructure has only one public IP assigned by the ISP. Routing with sub-interfaces/multiple public IPs doesn't make the setup much more complicated.

Next, we will configure the LAN eth2 for communication with LAN interface on DC router, LAN port groups on ESXi hosts and any other devices connected to this network.

#The interface for internal DC LAN (eth2)
allow-hotplug eth2
iface eth2 inet static
         address 172.18.18.2
         netmask 255.255.255.0
         network 172.18.18.0
         gateway 172.18.18.1
         broadcast 172.18.18.255

Pretty straight forward. We need the LAN IP address on DC router as our gateway. Remember to change these values according to your network.

Lastly, we are going to setup the eth1 interface. This is going to be our gateway address for the VM-LAN network so we are not adding the gateway address on this one.

#The interface for internal VM-LAN (eth1)
allow-hotplug eth1
iface eth1 inet static
         address 172.18.100.1
         netmask 255.255.255.0
         network 172.18.100.0
         broadcast 172.18.100.255

We can now save our changes by pressing CTRL+X and then selecting 'y' twice to confirm the edit & filename.

3.1. IPtalbes - Routing

Now the real fun begins. We reached the part where we have to configure the routing with the OS. That includes enabling packet forwarding, firewall rules and some initial routing (basically, what goes where and what shouldn't go anywhere).

To do that, we are going to use IPtables. I recommend doing that "the safe way", by creating a file which is going to hold to all the rules, but it is possible to enter these commands directly into the system.

So, let's start up by enabling packet forwarding by simply running the command:

> sudo echo 1 > /proc/sys/net/ipv4/ip_forward

This is going to enable the OS to "pass" packets to other hosts in the network according to the IPv4 header.

creating a file which is going to hold all the IPtables rules. The file can be located anywhere in the system and the rules within it won't affect the system until we run the iptables-restore command.

> sudo nano /etc/iptables.rules

Paste the following into this file. The following rules are going to enable NATing the received packets over the network.

*nat
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

The *nat marker indicates that the rule should be put into the NAT sub-table of IPtables.

In the next part, we are going to use the IPtables for configuring the firewall (or *filter sub-table).

3.2. IPtables - Firewall

Paste the following into the iptables.rules file:

*filter
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -j DROP
COMMIT

Explanation:

-A INPUT -i lo -j ACCEPT
Accepts all incoming traffic on loopback interface.

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
Accepts only "known" incoming traffic on the WAN interface. This prevents hosts from the internet from initiating a connection to any of our hosts.

-A INPUT -i eth1 -p tcp -m tcp --dport 22 -j ACCEPT opens up port 22 for incoming SSH connections from within VM-LAN (eth1).

-A INPUT -i eth0 -j DROP
Drops all other incoming traffic on WAN interface.

COMMIT
Writes changes to the system.

As the rules are being processed by Linux from top to bottom, we need to firstly "open up" what we need and them "close" all the others. Rules "lower" on the list do not overwrite the ones on "higher" positions.

Save the file with CTRL+X as before and enable the rules by typing in a command:

> sudo iptables-restore < /etc/iptables.rules

Lastly, we need to make sure the command above is going to be automatically run on each startup. There are many ways to do that - one of them is adding

pre-up iptables-restore < /etc/iptables.rules

into /etc/network/interfaces config file, just before #The main external WAN interface (eth0).

4. DHCP & DNS  (optional)

We can setup both DHCP and DNS by installing DNSmasq. To do so, run a command:

>sudo apt-get install dnsmasq

Finish up DNSmasq configuration by editing /etc/dnsmasq.conf file.

>sudo nano /etc/dnsmasq.conf

Paste the following for quick setup (replace IP values & domain with what's relevant to your network):

interface=eth1
listen-address=127.0.0.1
domain=lan.paulwilk.net
dhcp-range=172.18.100.50,172.18.100.100,12h

This will setup DNS & DHCP server on interface eth1. DNS is going to read the records from /etc/hosts file, while DHCP will hand out addresses from 172.18.100.50-100 range with a domain "lan.paulwilk.net" (12h lease).

III. Alternative setup for lazy people quick rollout

1. Script

If you can't be bothered with doing everything from the described above yourself, worry not! There is a workaround. However...

Ok, maybe you're in rush. I get it.

If you really don't want to do it all yourself, you might want to take a look at my github repo. 

This script needs to be run on a Linux VM with two NICs connected (eth0 - WAN, eth1 - LAN).

2. Freesco

Remember the Cisco virtual router I mentioned earlier? Well, some dedicated Linux admins decided to create something similar and share it for free - Freesco Virtual Router Appliance (download).

I think it's not good enough for any serious use, but you can use it for test lab or internal (secured by other firewall) network. Keep in mind this appliance wasn't updated in along time, so security is cr*p.

IV. End notes

I hope you found this tutorial useful. As it's the first one posted on this blog, I'm sure there are some errors, mistakes and typos. If you find any, don't hesitate to contact me using the comment section below.



That would be it then, I guess. I hope your OS won't crash today,
P.W.