hobby multi-isp serving routing scheme revisited

After my last two posts about multi-isp serving over 2 routers, some time has went by and one of my isp’s dsl-modem slash router changed and went ballistic with fixed settings. They decided that nobody ever wants a different subnet from 192.168.2.0/24, and therefor dump everything else to the outside interface. This didn’t only make a router inbetween very much needed (since I didn’t want to change my local network besides a /16 mask), it also made my network a lot more complicated.

Since the router would ditch everything going to 192.168.0.0, I decided to connect the secondary network interfaces (that happened to be configured to 192.168.2.0 as well) to the network as the serving interface. However, as usual this doesn’t matter when it comes to outgoing traffic. Anything it considers outside its network is send to the default interface instead, eventhough the incoming packets came from the other interface.

At some moment of clarity, I found that I could split up 192.168.2.0 into 2 networks as well, namely in this case (since the isp’s dsl-router has its local ip set fixed to 192.168.2.254) into networks 192.168.2.128/25 and 192.168.2.0/25. This allowed everything beneath 192.168.2.128 to be sent to the internal network, and everything above, including the dsl-router, to belong to the other side.

Basically like this:

The trivial stuff

interface FastEthernet0/0
 ip address 192.168.2.130 255.255.255.128
 ip access-group defaultallin in
 ip mask-reply
 ip nat inside
 ip route-cache flow
 duplex auto
 speed auto
 no cdp enable
!
interface FastEthernet0/1
 ip address 192.168.2.101 255.255.255.128
 ip access-group defaultallout out
 ip mask-reply
 no ip redirects
 ip nat outside
 ip route-cache flow
 duplex auto
 speed auto
!
ip classless
ip route 0.0.0.0 0.0.0.0 FastEthernet0/0 192.168.2.254
ip route 192.168.0.0 255.255.255.0 FastEthernet0/1
ip route 192.168.2.0 255.255.255.128 FastEthernet0/1
!
ip access-list extended defaultallin
 permit ip any any
ip access-list extended defaultallout
 permit ip any any

Note: default interface setting “no ip proxy-arp” is negated and thus on.

The headache stuff, yet in the end not that hard

The normal way of NAT is to e.g. “ip nat inside source list 1 interface FastEthernet0/0 overload”, to replace the source ip, with the ip of interface fa0/0, if the packet is permitted by list 1.
Where list 1 is often casually defined as “access-list 1 permit any”, allowing everything to be translated.

A line like “ip nat inside source static tcp 192.168.0.10 80 interface FastEthernet0/0 80″ is often used to change the destination ip/port to 192.168.0.10 if a packet arrives at the external interface fa0/0 on port 80.

In this particular case, we still want a “normal” NAT source ip translation from the inside to the outside (because we need to force our server to not send traffic to the default gateway), but not All traffic, since we still want to use the other ISP for regular Internet traffic when we fancy such a thing. Unfortunately a standard access-list doesn’t allow us to be very specific unless we want to define the source. Since the source ip could be anything (because the traffic is coming from the Internet), we can’t use a standard ACL.

We can however use a Extended ACL, which allows to be specific about the Destination IP and portnumber, which goes as followed:

ip nat inside source list ourspecialnat interface FastEthernet0/0 overload
!
ip access-list extended ourspecialnat
 permit tcp any host 192.168.2.2 eq www

And there you have it. A specific permission for destination 192.168.2.2 port=80, to be translated. Now all we need to do is to set a portforward from our ISP’s dsl-router from port 80 to 192.168.2.2. It routes traffic inside because the ip matches its 192.168.2.0/24 network, and our special nat router makes sure the packets are sent and received via  the same route due to obscuring the source ip.

PS. Perhaps it should be noted that this is just a funny exploit in search of better solutions. Most of the time you will want to know the actual IP address a visitor of your website is using, and not get a local IP instead. Chances are however that setting up either your server or your network to remember which route they should be going (with an IPv4 NAT-ted network) is considerably more difficult. When no traffic is translated in the first place (like for example when you have a large number of IPv4 addresses or a couple of IPv6 addresses), there wouldn’t be an issue of rejecting source/destination mismatches, because they would always be the same.

Yes, I was wrong

… you can’t make traffic go one way for one port, and another way for some other port.

Why? Because you’re most likely stuck with switched traffic that only sends traffic down one cable when it figures out a certain network-address is connected to that cable.

However, I did manage to subtract one router out of the equation with the wondrous discovery of selective NAT. Aparantly you can change the default source 0.0.0.0 ACL to a specific per port list. The ip’s and ports that don’t match the ACL won’t be translated. Which is exactly what I was looking for.

This does require that with every port forward from your isp’s router to be added to the ACL of the 2nd router, but that won’t be that much of a problem.

And when your internet breaks down…

… you start connecting that console link again and start typing to get that other internet connection to work.

In this post I explained I had made a network of 3 routers so that my web-/database-/whatnot-server was able to serve over

2 different external IP’s via 2 different ISP’s (one broadband, one dsl). It had a hickup however in that the 2nd ISP couldn’t be used for normal Internet surfing etc. This was because it has a NAT configuration upside down to avoid the default-router issue.

So now I have connected a 4th router to do normal routing between the internal 192.168.0.0 network and the other network 192.168.2.0 to get to the router that connects to the dsl isp (still following?)

It proved to be quite simple (feel a bit silly to use the router for such easy work). It just routes to and from both network interfaces, with the

exception that it blocks all traffic to and from the server (because that one should use the other router with the upside down NAT configuration).

And then it just works. (well, you need to manually set the gateway of your client ofc.) I could probably make the server be able to connect to the other isp as well as long as I make sure the proper ports are blocked… But that’s stuff for at a later time.

Serving over multiple external IP’s

Normal people tend to just rely on 1 single internet connection to do both websurfing and server hosting, and some might even wonder if 2 internet connections is even going to work.

Let me explain why you can’t use 2 external ip’s in a normal 2x modem-router-network configuration.

To make things easy, I’ll give the actors some names in this example;

• Alice is the person who’s visiting your website
• Fred is the name of your internal web server
• Lucy is the name of the first router, and is the default router for outgoing traffic
• Peter is the name of the second router

When you configure Peter to forward/translate a specific port (eg tcp 80 for http) you supply the internal IP address Fred, and everything just works. Peter copy/pastes Fred’s mac-address into the IP-packet, changes the port number if needed, and sends it to your network. The external source IP address of Alice is still there in the packet, and Fred will use that IP address to send packets back Alice containing your website and other things.

When Fred tries to figure out on how to reach Alice, he figures he should just use the default router that is supplied in his network configuration, so he puts the MAC address of Lucy into a packet along with Alice her IP address and sends it. Lucy has NAT to go onto the big bad internet, so instead of Fred’s IP, she puts her own external IP address into the packet and adds some reference material into her memory so she knows that returning packets will have to go to Fred.

However, when Alice finds out that the external IP address is from Lucy when she wanted to see the website that Peter promised, she hits the panic button and tells Lucy to leave her alone, and tries to talk to Peter again – ignorant of the fact that Lucy was just relaying the same website.

Solution 1

Get a very expensive router that has 2 seperate WAN ports that remembers which way a certain stream of packets was going and sends things back on the same path. Unfortunately I don’t have one of those, so I can’t tell you how that exactly works, nor if it actually does work.

Solution 2

Get a third router and set it up to serve with a not so very logical NAT configuration.

From the example given you can see one occasion where a router does actually remember information about previous packets. Lucy masks her outgoing packets to pretend the packets are coming from her external IP address instead of an internal IP address. And you can abuse this exact mechanism to force webserver Fred to send his packets to the right router.

How that exactly works:

• outside has an external IP
• inside has an internal IP
• NAT masks packets going from the inside to the outside

outside        outside
 Peter          Lucy
inside         inside
   |             |
   |             |
   |        192/168.0.0/24
   |             |
192.168.2.0/16   |
   |             |
   |             |
 inside          |
 Markus          |
outside          |
   |             |
    \            |
      \          |
      192.168.0.10
         Fred

When Alice requests a website via Lucy’s outside IP address, Lucy directly sends it to Fred, and Fred has Lucy has his default router and thus sends it back through that route.

When Alice requests a website via Peter’s outside IP address, Peter wants to send it to Fred, but he can’t find Fred. Markus however tells Peter to just give it to him, and he’ll give it to Fred. Then Markus masks Alice her IP address with his own outside IP address, and sends the packet to Fred. When Fred wants to send something back, he thinks Markus sends the packet, and just sends it to him. But Markus checks his memory and remembers he should send it to Alice. All Markus needs to know now is that he needs to send things through Peter to get to Alice.

The downside

The unfortunate by-product of using a third router is that you can’t connect the 2nd internet connection directly to your local network, because then traffic wouldn’t be masked. So to use the 2nd internet connection as a Backup internet gateway, you’ll have to use a 4th router with a very careful routing scheme to not conflict with the internal nat router. I haven’t tried that out yet, but I have another Cisco 1841 lying here I might give it a go with as soon as I can get my hands on another switch…

Cisco IOS DHCP Manual bindings for dummies

So I was screaming and ranting (not really, but dramatic effect always works… right?… what?.. it doesn’t? hmm…)

I use a Cisco 385 router (or is it 580, 850, 857, I can’t remember) to keep my network together and connect it to the outside world. It does a great job at most trivial things (except for IPv6 which is non-existant, and thus it will soon be replaced by a 1841 probably) and I have fun forwarding ports and whatnot.

The thing that however puzzled me was the fact that manual IP/DHCP bindings for computers in my local network were sometimes accepted, and sometimes not. I had “read” (as in read fast and inaccurate) the manual and came up with simple entries like:

ip dhcp pool afancyname
host 192.168.0.10 255.255.255.0
hardware-address 0012.3456.789a
client-name afancyname

And after fiddling around with settings in Windows for weeks, I finally started actually reading the manual and stumbled upon the difference between “hardware-address” and “client-identifier”.

The big difference (next to a simple 01 prefix to the mac-address) is that hardware-address is for BOOTP, and client-identifier is for DHCP. Client-Name is also BOOTP only, and should not be used.

So I looked up the Client-identifier via “sh ip dhcp bindings” and wrote the following config bit instead:

ip dhcp pool afancyname
host 192.168.0.10 255.255.255.0
client-identifier 0100.1234.5678.9a

Et voila, release and renew your IP address client-side, and you have your preferred IP address.

However, ofcourse this is only because Windows supplies the client-identifier argument with its DHCPRequest packet, and Linux for some reason does not. So for Linux hardware we need to fallback on hardware-address (unless you manually edited your dhclient.conf file it seems):

ip dhcp pool myfancylinuxpc
host 192.168.0.11 255.255.255.0
hardware-address 0012.3456.789b