Vendor Lock-in with public addresses

With IPv4 most companies have an internal  network with private IP addresses (RFC1918). On the edge of their networks they have routers and firewalls which perform NAT for the internal network. The public IP addresses, often assigned from their ISP, are mostly configured on the router and firewalls. When a company changes ISP and thus needs to change the public IP addresses they only need to change the router and firewall configuration.

With IPv6 this is a bit different. When you enable IPv6 on your network, you probably got an IPv6 subnet assigned  from your ISP. The big difference with IPv4 is that with IPv6 public” IP addresses are configured on all your devices in your network, not just on the router or firewall. Your routers and firewalls will not perform NAT, so each device will have an IPv6 address which is publicly available on the internet.

When you change ISP all IPv6 addresses which are configured on your servers will not work anymore. These addresses are present in the DNS server and cached on your clients and maybe even hard coded in configuration files on your servers. Changing IPv6 addresses on your servers can cause a disruption in your network.

The solution for this “vendor lock-in” problem is Unique Local, which is like RFC1918 for IPv6. You can configure your network with your generated “Unique Local” IPv6 prefix. A Unique Local address starts with FD. The second part consists of 40 bits which are generated as specified in RFC 4193. Together with FD this will make your Unique Local Prefix, for example FD31:DFC3:E7D9::/48.

Unique Local address will also be added to the DNS server. When clients ask for your server they get multiple AAAA records, one with the public IPv6 address and one with the unique local address. When your clients have an IPv6 address in the unique local prefix, they will automatically use the unique local address to connect. You can also use these unique local addresses in your configuration files. When you change ISP, and your public IP addresses will change, your unique local address stays the same for each device so there is no disruption in your local network.

Because a unique local prefix is ::/48 you can use it at all your (remote) locations throughout your organization. The reason that you need to generate a prefix with a special procedure is to be unique in the world. There are situations where you would want to connect different organizations, for example with an IPSEC site to site tunnel, and would like to use unique local addresses. The benefit of using unique local addresses is that you will be sure it will not be routed over the internet.

I’ve created my own Unique Local GlobalID Generator which you can find here.

IPv6 and IPv6-aware but disabled applications

If you enable IPv6 in your network please note that Windows Vista and above prefer IPv6 over IPv4. This means that when your client asks for a website running on your server and an AAAA record is present your client will try to reach the server over IPv6. When you enabled IPv6 in your network but did not enable IPv6 for all IPv6 aware applications clients can experience time delays or even messages that applications cannot connect.

A good example is a Windows 2003 server with IPv6. When you enable IPv6 on the Windows 2003 server the DNS server will hold an AAAA record for this server. When you connect from a Windows 7 client with remote desktop to the Windows 2003 server it will take a long time to connect. Why? Because the RDP client is an IPv6-aware application and because it gets an IPv6 address in the DNS query result, it expects the server listening on IPv6. After a time out it will know that the server is not listening on IPv6 and will try to connect with IPv4.

What if your client applications take 30 seconds or more to start because of IPv6? Your users will not be happy and complain. The “solution” here is to make your servers and clients prefer IPv4 over IPv6. This way you can enable IPv6 without applications slowing down. When only IPv6 is available IPv6 will be used though.

The reason that IPv6 is prefered over IPv4 is more a political decision then a technical decision. At all our client sites where we have custom applications and a mixed environment of servers (2003, 2008, 2008r2) we have enabled this “fix”.

You can set the following registry entry (which ofcourse can be set in a group policy):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\
DWORD (32-Bit): DisabledComponents  
VALUE: 0x20

The fix can be found here on the microsoft site: http://support.microsoft.com/kb/929852/en

Enable IPv6 on remote locations and in your tunnels

When IPv6 is enabled your DNS is filled with AAAA records. Clients with IPv6 enabled will by default prefer IPv6 over IPv4. This means that when your organization has multiple remote locations you should be aware that enabling IPv6 can have an impact on the remote locations. With IPv4 a lot of organizations have IPSEC tunnels between locations. When you enable IPv6 in your organization clients will try to use IPv6 to connect to the server. When IPv6 is not properly configured for the IPSEC tunnels the clients will try to connect over the internet.

So, the tip here is to be sure that when you enable IPv6 you should think of each aspect of your network!

Privacy extensions and IP Security

Windows Vista and Windows 7 have privacy extensions enabled by default. This means that a client has multiple IPv6 addresses, one of them being the temporarily address. The client will register its normal IPv6 address in the DNS server but for outbound connections the temporarily address is used. When you rely on logging and/or matching IP address with the DNS name you should disable privacy extensions for your clients (e.g. trough a group policy).

You cannot simple make something up. According to RFC 4193 you need to follow a special procedure to generate a uniqe local Global ID. Create an SHA1 hash from a EUI64 identifier and the NTP64 date. Then take the first 40 bits of the hash to add them to the 8 bits (FD). This will be your Unique Local Global ID.

An output example with the help:

Title : Unique Local GlobalID Generator
Author: Henk van Achterberg (henk@vanachterberg.org)
Ver.  : 1.1.0.0
Web   : http://www.vanachterberg.org/
Descr.: Generates Global ID's for FC::/7 (Unique Local)
      : according to RFC 4193 (http://tools.ietf.org/html/rfc4193)
Usage :

UniqueLocalGenerator.exe -n<amount to generate> -f[filename] -o[1: formatted, 0:unformatted -d[debug]

Example:

UniqueLocalGenerator.exe -n10 -fglobalids.txt -o1

And with for example 10 prefixes:

C:\>UniqueLocalGenerator.exe -n10
FD11:01AD:20D2::/48
FDAA:9B97:0133::/48
FDF1:64D8:2264::/48
FD7B:8CA0:3776::/48
FD42:586E:02EF::/48
FD60:7EE7:A024::/48
FD02:F773:3C06::/48
FD31:DFC3:E7D9::/48
FDF2:A590:1080::/48
FD40:F274:DFD4::/48

And if you do not want the nice IPv6 subnet output but plain hex:

C:\>UniqueLocalGenerator.exe -n10 -o1
FD4FDA8C857C
FDB0CF9745DE
FD45F881811E
FD59DF8567CE
FDFA7021FF85
FD1B527BD6DD
FD882DAC6B4E
FDB42643B85B
FD2E9A5BBA1F
FD835A8C476F

You can download the application here:

UniqueLocalGenerator.zip

If you use Unique Local GlobalID’s please register them at this site: http://www.sixxs.net/tools/grh/ula/ .

To solve this issue we disable DAD for Windows Vista / Server 2008 or Windows 7 / Server 2008 R2. We need to find the interface number of the interface where we want to disable DAD.

Fire up a command prompt and type this command:

netsh interface ipv6 show interfaces

You will receive a list with all your interfaces. Look for the interface you want to disable DAD for and note the ID in the IDX column. Run the following command to disable DAD on the interface:

netsh interface ipv6 set interface "<interface ID>" dadtransmits=0

You can look up the current value with this command:

netsh interface ipv6 show interface "<interface ID>"

I hope this information is helpful, it was sure for me!

Our ISP has given us the 2a00:1450:7000:200::/64 subnet for our LAN. Our WAN side is 2a00:1450:a1:1::/64 where 1 is the ISP router and 2 is reserved for us.

First we configure the outside interface, which is VLAN 2:

ASA(config)# interface vlan 2
ASA(config)# ipv6 enable
ASA(config)# ipv6 address 2a00:1450:a1:1::2/64
ASA(config)# ipv6 address fe80::2/64 link-local
ASA(config)# ipv6 suppress-ra

We specify two addresses, the public address and the link local address. The link local address is optional although I like to specify it anyway. We do not want to send router advertisements on the WAN side so we suppress them with the suppress-ra statement.

To allow our ASA to access the IPv6 internet we enter the default route on the outside interface to point to the IP of the ISP:

ASA(config)# ipv6 route outside ::/0 2a00:1450:a1:1::1

We can now ping IPv6 hosts on the internet:

ASA# ping ipv6.google.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2a00:1450:8005::93, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms

We configure the inside interface, VLAN 1 with the IPv6 address:

ASA(config)# interface vlan 1
ASA(config)# ipv6 enable
ASA(config)# ipv6 address 2a00:1450:7000:200::1/64
ASA(config)# ipv6 address fe80::1/64 link-local
ASA(config)# ipv6 nd prefix 2a00:1450:7000:200::/64

As you can see it does not differ much with the outside interface. The big difference is that we want to send out router advertisements with the prefix which can be used to generate IPv6 addresses for clients. We use the ipv6 nd prefix command for that.

When the ASA has a route for ::/0 the ASA will advertise itself as a default router on the LAN.

Again, IPv6 does not use NAT so please be careful and design your access lists well!

We have the following configuration:

Internet interface: Redundant Ethernet 8 (reth8), which consists of two physical interfaces

LAN interface: Redundant Ethernet 4 (reth4), which consists of two physical interfaces

We have two routing instances:

vrISP with interface reth8

vrLAN with interface reth4.100 (VLAN 100)

We will use the following IPv6 addresses:

WAN router ISP: 2a00:1450:8888::1/64

WAN router customer side: 2a00:1450:8888::2/64

Subnet that will be used for the LAN: 2a00:1450:9000:100::/64

Note: The ISP is routing 2a00:1450:9000::/48 (this space is given to the customer) to 2a00:1450:8888::2. We will use a /64 subnet for the reth4.100 interface.

At first we need to enable IPv6 globally at the SRX.

Enable IPv6:

root@SRX# set security forwarding-options family inet6 mode flow-based

When you enter the commit command you will receive a notication that the SRX needs to reboot before IPv6 will be active. When you are running a cluster make sure you reboot all your routers in this cluster.

When the SRX has rebooted we can assign the WAN IP address.

Assign Ipv6 address for reth8:

root@SRX# set interfaces reth8.0 family inet6 address 2a00:1450:8888::2/64

If you run a cluster please disable Duplicate Address Discovery. Otherwise the SRX will not function because it detects a duplicate on the line.

Disable DAD on interface reth8:

root@SRX# set interfaces reth8.0 family inet6 dad-disable

We now set the default route for the vrISP routing instance to the WAN ip address of the ISP. As you can see we specify the routing instance twice, first to indicate we work in the vrISP routing instance and secondly to indicate we want to add the route to the IPv6 routing table (inet6.0) of the vrISP routing instance.

Set the default route for vrISP to the ISP:

root@SRX# set routing-instances vrISP routing-options rib vrISP.inet6.0 static route ::/0 next-hop 2a00:1450:8888::1

We will now specify the IPv6 address for the LAN interface, I also did the dad-disable on the LAN interface because of the cluster.

Assign IPv6 addresses to reth4.100:

root@SRX# set interfaces reth4.100 family inet6 address 2a00:1450:9000:100::1/64
root@SRX# set interfaces reth4.100 family inet6 dad-disable

Optionally you can set the link local address.

Assign link local address to reth4.100:

root@SRX# set interfaces reth4.100 family inet6 address fe80::100:1/64

We want our hosts to know which prefix they can use to generate IPv6 addresses. We configure the routing advertisement on the LAN interface.

Enable routing-advertisements with the LAN prefix on reth4.100:

root@SRX# set protocols router-advertisement interface reth4.100 prefix 2a00:1450:9000:100::/64

We need to configure a default route for the vrLAN routing instance to the vrISP routing instance to be able to pass IPv6 traffic to the internet.

Set the default route from the LAN to the vrISP:

root@SRX# set routing-instances vrLAN routing-options rib vrLAN.inet6.0 static route ::/0 next-table vrISP.inet6.0

root@SRX# set security nat destination pool IPv6-LAN routing-instance vrLAN
root@SRX# set security nat destination pool IPv6-LAN address 2a00:1450:9000:100::/64
root@SRX# set security nat destination rule-set ISP from interface reth8.0
root@SRX# set security nat destination rule-set ISP rule IPv6-LAN match source-address ::/0
root@SRX# set security nat destination rule-set ISP rule IPv6-LAN match destination-address 2a00:1450:9000:100::/64
root@SRX# set security nat destination rule-set ISP rule IPv6-LAN then destination-nat pool IPv6-LAN

To allow traffic inbound to the LAN so you can access services, e.g. http, we need to create a firewall filter to redirect the incoming traffic to the correct routing instance.

Configure a firewall filter to pass incoming IPv6 traffic to the vrLAN:

root@SRX# set firewall family inet6 filter IPv6 term LAN from destination-address 2a00:1450:9000:100::/64
root@SRX# set firewall family inet6 filter IPv6 term LAN then routing-instance vrLAN

To allow other IPv6 traffic to pass to other interfaces via global routing, or at the reth8 interface we need to create a default action to allow traffic for IPv6 traffic.

Enable unmatched IPv6 traffic to pass:

root@SRX# set firewall family inet6 filter IPv6 term default then accept

This input filter needs to be applied to the reth8.0 interface.

Apply IPv6 firewall filter to reth8.0:

root@SRX# set interface reth8.0 family inet6 filter input IPv6

Please note that when you add a second subnet in this filter you will need to move the default statement AFTER the newly added statement otherwise the traffic will not be redirected to the routing instance.

We have now a working IPv6 connection. The only thing you should do is set up policies to allow only the traffic you want on the interfaces.

When I want to enable IPv6 (outbound AND inbound) for my lab network (vrLAB, vlan 200 on reth4) I just need to enter a few lines of configuration.

Add LAB configuration:

root@SRX# set interfaces reth4.200 family inet6 address 2a00:1450:9000:200::1/64
root@SRX# set interfaces reth4.200 family inet6 address fe80::200:1/64
root@SRX# set protocols router-advertisement interface reth4.200 prefix 2a00:1450:9000:200::/64
root@SRX# set routing-instances vrLAB routing-options rib vrLAB.inet6.0 static route ::/0 next-table vrISP.inet6.0
root@SRX# set firewall family inet6 filter IPv6 term LAB from destination-address 2a00:1450:9000:200::/64
root@SRX# set firewall family inet6 filter IPv6 term LAB then routing-instance vrLAB
root@SRX# insert firewall family inet6 filter IPv6 term default after term LAB

Album overzicht

Capi-World-Tour-Munchen

21 Photos

Capi-World-Tour-Frankfurt

13 Photos

Capi-World-Tour-Oslo

14 Photos

Capi-World-Tour-Helsinki

14 Photos

Capi-World-Tour-Johannesburg

41 Photos

Capi-World-Tour-Kaapstad

25 Photos

De volgende e-mail is 5 april 2011 aan mijn collega’s verstuurd:

Beste collega’s en vrienden

Ik wil iedereen heel erg bedanken voor de tijd die we samen hebben gehad. Bovertis blijft altijd een speciaal plekje in mijn hart houden.

Op dit moment ben ik niet meer op zoek naar een nieuwe uitdaging. Ik ga in augustus beginnen bij Atrato IP Networks. Mocht je zelf graag mijn oude job willen, kijk dan op bovert.is/zoekt.jou en neem contact met me op via henk@bovert.is.

Er zijn 4 servers die meegenomen worden in de back-up:

• DC
  Server 2008 R2 Standard, Exchange 2010, Fileserver, Endpoint server
• DB
  Server 2008 R2 Enterprise, SQL 2008 SP1, Hyper-V
• DC1-COPENHAGEN
  Server 2003 Standard, Fileserver
• DC1-Schiphol
  Server 2003 Standard, Fileserver

Alle servers met DC in hun naam zijn Domain Controllers.

Maandag 9 augustus 2010

Op maandag is een medewerker van een Symantec Partner, op locatie bij onze klant geweest om de back-up te configureren. Er is gekozen om met Symantec Backup Exec 2010 Continuous Protection Server (CPS) te gaan werken omdat dan éénmalig alle data naar de back-up server moet worden gekopieerd en vanaf dat punt alleen nog de wijzigingen. Elke nacht zou dan een volledige back-up naar tape worden geschreven. CPS is op back-up server geïnstalleerd en daarna gedistribueerd naar de vier servers. Tot dusver zag alles er goed uit en was alleen de Schiphol server nog niet gesynchroniseerd vanwege de hoeveelheid data.

Dinsdag 10 augustus 2010

Dinsdag ochtend is geconstateerd dat de DC een paar keer was herstart vanwege een BSOD. Om 9:18 is een e-mail naar de medewerker van de Symantec Partner gestuurd met daarin de bevindingen. Daarnaast is ook het Exchange gedeelte van CPS uitgezet omdat het idee was ontstaan dat daardoor de crashes werden veroorzaakt. Om 11:38 is er een e-mail naar dezelfde medewerker uitgegaan met de vraag of Symantec Support hier niet naar kan kijken. Om 11:45 wilde de DC, na de zoveelste BSOD, niet meer doorstarten.
De BSOD werd veroorzaakt door een conflict tussen Symantec Endpoint Security (anti virus) en CPS.

http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/711ddef758a372336525773b007d8327?OpenDocument

De oplossing in dit document kan helaas niet uitgevoerd worden omdat op de DC de SBS versie draait en deze heeft geen uitgebreide opties om het scan gedrag aan te passen.

Hierna begon de stress op te lopen. Maar er was een verwachting dat, omdat de laatste back-up succesvol geweest was, de back-up terug gezet kon worden. De verbazing was groot toen zowel Symantec als de medewerker van de Symantec Partner aanstuurden op een herinstallatie van de DC en deze opnieuw in het domein te hangen en dan handmatig alles er weer op te krijgen.

De plaats van verbazing werd ingenomen door grote paniek toen de back-up niet hersteld wilde worden. Buiten het feit om dat de documentatie elkaar tegen spreekt, voor de CPS agent heb je domein authenticatie nodig en om te herstellen mag je niet in een domein zitten, kreeg ik ook een foutmelding “Access Denied” bij het terugzetten van de system state via Backup Exec, terwijl de credentials juist waren, en een algemene error bij het terugzetten via CPS. Al met al hadden we dus een back-up maar het herstellen van die back-up bleek dus niet mogelijk. Uiteindelijk is besloten om de DC opnieuw te gaan installeren en opnieuw in het domein te gaan hangen om op die manier te proberen om de server weer in de lucht te krijgen.

Woensdag 11 augustus 2010

In de ochtend is de DC1-Schiphol omgetoverd tot Operations Master zodat deze server alle rollen in Active Directory (AD) kan uitvoeren. Daarna is de DC uit AD verwijderd via NTDSUTIL. Het laatste beetje hebben we via ADSI Edit moeten doen omdat er nog iets de DC Container aanwezig was. Daarna was het de tijd om de DC weer toe te voegen aan AD. Toen dat gebeurd was is Exchange 2010 in recovery modus geïnstalleerd en konden de mailboxen er weer aangehangen worden. Vanaf dat punt werkte Exchange 2010 weer. Helaas kon door een fout in AD geen instellingen worden aangepast in Exchange 1010. Hierdoor kregen de gebruikers een melding van een onveilig certificaat bij het starten van Outlook. De medewerker van de Symantec Partner, die rond 16:00 uur weer ter plaatse was, heeft een support call bij Microsoft ingeschoten om dit probleem op te lossen.

Donderdag 12 augustus 2010

Vroeg in de ochtend zijn de group policies opnieuw aangemaakt en zijn de gebruikers mappen en bedrijfsmappen opnieuw van rechten voorzien. Ook zijn de printers weer geïnstalleerd op de DC. Rond 13:00 uur was de klant weer voor 100% operationeel. Buiten het feit om dat de back-up niet ingesteld was. Microsoft heeft via het overnemen van de server geprobeerd het probleem op te lossen maar is daar niet uitgekomen.
Er is een e-mail naar onze leverancier uit gegaan met het verzoek een voorstel te doen om een back-up oplossing te configureren en daarnaast ook te laten zien dat het hersteld kan worden.

Vrijdag 13 augustus 2010

Op deze dag is er niets gebeurd, er heeft geen communicatie plaats gevonden met Microsoft, de Symantec Partner of onze leverancier.

Maandag 16 augustus 2010

Om 11:00 uur heeft Microsoft opnieuw de server van afstand overgenomen en heeft het probleem kunnen verhelpen door permissies te wijzigen voor de groep Exchange Servers op het root niveau van het domein. Daarnaast is er druk overleg met onze leverancier over de implementatie van de back-up en daaraan gelieerd ook een paar uur om een gemaakte back-up met succes te kunnen herstellen.

Samenvatting

Nog even puntsgewijs alles op een rijtje:

• De klant had een stabiel draaiende situatie.
• De back-up oplossing met Backup Exec is geïmplementeerd.
• De DC is vanwege een compatibiliteitsconflict met Symantec Backup Exec en Symantec Endpoint Protection (nota bena van dezelfde leverancier) gecrashed.
• Na een halve dag geen success gehad te hebben met het terug zetten van een back-up (welke succesvol is gemaakt) is besloten om maar te herinstalleren.
• Na herinstallatie kwam herconfiguratie van rechten, Network Policy Server (RADIUS), group policies en andere zaken (waaronder Exchange).
• Een permissie probleem heeft tot maandag 16 augustus voor (kleine) problemen gezorgd.
• De klant heeft nu nog steeds geen functionerende backup

Wat ook nog erg vervelend is, is dat ik door dit gedoe mijn kinderen 3 dagen niet gezien heb en mijn vrouw nauwelijks.

Ik werk dit artikel bij zodra er nieuws is.

Bedankt Symantec!

Klik hier om het artikel te lezen.