Tuesday, July 1, 2008

Cute little OSPF Problem crashing a Network!

This morning I received a ticket to check a network down issue...

According to the description, customer had an isolated site, represented in the diagram bellow by Router R3 (by the way, here I´m working in a fictitious network, to avoid showing our customer data):

OSPF Router ID

First things first... it could be a link failure (there´s only one  exit from R3 to the core´s network, and that´s S1/0, and as far as the site is isolated,  any engineer would check the link first) and that was exactly what I did. To my surprise link was UP!

Ok, so link is up... I was able to ping to other side of the link, also a good sign... just did a quick check at the router configuration, and found that it´s running OSPF.

Checked the neighbor relationship between it (R3) and it´s neighbor router R2. Again, everything looks pretty much fine so far!

The real problem began to reveal itself when I checked R3 routing table... R3´s routing table only had it´s connected network, plus the network 192.168.1.0... Hmmm... it seens to be an OSPF related issue!

R3#sh ip route
.

output omitted

.

Gateway of last resort is not set

    20.0.0.0/24 is subnetted, 1 subnets
C      20.3.3.0 is directly connected, Loopback1
    10.0.0.0/32 is subnetted, 1 subnets
C      10.3.3.3 is directly connected, Loopback0
O   192.168.1.0/24 [110/128] via 192.168.3.2, 00:00:02, Serial1/0
C   192.168.3.0/24 is directly connected, Serial1/0
    30.0.0.0/24 is subnetted, 1 subnets
C      30.3.3.0 is directly connected, Loopback2

Quick jumped to R2 to check it´s routing table... I´ve just found entries to R1´s network, plus it´s directed connected networks, nothing related to router R3.

R2#sh ip route
.

output omitted

.

Gateway of last resort is not set

     20.0.0.0/32 is subnetted, 1 subnets
O       20.1.1.1 [110/65] via 192.168.1.1, 00:00:02, Serial1/0
     10.0.0.0/32 is subnetted, 1 subnets
O       10.1.1.1 [110/65] via 192.168.1.1, 00:00:02, Serial1/0
C    192.168.1.0/24 is directly connected, Serial1/0
C    192.168.3.0/24 is directly connected, Serial1/1
     30.0.0.0/32 is subnetted, 1 subnets
O       30.1.1.1 [110/65] via 192.168.1.1, 00:00:02, Serial1/0

Ok, now I´ve found the problem... take a look at the neighbor relationship in R2 (using the show ip ospf neighbor command):

R2#sh ip ospf neighbor

Neighbor ID   Pri   State     Dead Time  Address      Interface
10.1.1.1       0    FULL/  -  00:00:33   192.168.3.3  Serial1/1
10.1.1.1       0    FULL/  -  00:00:37   192.168.1.1  Serial1/0

Can you see the problem?! Off course you can! Both routers(R1 and R3) have the SAME ROUTER ID!

I didn´t belived what I saw! Customer told me that "nobody" touched the router, at least for the last couple weeks!

Ok, just doing a debug ip ospf hello at R2 you can see the same issue... both routers (R1 and R3) using 10.1.1.1 as it´s Router ID.

R2#debug ip ospf hello
*Mar  1 00:11:16.039: OSPF: Rcv hello from 10.1.1.1 area 0 from Serial1/0 192.168.1.1
*Mar  1 00:11:16.043: OSPF: End of hello processing
*Mar  1 00:11:16.403: OSPF: Rcv hello from 10.1.1.1 area 0 from Serial1/1 192.168.3.3
*Mar  1 00:11:16.407: OSPF: End of hello processing

Taking a look at the routers ID, I´ve found R1 RID: 10.1.1.1, R2 RID: 10.1.1.2 and R3 RID: 10.1.1.1, asked the customer if I could change R3´s RID to 10.1.1.3 or if he had any other RID to be used, and received his GO-AHEAD to change it!

After entering the router-id 10.1.1.3 command in R3´s OSPF configuration, and off course, using a clear ip ospf process (in R3) things looked a lot better!

Take a look at R2´s neighbor table right now:

Rack1R2#sh ip ospf neighbor

Neighbor ID    Pri  State      Dead Time  Address     Interface
10.1.1.3       0    FULL/  -   00:00:33   192.168.3.3 Serial1/1
10.1.1.1       0    FULL/  -   00:00:37   192.168.1.1 Serial1/0

And finally, R2´s Routing Table with all posible routes:

Rack1R2#sh ip route
.

output omitted

.

Gateway of last resort is not set

     20.0.0.0/32 is subnetted, 2 subnets
O       20.1.1.1 [110/65] via 192.168.1.1, 00:00:05, Serial1/0
O       20.3.3.3 [110/65] via 192.168.3.3, 00:00:05, Serial1/1
     10.0.0.0/32 is subnetted, 2 subnets
O       10.3.3.3 [110/65] via 192.168.3.3, 00:00:05, Serial1/1
O       10.1.1.1 [110/65] via 192.168.1.1, 00:00:05, Serial1/0
C    192.168.1.0/24 is directly connected, Serial1/0
C    192.168.3.0/24 is directly connected, Serial1/1
     30.0.0.0/32 is subnetted, 2 subnets
O       30.3.3.3 [110/65] via 192.168.3.3, 00:00:05, Serial1/1
O       30.1.1.1 [110/65] via 192.168.1.1, 00:00:05, Serial1/0

So... with a 5 minute troubleshooting everything was fine! But this little issue kept our customer´s network down for a while, due to a mistake some "ghost" made in their router configuration!

You might be asking how the RID is selected / configured...:

1 - If you configure the RID with the command router-id <address> (example: router-id 10.1.1.1) under OSPF configuration mode, this manually configured RID is used;

2 - If no RID is configured, the router tries to use the higher Loopback IP Address it founds in the OSPF startup process;

3 - If no Loopbacks are configured, the routers uses the higher IP Address configured in any physical interface;

4 - If no RID is configured, and the router has no IP configured interfaces, the OSPF process cannot start!

Just be carefull while using OSPF RID, it has to be unique in your network! Otherwise, you can have huge problems! Keep your network well documented, and do not make changes in a live enviroment without studying the side-effects before actually doing it!

5 comments:

Anonymous said...

Yes and very similar to a situation I had to deal with.
Lets say you are not allowed to hard code the OSPF router ID and you have this problem , I could not find an answer.
Any idea
Thanks
Giorgio

Anonymous said...

Again why I missed that I could have put another Loo with an higher
IP address . Since there was no OSPF config on the R the previous ID was chosen automatically because of the same reason !
Sorry for the trouble.
By the way your blog is really useful and I am going to follow it
closely. I have failed the lab first time and I need a change my self.
Regards Giorgio

Caue said...

Hey buddy!

It´s nice when somebody comes with the problem and a solution! hahaha! Yeah! You´re absolutely right! If you cannot set the Router ID manually, set the Loopback, the highest Loopback address will be used as RID!

Just remember, you may need to clear the OSPF, or reload the router to take effect!

Thanks man! In fact, Internet is AWESOME, there are so many resources out there nowadays, blogs, presentations, I just love!

Sorry to hear that you failed your first time man! :( Have you used any Workbook/Vendor for your preparation?!

Where do you live?! Any new plan of attack?! Im in the beginning of my journey, it´ll take a while before my first attemp!

Let me know if I can assist you in anything! It´ll be a pleasure!

Take care man!
Bye!

Anonymous said...

I leave in UK but I am Italian.
I am using the IPexpert new ver 10 and I have material from Internetworkexpert.
The exam is doable no doubt , but the questions, very clear, are using the information in a way that shows if you understand it or not. No tricks from Cisco at all , but a very limited amount of basic stuff.THINKING is necessary all the time. Searching the Cisco Doc is useful but will not be required that much if you have a good and deep understanding.
Do not forget the S&R part of the exam is really where you pass or fail.You have to know the rest but not at the same level.
Anyway today I am rewieing the Doyle OSPF and I will do the OSPF volume 1 labs.
Thanks for your offer I am fairly isolated and would be nice to talk about this things.
Regards
Giorgio

Caue said...

Hello Giorgio!

Yeah... they told me that, we need to take care of every little aspect of the LAB, how one thing will affect the other and so on!

Good to hear that you´re going to check Doyle´s book! Those are the best, I have them all! :)

Please, send me a msg at: cauefw@gmail.com or add me on MSN: cauew6@hotmail.com or SKYPE: cauefw and we can talk better! For sure it´s always a pleasure to have someone to talk regarding the CCIE Studies and to keep each other motivated!

Good lucky in your studies! Wish to talk to you soon buddy!

Cheers,
Caue Wailemann