IDNetters Forums

Technical News & Discussion => IDNet Help => Topic started by: JamesAllen on Apr 25, 2013, 14:30:04

Title: Google down - routing issue at Telehouse?
Post by: JamesAllen on Apr 25, 2013, 14:30:04
Now, hopefully this will fix itself but I've just found that Google is 'down' and on doing a trace route I notice a failure on one of the nodes out of Telehouse. Here it is:

Tracing route to www.google.com [173.194.41.144]
over a maximum of 30 hops:

  1    <1 ms     1 ms     1 ms  192.168.0.3
  2    31 ms    31 ms    30 ms  telehouse-gw2-lo1.idnet.net [212.69.63.51]
  3     *        *        *     Request timed out.
  4     *     telehouse-gw6-gi3-401.idnet.net [212.69.63.182]  reports: Destination net unreachable.

Other sites are working (though I assume some will also be affected by this), for instance Facebook:

Tracing route to star.c10r.facebook.com [31.13.80.17]
over a maximum of 30 hops:

  1     1 ms     1 ms     2 ms  192.168.0.3
  2    31 ms    31 ms    31 ms  telehouse-gw2-lo1.idnet.net [212.69.63.51]
  3     *        *       31 ms  telehouse-gw6-gi4-400.idnet.net [212.69.63.246]

  4    31 ms    32 ms    31 ms  ge-2-25.r02.londen03.uk.bb.gin.ntt.net [213.130.48.161]
  5    31 ms    31 ms    31 ms  ae-4.r23.londen03.uk.bb.gin.ntt.net [129.250.5.40]
  6    38 ms    38 ms    38 ms  as-0.r20.parsfr01.fr.bb.gin.ntt.net [129.250.5.9]
  7    39 ms    38 ms    38 ms  ae-0.r21.parsfr01.fr.bb.gin.ntt.net [129.250.2.181]
  8    41 ms    39 ms    39 ms  po-2.r01.parsfr02.fr.bb.gin.ntt.net [129.250.4.178]
  9    40 ms    39 ms    39 ms  82.112.96.146
10    60 ms    39 ms    40 ms  ae1.bb02.cdg1.tfbnw.net [204.15.22.198]
11    39 ms    38 ms    39 ms  ae2.pr02.cdg1.tfbnw.net [204.15.23.15]
12    39 ms    46 ms    55 ms  po126.msw01.02.cdg1.tfbnw.net [31.13.27.93]
13    38 ms    39 ms    38 ms  31.13.80.17

Anyone else seeing this?
Title: Re: Google down - routing issue at Telehouse?
Post by: Bill on Apr 25, 2013, 14:31:58
Similar here- I can't connect to tbb, and a few other sites as well.
Title: Re: Google down - routing issue at Telehouse?
Post by: JamesAllen on Apr 25, 2013, 14:34:05
Thanks for confirming Bill - glad it's not just me, though didn't think it could be with the specific node failure in the trace route.

Hopefully IDNet will get this resolved quickly. Wonder if the turn it off and back on again on that node will sort it ;)
Title: Re: Google down - routing issue at Telehouse?
Post by: Technical Ben on Apr 25, 2013, 14:35:47
Seems to be falling over in some form.
Title: Re: Google down - routing issue at Telehouse?
Post by: jameshurrell on Apr 25, 2013, 14:36:18
Yep, Google and Thinkbroadband down, plus my client's VPN IP address and OWA address (cannot do any work  ;D ). i've rebooted here, but made no difference.
Title: Re: Google down - routing issue at Telehouse?
Post by: JamesAllen on Apr 25, 2013, 14:38:42
Not sure if this is a good or bad thing, but bing.com is working so we can at least search while Google isn't accessible.

Quite amazed as I haven't been on Bing for ages - they have totally ripped off Google in almost every way down to the layout.
Title: Re: Google down - routing issue at Telehouse?
Post by: patc57 on Apr 25, 2013, 14:39:32
Confirmed, seems major link/route is down, as I can't reach 3 out of 5 of my US based servers. A 100 packet test MTR to one currently looks like this:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              FTTC-IDNET -    0 |  100 |  100 |    0 |    1 |    2 |    1 |
|             telehouse-gw4-lo1.idnet.net -    0 |  100 |  100 |   25 |   31 |  117 |  117 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   23 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   24 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   22 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|        Destination network unreachable. -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   21 |    0 |    0 |    0 |    0 |    0 |
|________________________________________________|______|______|______|______|______|______|
Title: Re: Google down - routing issue at Telehouse?
Post by: miriam_idnet on Apr 25, 2013, 14:48:02
Hi All,

We are investigating an external routing/peering problem as we speak, keeping the status page updated.

Regards,

Miriam
IDNet Support
Title: Re: Google down - routing issue at Telehouse?
Post by: Bill on Apr 25, 2013, 14:51:41
Thanks Miriam :thumb:
Title: Re: Google down - routing issue at Telehouse?
Post by: Technical Ben on Apr 25, 2013, 14:59:21
One of the benefits of having the forum on the companies servers, is we get to it before any of these routings? all this time I thought it was magic that kept the forums working when everything else crashed.  :laugh:
Title: Re: Google down - routing issue at Telehouse?
Post by: karvala on Apr 25, 2013, 15:09:26
The BT Bletchley core is down according to the internet traffic report.

EDIT:  Actually, scrap that; I think they're using an outdated server name; looks there is never any response on that.
Title: Re: Google down - routing issue at Telehouse?
Post by: jameshurrell on Apr 25, 2013, 15:09:29
Fixed for me now.
Title: Re: Google down - routing issue at Telehouse?
Post by: JamesAllen on Apr 25, 2013, 15:12:42
I'm back up. Thanks for sorting that out quickly IDNet.
Title: Re: Google down - routing issue at Telehouse?
Post by: karvala on Apr 25, 2013, 15:13:00
Me too; nice and fast.  :thumb:
Title: Re: Google down - routing issue at Telehouse?
Post by: Bill on Apr 25, 2013, 15:16:03
Ditto here :thumb:

One or two US sites took a little while to wake up, but they got there!
Title: Re: Google down - routing issue at Telehouse?
Post by: Bill on Apr 25, 2013, 15:22:22
Interestingly, because the tbb main site is dual-stack and I use IPv6 I could access my BQMs during the whole outage, and they show very different effects:

(http://www.thinkbroadband.com/ping/share-thumb/5faacfd6b65e30184410a42e2247006e-25-04-2013.png) (http://www.thinkbroadband.com/ping/share/5faacfd6b65e30184410a42e2247006e-25-04-2013.html)


(http://www.thinkbroadband.com/ping/share-thumb/d374d31d29df346c9d4c4d157553631f-25-04-2013.png) (http://www.thinkbroadband.com/ping/share/d374d31d29df346c9d4c4d157553631f-25-04-2013.html)
Title: Re: Google down - routing issue at Telehouse?
Post by: brian_idnet on Apr 25, 2013, 15:23:36
Hi All,

This should now be resolved.

Regards,

Brian
Title: Re: Google down - routing issue at Telehouse?
Post by: zappaDPJ on Apr 25, 2013, 15:56:18
It does seem to be. Thanks for the update Brian :thumb: