IDNetters Forums

Technical News & Discussion => IDNet Help => Topic started by: c1hundred on Mar 15, 2011, 20:32:33

Title: lost service at 7:20pm
Post by: c1hundred on Mar 15, 2011, 20:32:33
my service has died, having to send thi by mobile :(
Title: Re: lost service at 7:20pm
Post by: Mouseroo on Mar 15, 2011, 20:44:41
Same here in Somerset. Lost service just after 8.
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 20:45:28
c1hundred, are you in Somerset too?
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 20:45:46
Call support and leave a message stating you have no service, someone should get back to you. The number is 0800 331 7000
Title: Re: lost service at 7:20pm
Post by: joll200x on Mar 15, 2011, 20:47:52
Ditto, bury st edmunds here. 01284.
Title: Re: lost service at 7:20pm
Post by: Gary on Mar 15, 2011, 20:48:35
I lost connection at the same time but it came back within a minute. 01243 Area code, seems fine now.
Title: Re: lost service at 7:20pm
Post by: gwidnet on Mar 15, 2011, 20:50:26
Same here in Northampton :(
Title: Re: lost service at 7:20pm
Post by: David on Mar 15, 2011, 20:52:04
no service here either
Title: Re: lost service at 7:20pm
Post by: c1hundred on Mar 15, 2011, 20:53:25
i am in aylesbury, buckinghamshire
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 20:53:37
I've emailed support with a link to this thread.
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 15, 2011, 20:54:38
Nothing showing yet on any status pages IDNet,AAISP,Zen and Plusnet. As Glenn suggested please contact support.
Title: Re: lost service at 7:20pm
Post by: ChickenHawk on Mar 15, 2011, 21:00:24
Sync and Authentication here, but no throughput on VDSL in Glasgow.    :mad:
Title: Re: lost service at 7:20pm
Post by: Liam on Mar 15, 2011, 21:04:39
Quote from: ChickenHawk on Mar 15, 2011, 21:00:24
Sync and Authentication here, but no throughput on VDSL in Glasgow.    :mad:

Same here, I smashed my router up with rage thinking it crashed on me during a heated game of TF2 lol
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 21:06:39
No service here in London - Cricklewood exchange. Been trying to get router to connect for past 2 hours or so...
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 21:16:42
I have just had an email back from Simon, There is a problem with one of the BT Host Link connections, BT are investigating.
Title: Re: lost service at 7:20pm
Post by: ChickenHawk on Mar 15, 2011, 21:17:50
Status page has been updated...
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 21:21:37
Service: Announcement
Updated: 2011-03-15 21:10:47
State: Sub-Optimal Service
Message: There is a problem with one of our connections into BT. They are investigating.
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 21:29:34
Shouldn't there be a backup onto a different network provider???  Say C&W (purely as a last resort of course)?  This happens far too frequently!  V-ANGRY/MAD.

p.s. Welcome to the forum imbimp!  :)
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 21:42:32
Thanks Conrad...

Glenn...

What's the SLA with BT?

How often do they provide updates on this issue - I presume often since it seems like a fairly widespread problem...
Title: Re: lost service at 7:20pm
Post by: ou7shined on Mar 15, 2011, 21:46:10
Quote from: ChickenHawk on Mar 15, 2011, 21:00:24
Sync and Authentication here, but no throughput on VDSL in Glasgow.    :mad:
Same here in Aberdeen.  ::) ::)
Title: Re: lost service at 7:20pm
Post by: ChickenHawk on Mar 15, 2011, 21:48:14
Quote from: Conrad on Mar 15, 2011, 21:29:34
Shouldn't there be a backup onto a different network provider???  Say C&W (purely as a last resort of course)?  This happens far too frequently!  V-ANGRY/MAD.

p.s. Welcome to the forum imbimp!  :)
I think thats part of the problem.  The email I got back say they're also investigating why it didn't fall over to the other link.
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 21:48:21
Quote from: imbimp on Mar 15, 2011, 21:42:32

Glenn...

What's the SLA with BT?

How often do they provide updates on this issue - I presume often since it seems like a fairly widespread problem...

Ihave no idea in their SLA's. In the past an update has been issued when there is something to say, rather than on the hour.
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 15, 2011, 21:53:01
imbimp :welc: :karma: sorry it's not in better circumstances.
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 15, 2011, 21:55:00
 :welc: :karma: imbimp.
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 21:55:18
My connection is down too. Sync was up and down like a yo yo for an hour and then when I thought it had sorted itself out it's got the bad ip address.
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 15, 2011, 21:58:16
Are these all WBC outages? Anyone on adslmax with no connection?
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 21:58:58
Just had a callback from support (thanks Simon!)...  Nothing new though :(

BT are looking into it...  I bet they are saying things like: "High Order Failure", "Major Service Outage", "We have engineers en-route" etc...  BT make me LOL! - and NOT in a funny way!


Steve, ADSL2 here.
Title: Re: lost service at 7:20pm
Post by: shorn on Mar 15, 2011, 22:01:24
No service here, Southampton, Hampshire. Fibre connection. Poor stuff, been out for a couple of hours now. Poor customer service.
Title: Re: lost service at 7:20pm
Post by: lozcart on Mar 15, 2011, 22:03:34
I'm so far unaffected.

Birmingham, ADSL2
DSL4
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 22:05:30
ADSL here...

Not surprising I see so many smashed phone boxes on my way to work...

Can't BT just re-boot their giant Internet router or kick some switches?
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 15, 2011, 22:06:59
Unfortunately BT have a habit of kicking the wrong switches allegedly.
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 22:11:08
I'm on adsl2+. Wish BT would pull their finger out.

Thanks god I saved an ear bashing from the mrs by setting up her htc desire as a access point
Title: Re: lost service at 7:20pm
Post by: Niall on Mar 15, 2011, 22:16:39
I don't seem to be effected at all, here in Wrexham.

BT seem to be utterly pathetic recently. Something really needs to be done about them and their, quite frankly piss poor treatment towards ISPs. I imagine IDnet will lose business and money over this again, and once again BT will no doubt just shrug their shoulders and blame an engineer for deleting a config again.
Title: Re: lost service at 7:20pm
Post by: JamesAllen on Mar 15, 2011, 22:22:02
Same problems here, though lost service about 20:50 or so.

Router re-synced, but obtained the wrong IP, gateway IP and DNS.

On ADSL2+ in Nottingham.
Title: Re: lost service at 7:20pm
Post by: Simon_idnet on Mar 15, 2011, 22:25:35
One of our Host Link connections is down. Around 90% of the traffic has successfully failed-over to the other link but some circuits have not. That's makes two faults and we have BT investigating both!
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 15, 2011, 22:28:54
Thanks for the update Simon.
Title: Re: lost service at 7:20pm
Post by: Simon on Mar 15, 2011, 22:28:58
All seems fine here in Sussex, on ADSL Max.  

Thanks for the update, Simon.  :thumb:
Title: Re: lost service at 7:20pm
Post by: psp83 on Mar 15, 2011, 22:30:54
All ok here in Wiltshire on ADSL2  ;D
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 22:33:04
Still cr*p here in Cricklewood...Internet doesn't work either
Title: Re: lost service at 7:20pm
Post by: Ray on Mar 15, 2011, 22:34:35
No problem here on ADSL max in South Leicestershire.
Title: Re: lost service at 7:20pm
Post by: bridgej on Mar 15, 2011, 22:47:27
No service in stratford east london
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 22:49:36
So nearly another 30 minutes have passed Simon...

What's the plan?

Have BT given up for the night?
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 23:05:27
nearly 2 hours since the announcement...  This is doing my head in!!! 
Title: Re: lost service at 7:20pm
Post by: imbimp on Mar 15, 2011, 23:08:33
Time for an update I think...

Or do a few successes equal let's give up?

Slightly confused by the amber status...no service ought to be red, no?

When is this going to be fixed?
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 23:11:52
The reason for no update will be because there isn't anything. BT will be investigating.

The reason fir Amber rather than red is because only 10% or so are affected, not the majority. 
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 23:18:11
Don't know if this helps diagnose the fault, but my router has picked up the following (incorrect) IP details:
IP: 172.16.16.79
DNS: 62.6.38.125

Hope this helps to fix the problem... LIKE NOW! O.o

Title: Re: lost service at 7:20pm
Post by: ou7shined on Mar 15, 2011, 23:20:36
Quote from: imbimp on Mar 15, 2011, 22:49:36
So nearly another 30 minutes have passed Simon...

What's the plan?

Have BT given up for the night?
Buckle up. Last time this happened my connection was out for 3 days.
Let's hope it's not that bad again. :fingers:
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 23:21:59
Mine has picked up something similar. It's some sort of internal BT address.
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 23:28:41
it's changed now to 172.16.16.27.  Is someone actually trying to fix this, or are BT doing their "standard care - won't even look at it until the morning" BS???  I need to know as I am tired and could do with some sleep.  If I know that nothing is going to be done about this tonight, then I can hit the sack (albeit fuming).

Ta.

...now 172.16.16.58...
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 23:33:51
Bt will be working on it now, and probably have an engineer on site or working on the kit remotely.
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 23:35:44
That said, it doesn't mean that it will be sorted any time soon. Could be 5 mins or could be 5 hours. Until we get an eta from BT via support we won't have a clue.
Title: Re: lost service at 7:20pm
Post by: Conrad on Mar 15, 2011, 23:45:28
Okay.  So BT haven't actually given an update, you're not sure whether anyone is physically working to resolve the fault or not, either on-site (wherever that might be) or remotely, and an expected time of resolution could be anywhere between 5 mins and 5 hours (despite surely having SLAs in place no?)...  I'm so glad that's been cleared up.  I'm off to bed.

O.o

Night all.
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 15, 2011, 23:48:51
I'm not a member of staff, just another customer. It's just i've seen this sort of thing and have a good idea of what will be happening.
Title: Re: lost service at 7:20pm
Post by: ou7shined on Mar 15, 2011, 23:56:24
Mine's back up. :thumb:

(http://www.thinkbroadband.com/ping/share-thumb/f8446b1946658787467cfbefa11e2585-15-03-2011.png)
Title: Re: lost service at 7:20pm
Post by: Liam on Mar 15, 2011, 23:59:25
Guys, calm down. yes its frustrating as hell but IDNET are at the mercy of BT.

I have a mate that used to work for BT and the stories he has told me about guys he worked with is unreal, they make changes willy nilly on the network without doing any documentation so that when things do go wrong they struggle to undo the damage they've done.

anyway, things seems to be going back to normal now.
Title: Re: lost service at 7:20pm
Post by: .Griff. on Mar 16, 2011, 00:01:26
Is this affecting all IDNET customers or have I been lucky?
Title: Re: lost service at 7:20pm
Post by: JamesAllen on Mar 16, 2011, 00:03:53
Yay!

All back.

Not too bad really as my Nokia N95 worked nicely via Bluetooth (during the downtime) - giving the laptop a pretty decent 3G connection.

Glad it happened out of working hours though.

Hope everyone affected is back very soon.
Title: Re: lost service at 7:20pm
Post by: Simon_idnet on Mar 16, 2011, 00:10:02
BT have resolved the problem. Now we want to find out what caused it...
Title: Re: lost service at 7:20pm
Post by: esh on Mar 16, 2011, 00:46:41
All in the clear here, ADSL-Max.... didn't even know there was a problem.

You'd think after the 3rd problem of this nature they'd get the fail-over working properly. Oh well, at least this time it was only a *few* hours, I guess.

For what it's worth Simon, I think I feel your pain, even though this one hasn't affected me... If this turns out to be another "pulling the wrong cable out of the cabinet"/"configuration deletion" cause, I really don't know what can be said... Yet some part of me would not be surprised at all.
Title: Re: lost service at 7:20pm
Post by: Lance on Mar 16, 2011, 06:39:10
Quote from: .Griff. on Mar 16, 2011, 00:01:26
Is this affecting all IDNET customers or have I been lucky?

Affected only the minority, approx 10% who didn't get switched over to the backup link automatically.
Title: Re: lost service at 7:20pm
Post by: JB on Mar 16, 2011, 08:08:45
Quote from: Simon_idnet on Mar 16, 2011, 00:10:02
BT have resolved the problem. Now we want to find out what caused it...

I hope you have more success with BT than after the last outage Simon. I was one of the lucky ones on 8Mb ADSL who kept a normal connection.

I'm sure it is co-incidence but it looks like BT have a contract out on IDNet.
Title: Re: lost service at 7:20pm
Post by: Gary on Mar 16, 2011, 08:12:49
I lost sync for a few seconds so I was also Lucky this time, but it does seem like IDNet is not paying their protection money to BT.  :evil:
Title: Re: lost service at 7:20pm
Post by: Steve on Mar 16, 2011, 08:40:34
Well at least the fall over to the back up host link worked better than last time.
Title: Re: lost service at 7:20pm
Post by: JB on Mar 16, 2011, 08:41:03
True.
Title: Re: lost service at 7:20pm
Post by: Gary on Mar 16, 2011, 08:56:32
Quote from: Steve on Mar 16, 2011, 08:40:34
Well at least the fall over to the back up host link worked better than last time.
Very true, BT need to be taken out the back and beaten.
Title: Re: lost service at 7:20pm
Post by: sobranie on Mar 16, 2011, 10:11:19
Quote from: Steve on Mar 16, 2011, 08:40:34
Well at least the fall over to the back up host link worked better than last time.

Didn't here I'm afraid, off for 4 hours on (01326).  May I join the back up link club please?

jftr I rang IDNet late last night and Simon rang me back within 10 minutes to advise that there was a problem. Well done Simon and thanks for the call.

Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 10:19:39
Quote from: 6jb on Mar 16, 2011, 08:08:45
I'm sure it is co-incidence but it looks like BT have a contract out on IDNet.

I feel the same, JB. The identical fault to last time which BT assured IDNet they had resolved. :mad:
Title: Re: lost service at 7:20pm
Post by: .Griff. on Mar 16, 2011, 13:11:26
I'm not sure what happened but my connection seemed to perform the opposite to everyone else's. While everyone on here was posting about their connection being down mine remained fine. However when you all confirmed your connections were back up mine went down and remained down until earlier this morning.  ???
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 13:19:02
On the plus side my download speed has doubled  :dunno:
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 16, 2011, 13:20:19
A silver lining, David.  ;D
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 13:22:51
I wont complain Glenn  ;D
Title: Re: lost service at 7:20pm
Post by: Simon on Mar 16, 2011, 13:49:50
That must have been it.  Everyone was down so they could upgrade David's connection.  ;D
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 13:58:50
 ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 14:08:07
 :laugh:

Stranger things have happened.
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 14:26:15
if its to do with me you promised never to mention that incident Rik.....it was never proved  ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 14:31:09
Only because it fell under Scottish jurisdiction.  :evil:
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 14:45:18
Thats because a prod isnt classed as a probe up there  ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 14:46:24
It depends what you prod. ;D
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 15:23:49
And where.... this was not in the street  ;D
Title: Re: lost service at 7:20pm
Post by: Simon on Mar 16, 2011, 16:29:17
'Up there'?  Up where, David?   :eek4: ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 16:37:02
Pompeii?
Title: Re: lost service at 7:20pm
Post by: David on Mar 16, 2011, 16:41:18
ohhhhhhhhhhhhhhhhh misssus  ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 16, 2011, 16:42:27
 ;D
Title: Re: lost service at 7:20pm
Post by: Ray on Mar 16, 2011, 16:42:52
 :lol:
Title: Re: lost service at 7:20pm
Post by: hells bells on Mar 16, 2011, 21:27:16
My connection went down last night and I phoned out of hours. Simon got back to me quite promptly (thanks for that Simon!)
Through past experience I know how long BT engineers can take with these things so I went to bed with a good book instead.  ;D

Checked the status this morning and saw everything was sorted and I know that Simon and the crew will be smacking knuckles to get answers out of BT about this.

Just so you know, we know its not IDNet at fault here and we still appreciate the good service you give us!
Keep up the good work! :)
Title: Re: lost service at 7:20pm
Post by: hells bells on Mar 16, 2011, 21:28:56
Unfortunately my connection speed didn't double over night though!  :'(
There again, it didn't decrease either!  :thumb:
Title: Re: lost service at 7:20pm
Post by: ou7shined on Mar 16, 2011, 22:21:56
Well my connection's back to normal.  ::)

(http://www.thinkbroadband.com/ping/share-thumb/f6606d1259dcc0c4dc442a05c26107bf-16-03-2011.png)
Title: Re: lost service at 7:20pm
Post by: Simon_idnet on Mar 22, 2011, 11:12:27
We have now discovered that one of our Host Link circuits was down due to a BT PEW (Planned Engineering Works) that they hadn't bothered to warn us about!

More to the point, the service should be resilient such that customers automatically fail-over to the other Host Link circuit. We have now discovered that the failure of the fail-over was all our fault, according to BT - there's a surprise! Apparently the configuration of our routers that talk to their routers is wrong and should have been setup differently - I suppose we are guilty of not being clairvoyant!

This is the best bit: when one circuit fails BT see that no traffic is being sent down that circuit and their "intelligent" load-balancing algorithms see this and so try to balance the traffic by sending all new customer logins to the "under-utilised" circuit - that one that's broken!!

The way around this is a complicated bit of BGP trickery which lets our routers force BT's load-balancer to send all traffic to us via the circuit that is not broken. A very clever and helpful chap at BT Operate has assisted us in implementing their BGP fix on our Cisco LNS routers. The configuration has now been verified and we will be testing it during a quite period.
Title: Re: lost service at 7:20pm
Post by: David on Mar 22, 2011, 11:15:25
 :eek4: must you ? v ;D

Thanks Simon  :thumb:
Title: Re: lost service at 7:20pm
Post by: Simon on Mar 22, 2011, 11:23:27
Thanks for the update, Simon.  I guess you need to polish your crystal ball.  ;)
Title: Re: lost service at 7:20pm
Post by: zappaDPJ on Mar 22, 2011, 11:25:55
Quote from: Simon_idnet on Mar 22, 2011, 11:12:27
I suppose we are guilty of not being clairvoyant!

Mystic Meg to the rescue! I've heard she can foretell planned engineering works better than BT  ;D

Thanks for the update Simon :)
Title: Re: lost service at 7:20pm
Post by: Technical Ben on Mar 22, 2011, 14:48:39
Quote from: Simon_idnet on Mar 22, 2011, 11:12:27
...

This is the best bit: when one circuit fails BT see that no traffic is being sent down that circuit and their "intelligent" load-balancing algorithms see this and so try to balance the traffic by sending all new customer logins to the "under-utilised" circuit - that one that's broken!!
...
(emphases mine)
I laughed out loud (very much so) to that little snipped. Better than any jokes I've heard in ages.... oh wait, it's not a joke.  :shake:

PS. BT need a "If [router down] = true, then [swap to other router] = true" switch or something in their coding.  ;)
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 22, 2011, 15:30:30
BT - how to cock up the world in one easy lesson. :(
Title: Re: lost service at 7:20pm
Post by: David on Mar 22, 2011, 15:46:06
Buzby has a lot to answer for 
Title: Re: lost service at 7:20pm
Post by: Glenn on Mar 22, 2011, 15:47:44
I think he's dead by now, David.
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 22, 2011, 16:01:34
And buried. :)
Title: Re: lost service at 7:20pm
Post by: David on Mar 22, 2011, 16:08:12
Not another one... ;D
Title: Re: lost service at 7:20pm
Post by: Rik on Mar 22, 2011, 16:10:06
There's always another one. :)
Title: Re: lost service at 7:20pm
Post by: David on Mar 22, 2011, 16:38:00
Cheerful Charlie  ;D
Title: Re: lost service at 7:20pm
Post by: Gary on Mar 23, 2011, 07:38:30
Quote from: David on Mar 22, 2011, 16:38:00
Cheerful Charlie  ;D
Yep he is bi-winning  ;D