QuoteDear All,
We have suffered several BT faults in the past 10 days. All appear to be due to "human error". Whilst we accept that equipment breaks and, indeed, people make mistakes, it is important that lessons are learned and processes corrected to prevent recurrence. Most important is that mistakes are corrected quickly and we feel that BT has been far too slow on too many occasions when correcting mistakes. Moreover, on each occasion we have given them timely and accurate synopses of the likely root of the fault which has been ignored in favour of following pre-set procedures. In our opinion valuable time was wasted in following unintelligent fault flow-charts fruitlessly. We want to make them listen to us!
Needless to say we would like to sincerely apologise to all of you for these recent outages. Although the outages have been out of our hands, we as the Service Provider have to take responsibility and reassure you, our very valued customers, that we are taking all the measures that we possibly can to avoid the chances of this happening again.
We have requested a meeting with BT Management for these reassurances and we will keep you posted on our discussions. However, it's clear now that we can't rely on BT alone - the risk to our business is too great. Therefore we are in the process of opening discussions with additional suppliers and we will keep you posted on how these discussions progress.
In the meantime, please bare with us, we continue to be committed in providing you with the very best levels of service, and we sincerely believe that despite being repeatedly let down by BT, we are now taking the appropriate measures, to protect us and you, going forward.
Regards
Tim
Firstly let me say a big thank you for making the preceding statement and keeping all your customers informed.
Regarding the following -
QuoteHowever, it's clear now that we can't rely on BT alone - the risk to our business is too great. Therefore we are in the process of opening discussions with additional suppliers and we will keep you posted on how these discussions progress.
I'm extremely happy to see that you're looking into other suppliers. Granted nothing may come of it but it shows you have customers interests at heart and I can only encourage you to pursue this with complete support from me as a customer who's been an LLU user for the past 6 years before joining IDNet and experiencing the lack of end user care BT Wholesale seems to demonstrate on a daily basis.
You obviously know a lot more of the industry than me but I'd encourage you to speak with Be Unlimited who resell their network, AAISP being one example, and anyone who's been a Be customer, directly or indirectly, will vouch for their network.
But wouldn't that mean IDNET would loose a whole raft of customers that like me don't have Be LLU in their exchange?
I wondering whether it would be a similar model to AAISP i.e those who have an alternative LLU supplier such as BE at their local exchange would be able to choose that as an alternative carrier to BT.
"In the meantime, please bare with us,..."
Personally I'm keeping my clothes on... It's cold out there.
Quote from: sn on Oct 21, 2010, 09:43:27
But wouldn't that mean IDNET would loose a whole raft of customers that like me don't have Be LLU in their exchange?
No, it means that providers outside of BT provide a link to the net.
But a BE connection fron AAISP is a lot more expensive than a BT one.
But it gives a choice for those who may need it perhaps for reasons of speed or reliability.
For myself I only have the option of Talk Talk LLU or Kijoma WiFi which gives speeds of about 40mb and 20mb down, but that means aerials etc and they are not the easiest company to get in contact with, Justina also is not keen on a large aerial on the chimney, so I can count both those options out :sigh:
I just want to say how much I appreciate the way IDNet handles problems when they do arise, as has happened over the last week or so with the two BT faults. In the past, when I had another provider and things went wrong, I would be left in limbo, anxious and tense, not knowing when or even if my connection would be restored (or when my phone call to support would be answered). Now, on the very rare occasions when I lose my connection, I feel pretty relaxed, thinking IDNet a) knows it's happening, b) is doing something about it, and c) will make sure it works.
It's so refreshing to deal with a company that shows a genuine commitment to providing an excellent service to its customers. I just hope that Simon and Tim never get fed up and sell the business!
Quote from: krysia on Oct 21, 2010, 10:31:55
I just want to say how much I appreciate the way IDNet handles problems when they do arise, as has happened over the last week or so with the two BT faults. In the past, when I had another provider and things went wrong, I would be left in limbo, anxious and tense, not knowing when or even if my connection would be restored (or when my phone call to support would be answered). Now, on the very rare occasions when I lose my connection, I feel pretty relaxed, thinking IDNet a) knows it's happening, b) is doing something about it, and c) will make sure it works.
It's so refreshing to deal with a company that shows a genuine commitment to providing an excellent service to its customers. I just hope that Simon and Tim never get fed up and sell the business!
It has to be better than talking to someone in India reading from a script and trying desperately to convince you that the fault is your end or you computer I'll give you that and full marks to Idnet for coming forward and being upfront with the problems. That said running a business myself I would be looking for some serious explanations from my supplier had they upset hundreds if not thousands of my customers by clearly being responsible for the cause of such (intentionally or not). To put a clearer picture if my supplier had upset 50 of my customers to the extent of taking their business elsewhere I would be finished, of that I have no doubt.
Monopoly or not BT have to be dealt with, Idnet I will stick with for now but more outages like these, well who knows.
At the end of the day we are Idnet's customers and therefore they have to keep us happy with the end service they provide, as without income they like me and many others are finished and that's the reality of it.
Idnet have to push BT for some answers and I would suggest some form of compensation, but that's the job of their directors not mine. They have a fine product but need to protect it. And I sincerely hope that they do. I will back them in any way I can.
On another note this forum is a godsend and Rik, Simon and the other guys (sorry if I didn't name you's) do a fantastic job really on Idnets behalf so heres a big thank you to them from me and I guess many others on this forum,
THANKS GUYS!
Supposedly I can get a Be connection at 13M, which is a little odd since I thought my exchange was only getting DSL2 enabled Q1 next year. Either way, if IDNet do go the way of a Be option it would be tempting for sure.
Quote from: Rudds on Oct 21, 2010, 10:56:28
On another note this forum is a godsend and Rik, Simon and the other guys (sorry if I didn't name you's) do a fantastic job really on Idnets behalf so heres a big thank you to them from me and I guess many others on this forum, THANKS GUYS!
Thanks, we appreciate that. :)
Thanks, Rudds, greatly appreciated, :thumb: :)
I prefer to stay with Idnet where I am kept informed promptly as to what is happening in plain English with none of the waffling that I get from some of other companies I have dealt with. Also many thanks to those core forum Idnetters who add that extra bit of info'.
I agree with you, Mike, regarding IDNet, though their attempts to communicate yesterday were hampered by BT's foul up. We try our best to extract and share information here, but at times, our best efforts come to nothing if BT act as idiotically as they did yesterday.
I'd also thank IDNet for looking at the bigger picture, and what is possible to help in the future. Even if the only result is a better relationship with BT. If they can help BT to streamline their fault finding. IE
If it quacks like a duck, check the ducks FIRST. Then, if that is wrong, use the trouble shooting list. However, going straight to the pig pen, then the chicken coop, then the cows, is wasting time when the ducks have flown the coop. Also, BT could do with a double checking system.
When in the bank, and even in other lesser important offices, we had double checked systems. I want to send a large amount of money between customers? It needs to be signed/password actioned by a manager, just to check it has a customer request, correct amount, and is legit. Granted, I've not worked in IT. But having an "undo" button is a godsend. ;)
But I guess, with cutbacks, and rushed jobs, mistakes happen. Unless BT are willing to fork out more on engineers, I don't see how to stop it.
If IDNet do look at other providers of the backend/hardware, it would be interesting. As said, BE are already reselling their equipment in a similar, but more costly, fashion to BT. It's a pity that Talktalk do not resell their LLU connections. As they have a very wide coverage (largest AFAIK) and also supply my exchange. However, Talktalk are on par with BT for service, if you get what I mean. ;)
I suppose the correct way to manage this, is to find a way to grow the relation ship and communication and trust with BT. So that they do their best to not allow this to happen in the future. However, if it's not in BTs interest, or they have no desire to, then we are stuck with them.
I'd imagine bad labelling came into play here somewhere, maybe the live fibre circuit needs a big yellow tag which says 'caution, live customer serving link, DO NOT disconnect'
The amount of money IDNet spend with BT would, you would have thought, got them some serious attention when things go wrong, but it seems BT can't be bothered or don't know how to deal with issues these days, Ben. :(
Quote from: pctech on Oct 21, 2010, 12:51:55
I'd imagine bad labelling came into play here somewhere, maybe the live fibre circuit needs a big yellow tag which says 'caution, live customer serving link, DO NOT disconnect'
The one which should have been live, but wasn't, probably did, Mitch. ;)
Quote from: Rik on Oct 21, 2010, 12:52:42
The amount of money IDNet spend with BT would, you would have thought, got them some serious attention when things go wrong, but it seems BT can't be bothered or don't know how to deal with issues these days, Ben. :(
A few years ago I had an idea to start my own ISP so started looking at the costs.
Looking then at the costs of the 622Mbps Centrals made my eyes water as they were at least 500 quid a month each and so I dropped the idea.
I know what you mean, Mitch, but having a whole gig hostlink to yourself does sound fun. ;)
;D
As I say I didn't get onto buying the routers and upstream links.
Quote from: Rik on Oct 21, 2010, 13:06:01
I know what you mean, Mitch, but having a whole gig hostlink to yourself does sound fun. ;)
Not after the last 2 weeks. Unless you like that kind of fun! ;D
On a side note. This is bad publicity for BT too. I'm in two minds about it really. Part of me says "IDNet can be honest, without embarrassing BT by exposing their faults" as this probably does not strengthen their relationship with BT. As BT would want to cover over (in a professional, not underhanded) such errors, fix them quickly, and learn from their mistakes. After all, we don't like it when people point out our faults as we walk down the street.
However, the other part of me says "But BT need to pull their socks up, perhaps making it clear what the fault is, is the only way to get them to do this". This will no doubt damage the relationship with BT. Sometimes the truth hurts, unless your willing to accept it. After all, companies that turn complaints into successes do better than those who don't even get complaints in the first place.
Basically, I'd understand if IDNet just said "BT has had a fault, we are investigating". But they currently go into a lot of detail. Which we are thankful for. I'm just not sure if BT are. :rofl:
BT probably regard IDNet as pussycats compared to AAISP, Ben, though the latter seem to have moderated their style lately.
As long as they are in our court! All the better. We can play Good ISP, Bad ISP. >:D
You can be both polite, calm, reasonable, and ROCK SOLID in your approach to complaints. I'm sure IDNet is. They would not be here if they were push overs. There are too many cowboys out there looking to rustle up some customers. So IDNets done a good job so far.
AAISP didn't seem to beat around the bush (neither do I) and I used to find reading their status page and the Director's own blog very amusing but yes they have toned it down.
I don't think BT care whether its good or bad publicity as the fact is they are in a win win situation as even LLU operators rely on them to service the copper loop, the so-called 'last mile'
At the end of the day no one really knows how the ISP talks to BT behind closed doors.
Take AAISP for example, what they say on the status messages could be a hard man act for the public but behind closed doors they could be kissing BT's arse!
Same could go for IDnet, IDnet could be taking the more gentle approach in the public but could be beating the cr*p out off the BT management as you read this ;D :evil:
I wouldn't mind 5 minutes with BT management and a baseball bat.
Quote from: Technical Ben on Oct 21, 2010, 13:16:22
Not after the last 2 weeks. Unless you like that kind of fun! ;D
On a side note. This is bad publicity for BT too. I'm in two minds about it really. Part of me says "IDNet can be honest, without embarrassing BT by exposing their faults" as this probably does not strengthen their relationship with BT. As BT would want to cover over (in a professional, not underhanded) such errors, fix them quickly, and learn from their mistakes. After all, we don't like it when people point out our faults as we walk down the street.
However, the other part of me says "But BT need to pull their socks up, perhaps making it clear what the fault is, is the only way to get them to do this". This will no doubt damage the relationship with BT. Sometimes the truth hurts, unless your willing to accept it. After all, companies that turn complaints into successes do better than those who don't even get complaints in the first place.
Basically, I'd understand if IDNet just said "BT has had a fault, we are investigating". But they currently go into a lot of detail. Which we are thankful for. I'm just not sure if BT are. :rofl:
Appeasing BT would be absolutely the wrong thing to do, period. It means they wouldn't take their responsibilities towards IDNet seriously, or I might say even less seriously than they do now. It's imperative that IDNet stand up to BT and hold them to their commitments. I deal with consumer rights issues on a daily basis, and I can tell you that there are two quite distinct corporate attitudes. One, from a company such as IDNet, is a genuine concern for their customers, a desire to keep people happy. When dealing with those companies, a flexible co-operative approach works very well, and is much better than blowing off steam and creating a bad atmosphere. The other, from a company such as BT, is very little concern for their customers (direct or indirect). This comes about due to a combination of poor management, unrealistic expectations and targets which force shortcuts to be taken, and a corporate structure that allows any one individual to avoid responsibility. Those companies operate on a line-of-least-resistance approach, and a co-operative and friendly attitude when dealing with them is taken as evidence that you can walked over without too many adverse consequences for them; they need to screw someone, and you're likely to make less fuss than the next man, so you're going to get it, regardless of whether or not you deserve it. In that situation, you have to find where the company is vulnerable, and you have to apply pressure there which unambiguously conveys the message that it's not in their best interests to screw you over; it will cost them more to do that than to do it right.
I think if anything, IDNet have been too co-operative towards BT historically. That's understandable, because Simon and Tim are basically nice guys who want to maintain good relations with their business partners. I think it's becoming evident even to them, however, that goodwill counts for nothing when dealing with a company like BT. It's only how much pain they believe you can cause them that counts in the end.
Quote from: pctech on Oct 21, 2010, 13:23:42
AAISP didn't seem to beat around the bush (neither do I) and I used to find reading their status page and the Director's own blog very amusing but yes they have toned it down.
I don't think BT care whether its good or bad publicity as the fact is they are in a win win situation as even LLU operators rely on them to service the copper loop, the so-called 'last mile'
They care about it only insofar as the dear old regulator is only ever motivated to do something by overwhelming public opinion applied via government pressure. Enough Daily Mail headlines, or irate punters in a DTI minister's constituency surgery, and the regulator is forced to overcome his natural inertia and take some action against BT, which they prefer not to happen. And make no mistake, the regulator will do nothing whatsoever without such pressure.
Quote from: Rudds on Oct 21, 2010, 10:56:28On another note this forum is a godsend and Rik, Simon and the other guys (sorry if I didn't name you's) do a fantastic job really on Idnets behalf so heres a big thank you to them from me and I guess many others on this forum, THANKS GUYS!
WHS - I may not use the forum very often, but the contributions of Rik, Simon and the rest of the crew are the other major reason for me staying with IDNet. A big
thank you from me too :thumb:
Quote from: DorsetBoy on Oct 21, 2010, 09:54:44
No, it means that providers outside of BT provide a link to the net.
If you could expand on your one liner I would be grateful.
Quote from: Steve on Oct 21, 2010, 09:58:33
But it gives a choice for those who may need it perhaps for reasons of speed or reliability.
But if you are on a small rural exchange only served by BT IPstream resellers, there's not much can be done, I take it. :(
What Dorset was saying was that, BE, like BT can provide a link to deliver traffic to an ISP but if BE don't have presence in an exchange (A point of presence) then it would require a BT ADSL link to take the traffic from the customer to a POP and then onward to the ISP.
My understanding as regards BE/O2's configuration is that, if the customer's exchange does not have a POP, the traffic is delivered to the core part of their ISP network via a BT hostlink, just like IDNet which would suggest the BT backhaul to a local POP isn't provided under a wholesale agreement.
I agree, Ben, I was just trying to emphasise that, because they don't publicly rant and rave doesn't mean that IDNet don't lean heavily.
Quote from: Rik on Oct 21, 2010, 14:50:54
I agree, Ben, I was just trying to emphasise that, because they don't publicly rant and rave doesn't mean that IDNet don't lean heavily.
Oh, I'm sure they do. I was just hoping BT don't throw a hissy fit over it all. ISP:"Go clean up your phone exchange/link" BT:"But I don't wanna! :bawl:"
Oh, and karvala, I've seen the "poor management, unrealistic expectations and targets which force shortcuts to be taken, and a corporate structure that allows any one individual to avoid responsibility." That kills a company and it's staff. Drains it of it's real resources. Sad so many take that approach.
Looking at other suppliers is fine but there is a huge caveat: act in haste, regret at length.
I have just migrated away from an ISP which changed wholesale supplier after some episodes of unreliability with their original supplier. The service then deteriorated in aspects apart from reliability of backbone to the point where I felt no option but to leave.
I am prepared to accept some outages. Always, I think it is right to judge an ISP by how they deal with problems rather than slam them for the occurrence of problems. I have had several ISPs who either did not respond adequately or who lied about it ("there is a fault affecting a very small minority of customers" etc.)
Faults will occur on any system. The only way to guard against faults is to employ multiply redundant configurations. Several concurrent wholesalers with duplication of routes. All the hi-tech kit duplicated several times with any one copy capable of carrying all traffic without degradation.
That can be done but not at £20 to £50 per month per customer.
If someone needs a guaranteed always-on service, they can pay for a dedicated adsl line with a SLA, and pay dearly.
So I think it is important not to overreact and kick BT too hard. Yes, they make mistakes. No they should not. Yes they should put some procedures in place to reduce the probability of a recurrence. But whatever they do and whoever supplies a service, no collection of humans can ever guarantee a 100% success record. The costs of going from 95% to 100% escalate dramatically with each increment of reliability and 100% is not attainable.
There is nothing in the laws of probability to say that a failure rate has to be evenly spread over time. Faults can arrive together, like buses.
I am not apologising for BT or idnet. I hate BT's help call centre as much as anyone. And I am just as annoyed when I lose my internet connection. Kneejerk reactions are never the way to go. By all means do the research. But ask customers for their opinion after, say, six months of faultless service and then see how keen people are to jump to a different provider.
Nicely put. :thumb:
On the whole, I would agree. However, I would like some resilience, though. Two outages in ten days has caused me work problems. I am thinking about a backup ISP or maybe investing in a 3G wireless router.
How about a PAYG mobile dongle, John?
Like all good ISPs IDNet have redundent upstream connections. (at least one American transit provider which is Cogent I believe plus links to several internet exchanges) so failure in an upstream link would result in automatic route re-selection via another link.
Unfortunately the same is not true regarding bringing customer traffic in, while BT can (in theory) re-route traffic between two nodes on its network in the case that for example, someone puts a digger through a fibre, the physical hand-off to the ISP is only as resilient as the number of links they rent so if they rent one and it goes down then thats tough until BT fix it.
As has been explained in IDNet's case they had two so the other should have taken the strain.
Now I wonder if the two links were fed by the same piece of kit and whether it maybe best to connect at another physical location in addition to what they have now?
Assuming BT offer them that option, Mitch? We shouldn't lose sight of the fact that for many customers, the switchover happened seamlessly as it was meant to (until some engineer decided to disconnect the working circuit!). Those that couldn't connect, 21CN customers, were, I suspect, victims of October 10 - I have a gut feeling that the configuration wiping wasn't fully corrected and, as a result, showed itself again yesterday. Purely my opinion, I have no information from IDNet on the issue yet - they are still waiting for explanations from BT.
Yes true, I don't know the physical topology of IDNet's set up but just maybe if they had one router in another DC (BT have presence in most if not all) then maybe the engineer would have thought before travelling as its a bit of a no brianer if two connectors are side by side?
I honestly don't know the answer, Mitch, but BT on both outages insisted on following flowcharts rather than engaging brains.
Perhaps the flowchart is the only method possible for the current crop of BT engineer, especially if those with knowledge and experience have now left for one reason or the other.
Sounds just like work.
Quote from: Steve on Oct 21, 2010, 19:13:07
Perhaps the flowchart is the only method possible for the current crop of BT engineer, especially if those with knowledge and experience have now left for one reason or the other.
Sadly, I think you've hit the nail on the head, Steve. The experience and skills have left the building, they were too expensive for BT. :(
Quote from: Rik on Oct 21, 2010, 18:53:44
How about a PAYG mobile dongle, John?
What I was thinking was along the lines that Dorset suggested. Something like a Draytek (http://www.draytek.co.uk/products/vigor2820.html) router that I can use for my regular connection, but that kicks in using a 3G modem if there is an outage.
However, the further problem is that here, mobile signals are a distant dream, but what seems possible is to invest in a Vodafone Sure Signal (http://online.vodafone.co.uk/dispatch/Portal/appmanager/vodafone/wrp?_nfpb=true&_pageLabel=templateCClamp&pageID=PPP_0161) and use it with the router and a mobile dongle.
I keep hearing about PAYG dongles, but I can't actually find one - they all seem to be contract. Given that I only want it for resilience, I don't really want a contract.
This aside, do you think this is all do-able?
Alternatively, I could just broadband enable my second phone line and put a cheap connection on it - but dealing with Plusnet, BT or Talk Talk etc. in order to achieve it, is really anathema to me. Also, I would have to mess around between routers when I want to use the second connection, albeit hopefully rarely.
John, a sure signal requires a working fixed ADSL connection so would kind of defeat the object.
:slap:
Why did you have to let a little detail like that get in the way of a good theory. ;D
Take a look here, John, top item:
http://threestore.three.co.uk/payg/?modem=1
Quote from: JohnH on Oct 21, 2010, 18:50:54
On the whole, I would agree. However, I would like some resilience, though. Two outages in ten days has caused me work problems. I am thinking about a backup ISP or maybe investing in a 3G wireless router.
This is the key for me as well. I realise that for many people occasional downtime presents no more than a minor inconvenience but for others it can be quite devastating. The recent outages have have had repercussions on my daughter's school work. Her computing course is completely dependent on having access to her school's on-line services and her tutors will not accept loss of connectivity as an excuse for failure to complete a task on time. My wife brings work home which requires Internet access. Last night's outage had her travelling back to do 30 minutes work, a three and a half hour round trip. Both my wife and daughter are looking firmly in my direction as to why this is happening and I think most people here know how much I personally rely on the Internet.
I guess I've been hit harder than most, there appears to be never ending exchange work and other work on the infrastructure that continually takes me out, especially after midnight. There should at least be a system in place for warning people of planned outages, there certainly used to be. In the 5 years or more I was with Pipex (pre Tiscali), I was always informed via email of any planned work that might effect my line and in all those years I had one solitary outage. Why is there no system in place to do this now?
I don't know what to do any more, I'm at my wits end to be honest.
Quote from: zappaDPJ on Oct 21, 2010, 19:35:49
Why is there no system in place to do this now?
There is, Zap, if BT tell ISPs in time. Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out. :(
Quote from: Rik on Oct 21, 2010, 19:45:49
There is, Zap, if BT tell ISPs in time. Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out. :(
Been there. Our systems use to go live on a Monday/Tuesday. Our weekly memos were sent out Wednesday. :slap: We did occasionally get a weeks notice.
;D The world conspires against us, Ben. :)
Quote from: Rik on Oct 21, 2010, 19:45:49
. . .Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out. :(
This is a bit of a diversion :) but I have been reading with interest of a puzzle in Cambridge city centre - a few days ago Anglian Water dug up a road to deal with a sewer collapse, and found a mysterious "subterranean chamber":
Chamber of secrets at site of sewer work (http://www.cambridge-news.co.uk/Home/Chamber-of-secrets-at-site-of-sewer-work.htm)
"
It is this, rather than a collapsed sewer, that caused a depression to appear in the road. . .
Whoever built the concrete chamber also sliced a sewer pipe in half – taking away six metres of the tube – and then attempted to repair it using a hose.
Water company bosses said the chamber had been built "right on top of a Victorian clay pipe" and that it had been constructed "recently".
John Clare, a spokesman for Anglian Water, said: "It's very mysterious. We don't know who built this. The engineers are baffled. The sewer has not collapsed as we thought – it has been effectively removed by a third party."
Today: Mystery of cavern solved: (http://www.cambridge-news.co.uk/Home/Mystery-of-cavern-solved-but-repairs-will-last-weeks.htm)
"
BT has now admitted that it was responsible for the cavern – which is full of fibre optic and phone lines, as well as several feet of water. It was built during the construction of the Grand Arcade in 2004."
It seems BT forgot to tell anyone ... :not:
Look at the link for a photo of the hose BT used to connect the two sections of severed sewer
Marvellous story Sarah. Are we surprised though at the guilty party. :karma:
This is a big story as the site in question is probably the busiest bus, taxi and pedestrian intersection in the centre of the old city. Any subterranean earthworks here are a public spectacle, and everyone takes an interest :)
That's certainly criminal damages, no?
It's unbelievable that BT behaved in such a cavalier way - they seem to think they are outside all man made controls. :shake:
Quote from: Rik on Oct 22, 2010, 14:49:50
It's unbelievable that BT behaved in such a cavalier way - they seem to think they are outside all man made controls. :shake:
And those above man. ::)
;D
We've had an Interim report from BT concerning the outage on the 10th. We are forbidden from publishing it. My interpretation is that BT were running an internal clerical programme to delete old configuration data that was no longer active. Our config was deleted in the management database and that this dataset change was automatically pushed out to the routers that our Broadband Gigabit Host Links are connected to. We have been advised that this programme has been halted whilst a root cause analysis is undertaken.
We understand that humans can make mistakes though with proper training and supervision followed by sanity checks something this calamitous should have been avoided. The big issue here is the length of time that it took to be resolved despite the information that we provided during the course of the outage. We are arranging a meeting with BT Management to review the two outages and we expect them to follow this with a review of their fault investigation procedures.
Simon
Thanks for the update, Simon
I thought they are likely to be prevented from making it public.
Thanks for that, Simon. Classic BT forethought. :sigh:
BT= Ballsup Telecom ?
That's the polite version.
Indeed, I really feel quite sorry for the IDNet staff, I imagine they got an earful from a lot of customers.
Not to mention MAC requests I'd imagine, Mitch.
Sadly yes Rik.
Seems to me these outages could happen to any ISP. IDNET have done a good keeping us in the picture. See no point in beating them up or moving ISP. Certainly see no reason to dish it out to Tech Support - ever - it does just not help anyone. Would never dream of using essentially a BT domestic service for mission critical work, or really any homeworking that relies on it. I barely noticed the various outages and frankly consider them a plain fact of life with a fragile technology and not worth the excitement generated.
have they already started with the new apprentice scheme and let some kid out of school loose on the system unassisted
thats what they cant tell us ::)
It feels like it, Baz. :(
'Pressing that big red button makes the connection go faster, innit'
;D Too near reality for comfort, Mitch.
I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. :slap:
It does paint a sorry picture, doesn't it.
Quote from: Rik on Oct 26, 2010, 16:28:00
Not to mention MAC requests I'd imagine, Mitch.
I'm still here ;D
And to be fair my connection hasn't dropped once since the last major outage... wood... touching... :pray:
Thanks for the update Simon.
;D
We glued your cables in, Zap. :)
Thanks ;D
Quote from: Den on Oct 26, 2010, 18:30:32
I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. :slap:
Oh dear,
Alarm engineers seem to know more about the kit than BT, then.
I suppose cos ones an engineer and the other's a Bloody Technician :whistle:
;D
Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers? :slap:
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors. ???
I think BT regards its customers as the testers, Ben. :(
Ironically I did have an outage last night at around 12.15am. Luckily it was very short lived and given the timing I'd guess at yet more exchange work.
BT can't help themselves. ;)
Quote from: Technical Ben on Oct 26, 2010, 21:28:35
Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers? :slap:
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors. ???
If a Cisco router can't find a startup config it will search the network for a TFTP server containing a confg file.
We've still not received an Incident Report from BT covering the outage on the 20th Oct - 4 weeks ago now! We have been informally advised that the repair work undertaken on the 10th Oct was incomplete such that the fibre break on the 20th exposed the faulty repair of the 10th (we'd already worked that out). We've been chasing weekly for a full report but apparently the only two guys who can perform a full audit are busy with the current BRAS upgrade programme because it is "not going very well at all" ...!
I think we'd noticed the latter, Simon. They're probably camped out in Edinburgh. :(
Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care. :shake:
Haha, good grief. That is just amazing. Good to know the BRAS upgrades are going as well as your fixes :whistle:
You get top marks for effort :)
Incredible! :shake:
Thanks for the update Simon.
Quote from: Rik on Nov 17, 2010, 15:47:11
Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care. :shake:
Bang on Rik, sadly IDNet is just another BT BTnet customer.
To be treated with total disdain. >:(
No matter what they may say, I reckon BT Retail probably gets the best treatment.
I wouldn't be at all surprised, Mitch. Proving it might be more difficult, of course.
We've finally had an Incient Report from BT covering the outage on 20th October.
There are some glaring inaccuracies regarding the timeline which we will be taking up with BT Management as, at the end of the day, our main issue is with the length of time that it took them to resolve the fault. Frustratingly, due to BT personnel availability, this meeting cannot now take place until the New Year.
To recap, one of the Gigabit links suffered a fibre break. All traffic should have failed over to the other Gigabit link but only 20CN (ADSL Max) connections successfully failed over. None of the 21CN (ADSL2+ and FTTC) traffic that had been on the broken fibre circuit managed to fail over. This was because the BT software configuration had not been reinstated correctly after the outage on the 10th October.
On a positive note, BT have now audited all other ISPs connections to check that no one else has been misconfigured similarly and they have also reviewed their configuration proceses and training and have implemented a revised template which includes quality checks throughout the setup.
Sounds like progress, Simon, but it's small consolation for the grief it caused you. :( Are they also going to train engineers not to disconnect the working circuit? ;D
Quote from: Den on Oct 26, 2010, 18:30:32
I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. :slap:
Does not surprise me I was in telecoms moved to alarms for a few years classic i remember was getting a call to a direct line site.
BT eng had dissed it and said to me he was going to trace it back as it had no dial tone so was faulty.
took me about 10 minutes to compose myself (stop laughing) and point out the error of his ways.
direct lines are what they say they go direct from customer to central monitoring station albeit provided by BT but do not have any BT equipment on them.
Simon at your meeting could you bring up the fact of BT help and support staff should be back in this country and given much better training and possibly look at your system which is a becaon in the dark lands of help and support. :) Then again they wouldn't listen which is they way they have been trained. :evil:
I had a meeting last week wih the new man at BT who is in charge of WBC. I went through the Incident Reports with him of our two major outages in October last year. He understood my belief that resolutions to both indicents should have been resolved much quicker and that the delays were caused by systemic process errors within BT and has promised to investigate and report back.
Promises, promises... Thanks for the update, Simon. :)
Glad that things are moving forward. Just imagine if it was Sir Alan Sugar involved on one or the other side. :eek4: