
According to this January 2011 "System Safety Program Plan," operational "reliability is provided for the ROCC [Metro's nerve center] systems by a back-up computer, which automatically activates if the primary control computer malfunctions."
There was no sign of any automatic, redundant system functioning this weekend, and operations for an entire metropolitan subway system were halted two times, once for nearly an hour.
According to Metro, "the computer problem affected an information management system that allows controllers in Metro's Rail Operations Control Center to see where trains are on a dynamic map and to remotely control switches."
It's called a code 34, and many riders reported hearing that throughout the system last night.
"code 34" is in effect on @wmata right now. Entire system is at a standstill... does anyone know what a code 34 means? @unsuckdcmetro
— Ohmygoshi (@Ohmygoshi) July 15, 2012
Recently, there have been at least two documented code 34 events, one this past March, and another in October of 2010. Metro sources tell me there have been more.
The weekend computer failure has some Metro workers scratching their heads because Metro recently built a back-up system for bus and rail OCC systems costing millions of dollars in Landover. It is unclear whether that particular back-up system is for cases like this weekend's, for a destructive event such as a fire--or both.
The Landover back ups were installed after the authority's inspector general criticized Metro for lacking IT contingency plans in a September 2010 internal audit.
One would think a fundamental requirement of any back-up system would be to avoid the need to completely stop operations.
Metro needs to explain to riders whether its back-up systems work, and if they do, why aren't they good enough to prevent the entire system from shutting down two times in less than a day?
It should be noted that while the weekend's events are alarming, sources confirm Dan Stessel's comments to WTOP that "the signal system, the system that keeps trains properly spaced from each other, did remain operational. Those systems were up and running at all times."
Other items:
Accountability lacking in Metro's IT department (Examiner)
@wmata · 662 weeks ago
The Anti-TB Guy · 662 weeks ago
The whole situation is beginning to sound like it's gone Code Blue...
Guest · 662 weeks ago
Metro wants more money. Anytime Metro inconveniences the customers and the system glitches become absolute failure like this....Metro is trying to get more money for repairs, trains escalators, salaries, managers salaries, new computers, new radios.
HOW ABOUT CUTTING THE 3 FIGURE SALARIES of these EMPLOYEES and MANAGERS!!!!!!!!
THE employees are causing the continual breakdowns and failure by Metro having a need for begging for more money!!!
GET READY........metro will soon ask for more money!
Cary · 611 weeks ago
Anon · 662 weeks ago
wesindc · 662 weeks ago
HAL: Sorry Dan I can't do that.
Ever n Anon · 662 weeks ago
SEJOJO · 662 weeks ago
fsdkljflksjd · 662 weeks ago
anon · 662 weeks ago
Heck, they're losing money now because people are leaving them whenever possible.
Andrew · 662 weeks ago
jkuchen · 662 weeks ago
Bitter Brew · 662 weeks ago
Kara · 662 weeks ago
If he is only doing tweets though, no, it does not make sense.
Bitter Brew · 662 weeks ago
"based on the need for better technology for our communications outreach."
That sounds like Twitter, not art/design to me. Anyway, didn't we learn that they outsourced the $400,000 of over-designed and worthless Rush+ brochures?
Kara · 662 weeks ago
My fave was that they needed to develop inventory software. That is an off-the-shelf product, not something you reinvent each time.
@VeggieTart · 662 weeks ago
And having TWO frozen escalators in a bank of three, with one going down is inexcusable. I'm just glad I caught the bus.
@suchfail · 662 weeks ago
Seriously though, inexcusable that this delayed the ENTIRE system. It s hows OCC does note trust their operators to provide VALID POSITION REPORTS. The signals still work, so you have your red and green signals -- Thats ALL you need. More incompotence.
Kid Charlemagne · 662 weeks ago
basic · 662 weeks ago
>20 GOTO 10
unsuckdcmetro 92p · 662 weeks ago
jkuchen · 662 weeks ago
>5 CLS
>10 PRINT "EXPECT DELAYS"
>20 GOTO 5
and you've got a TRS-80 classic. When you'd do an instant clear-screen like that, the letters would start to phase in and out or flash. I wonder if you could do something like that with the PIDs? :-)
Vienna Bound · 662 weeks ago
How often are these back up systems checked to ensure tat they are functioning propertly?
Jason · 662 weeks ago
This sounds suspiciously like preventative maintenance. You're not suggesting that Metro would do anything BEFORE somebody dies, are you?
Vienna Bound · 662 weeks ago
Neenya · 662 weeks ago
fsdkljflksjd · 662 weeks ago
guest · 662 weeks ago
Whoops · 662 weeks ago
You put "nefarious" instead of "cunning."
Mariterri · 662 weeks ago
guest · 662 weeks ago
http://wtop.com/654/2944758/New-rail-transit-safe...
Ritt Momney · 662 weeks ago
Jazzie B · 662 weeks ago
monitors the position of every train in the Washington, D.C., Metrorail
system failed on the morning of 24 Sep 1999. The backup involved personnel
with walkie-talkies along 96 miles of track monitoring trains. As a result,
the morning startup was delayed by half an hour.
[The cause was not identified. But this was reportedly the first time in
23 years that the start of daily service was delayed!
"A graphics generating device for Metro's finicky central computer
system froze about 3:20 a.m., sending Metro managers racing to fix it
before the scheduled start of daily service at 5:30 a.m. But they
couldn't restore it until 5:46 a.m., which meant normal passenger
service didn't begin rolling until about 6:15 a.m." -- resulting in
45-minute delays to start the day. "Last fall, the system crashed
several times during rush hour, including one episode similar to
yesterday's when computer-generated views of sections of the system were
blacked out for nearly two hours. In the 15 months after the system was
installed by McLean-based BDM International, it crashed 50 times."
Source: Computer Failure Puzzles Metro Opening Delayed, Rush Hour
Slowed, by Lyndsey Layton, *The Washington Post*, Saturday, September 25,
1999, Page B01, courtesy of Keith Rhodes; PGN-ed http://www.washingtonpost.com/wp-srv/WPlate/1999-...
mike · 662 weeks ago
hrh king friday 13 · 662 weeks ago
mike · 662 weeks ago
UnSuck Fan · 662 weeks ago
Events are too tame. Perhaps when The Apocalypse is upon us it will work.
Tommyboy74 · 662 weeks ago
Unfortunately, we all know what Jackie and the ATU would say if that were to ever be proposed let alone become reality...
Anonymous · 662 weeks ago
Anon · 662 weeks ago
unsuckdcmetro 92p · 662 weeks ago