Monday, July 16, 2012

OK, Computer!



According to this January 2011 "System Safety Program Plan," operational "reliability is provided for the ROCC [Metro's nerve center] systems by a back-up computer, which automatically activates if the primary control computer malfunctions."

There was no sign of any automatic, redundant system functioning this weekend, and operations for an entire metropolitan subway system were halted two times, once for nearly an hour.

According to Metro, "the computer problem affected an information management system that allows controllers in Metro's Rail Operations Control Center to see where trains are on a dynamic map and to remotely control switches."

It's called a code 34, and many riders reported hearing that throughout the system last night.



Recently, there have been at least two documented code 34 events, one this past March, and another in October of 2010. Metro sources tell me there have been more.

The weekend computer failure has some Metro workers scratching their heads because Metro recently built a back-up system for bus and rail OCC systems costing millions of dollars in Landover. It is unclear whether that particular back-up system is for cases like this weekend's, for a destructive event such as a fire--or both.

The Landover back ups were installed after the authority's inspector general criticized Metro for lacking IT contingency plans in a September 2010 internal audit.

One would think a fundamental requirement of any back-up system would be to avoid the need to completely stop operations.

Metro needs to explain to riders whether its back-up systems work, and if they do, why aren't they good enough to prevent the entire system from shutting down two times in less than a day?

It should be noted that while the weekend's events are alarming, sources confirm Dan Stessel's comments to WTOP that "the signal system, the system that keeps trains properly spaced from each other, did remain operational. Those systems were up and running at all times."

Other items:
Accountability lacking in Metro's IT department (Examiner)

Comments (41)

Loading... Logging you in...
  • Logged in as
Customers should be advised that there were no delays, as the outage lengths were within acceptable margins according to our metrics. Thank you for your patience as we continue to downgrade and destroy the Metrofail system.
The Anti-TB Guy's avatar

The Anti-TB Guy · 662 weeks ago

Code 34?? I wonder what failure codes 1 through 33 stand for?
The whole situation is beginning to sound like it's gone Code Blue...
2 replies · active 611 weeks ago
OCC can give absolute blocks to 10 feet of the train ahead of you. This did not have to be a stand still.
Metro wants more money. Anytime Metro inconveniences the customers and the system glitches become absolute failure like this....Metro is trying to get more money for repairs, trains escalators, salaries, managers salaries, new computers, new radios.

HOW ABOUT CUTTING THE 3 FIGURE SALARIES of these EMPLOYEES and MANAGERS!!!!!!!!
THE employees are causing the continual breakdowns and failure by Metro having a need for begging for more money!!!
GET READY........metro will soon ask for more money!
There are NO failure codes 1 through 33. Code 34 is meant to be a deployment of personnel at key interlockings (i.e., intersections for automobiles) to perform manual block operations of the trains safely, like a traffic cop, for automobiles.
Has the ROCC computer joined the union?
Dan Stessel: Please keep the trains running HAL.
HAL: Sorry Dan I can't do that.
1 reply · active 662 weeks ago
Ever n Anon's avatar

Ever n Anon · 662 weeks ago

My vote for post of the day! LOL!
Sat, 07/14, 2:00 Hour, Potomac Ave Station, waited 10-15 minutes, no announcements of what was happening, crowd kept growing, we gave up and exitied where SmartCard, METRO charged $1.70 for Inconvenience of being stranded on a Steam Bath Platform. WTH??? METRO really sucks to charge people that METRO FAILED to provide Services to. METRO should review records where people entered & exited stations within 30 minutes and credit back the money METRO stole.
3 replies · active 662 weeks ago
fsdkljflksjd's avatar

fsdkljflksjd · 662 weeks ago

It wouldn't be very hard to have a system in place where fares were not charged if the same card was used to exit w/in a short time frame, say two minutes. None of this matters though because metro knows it's easier and they benefit by not having such a system in place. If metro lost money you can believe they would act on it.
Oh, right, like how they acted after learning they were being scammed for thousands of dollars at their parking lots? It took them literally years to figure out they were losing money, and it wasn't chump change. They're too dumb to know when they're losing money.

Heck, they're losing money now because people are leaving them whenever possible.
ditto for the two revenue collectors that stole hundreds of thousands of dollars over many years. They only got found out when they the convenience store clerk got tired of them buying lottery tickets and paying in quarters every time. Metro had no idea money was walking right out of their front door.
I have many fond memories of the TRS-80 Model 1 computer--it was a good computer for its time. Of course, that time was 1980, and the monitor made watching Channel 3 nearly impossible for the family members nearby. Ah, there's the analogy for Metro!
Bitter Brew's avatar

Bitter Brew · 662 weeks ago

@Stan Dessel: You bought $11,000 of Macs for your office? What is that, $100 / tweet / year?
3 replies · active 662 weeks ago
If they do design of any sort it might make sense. They are the favored device of designers and other types of artists.

If he is only doing tweets though, no, it does not make sense.
Bitter Brew's avatar

Bitter Brew · 662 weeks ago

Metro's defense - it was:
"based on the need for better technology for our communications outreach."

That sounds like Twitter, not art/design to me. Anyway, didn't we learn that they outsourced the $400,000 of over-designed and worthless Rush+ brochures?
Right, Dan has no need for it. My point was there could be a legit case for the relevant department and I often see that done.

My fave was that they needed to develop inventory software. That is an off-the-shelf product, not something you reinvent each time.
Kid Charlemagne's avatar

Kid Charlemagne · 662 weeks ago

I picked the wrong week to stop sniffing glue!
>10 PRINT "EXPECT DELAYS";
>20 GOTO 10
2 replies · active 662 weeks ago
Hahahah! That's about as far as my "programming" ever got.
Make it

>5 CLS
>10 PRINT "EXPECT DELAYS"
>20 GOTO 5

and you've got a TRS-80 classic. When you'd do an instant clear-screen like that, the letters would start to phase in and out or flash. I wonder if you could do something like that with the PIDs? :-)
Vienna Bound's avatar

Vienna Bound · 662 weeks ago

I hope there will be a system-wide review of all processes and the back-ups. What if a computer 'glitch' caused all the signal lights to be green and there were multiple collisions? What if a computer 'glitch' caused all communications to fail? What if a computer 'glitch' occurred during a freak summer storm that took out cellular communication centers and 911 call centers so that operators couldn't communicate with central control with their personal cell phones? We already know that the K-Mart blue light special phones along the tracks may or may not work depending upon which department fixed the door panel or phone first.

How often are these back up systems checked to ensure tat they are functioning propertly?
3 replies · active 662 weeks ago
"How often are these back up systems checked to ensure tat they are functioning propertly?"

This sounds suspiciously like preventative maintenance. You're not suggesting that Metro would do anything BEFORE somebody dies, are you?
Vienna Bound's avatar

Vienna Bound · 662 weeks ago

Oops! My bad. It's Monday. I forgot that Metro's slogan is "Honey Badger don't care."
Actually, I think it's more like 'if you believe in caring, you believe in slavery'
fsdkljflksjd's avatar

fsdkljflksjd · 662 weeks ago

My parents are in town visiting and they wanted to take metro to go to the zoo. Shady Grove to Woodley Park would cost the six of us $67.50 using paper farecards and including one peak trip. My parents asked if I was joking. Ahh, the memories I have of a nice smooth train ride, not being jerked around, literally, and the 10% bonus for putting $20 on a card, oh that glorious 10% bonus, gone forever!
2 replies · active 662 weeks ago
hmm, maybe all these wmata problems are a nefarious scheme by the cabbies to get more people into taxis? :-)
I noticed a typo.
You put "nefarious" instead of "cunning."
Mariterri's avatar

Mariterri · 662 weeks ago

I just think that we are at the point of no return with Metro. The system is too broken to fix; it can only just sputter along and we can be grateful that we made it to work alive if only a little sweaty. I have been riding metro since 1989, and my take is that until the DC, VA, MD and federal governments figure out a cohesive method to enforce accountability in both safety and funding, we are screwed. Too many cooks in the kitchen with too many differing agendas and too little accountability beyond the latest soundbite or twitter feed. Heavy sigh with little hope (I mean, even the deaths of riders and workers have done little to change the system).
Ritt Momney's avatar

Ritt Momney · 662 weeks ago

Metro fixed the computer problems retroactively.
According to the AP, the 3-year old $20M central computer system that
monitors the position of every train in the Washington, D.C., Metrorail
system failed on the morning of 24 Sep 1999. The backup involved personnel
with walkie-talkies along 96 miles of track monitoring trains. As a result,
the morning startup was delayed by half an hour.

[The cause was not identified. But this was reportedly the first time in
23 years that the start of daily service was delayed!
"A graphics generating device for Metro's finicky central computer
system froze about 3:20 a.m., sending Metro managers racing to fix it
before the scheduled start of daily service at 5:30 a.m. But they
couldn't restore it until 5:46 a.m., which meant normal passenger
service didn't begin rolling until about 6:15 a.m." -- resulting in
45-minute delays to start the day. "Last fall, the system crashed
several times during rush hour, including one episode similar to
yesterday's when computer-generated views of sections of the system were
blacked out for nearly two hours. In the 15 months after the system was
installed by McLean-based BDM International, it crashed 50 times."
Source: Computer Failure Puzzles Metro Opening Delayed, Rush Hour
Slowed, by Lyndsey Layton, *The Washington Post*, Saturday, September 25,
1999, Page B01, courtesy of Keith Rhodes; PGN-ed http://www.washingtonpost.com/wp-srv/WPlate/1999-...
The LCARS system wouldn't have never crashed
2 replies · active 662 weeks ago
hrh king friday 13's avatar

hrh king friday 13 · 662 weeks ago

Isolinear chips, eh?
or some Dilithium Crystals
UnSuck Fan's avatar

UnSuck Fan · 662 weeks ago

"It is unclear whether that particular back-up system is for cases like this weekend's, for a destructive event such as a fire--or both."

Events are too tame. Perhaps when The Apocalypse is upon us it will work.
Tommyboy74's avatar

Tommyboy74 · 662 weeks ago

Here's an idea, and a perfect way to bust the messed up unions in the system. Some cities have outsourced rail operations and even repair functions. Example- in Toronto, GO Transit was recently outsourced to Bombardier, who is responsible for staffing and operating the trains. In London, Alstom is responsible for railcar maintenance and operations. And guess what- reliability and on time performance improved in those cases.

Unfortunately, we all know what Jackie and the ATU would say if that were to ever be proposed let alone become reality...
Made do without a car for six years in DC, Metro finally drove me to it and couldn't be happier. Now only take metro to and from work
Anyone know what happened to the Post's Metro reporter? Recent stories on the system have been written by random reporters. And I'm not just talking about the fluff piece about the Hispanic man in Metro's private printing plant that "works harder than the new generation."
1 reply · active 662 weeks ago
I think she had a kid.

Post a new comment

Comments by

 
Creative Commons License
This work is licensed under a
Creative Commons Attribution-Noncommercial 3.0 Unported License.
Site Meter