“when a Metro subway train rammed into another train in Washington, D.C. last June, designers had to confront the unpleasant reality that automation may have been the cause.
The accident, which killed nine people and injured 80, may have been rooted in a computer malfunction and the operator’s inability to manually apply the brakes quickly enough.
The Metro train accident lies at the heart of what human factors experts refer to as the “automation paradox.” As automated systems become increasingly reliable and efficient, the more likely it is that human operators will mentally “switch off” and rely upon the automated system. And as the automated system becomes more complex, the odds of an accident or mishap may
diminish, but the severity of a failure is often amplified.”
“Making Automation Work“, di Samuel Greengard, in Communications of the ACM, dicembre 2009