Given the move toward driverless cars, which includes the more short-term goal of driving assistance, what the appropriate shared authority and interaction paradigms should be between human drivers and the automation remains an open question until more principled research and testing has occurred. It is unclear at this time how robust driverless cars are to system failures (including human failures) and operations in degraded sensor environments. Automation onboard such vehicles is inherently brittle and can only account for what it is programmed to consider. Communication
between what is technically a very complex system to a human population of extreme variability in driving skills and attention management will be difficult, since the driver will need to be appropriately informed of the state of the system, including limitations, and will need to build appropriate trust in the automation’s capabilities (neither too much or too little). Further complicating this problem is the significant body of research demonstrating that automated systems can lead to boredom, which encourages distraction. This leaves operators unaware of the state of the vehicle (aka, mode confusion) and ill-suited to respond quickly and appropriately in case of a potential accident. Over time, operator skill degradation due to automation use can further reduce the human ability to respond to emergent driving demands, and will likely lead to risk homeostasis even in normal operations. Each of these issues are well-known to the human systems engineering community, but it is unclear that these issues are being considered by driverless car designers or that manufactures are conducting human-in-the-loop tests with representative members of the driving population. Until these tests show that the vehicles account for the aforementioned issues, driverless cars will not be safe for unrestricted access and use on U.S. roadways. Moreover, there are significant socio-technical considerations that do not appear to be a concern in the push to introduce this technology on a wide scale. The utilitarian approach quoted by many in the press, i.e., that driverless cars will eventually kill people but that this should be acceptable due to the likely
reduction in overall deaths (which is not yet proven) demonstrates an insensitivity to a deontological perspective that causes many people to be uncomfortable with such a significant shift in responsibility and accountability to computers
