In my opinion a "kill switch" (or any similar solution) is not the answer and shall not be the answer. Instead, proper application of existing functional safety standards is what would do the job. Tell you why: Self driving cars differ from human drivers in one very important regard. Their software does not change over time, as it may be the case with human drivers (if you allow me to name human capabilities "software"). This fact enables you to design a self driving car with a proper level of a) safety and b) safety integrity and assume with good reason, that (due to proper safety integrity) your safety will stand the test of time, even though parts of your hardware may fail. You cannot do this with human drivers. These you would have to "re-test" every now and then to make sure they are "still safe". In addition to that, human drivers perform very differently when tired, stressed, distracted, ...

This being said, why do self driving cars have these accidents? From my point of view, there are two main reasons:

1) (referring to a) above) Anticipation of safety problems and the actions taken by car makers are not yet at a stage that can prevent accidents safely enough. There are still problems with different types of sensors (camera, radar, laser scanners) who all have their drawbacks and their blind spots. Furthermore, the scenarios how cars can fail are not evaluated and / or taken care of to the extend they have to be to ensure the safety we all wish for.

2) (referring to b) above) Existing standards for safety integrity (e.g. ISO 26262) are often not applied the way they should be to make the car reliable enough and make it cope with hardware failures and environmental changes.

And then there is another factor, which is sometimes neglected by many people: There is no 100% safety. In critical systems we may come close to that (when safety and safety integrity standards are properly applied), but there will always be accidents. The question is how severe and often they will be. At the moment self driving cars drive significantly safer than human drivers (even though people come to harm), we have already gained at least something. Still, I think that they could be safer, even with the technical challenges we face today, if car makers were more sensitive to proper safety / safety integrity development, which unfortunately is an expensive discipline.

Yet, cars with proper safety design and safety integrity will not need to be decommissioned. If you do your job right, you know how safe a system is (and with what chance it will stay safe over time) before you put it on the road.

Anyhow, if cars have severe problems nowadays, the manufacturers usually are calling them back and remedy the problem. If it is not too severe, they update the car during the next inspection / repair.

Hi Oliver,

Some good thoughts in the reply, you raise many points.

I question the commercial success of driverless vehicles based on the impact on multi-vehicle users such as taxi firms and large scale hauliers, the audience likely to make the most operating cost savings if the equipment is reliable.

In terms of technical aspects, unless someone can identify every conceivable situation that a vehicle can get into (which we both agree is not a realistic prospect) there will need to be an ongoing development process which is almost certainly going to involve AI and a learning concept. The control system will be self modifying and will change over time. It will probably conduct on-air updates (in the same way that Tesla have done), the updates may be in "Torrent" style directly from other connected vehicles (shared learning) mainly because the scale of a central repository for a production scale global deployment is massively different to the c.200 self driving Ubers that are currently under test.

As for vehicle call back, this is exactly one of my points. A "total loss of steering control" or "braking system will fail to operate near pedestrians with bicyles" recall is not in the same category as a "recall for possible unexpected airbag deployment". If a critical fault is identified, as may be the case with the Uber accident, how do you prevent all production vehicles from making the same "mistake". For Uber it was relatively easy, just tell the 200 existing drivers that they should not use autonomous mode, this is probably not possible for e.g. all Volvo vehicles across the world. How would you inform them? how would you make sure they received the message? How do you make sure that they comply? How would you update them simultaneously? This could be many hundreds of millions of vehicles all on the same day. Don't forget that if the dream comes true, none of these vehicles will have a driver to actually take control.

If a manufacturer advises you (as a person) of a recall stating that you should not drive your vehicle and you choose to ignore the advice then you (as the control system) are taking the risk (operating a dangerous vehicle). Similar rules will almost certainly apply to manufacturers if they choose to allow the driverless control system to continue to operate. Apart from everything else, how inconvenient will this be for all the tired, drunk, drug impaired, blind, underage, unlicensed, ... vehicle occupants at the time that the decision is made?

As for safety, which is another separate issue that I tried to avoid in my original comment, Uber (and others) imply how safe they are. Uber stated recently that they have driven a million miles in a driverless mode which sounds impressive if you have no other information. Government Accident figures for the UK in 2013 (according to pdf "Facts on road fatalities - Gov.uk") show that there were 5.6 (five point six) "all causes" road deaths per Billion miles (2016 is similar). The rate is higher in the US, but not by orders of magnitude.The "private hire / taxi" fatality rate is much lower than for most other vehicle categories. Given this information you can make comparisons. Uber are at around 1000 fatalities per billion miles in a very restricted operating regime with a safety co-pilot and they do not even know why. Safer? Really? How long can you tolerate a 200x increase in road deaths in the hope that one day it will become safer?

Hi James,

thank you for that answer, I appreciate these details a lot. With these you point at some problems that I am about to push forward within our company: human factors engineering (taking into account how people do behave and will behave under certain circumstances) and safety in machine learning. Furthermore the close relation between security (especially during remote update) and safety is crucial.

All I wrote in my first answer is from a "traditional" safety perspective along the processes of ISO 26262. The two problems are:

1. Car makers do not apply these to the extend they should (my personal opinion, anyone may feel free to show me the opposite). No matter how many miles a car (or many of them) drive without accident - this is a poor indicator for safety, because all these many cars along all these many miles may never have seen the really safety critical situations. These can only be found systematically and this is the gap I see.

2. These processes do not cover machine learning at all and "safety and machine learning" is still under discussion in many papers and all the companies. There is progress, but we are not there yet.

So I agree, if a cars behaviour can change over time due to updates and learning ("intelligence") we need a completely different approach to safety, not "only" application of current development and testing standards. And that will probably imply a quick fallback solution in case of systematic failures, whith all the critical implications you brought up. I will take these thoughts into my presentation and raise these questions in our safety community (company-internal). A collegue just tries to raise funds for an internal project concerning AI and safety, so we may (hopefully) have some resources to work on that.

I will bookmark the link to this discussion and post updates!

Hi Oliver, I have spent quite a lot of time considering the human factors approach to engineering, it is a fascinating area to work in. Simulating people related road events is very difficult as I am sure you are aware. Covering the "once in a lifetime" events realistically is a near impossibility.

I too will keep a check on this post.

International collaborations between academic and industrial partners are usually quite a good way to attract external funding if you are interested.

Please feel free to contact me directly through email, my address is easy to find on the www.derby.ac.uk website or from the heading of my RG submissions e.g.

Conference Paper Available Forward Road Capacity Detection Algorithms to Redu...