Statistically, a perfected self-driving car is safer than a human driver, so it really comes down a matter of opinion and perception.
At some point, this technology will become so advanced that lawmakers will be forced to debate whether or not to outlaw manual driving, according to Tesla CEO Elon Musk. There's an argument for it: Deaths and injuries from auto accidents are bad. They would decrease drastically if human error were to cease to be a liability. Plus, you can't drive drunk if you can't actually drive. The counterargument against this perspective, outside of the rah-rah "freedom" appeals, would necessarily center on security. Could hackers take control of your car while it's on the road?
Automating vehicles creates a whole new sets of risks.
“The motor industry is really strong on safety but if someone tries to interfere with the vehicle, tries to hack it and disrupt it, then these don’t fall under the typical safety issues. ⠀⠀⠀⠀⠀⠀⠀⠀ “Unfortunately living in the world today people do try to tamper with technology. The industry is only just starting to recognise this.” ⠀⠀⠀⠀⠀⠀⠀⠀ He also said that software would have to be reliable and bug free. “Recent reports analysing software show that 98% of applications have serious defects and in many cases there were 10-15 defects per application,” he said.