Thanks to a self driving car's numerous sensors, a fully functional self-driving car would be safer than a human's driving. Even some of the cars being tested already see lower accident rates than the national average. Keep in mind that there's still a lot of work to be done to bring these cars to market—once they're perfected, they'd be even safer.
Besides the actual numbers, many on the pro self-driving side don't see much of a difference between jumping into an Uber or Lyft with a stranger and trusting a computer to drive. At the end of the day, you don't really know what their driving capabilities are like but you trust them anyway. At least with self-driving cars, you know a lot of engineering and money went into the thing to get it right.
Self-driving detractors say there's a fundamental gulf of trust that's impossible to overcome. People spend their entire lives driving. Some people don't trust anyone but themselves to drive. Then there are people who genuinely enjoy the freedom of the road. There will always be hold outs who won't trust self-driving cars.
Besides, who can blame them when the companies that are making self-driving cars a reality keep on messing up in very public ways.