Kavanaugh: A Probability Poll

There’s some literature suggesting that people are more careful when they think in probabilities. If you ask them for a definite answer, they might give it and sound very confident, but if you encourage them to think probabilistically they might admit there’s more uncertainty.
I wanted to look into this in the context of the recent Supreme Court confirmation hearings, so I asked readers to estimate their probability that Judge Kavanaugh was guilty of sexually assaulting Dr. Ford. I got 2,350 responses (thank you, you are great). Here was the overall distribution of probabilities. Horizontal axis is percent chance he did it; vertical axis is number of people who responded with that percent:

This looks weird because people were most likely to give numbers rounded off the the nearest ten.
I separated responses into bins from 0 – 9%, 10 – 19%, and so on to 90 – 100%. Keep in mind that the last bin is slightly larger than the others, so it might make it unfairly look like more people gave extreme high answers than extreme low answers. I also switched the vertical axis to percent of responses in each bin. Smoothed out, it looks like this:

This looks pretty balanced, and it is: the average probability is 52.64%. This is probably a fake balance based on all the different demographic skews involved cancelling out: 2.5x as many Democrats as Republicans answered the survey, but 9x as many men as women did.
Here are the results broken down by party (blue is Democrats, red is Republicans):

And here are the results broken down by gender (blue is men, pink is women):

There should be an interaction between party and gender, because men are more likely to be Republicans, and women Democrats. I didn’t have enough data to investigate this too carefully, so I can’t say whether gender controlled for party remains significant or not.
I asked two questions to assess participants’ level of background knowledge: where did Kavanaugh go to law school? (correct answer: Yale) and what is Kavanaugh’s wife’s name? (correct answer: Ashley, but a shout-out to everyone who wrote “Mrs. Kavanaugh” and to the one person who wrote “beer”). Here are the probabilities of people who got both questions right (gold) vs. both questions wrong (green):

People with high background knowledge were more extreme in their answers, and slightly more likely to think Kavanaugh is innocent. I worry that I made a mistake in the questions I chose, since people who are more sympathetic to Kavanaugh might be more likely to know about his family. In retrospect, I should have asked at least one question about Dr. Ford.
What about neutral people? Do such people exist? I looked at people who were neither registered Democrats nor Republicans, and who rated their liberal-vs-conservative ideology, on a one to ten scale, as 4, 5, or 6. These people looked like this:

This chart makes it look like they’re slightly leaning towards guilt, but I think that might be a function of the binning; their mean probability of guilt was 49.85%, about as close to “totally uncertain” as you can get. Neutral people with more background knowledge were, again, more likely to lean innocent, with a mean of 41%.
I asked people whether they felt the evidence that Kavanaugh may have committed sexual assault was sufficient to reject his nomination to the Supreme Court, regardless of any other reasons to vote for or against him (like his legal opinions). 55% of respondents thought that yes, his nomination should be voted down; 45% still supported him. Here is a list of support for confirmation by probability of guilt:

Of people who thought there was only a 0 – 9% chance Kavanaugh was guilty, 98% thought he shouldn’t be rejected from the Supreme Court on this basis alone. Of people who thought there was a 90-100% chance of guilt, 96% thought that was sufficient to reject the nomination. Of people who thought there was 50-50 chance of guilt, about 50% still supported him and 50% opposed him.
This question suggests there is no real consensus about how plausible an accusation has to be before it means someone should be denied nomination to the Supreme Court. People generally agreed that if there was below a 25% chance the accusations were true, he should definitely be confirmed, and if there was above an 80% chance, he definitely shouldn’t be. But between 25% and 80%, people were pretty split on whether the Senate should err on the side of not confirming a potential assaulter, or wait until it was beyond a reasonable doubt. If we were trying to make these answers into a guideline for how a Senator should vote, it looks like they would be satisfying the most people if they voted to confirm if they thought the accusations had a less than 50% chance of being true, and to reject if they thought they had a more than 50% chance. I wonder how many people would endorse this rule as written.
I also asked people whether they would reject Kavanaugh in the hypothetical universe where he had immediately admitted to the accusations, then apologized for his actions and said he had changed as a person since then. About 55% of people said they would accept him in this scenario, meaning he gains about 10% support in the SSC demographic compared to the real-world situation. But the question was poorly worded and I’m not sure how many people answered yes they would reject him, accidentally meaning to say yes they would accept him.
Here is a graph of how people answered this hypothetical compared to how guilty they thought he was:

There’s no correlation. This makes sense: how guilty you think he is in this universe shouldn’t affect your opinions about a hypothetical universe where you know he’s guilty – but for some reason I’m still surprised. I guess I expected people’s partisan biases to sneak in, even if they didn’t make sense. Maybe the question was so confusing that answers to it are basically random.
Overall, when asked to use probabilities people were able to admit to a little bit of uncertainty in their answer. They could give probabilities that were well-formed and self-consistent. But none of this came close to removing the partisan bias and the strong difference in opinions. There is no consensus in the general SSC demographic, and even unbiased people as a group are unable to send a coherent signal. This is not a good way to get beyond confusion and disagreement on an issue like this one.
You can download the raw data (slightly cleaned up) here.