February 5, 2022
In the previous article I “bought” Jgalt’s track record and talked about its peculiarities. Since then I’ve wanted to get my hands on another user’s track record and compare the two. I finally got enough tachyons to do this, and I decided to check out datscilly, the 1st rank Metaculus user.
I’ve decided the initial prediction plot was misleading, so I’m going to start things out with the final predictions this time.
I looked at datscilly’s calibration plot, and it showed that he tends to be a little underconfident at the extremes. I think this is just because he isn’t updating all the time.
And I finally decided to do an animated plot. Here’s one that shows how datscilly and Jgalt change their predictions over time.
The animation works like this: Each dot is a question, the beginning of the animation is the publication time, and the end is the closing time. This is why some dots appear in the middle of the video; datscilly and Jgalt sometimes don’t predict on questions until long after they’ve opened.
And since I mentioned it, here’s a plot of when datscilly and Jgalt like to make predictions.
The x-axis shows how long-lived the questions are, and the y-axis shows when each user made their first prediction. A relative time of 0 means “pretty much right when it opened,” and a relative time of 1 means “right before it closed.” Since the x-axis is on a log scale, I’ve added some labels to indicate important time periods. Lots of questions are roughly one year long, and most are between a month and a year. What we can clearly see here is that datscilly tends to predict early, and this has surely helped him get a higher average Metaculus score.
By the way, datscilly has a Metaculus skill score of 0.966. Looking at this chart, I don’t think anyone with a large number of resolved questions exceeds 1, which is a bit disappointing. Also, based on that chart, I’m thinking my next victim will be alexrjl. He has a lot fewer resolved questions, but it’s still well over 200. Also, if I’m being honest, Jgalt and datscilly give me an “extremely online” vibe and I’d like to get a more normal person involved here. But that will have to wait a few weeks.