How to score 50% predictions

[jack]

I've recently been experimenting with removing stress from my daily[1] todo list by listing things I hoped to do but putting a likelihood on them, like "90% foo, bar, blah; 75% other thing" etc. I know this seems overcomplicated, but I find failing things I'd planned to do REALLY REALLY kills my motivation, so it's worth arranging things such that even if they go better or less well than expected, they fall into the broad range of "what I planned for". And it also means that I'm more pushed to put small, comparatively important things first, rather than starting with the difficult things and never getting to anything else.

I don't know if I will keep it up, but even just trying it raised several interesting questions.

Slatestarcodex sometimes posts predictions like this, usually for an upcoming year, to test where he's being honest with what he expects and where he isn't (usually about external factual things like politics, but some of himself). A question arises, how to score this? Especially the 50% ones.

You can cobble together some score which is maximised when 90% of the 90% predictions are true. I think there's some particular baysian probability thing that measures, given those expectations, how unlikely a particular outcome is (which equates to 'how wrong you are' which you try to minimise).

But the bit that gets confusing is, how to rate 50% predictions? For my system, I'm predicting what *will* get done, so I feel like it's clear if I'm over-estimating or under-estimating. But Scott had the problem, that it seemed arbitrary if he said "X will do Y" or "X will do not-Y", so the 50% predictions should be random even if they're comically bad, which logically makes them impossible to score. And yet, you feel that if they're really bad, you should be able to recognise that in a systematic way. Maybe they should all be stated relative to the status quo? Or something else?

[1] Freudiano 'faily' :)

Comments

[jack]

Mm, possibly. I've only been trying the technique a little so I'm not sure how it will work out. But if the time estimate is variable, even if I plan in descending order of importance, there's going to be *some* task at the 50/50 point.

[seekingferret]

No, I think I've been thinking about this wrong. SSC's predictions are different than yours- SSC is trying to make binary predictions about the world- either a thing will happen or a thing won't happen, so a prediction that a thing has a 40% chance of happening this year is the same as a 60% prediction that it won't happen, for his prediction calibration purposes. That's why 50% predictions are confounding for him, because they signal a lack of knowledge about the outcome.

But you know your schedule and your capabilities so you probably have a pretty good sense of what needs to happen for you to accomplish some task. And you only care about success rate, not about the probability of a thing happening vs. it's opposite. So a 50% task doesn't mean it's a task where you can't predict whether or not you'll succeed, it's a task that you know you have a 50% chance of completing. So of course you'll have 50% tasks! And you'll have 40% tasks and 30% tasks and 20% tasks and 10% tasks. And also some of your 10% tasks will be things that you have a 100% chance of completing within a week, it's just lower priority today and you'll only get to it contingent on the results of higher priority tasks.