Meta-quiz Meme

[jack]

Related to several places, most recently stolen from God Plays Dice here, from a book. The point being not what people know, but how good they are at knowing what they know.

"For each of the following ten questions, give a range that you are 90 percent confident contains the correct answer. Your goal is to get exactly nine of these right[1]. Yes, I know that sounds weird! But the point is that if you get all ten right, you're proabably underestimating your own abilities to predict things. If you get eight or less, you're probably overestimating them."

Assign a range to each question in a comment. Look up the answers and see how many you got right. Post it if you like. GodPlaysDice said to repost it if you liked, and to email him the answers (izzycat AT gmail DOT com) if you like; I assume he wishes to informally gauge something.

Here are the questions:
1. How old was Martin Luther King, Jr. at death?
2. What is the length of the Nile River?
3. How many countries belong to OPEC?
4. How many books are there in the Old Testament?
5. What is the diameter of the moon?
6. What is the weight of an empty Boeing 747-400?
7. In what year was Mozart born?
8. What is the gestation period of an Asian elephant?
9. What is the air distance from London to Tokyo?
10. What is the depth of the deepest known point in the ocean?

Although what interested me was that it simply meant you could have a quiz where people who don't know much about it (or who know too much about it) can play too. I'm curious to see how big the ranges are -- mine are embarrassingly wide, generally between a factor of two to a factor of ten, though of course, I know several much more precisely now.

[1] It would be more precise to say "and not know which one you got wrong". The idea being you should be pretty certain about all of them, not guess "0-1000,000" on nine and "-315.17" on the last one :)

Comments

[liv]

Interesting, and a good selection of questions to estimate.
My guesses:
1. 30-50
2. 500-3000 miles
3. 6-20
4. (I know the answer, it's 39)
5. 200-1500 miles (ugh)
6. 50 tonnes-200 tonnes (also pretty wild)
7. 1690-1780
8. 6-30 months
9. 5000-8000 miles
10. 1-5 miles

score: 1 yes, 2 no, 3 yes, 5 way wrong, 6 yes (to my surprise), 7 yes, though I chose probably an unfairly wide range, 8 yes, 9 yes, 10 yes. 7/9, which as the comments to the God Plays Dice post pointed out, isn't totally horrible for a 90% confidence interval. I feel stupid about 5 though.

[cartesiandaemon.livejournal.com]

1. 25-50 yes 39
2. 400-4000mi no 4100mi
3. 5-15 yes 12
4. 20-25 no also 39 (oops)
5. 2000-5000mi yes 2160mi
6. 5T-30T no 58T
7. 1650-1850 yes 1756
8. 6m-1y no 22m (ouch, this fact rang a bell when I looked it up)
9. 3000-6000mi no 9500mi
10. 3-20 mi yes 11000m

5/10. Hm.

[woodpijn.livejournal.com]

That is indeed a really interesting exercise. My reaction to a lot of those it "argh, I have no idea!" but that doesn't really matter - it just means I have to give a big range.

OK, then:
1. 35-48
2. 300-700 miles
3. 4-10
4. 39 (This is the only one I know the actual answer to, so I can't satisfy the "90% confidence" requirement :)
5. 1000-3000 miles
6. 30-100 tonnes
7. 1780-1820
8. 9-18 months
9. 4000-6000 miles
10. 1000 feet-10 miles (blarg, really no idea)

Some are definitely harder than others. Most of the questions reminded me of the stereotypical "how many petrol stations in the country"-type interview questions. You can't expect to get those quite right, but you can estimate various things and multiply them together and get a plausible answer. But for some of them (like the last one) I really felt I had nothing to go on.

[woodpijn.livejournal.com]

And based on the correct answers in your post:
1. Correct
2. No, way out
3. No
4. Correct
5. Correct
6. Correct
7. No
8. No (I knew it was long, but underestimated quite how long)
9. No
10. Correct (although your post says "11000 mi", so I thought I must be way out, but then realised that was a typo for "11000m")

So, 5/10. Yeah, I should have given bigger ranges.

[cartesiandaemon.livejournal.com]

although your post says "11000 mi", so I thought I must be way out, but then realised that was a typo for "11000m")

Yeah, my brain was confused between three different systems of measurement, and came out with that, which is significantly further than the distance all the way to the other side of the earth :)

[cartesiandaemon.livejournal.com]

This is the only one I know the actual answer to, so I can't satisfy the "90% confidence" requirement :)

:) Yeah, it's unfortunate that the one people are most likely to know precisely is also the one you can't give a more precise answer to if you want to :)

[rochvelleth.livejournal.com]

Oh dear, I'm going to do so badly at this! But I'll have a go...

1. 35-50
2. Um, 1,000-5,000?? It's very long, put it that way.
3. 10-30
4. 39
5. Um, OK, it's a lot smaller than Earth so... let's say 800-3,000
6. I'm crap at weights. 1-100 tonnes
7. 1756
8. 15-25 months (though I think it's just under 2 yrs, but could be wrong)
9. I think we're talking somewhere near half the Earth's circumference, and that's about 20,000 miles, so I'll say 7,000-12,000 miles
10. God, I've no idea. I don't even know what to do it in. Feet? Let me see, I suppose there's potential for it to be as the highest point on land is high, IYSWIM. I'll guess 5,000-20,000 feet.

*goes to check answers*

Oh, how cool, I got them all. But only by seriously hedging my bets on 6 and 10, though! It was mostly just logical guessing, though I knew two of the answers exactly.

It strikes me that it would be possible to do some sort of scaling thing with this to give you quite a precise score on how you did, which would be cool. And indeed would remind me of the Professionals episode I was watching last night with old computers and assessments of agents' mental agility.

[cartesiandaemon.livejournal.com]

7. 1756

:) Feel free to give months and days if you feel the need :)

[cartesiandaemon.livejournal.com]

Ah! I thought you'd do fairly well, because I discovered you know some general knowledge :) Even your wild guesses are in the right ball-park, rather than "I don't know the order of magnitude of the order of magnitude" :)

[alextfish.livejournal.com]

I fear these will be embarrassingly wide too... (Written without looking at anyone else's comments)

1. 33-45
2. 80-300 miles
3. 5-20
4. 38-40 (this is one I'm pretty confident of)
5. 500-1500 miles
6. Assuming by "empty" you mean empty of people, but full of equipment etc... even so, I feel I just don't even know what order of magnitude to guess here. To reach 90% certainty, I have to say something absurdly wide like 100 - 10000 tons :(
7. Owch. Painfully historically uneducated, I guess 1650-1800.
8. 8-16 months
9. 3000-8000 miles
10. Not until this question do I realise how little intuition I have for heights and depths... I can't remember what the numbers on contour maps are or anything. 8-40 miles?

[alextfish.livejournal.com]

Which is: 1 yes, 2 waaayy out, 3 yes, 4 yes, 5 no, 6 no (owch - but then, I really have never in my life had any intuition for weights at all), 7 yes, 8 no, 9 no, 10 close but no. An embarrassing 4/10.

I think a few too many of the questions are/could be based on an estimate of the size of the Earth. Distances on a scale of miles rather than metres, millimetres or light years.

Discussing it with Rachael, we conclude that the low scores aren't really an expression of overconfidence (as our hesitancy comments), but just a reluctance to post a vastly large range.

[cartesiandaemon.livejournal.com]

Discussing it with Rachael, we conclude that the low scores aren't really an expression of overconfidence (as our hesitancy comments), but just a reluctance to post a vastly large range.

Yeah, I think so. I think I was actually having fun answering at 70% or similar, regardless of what I was supposed to be doing. Which might actually be better, if you don't really know an answer -- of a normal curve, 90% is, I think, quite a bit, and people prefer to have a good chance of being fairly close to right, than a very good chance of being within several orders of magnitude :)

[naath.livejournal.com]

Heh, I just went for (IMO) humungous ranges... and got 9/10, so obviously the problem is "admitting that I don't have a clue about how good my guesses are".

We did this exercise in Dr MacKay's lectures once - most people were very overconfident about their guesses, or unwilling to be truthful about their lack of confidence (I on the other hand was entirely confident that I had no idea and most of the questions, and my biggest problem was guessing what Large Number would be a plausible upper bound (er, it looks silly when you put "entire size of universe" kind of scale numbers for a number of objects present on Earth), I'm terrible with visualising big numbers).

[cartesiandaemon.livejournal.com]

And, indeed, only a couple of guesses were really wild. And obviously, you were much better than anyone else at guessing 90%. That should actually be a figure where, if you get it wrong, it would really really surprise you, whereas I (and most people) couldn't be that broad with the bigger numbers.

[midnightmelody.livejournal.com]

I dispute that 4 has one correct answer! (It depends whose canon we're using.)

[naath.livejournal.com]

2 also has two correct answers.

[cartesiandaemon.livejournal.com]

Apparently so. Do you know which are most common? I knew it would be controversial, but just picked an answer off wikipedia, but apparently Chris's test (below) used a different one. A check on wikipedia shows plausible answers would be:

* From the basin to lake victoria
* From the basin the head of the blue nile
* From the basin to the head of the longest stream which feeds lake victoria

But I don't know which two are 4100 and 3400 miles :)

[naath.livejournal.com]

It's the length from the start of the Blue Nile to the Med or the start of the White Nile to the med (that are the given measurements, aiui).

[cartesiandaemon.livejournal.com]

Thanks. And you never know, now I might actually remember it, there's nothing like getting a question mildly wrong to sear it in your memory forever :) (My first estimate was "How long can it be? Well imagine it's in Asia and really wiggly, say that's 1/6 of the circumference of the earth wide...")

[cartesiandaemon.livejournal.com]

:) Conveniently, if you mark your own, it doesn't actually matter. This only occurred to me when I went to look up my answer, that it might vary, but I just said "OK, then my answer refers to the King James Bible" and checked for that.

I guess that would also allow people to give a more precise answer, if you know for a fact which canons have 39, you can go on giving information until you're only 90% certain :) (Although that's not really very practical :))

[gareth-rees.livejournal.com]

This was the idea behind Chris Lightfoot's 2004 estimation quiz. He applied his usual high-powered statistical analysis to the results in this blog entry.

[cartesiandaemon.livejournal.com]

Ooh. Yes, exactly. That's really interesting.

[cartesiandaemon.livejournal.com]

Ironically, the two different tests have different lengths for the Nile, so after being able to give an answer to within a hundred miles on Chris's, it was wrong :)

[naath.livejournal.com]

I'm completely confident that I have pretty much no idea of the answers to any of these...


Guesses
1 - adult, so more than 20 but less than 100 because few people get older than that
2 - long (heh), er, more than 50 miles but less than 5000, actually I have no idea what 5000 miles looks like; so I pulled that number out of my head. also which, there are two branches...
3 - more than 1 and less than "all countries" which is O(100) or so.
4 - more than the 7 I can name and less than 100 which would make for a really fat book.
5 - help, more with the "visualise big numbers" problem. Well, more than 100 miles and less than 100,000 miles.
6 - more big number. Between 1000 and 100,000 kg
7 - well, Mozart wasn't Medieval or Modern... so, er, somewhere between 1500 and 1900.
8 - I remember that this is "long"... more than 8months and less than 2 years
9 - argh! bignumbers. More than 100miles and less than 20,000 miles.
10 - Mariana Trench (heh, I knew that), I also think that it's deeper than Everst is high, but that doesn't help because I don't know that number either! More than 1mile but less than 100 miles.

(Surprisingly this is actually 9/10 within range, 6 is too low; although as I suspected my grasp of how big the big numbers actually are is hugely off and most of the length answers are at the lower end of my range)