I would hesitate a great deal before entering an argument with either Larry Wasserman or Eliezer Yudkowsky, but here goes.
You're right that if, for whatever reason, someone is fascinated by "coverage" then confidence intervals will answer their questions better than Bayesian posteriors. But I think Eliezer's right that there's something very wrong with thinking that "coverage" in this sense is what matters.
Let's consider your example again. In what circumstances is the following actually a useful problem to solve? "Given an observation of one thing from a box, tell me a set of box-types in such a way that for each box-type you'll choose a set including the right one at least 70% of the time."
I can think of some. For example: a mad scientist starts sending you boxes, with instructions to start guessing; he's going to monitor your results on each box-type and if he sees you getting any type of box wrong more than about 30% of the time he'll kill you. Otherwise he'll reward you for nominating fewer box-types each time. But (1) that's a desperately contrived situation and (2) the most diehard Bayesian, in that situation, will produce something like "confidence intervals" because that's what Bayesian decision theory says to do.
Is there any not-so-contrived situation where the problem solved by confidence intervals is actually an important one?
By the way, my best guess about the Rule / Theorem thing is that he's distinguishing between a theorem about conditional probabilities, and a normative rule saying "when you get new information, update your beliefs like so".
> the most diehard Bayesian, in that situation, will produce something like "confidence intervals" because that's what Bayesian decision theory says to do.
I disagree that this is what "Bayesian" decision theory says to do. It's what decision theory says to do, and it's what math says to do, and it's what the constraints require. It's not particularly "Bayesian" -- it's just what you have to do.
If everything that happens to be the correct answer (including frequentist confidence intervals when called for) is now described as Bayesian, then the term has no meaning and we are all Bayesians. :-)
> Is there any not-so-contrived situation where the problem solved by confidence intervals is actually an important one?
What would you do in the case of my 100 robots, where you want 70 of them to come to the correct decision, and they have to make their decisions independently? Having them all calculate a posterior independently works terribly (as I showed, 80% of them come to the wrong conclusion with >73% belief). Confidence intervals work a heck of a lot better.
The optimal approach would consider what single algorithm works best when run independently. Finding these solutions (on, e.g., a decentralized POMDP) is an open problem.
I called it Bayesian because it describes what Bayesians will do. It does indeed also describe what anyone else sane will do. The point is that in order to make confidence intervals the right answer you need a situation weird enough to make even Bayesians use confidence intervals.
In the case of your 100 robots, why am I supposed to want 70 of them to come to the correct decision? This seems just like my mad-scientist example: contrived to force confidence intervals (or something very like them) to be the right answer. Can you explain in what sort of situation this would be a sensible thing to care about?
So explicitly state the property you do want to optimize for in your robot example, and state your prior belief, and then crank the handle on the Bayesian reasoning machine to obtain your optimal answer.
You may get an intractable problem that you can't solve exactly, and for a carefully cherry picked objective the frequentist answer might even be a good approximation.
What if you're an Italian seismologist, and you want to produce a prediction which is in some way useful, but still expresses an appropriate level of doubt to a lay audience?
You're right that if, for whatever reason, someone is fascinated by "coverage" then confidence intervals will answer their questions better than Bayesian posteriors. But I think Eliezer's right that there's something very wrong with thinking that "coverage" in this sense is what matters.
Let's consider your example again. In what circumstances is the following actually a useful problem to solve? "Given an observation of one thing from a box, tell me a set of box-types in such a way that for each box-type you'll choose a set including the right one at least 70% of the time."
I can think of some. For example: a mad scientist starts sending you boxes, with instructions to start guessing; he's going to monitor your results on each box-type and if he sees you getting any type of box wrong more than about 30% of the time he'll kill you. Otherwise he'll reward you for nominating fewer box-types each time. But (1) that's a desperately contrived situation and (2) the most diehard Bayesian, in that situation, will produce something like "confidence intervals" because that's what Bayesian decision theory says to do.
Is there any not-so-contrived situation where the problem solved by confidence intervals is actually an important one?
By the way, my best guess about the Rule / Theorem thing is that he's distinguishing between a theorem about conditional probabilities, and a normative rule saying "when you get new information, update your beliefs like so".