In Bayesian analysis you have some set of prior theories with associated probabilities, you observe data, you alter your estimate of the likelihood of those theories, and then your next prediction becomes a weighted average of those predictions.
Your analysis is a reasonable first cut, but the above question is a good one, and your reply is rather incorrect.
First, you haven't really done a Bayesian analysis, for several reasons.
The easiest problem to fix is, you didn't specify any priors. We could validate your style of calculation by assuming single-engine failures are IID with parameter "p," uniform on the unit interval. (A beta distribution would be the standard conjugate prior.)
If we go ahead and make the uniform-p assumption, then what you've calculated is a most-likely posterior value of "p" (35/36). (This is the maximum-likelihood estimate for p.) But in a Bayesian world, "p" has a full posterior distribution, not just a most-likely value.
So, still in the Bayesian world, the probability of failure (P(N_fail > 2)) must be calculated on the basis of not the most likely p, but the posterior of "p". You didn't do this; you just used the most-likely p.
Which brings us to the second problem with your reply. You really can get a confidence value on P(N_fail > 2). You can get a full posterior distribution! It will be a 1-dimensional density on [0,1]. And you could calculate this, either by simulation or by an analytic procedure (because it's a low-dimensional problem).
This posterior on P(N_fail > 2) would be the answer to the parent's question. It would probably be rather "fat", validating the intuition that we don't have much data.
My reply might be a little smarty-pants. Sorry if it is! As I said, your analysis is a reasonable first cut.
The easiest problem to fix is, you didn't specify any priors.
Read the whole thing more carefully. I started with the calculation for the maximum likelihood estimate, but I ended with a prior with equal a priori likelihoods for failure rates 0%, 0.01%, 0.02%, ..., 99.99%, 100%. This is a reasonable discretization of uniform on the unit interval.
My reply might be a little smarty-pants. Sorry if it is!
I would suggest that before indulging a tendency to be a smarty pants, that it is good to read the whole thing.
You're right, the second half of your post in effect puts a two-atom prior on "p", with zero everywhere else, and then goes on to use a sequence of such atoms, which would approximate a uniform prior. It's more standard to use smooth priors, because we don't have precise information, but you are right, I was not reading carefully.
You are still in error that there is not a way to describe the uncertainty in your estimate of the posterior probability of system failure. It has a posterior distribution, like everything else in a Bayesian analysis. You would compute it as I described -- Monte Carlo would be easiest.
The question _does_ make sense - you have some probability distribution that represents your belief, you observe data and form an updated probability distribution (your posterior) that represents your new belief incorporating the observed data, and the question asked (in a roundabout way) what the variance of the posterior distribution was.
I interpret the question along the lines of "if they do a hundred launches, what's your probability distribution on the number of them that fail?"
If I have a normal coin and a coin which is double-sided (but I don't know which side), I'll give 50% for both of them coming up heads next time I toss. But if I toss them a hundred times each, my probability distributions for them look totally different.
I suspect this information is encoded in your priors, but I don't know offhand how to access it.
So thinking about this some more: we're assuming engines have some "true failure rate" which we're trying to divine from evidence. 2% is currently the mean of your distribution on TFR, but the relevant question is what's the variance of your distribution?
Uniform prior, updated on the evidence that 1/36 engines have failed, I think this gives P(TFR = x) = x(1-x)^35 / (int x(1-x)^35 dx from 0 to 1). Apparently the integral is 1/1332, so P(TFR=x) = 1332·x·(1-x)^35. But that seems to have a mean of 5%, compared to your value of 2%, so I may have done something dumb?
You can answer that question for each prior. The probability of, say, 3 failures is the sum over all priors of the probability that that prior is true, times the probability that it would leave you with 3 failures.
I leave writing a program to calculate this as an exercise to the reader. I've put enough time in on this one already, I have paying work to get back to.
Not really, if you'd built your model from a billion launches and 20 million failures, a dozen failures in a row won't change your priors much at all. You've built your model off of far less data.
How many failures are required to change your model from 2% to, say, 4%?
It is easy enough to modify the linked script for any scenario that you want.
If 2 engines failed on the next launch, the model would predict a 4.8% chance of failure on the following launch.
As for your "build your model from a billion launches and 20 million failures" comment, if I had that much data, then it wouldn't much matter what reasonable set of priors that I started with, I'd wind up convinced that the true failure rate was very close to 2%.
Note that the prior that I am talking about is the distribution of possible theories before I saw ANY data.
I could. But that would be inappropriate here. The lower bound of any such interval would be astonishingly close to a failure rate of 0, while the maximum likelihood estimator is itself fairly close to 0. Therefore any reasonable set of priors usually puts you in the upper tail.
For instance one such interval has a MLE of an engine failure rate of 2.78%, a lower bound of 0.1% and an upper bound of around 15%. (Corresponding rocket failure rates range from 1 in 10 million to about 9.9%.)
In Bayesian analysis you have some set of prior theories with associated probabilities, you observe data, you alter your estimate of the likelihood of those theories, and then your next prediction becomes a weighted average of those predictions.
That number has no attached variance.