Good stuff OP, nice to see Bayesian getting more attention recently. However, after reading your blog post, I can't help get the feeling this is how a natural frequentist would approach the problem. This method is effectively relying on known, statistical record of past events to form the priors.
Often a more useful and appropriate construct in the Bayesian world, is the use of a belief network or Bayesian Network. This is a probabilistic directed acyclic graph (DAG) that encodes priors, often in the form of subjective beliefs (yes subjectivity can be useful), including specific domain knowledge.
Common example: Consider a naive Bayesian classifier (a specialized form of belief network) that identifies individual pieces of spam. Do we arrive at the spam score by entering the probability of past events into a simple model based of the Bayes theorem formula?
No, it's trained using the vast amount of domain knowledge and pattern recognition (through our experience and own estimation of what 'spam' is) encoded in our minds, that provide the priors. Thus, even though there is a large amount of subjectivity involved, the overall result can objectively be measured, within a given utility function. Incidentally, this is often what makes many hardcore empiricists 'nervous', and hence avoid belief networks altogether.
Coming back to the Falcon 9: A piece of prior information outside the scope of historic safety records, for example, one of the lead engineers having a nagging doubt about a particular technical risk based on some observed phenomenon, could have an impact on the real world probability of the next event being a failure. (Which is a pretty useful thing to know!)
In fact, this exact scenario happened in 2003 with the disastrous destruction of the Space Shuttle Columbia. [1] An engineer spotted something wrong on previous flights, but management failed to heed the warning[2]. This could quite possibly have been averted, if a risk mitigation model were in place to account for such evidence.
Looking forward, it's quite possible to imagine a future where this decision making has been outsourced to a sophisticated AI based off a Bayes net, with far more accurate real world modeling of risk and failure probabilities, outclassing the amount of evidence and a human or committee could possibly hope to compete with.
While I've nothing against frequentist approaches (albeit Bayesian naturally makes more intuitive sense to me), a minor drawback is the reliance on the past to predict the future. For example if you had safety records on 1 million previous flights, then one might be tempted to say, "well that's that then, we now know objectively the probability of failures in the future -- end of story". But, the 1 000 001 flight may have been designed to fly on a completely different type of technology, that will change significantly change the safety record of space flight going forward for the next "x" years. Thus using a Bayesian approach account for all relevant priors, it would in theory be possible to reflect a more accurate probability for the 1 000 001 flight, before it took place.
Lastly Bayes nets are not the best tool for every job, and do have drawbacks in certain situations. They are vulnerable to things like Bayesian poisoning or confirmation bias. A Bayesian approach is only as useful as the ongoing real world relevancy and accuracy of the priors. As the old adage goes, GIGO - garbage in, garbage out.
> for example, one of the lead engineers having a nagging doubt about a particular technical risk based on some observed phenomenon, could have an impact on the real world probability of the next event being a failure. (Which is a pretty useful thing to know!)
You should know that there's no such thing as "real world probability". The rocket will crash, or it will not, period.[1] Probability, as a measure of your own ignorance, is subjective.[2] Your main point still stands though: knowing about the uncertainty of that lead engineer certainly should influence your assessments of the risks involved.
[1] What will actually happen is, the universe splits into many "worlds" (blobs of amplitude in configuration space), a fraction of which will have the rocket crash, and the rest won't. That's the closest thing we have from "real world probability", though it really isn't: the laws of physics as we currently know them are still deterministic.
Indeed. We pretty much agree then. If you re-read the "real world probability" in the context, I was talking specifically about a belief network. The degree to which a justified belief, in an outcome will occur. All beliefs are by their definitions 'subjective' and occurring in a mind.
Actually my current thinking over the last decade mostly aligns with what could be described as physicalist view of the reality, so even 'subjective' thoughts, ideas, concepts etc exist objectively in a physical sense as well (glia cells, neurons etc). (but that's a whole other topic ;)
I simply worded it 'real word' because I was attempting (perhaps ineloquently I will concede), to differentiate between frequentist and the Bayesian understanding of the term probability, because they differ [1].
Bayesian favors bringing in a priori beliefs into the model whereas a posteriori consideration of a problem, as occurs in frequentist approaches, favor isolation of the model.
>What will actually happen is, the universe splits into many "worlds"
Interesting, you state that so.. assertively :p I'd give the chance of a many worlds interpretation corresponding well with our physical reality, a low probability event, with a pretty high credibility interval ;)
I'd give the chance of a many worlds interpretation corresponding well with our physical reality, a low probability event, with a pretty high credibility interval ;)
There is a theorem that if an experiment and observational apparatus are both quantum mechanical systems, then the many worlds hypothesis describes what happens when that experiment is observed with that apparatus. If quantum mechanics is merely a good approximation of some better theory, then to whatever extent it is a good approximation of the system, the many world hypothesis remains a good description of that interaction.
Therefore your confidence that the many-world's hypothesis is an inaccurate description of what happens when you observe the outcome of a quantum mechanical experiment is an insistence that your brain and body are not well-described by the best theory that physics has for how the world works.
I think most physicists agree that at the bottom, we have a distribution of "complex amplitude"[1] over a "configuration space"[2]. But as you can see from my second link, many (most?) physicists insist that we can derive a "probability" from a complex number. Note that such probability would then be an actual real world probability, where the universe itself is uncertain about what to do. True non-determinism.
It's only natural. At the experimental level, the researcher does observe Born statistics. Same setup, different results, so there is probability in the territory after all.
There's a problem with that however: The equations, which make such wonderfully accurate predictions, (i) are dederministic, and (ii) do not state at any point that the blob of amplitude we don't see disappear in a puff of smoke. They merely say that the blobs eventually stop interacting. The same way that if you launch a photon to outer space, never to meet it again, it won't disappear the instant it reaches the boundary of our observable universe. If you insist on a mono world, you have to assert that the other blob, despite being predicted by those otherwise accurate equations, somehow doesn't exist when you don't see it.
One way to do it is to believe that, contrary to what the equations say, the blob you don't see does disappear in a puff of smoke. Its amplitudes are literally zeroed out behind your back. In hindsight, this one looks nuts to me. I mean, how can we justify distrusting accurate equations in a way that doesn't even make experimental predictions?
Another way is to call the square moduli of those amplitude "probabilities", and pretend that because it's probabilities, the blob you are not in isn't real. But the equations do not make any difference between the two blobs. Then how come the other blob is less real than our own?
To me, those two explanations really feel bizarre. You have to start from a mono world assumption to come up with that. An easy mistake to make, since personal experience is telling us all the time that there is only one world. A bit like a leaf in a binary tree: its ancestors form a line, not a tree. But Kolmogorov complexity says a literal interpretation of the equations (which means many world) is simpler than anything else we currently know about. So to hell with personal experience (which by the way is responsible for much worse whackery than mono world).
Now there is a way out: we can admit that current physics imply many worlds, but insist that real physics probably don't. Current physics are not complete after all. We may have big surprises. This argument is certainly be much saner than the Copenhagen interpretation. So much that it does lower my probability for many worlds somewhat. Just not enough to squash my confidence. :-)
You are criticizing me for trying to rely on data rather than a complex subjective model based on information from the beliefs of people that I have never met and have no input from? There is no way for me to attempt that approach that does not come down to some form of "making shit up".
More generally you are right that I prefer to work off of data rather than subjective opinion. Data I understand. Subjective opinion is valuable, but suffers from major potential biases. Correcting for that can be very hard.
Often a more useful and appropriate construct in the Bayesian world, is the use of a belief network or Bayesian Network. This is a probabilistic directed acyclic graph (DAG) that encodes priors, often in the form of subjective beliefs (yes subjectivity can be useful), including specific domain knowledge.
Common example: Consider a naive Bayesian classifier (a specialized form of belief network) that identifies individual pieces of spam. Do we arrive at the spam score by entering the probability of past events into a simple model based of the Bayes theorem formula?
No, it's trained using the vast amount of domain knowledge and pattern recognition (through our experience and own estimation of what 'spam' is) encoded in our minds, that provide the priors. Thus, even though there is a large amount of subjectivity involved, the overall result can objectively be measured, within a given utility function. Incidentally, this is often what makes many hardcore empiricists 'nervous', and hence avoid belief networks altogether.
Coming back to the Falcon 9: A piece of prior information outside the scope of historic safety records, for example, one of the lead engineers having a nagging doubt about a particular technical risk based on some observed phenomenon, could have an impact on the real world probability of the next event being a failure. (Which is a pretty useful thing to know!)
In fact, this exact scenario happened in 2003 with the disastrous destruction of the Space Shuttle Columbia. [1] An engineer spotted something wrong on previous flights, but management failed to heed the warning[2]. This could quite possibly have been averted, if a risk mitigation model were in place to account for such evidence.
Looking forward, it's quite possible to imagine a future where this decision making has been outsourced to a sophisticated AI based off a Bayes net, with far more accurate real world modeling of risk and failure probabilities, outclassing the amount of evidence and a human or committee could possibly hope to compete with.
While I've nothing against frequentist approaches (albeit Bayesian naturally makes more intuitive sense to me), a minor drawback is the reliance on the past to predict the future. For example if you had safety records on 1 million previous flights, then one might be tempted to say, "well that's that then, we now know objectively the probability of failures in the future -- end of story". But, the 1 000 001 flight may have been designed to fly on a completely different type of technology, that will change significantly change the safety record of space flight going forward for the next "x" years. Thus using a Bayesian approach account for all relevant priors, it would in theory be possible to reflect a more accurate probability for the 1 000 001 flight, before it took place.
Lastly Bayes nets are not the best tool for every job, and do have drawbacks in certain situations. They are vulnerable to things like Bayesian poisoning or confirmation bias. A Bayesian approach is only as useful as the ongoing real world relevancy and accuracy of the priors. As the old adage goes, GIGO - garbage in, garbage out.
[1] http://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster
[2] http://www.guardian.co.uk/science/2003/jun/22/spaceexplorati...