Tuesday, July 29, 2014

Noahopinion

Noah Smith is having fun with automatic priors, pointing out that the reasonable person's prior on personal immortality is that you got it because you ain't dead yet.

Consider Proposition H: "God is watching out for me, and has a special purpose for me and me alone. Therefore, God will not let me die. No matter how dangerous a threat seems, it cannot possibly kill me, because God is looking out for me - and only me - at all times."

Suppose that you believe that there is a nonzero probability that H is true. And suppose you are a Bayesian - you update your beliefs according to Bayes' Rule. As you survive longer and longer - as more and more threats fail to kill you - your belief about the probability that H is true must increase and increase. It's just mechanical application of Bayes' Rule:
Frequentists may disagree, bodycounts tell a bunny something, Noah points out that while there are other reasonable priors, how to determine which one should be used appears to be a religious matter.  OTOH, there are people, particularly teenage boys who use Proposition H daily. 

Eli and Socrates debated similar issues a while ago and Andrew Gelman about eight years ago pondered same
The fundamental objections to Bayesian methods are twofold: on one hand, Bayesian methods are presented as an automatic inference engine, and this raises suspicion in anyone with applied experience, who realizes that di erent methods work well in different settings (see, for example, Little, 2006). Bayesians promote the idea that a multiplicity of parameters can be handled via hierarchical, typically exchangeable, models, but it seems implausible that this could really work automatically. In contrast, much of the work in modern non-Bayesian statistics is focused on developing methods that give reasonable answers using minimal assumptions.

The second objection to Bayes comes from the opposite direction and addresses the subjective strand of Bayesian inference: the idea that prior and posterior distributions represent subjective states of knowledge. Here the concern from outsiders is,  first, that as scientists we should be concerned with objective knowledge rather than subjective belief, and second, that it's not clear how to assess subjective knowledge in any case.

Beyond these objections is a general impression of the shoddiness of some Bayesian analyses, combined with a feeling that Bayesian methods are being oversold as an allpurpose statistical solution to genuinely hard problems. Compared to classical inference, which focuses on how to extract the information available in data, Bayesian methods seem to quickly move to elaborate computation. It does not seem like a good thing for a generation of statistics to be ignorant of experimental design and analysis of variance, instead becoming experts on the convergence of the Gibbs sampler.
Lars Syll riffs off Smith with a little less patience noting that as a religion Bayesianism is dangerous.  The right to carry automatic priors can produce nonsense.  James Annan showed this with Myles Allen's way to wide uniform prior, and the recent discussion about Nic Lewis' attempt to mangle C14 dating priors should caution the Bayesians.

Long live frequentism, at least when there is enough data. 

11 comments:

David B. Benson said...

What is 'objective knowledge'?
https://en.wikipedia.org/wiki/Objectivity_%28philosophy%29
ain't quite it.

Sir Karl gave a stab at it in Objective Knowledge
A Realist View of Logic, Physics, and History
with one chapter reproduced in
http://marxists.org/reference/subject/philosophy/works/at/popper.htm
in which one discovers the tentativeness of his 'objective knowledge', always subject to refutation. (He writes much better than this, I think, in his Proof and Refutation.)

Whatever that stance, it is not the classical meaning of 'knowledge': justified true belief. A belief is certainly subjective and by this criterion all knowledge which might exist is 'subjective knowledge'. Do not let the language used by some Bayesians mislead you.

The hard part is the true. So maybe we should go along with Sir Karl to replace that by tentatively true: 'objective knowledge is justified, tentatively true belief? If so, then this is weaker than just 'knowledge', i.e., 'subjective knowledge'.

In considerations of statistical methods, it might well be best to simply avoid the word 'knowledge'. After all, one is only manipulating mere correlations.

David B. Benson said...

E.T. Jaynes, "Probability Theory: the logic of science" is surely the most entertaining way to learn Bayesian methods.

I prefer using the Bayes factor as a way of comparing two hypotheses, no bias toward either, given the data. One then uses AIC or even BIC to see if the difference between the two hypotheses is large enough. There are again three possible outcomes: H0 is better; H1 is better; equivalent explanatory power.

Fernando Leanme said...

In my career I had to use risk analysis and probability estimates all the time. I became really excited when I realized this could be used in value of information exercises. This in turn justified funds to get more data. The data allowed us to refine our analysis, but there was always risk.

What I found was that as we gathered and reanalysed all the data (old, new), we couldn´t impact decisions anyway.

The guys in charge had their own notions and mental baggage, and they refused to acknowledge what they "knew" had changed.

Amazingly, they kept approving budgets to get more data, even if they ignored the results we got.

So I went to visit a statistical guru, and asked him to explain to me how I could incorporate flawed decisions by people who refused to accept incoming data. He told me he was a statistician and not a psychologist, and that he had better things to do with his time.

That´s when I decided we had enough ice data in the Russian Arctic, and what we had left to do had little to do with data. It was mostly political, and brute force would win the day.

Eventually I left Russia and moved to Venezuela, and kept finding decision makers who didn´t accept new data very well. But they kept me around buying more data, analysing the data and paying our team a ton of money.

Today our team has moved on from Venezuela after seeing the mess Chavez was making. Some are in the UK, others are in Canada. The youngest is in Alaska, and I´m on the beach in Spain. That´s the way things work in life, I guess.

J Bowers said...

Fernando, when you analyse data and conclude the world isn't warming but cooling because sea level dropped for a short while (La Nina, by the way), do you not think there could be a sound reason for your previous analyses to have been ignored?

Anonymous said...

I know Eli loves to pretend he's one of the gang, but Eli should leave statistics to real statisticians.

Thomas Lee Elifritz said...

leave statistics to real statisticians.

Perhaps he's too busy thinking about reality and nature, designing the experiments, building the hardware and producing hard data.

corey said...

Richard, is that you?

a_ray_in_dilbert_space said...

Anonytroll loves to pretend he can find his ass with both hands and a GPS. He should continue to do so, as his delusions of adequacy are entertaining.

THE CLIMATE WARS said...

How does the absence of indignant comments from dead Bayesians figure in the modal logic?

Fernando Leanme said...

J. Bowers, I thought it would be of interest to mention my experience with the way we saw data ignored in real life. Whether you wish to discuss my personal experience or not it's up to you. However, inventing ideas about my supposed positions in a somewhat irrelevant subject matter doesn't contribute to a polite chat. Do you feel unusually put off by what I wrote?

David B. Benson said...

Cowards...