Wednesday, July 8th, 2009

Quote of the day

No atheist should call himself or herself one… A more appropriate term is “naturalist”, denoting one who takes it that the universe is a natural realm, governed by nature’s laws. This properly implies that there is nothing supernatural in the universe. […] People with theistic beliefs should be called supernaturalists, and it can be left to them to attempt to refute the findings of physics, chemistry and the biological sciences in an effort to justify their alternative claim that the universe was created, and is run, by supernatural beings.

A.C. Grayling, Against All Gods

(Leave a comment)

Friday, July 3rd, 2009

Homeopathic A&E

(In case of embedding problems, click here.)

(4 comments | Leave a comment)

Thursday, July 2nd, 2009

Why Evolution Is True; and cumulative selection

The other day, I picked up Why Evolution Is True by Dr. Jerry Coyne from the library. A few days later I had finished it, and a few days after that, I want to write a few lines on what I think about it.

The book’s aim—as very straightforwardly implied by the title—is to lay out in concise form a reasonably comprehensive (and comprehensible) body of evidence for evolution. As such, it spans a pretty wide range of areas—biogeography, palæontology, genetics, and so forth. At only ~300 pages, it has to go at a pretty good pace, and it does—but it’s largely a good thing. The book is accessible, but not dumbed down; it is brief and concise, but not superficial. It lays out a huge breadth of evidence with plentiful references (many internet references) for those who want more depth.

My brief opinion is: This is one of the best, and possibly the best book I have read in terms of laying out precisely what the title claims: Why Evolution Is True.

The funny thing is, when I put it down, my mind was actually full of gripes. I was constantly wondering about the tone—it wasn’t very technical, but couldn’t it have been simplified in places? I now think that, yes, it could have, but I don’t think it would have been to its advantage. It’s simple enough to be accessible to laymen, and that is enough. Let’s not pretend that it isn’t science, don’t give the impression of condescending, and don’t sacrifice precision by avoiding scientific terminology altogether.

I also found one argument missing that I might have liked to see—one that Richard Dawkins has made wonderfully lucid in more than one book—that of the difference between “single-step” and cumulative selection: The counter to the old “747 in a junkyard” argument¹. In fact, its omission irked me very greatly because I think it is such an excellent counter to fairly common creationist/cdesign proponentsist objections to evolution by natural selection as being statistically impossible.

However, I think that the reason why this irked me so very greatly may be because virtually every other persuasive argument is either explained or alluded to; and the focus of the book is, after all, on evidence rather than argument. If someone near you suffers under the delusion that evolution is not a fact, and the neo-Darwinian synthesis is not a very solid scientific theory, you could scarcely do better than to recommend this book to them—perhaps with an explanation of cumulative selection to solidify the deal; or have them graduate to Dawkins, e.g. The Blind Watchmaker, which takes a complementary approach of theoretical argument (though on a very accessible level!) as contrasted to Coyne’s straightforward presentation of evidence.


¹ The “747 in a junkyard” argument stems from this quote by astronomer Fred Hoyle:

A junkyard contains all the bits and pieces of a Boeing 747, dismembered and in disarray. A whirlwind happens to blow through the yard. What is the chance that after its passage a fully assembled 747, ready to fly, will be found standing there? So small as to be negligible, even if a tornado were to blow through enough junkyards to fill the whole Universe.

Hoyle was not a creationist—but never mind his motivation. Creationists have hijacked this quote and use it to point out a perceived implausibility of evolution. The chance of something so complex as an eye, for instance, arising by chance, is of course minuscule. How can “Darwinists” claim that it arose purely by chance? The answer is, of course, that they don’t, because nobody thinks that the eye sprung forth fully formed from a single mutation, but rather incrementally, and if it was improbable, it was a matter of cumulative selection.

What do I mean by “cumulative probability”? I mean that we can build up on past successes. Take, for example, a coin flip. The odds of getting heads on a single flip is ½. The odds of two flips simultaneously resulting in heads are ½×½ = (½)² = ¼. Three heads at once? ½×½×½ = (½)³ = ⅛. —And so on. The odds of, say, 100 heads all at once are 1 in 2100: Less than one in a thousand billion billion billion. If we flip our 100 coins once a second, it will take us on the order of a million billion billion years to flip all 100 heads at the same time. That’s about 100,000 billion times the age of the universe. This is single-step selection: We’re looking for a specific result, and we need to get it in a single step: The simultaneous flip of 100 coins.

But natural selection doesn’t require this. The theory of evolution by natural selection predicts that any helpful change will be “saved up” and passed down to further generations—it doesn’t have to be perfect—it just has to be an improvement, however small. If we flip 100 coins, we’ll almost certainly get some heads—the odds of getting 0 are the same as getting 100, and that will virtually never happen. We’ll probably get about 50 heads. Now we’re allowed to save them, and only have to re-flip the 50 tails. Probably about half of them will be heads. —And so forth. If we assume that we get half heads, half tails every time, we’ll have 100 heads—on average—after 7 flips or so.

You will note that 7 is rather less than a thousand billion billion billion. We can now accomplish the task of flipping 100 heads in about 7 seconds rather than 100,000 billion times the age of the universe (if we can sort through them quickly enough…). The argument is vastly simplified, and obviously none of this applies at all closely to biology.

What should be clear—and the point of the argument—is that there is a huge (in fact, a geometrical) difference between single-step and cumulative selection.

(Leave a comment)

Tuesday, June 16th, 2009

Not a great threat

It would be disingenuous to imply that non-vaccination might not lead to an increased incidence in vaccine-preventable illness. It would be equally disingenuous to state that this possibility poses a great threat to America's children.

Dr. Jay Gordon, quoted at Respectful Insolence

It would be…disingenuous to state that this possibility poses a great threat to America’s children.

Never mind polio, which killed or crippled thousands of children every year before it was eradicated by vaccines, the fear of which ruled some people’s childhoods.

Never mind smallpox, an epidemic disease with an average fatality rate of 30%, also eradicated by vaccines.

Never mind Hemophilus influenza type b (HiB), a disease now nearly forgotten in pediatric wards thanks to vaccination, but which used to cause disease in one of every 200 children under the age of 5—whereof ½–⅔ developed meningitis, with a mortality rate of 5% and rate of permanent brain damage of 30%.

No—none of these, nor any of the other among the dozens of vaccine-preventable diseases now eradicated or dramatically reduced, pose a great threat; thus, because there’s no great threat, we should cautiously withhold vaccination just in case we ever find evidence that they cause any harm. We have no such evidence, but why jump the gun? It’s not like they prevent any great threat.

(Leave a comment)

Friday, June 12th, 2009

Mere Christianity

Over the course of many a fruitless religious debate, one book that my ‘opponents’ have often urged me to read is Mere Christianity by C.S. Lewis. I had never done so, but when I found out that the whole thing was available online (here), I went ahead and read it—in stolen snippets of two days, at that; it’s short and a light read. My very brief conclusion is that C.S. Lewis is an entirely different brand of apologist from the raucous, idiot, Ray Comfort kind to which I have previously been exposed. I get every impression that he was being quite sincere and honest. He may also very well have been intelligent. —I say “may” because this book provides no evidence that he was, but nor do I think that it provides strong evidence that he wasn’t.

That said, in the early chapters of Mere Christianity, comes off as honest, sincere, quite possibly intelligent, and completely unconvincing and to all appearances dead wrong. (This review originally contained a part explaining why I consider it coherent to be intelligent, honest, and completely wrong; that aside grew into this.) So unconvincing and so wrong, in fact, that while I consider it entirely possible that he was intelligent, and while some of his fans may very well be very intelligent (with the same rationale), anyone who was convinced by it must have had their critical thinking faculties shut off for the day. Much as a palæontologist accepts a single fossil or a physicist a single relativistic experiment, you may accept Mere Christianity as fitting into a worldview, but it is no more sufficient to build a complete theory upon. Unlike fossils and physical experiments, however, Mere Christianity attempts logical arguments, and—well, we shall see how it succeeded.

The book is written in a compelling way—easy, conversational language, and a structure where each chapter builds directly and explicitly on the one before it. Thus, he starts off by establishing a universal moral law; shows that the universal law must reflect some underlying reality; shows that this underlying reality must be an Intelligence; shows that it must be an Intelligence rather like the Christian God—and so forth. He is not mealy-mouthed, nor needlessly offensive, nor does he sound insincere. All of this gives me a rather favourable view of him as a person.

As a logician and persuader, however, I can’t give him much respect. My initial reaction to the first few chapters was that, with some minor restructuring, they could easily be retitled according to which logical fallacy he built each chapter’s claim upon. Thus one early chapter took St. Anselm’s failed Ontological Argument and applied it to moral law: We can conceive of a moral law better than our own; therefore there must be a Perfect moral law. (Not true: We might have and fully grasp the ultimate moral law but fail to recognise that it’s perfect.) Another was based on Equivocation (descriptive natural laws with prescriptive moral laws). Another, while not a formal fallacy that I’m aware of, was based on equivocating percepts with objects: That is, he went from All humans feel that there is something rather like X to Therefore, there exists an X with some sort of independent reaction. (Nonsense! If we find that all humans feel X we have indeed discovered a fact, but it’s a fact about human brains, not about the world outside them.) These percepts, once reified, were deified in short order.

Unfortunately, the book went rather downhill from this point. In the early chapters, I can really respect what Lewis was trying to do. Of course, I find that his arguments were not in fact valid, but he clearly believed the premises were true, he obviously believed in his conclusion, and as I have said before and will gladly repeat, it is often very difficult to find flaws in your own inferences when they make a path whereby, as far as you can tell, you get from the right starting point to the right end point. And in these early chapters, I am inclined to agree that if his arguments had been valid and sound, as he believed, then he had some very right and very valuable things to say; and he does lay out his arguments, however flawed, clearly and lucidly.

But this, alas, was not to last. Having once established (in his mind) that there must be a deity that shares some important, basic traits with the god of Judeo-Christian mythology, he went on to implicitly assume a whole slew of Christian dogma, and he did it so suddenly and unselfconsciously that it took me a chapter or two before I went Hang on a minute…! It is as though, once you accept a good, omnipotent creator deity, Moses, the Ten Commandments, Jesus, Judas, and the whole cabaret just followed naturally. This was a huge disappointment—he didn’t even try to show his work in this part of the examination.

The redeeming aspect of this part of the work was that if you once accept his assumptions, a lot of the things he says are very cogent and sensible. But that is not much help if you haven’t accepted those assumptions! He also argues an awful lot by metaphor. This is fine—he manages to explain a number of very weird things in Christian dogma in a way that made a lot of sense to me. So far, so good. However, a critical feature of an explanation by metaphor is that you have to be able to show how it reduces back to the real issue. Again, Lewis doesn’t fail to do this—he never even attempts it. It felt very much as if it never occurred to him that this had to be explained.

And I found this very peculiar, because C.S. Lewis was by all accounts an atheist, and he was brought to believe in all these things. How did this happen? I feel as though he must have had more of a story to tell, because the argument he lays out is completely insufficient to take an intelligent person from atheism to Christianity. Even if his initial arguments had been sound, there just wasn’t a chain of logic available to bring an atheist any further than a sort of nebulous proto-Judeo-Christian monotheism with no specifics of ritual or dogma, let alone such esoteric notions as the Trinity (which, by the way, he explains in lucid, wonderful metaphor that he completely neglects to show to be equivalent to any underlying reality). I supppose Lewis, if he was an atheist before, must not have reached that point by skepticism so much as more specific disappointment with points of dogma.

The part of the entire book that I found the most rewarding to read was, and this might surprise you, the two chapters on Faith. Now, I make it no secret that I regard the concept of faith with derision and contempt—faith, as I generally see it used and defined, refers to belief without evidence, and in some circles (particularly US fundamentalists) even belief in spite of evidence, which is lunacy and the least ethical and virtuous thing you can possibly do without involving others. However, C.S. Lewis defines faith very differently. I can do the concept no better justice than to quote him:

Roughly speaking, the word Faith seems to be used by Christians in two senses or on two levels, and I will take them in turn. In the first sense it means simply Belief—accepting or regarding as true the doctrines of Christianity. That is fairly simple. But what does puzzle people-at least it used to puzzle me—is the fact that Christians regard faith in this sense as a virtue. I used to ask how on earth it can be a virtue—what is there moral or immoral about believing or not believing a set of statements? Obviously, I used to say, a sane man accepts or rejects any statement, not because he wants to or does not want to, but because the evidence seems to him good or bad. If he were mistaken about the goodness or badness of the evidence that would not mean he was a bad man, but only that he was not very clever. And if he thought the evidence bad but tried to force himself to believe in spite of it, that would be merely stupid.

Well, I think I still take that view. But what I did not see then—and a good many people do not see still—was this. I was assuming that if the human mind once accepts a thing as true it will automatically go on regarding it as true, until some real reason for reconsidering it turns up. In fact, I was assuming that the human mind is completely ruled by reason. But that is not so. For example, my reason is perfectly convinced by good evidence that anaesthetics do not smother me and that properly trained surgeons do not start operating until I am unconscious. But that does not alter the fact that when they have me down on the table and clap their horrible mask over my face, a mere childish panic begins inside me. I start thinking I am going to choke, and I am afraid they will start cutting me up before I am properly under. In other words, I lose my faith in anaesthetics. It is not reason that is taking away my faith: on the contrary, my faith is based on reason. It is my imagination and emotions. The battle is between faith and reason on one side and emotion and imagination on the other.

With this second definition of the word faith, it actually makes sense. What this teaches me is that when I next meet someone extolling the virtues of faith, I need to explicitly establish what, precisely, this person means, because he or she may not be referring to it in the sense that I am used to encountering it. If someone believes in the virtues of faith¹, they are beneath being reasoned with. Faith², on the other hand, is in fact a positive thing! I do not need to be persuaded of its virtue; I agree with it! On the other hand, faith² is not a way in which religion can be reached. If somebody tells me that You won’t find God by evidence; you just have to have faith, they are using faith¹ and I will continue to dismiss them. If they take offence at this, I can now not only explain why, but also point out that C.S. Lewis regarded that claim as stupid.

(Leave a comment)

Reason and error

A reasoned belief is one that is founded on empiricism and a logical argument. Hopefully, we’ll all agree that logic is sound. If you argue that logic doesn’t work, then there’s no point in discussing anything at all with you, because no chain of reasoning can—well, reasoning depends precisely on logic! Thus, I will presuppose that we agree on logic, though you may or may not agree that empiricism is necessary, and some would even claim that empiricism is not epistemologically sound.

First, let me define what I mean by empiricism (I am no philosopher; there may be more precise terms). I do not mean that what I see is necessarily reality (au contraire, I am well aware that our senses are flawed and our brains are prone to certain types of delusion). What I mean by empiricism is simply the following assumption: There exists a systematic relationship between external reality and the percepts of a healthy brain. I must define the brain as healthy: If it is not, it may not follow logic, and it may be plagued by hallucinations to the point where it cannot follow any sort of external reality. If so, alas, I posit that this brain is beyond help. It is not, I admit, impossible that this applies to any given brain, including my own; but absent evidence to this fact, it cannot serve me to believe it or to behave as though it were true, so I will assume that the percepts in my brain do systematically reflect an external reality. I do not, however, need to assume that the relationship is perfect—strictly speaking, all I need is statistical significance.

If I am allowed to assume both logic and empiricism (in the sense above), I can build up a consistent and coherent world view. It doesn’t matter (in principle) that the system is noisy—that some of my logic will be faulty and some of my perceptions incorrect. The assumptions suffice to formulate experiments, which allow me to verify my logic against observed reality, and cross-check my perceptions as much as I want. Repeated experiment lets me overcome the effects of noise in both argument and perception.

I will even take a controversial step and claim that logic needs empiricism for validation—the two cannot be extricated from each other. You cannot, after all, use logic to prove that logic is true—it’s circular (it only works if logic is true to begin with). If you are mathematically inclined, you may note that logic can be represented as a form of mathematics—I wonder if perhaps Gödel’s Incompleteness Theorem can provide a formal version of this verbal argument?

In any case, empiricism supports logic. The reason is as follows: If you assume both empiricism and logic, you can formulate experiments so that, given percept A, you can make a statistical expectation on percept B. This, however, presupposes logic. If we don’t have logic, we have no reason at all to suppose that B will follow A with any degree of certainty. Because we can empirically observe that experiments do bear out, this supports the logical reasoning that we used to make the predictions.

Of course this is far from iron-clad (and even in its weak form does also presuppose logic), but then we can’t really expect too much of an argument that tries to provide evidence for logic itself, now can we?


Having explained why I think that empiricism is a necessary assumption to make any sense of the world whatsoever, I suppose I should mention—however briefly—why I dismiss alternatives. The most obvious alternative is solipsism, the notion that none of the external world has any reality to it and all you can really know is your own mind. That’s not exactly nonsensical, but it’s not worth considering because it tells you nothing—it won’t get you anywhere. It provides no epistemological framework useful for interacting with anything (if everything you interact with is in your own head, why expect it to behave systematically?). It provides no reason to take logic seriously. It allows you no conclusions.

And, quite frankly, I think that all systems that reject empiricism and scientific thinking suffer of different degrees of the exact same thing. What you claim to intuitively know I may very well intuitively doubt, and if we are to settle it independently—well, we need logic and empiricism. If you claim that reality is somehow subjective and depends on your point of view, that your reality is not necessarily the same as mine, we lack a framework to interact, and it is self-defeating because you have no standing to declare that my view of reality as objective isn’t right (if you do so declare, you are making a distinctly universal and objective claim).


A logical argument, in its most basic form, looks like AB; A; ∴B. In English: “If A is true, then B must be true; A is true; therefore B is true.” A and B are both propositions, roughly “truth claims”. A is the premise. AB is the inference that drives the argument. B is the conclusion. Now, there are four ways to be wrong:

  1. You believe in proposition B without any logical or empirical reason. This is just silly.
  2. Your premise is correct (A really is true), but your argument is not validA doesn’t necessarily imply B.
  3. Your argument is valid, but not sound: Your premise, A, is not actually true.
  4. Your premise is false and your argument is invalid.

Note that it is quite possible to go from false premises to a true conclusion, or true premises to a true conclusion via an invalid argument. Reaching a correct conclusion is not proof of sound thinking!

The point of this discussion is that if once you believe in a set of premises and in a conclusion, it’s pretty easy to overlook flaws in the inference. If I know I believe B because A is true, and nothing occurs to gainsay either A or B, I’m not likely to revisit the inference AB with a very critical gaze, because clearly, it worked. However, this is not a reasonable thing to do if this argument is my only reason for believing in B—and since I may have made a mistake in any argument, I should try to be critical of all of them (it may not be my only reason for believing something, but the other reasons may be unsound arguments, so I should treat each one as important). To me, critical thinking lies in scrutinising the premises, but especially of watching inferences very carefully. I pay less attention to conclusions (in a debate, I am unlikely to attack them), because they will flow naturally from the argument if once a sound argument is established.

(3 comments | Leave a comment)

Thursday, June 11th, 2009

$2,500,000,000 US tax dollars tells us “Sorry, it was a waste of time”

The US National Institute of Health department, the National Centre for Complementary and Alternative Medicine (NC-CAM), whose aim is to find evidence for alternative medicine, found to its chagrin that alternative medicine doesn’t work. Key snippets:

Ten years ago the government set out to test herbal and other alternative health remedies to find the ones that work. After spending $2.5 billion, the disappointing answer seems to be that almost none of them do.

Echinacea for colds. Ginkgo biloba for memory. Glucosamine and chondroitin for arthritis. Black cohosh for menopausal hot flashes. Saw palmetto for prostate problems. Shark cartilage for cancer. All proved no better than dummy pills in big studies funded by the National Center for Complementary and Alternative Medicine. The lone exception: ginger capsules may help chemotherapy nausea.

As for therapies, acupuncture has been shown to help certain conditions [though if I read it aright, That finding was called into question when a later, larger study found that sham treatment worked just as welled.], and yoga, massage, meditation and other relaxation methods may relieve symptoms like pain, anxiety and fatigue.

…Critics say that unlike private companies that face bottom-line pressure to abandon a drug that flops, the federal center is reluctant to admit a supplement may lack merit — despite a strategic plan pledging not to equivocate in the face of negative findings.

"There's been a deliberate policy of never saying something doesn't work. It's as though you can only speak in one direction," and say a different version or dose might give different results, said Dr. Stephen Barrett, a retired physician who runs Quackwatch, a web site on medical scams.

Critics also say the federal center's research agenda is shaped by an advisory board loaded with alternative medicine practitioners. They account for at least nine of the board's 18 members, as required by its government charter. Many studies they approve for funding are done by alternative therapy providers; grants have gone to board members, too.

[The Centre’s methodology] is opposite how other National Institutes of Health agencies work, where scientific evidence or at least plausibility is required to justify studies, and treatments go into wide use after there is evidence they work — not before.

In a federally funded pilot study, 30 dieters who were taught acupressure regained only half a pound six months later, compared with over three pounds for a comparison group of 30 others. However, the study widely missed a key scientific standard for showing that results were not a statistical fluke.

In other words, NC-CAM, which was founded with the intent of finding evidence for the quackery that the sponsoring Senators were already convinced by (to look for a yes, in other words, rather than objectively assessing credibility), is perfectly happy to spend millions upon millions of US tax dollars on investigating ludicrous fantasies like distance faith healing, energy healing, and homeopathy (dollars that could be spent on valid research), is biased by a board of proponents, tends to publish lackluster studies with missing controls…and still can’t come up with a single positive result beyond noting that ginger may (may) help with nausea.

If that’s the best they can come up with the cards stacked unreasonably in their favour, then it’s time to pull the plug and spend the next $2.5 billion dollars on something useful.

(1 comment | Leave a comment)

Thursday, June 4th, 2009

Excerpt from a paper

Virtually all commercially produced pharmaceuticals have their source and/or intellectual roots in active compounds isolated from plants. It is ironic that ‘herbal’ sources themselves are often considered ‘safe’, but that commercially produced derivatives – the compounds that are forced to undergo toxicity testing – are not. The finding that 1/5 Ayurvedic medicines purchased on the Internet have detectable and often toxic levels of lead, mercury and arsenic (Saper et al., 2008) should give those who think CAM interventions can do no harm serious pause.

Homeopathy and the curse of the scientific method
Karen L. Overall, Arthur E. Dunham

Indeed.

(1 comment | Leave a comment)

Vaccines and the...what epidemic, now?

Although people all over the spectrum—layman bloggers like me or medical experts like those at the American CDC—all agree that the link between MMR and other vaccines on the one hand and autism on the other are spurious, many people still ask: If not this, then how shall we explain the autism epidemic?

Well, the best answer I have seen is in this, one of Orac’s most succinct articles. I do not, myself, have much to add, so I shall merely provide that link and provide a summary for the most impatient among you (though if you are that impatient, why read someone so wordy as me?).

The one piece of irony I wish to add is that I have seen pre-emptive protests by those who do buy into this stuff that claiming that diagnostics have improved radically in the past few decades won’t cut it—irony, because I have never seen anyone claim that we are better now than previously at diagnosing autism. Instead—and here I go into brief summary mode for that article—what has happened is that the diagnostic criteria have changed. In a very real sense, the definition of autism has changed.

What seems to have happened is this: Various sources of statistics, like those for students with any kind of significant learning disabilities, classify those students by their primary diagnosis. A couple of decades ago, autism wasn’t even a category. By sixteen years ago, people were diagnosed as autistic if they met a specific set of criteria. More recently, the criteria have expanded, autism has been expanded into autism spectrum disorder (and many who are diagnosed on that spectrum are defined as high-functioning: They may have ‘peculiarities’, but are not ‘disabled’ in any serious sense)…and of course the number of people diagnosed as autistic have gone up.

Well, of course they have! Thirty years ago they’d have been diagnosed as something else entirely. And this is not because doctors have become better at making the diagnoses: No one is claiming that. Instead, the medical community has changed the definition of what it means to be autistic. (This may very well be for good reason: Unifying similar conditions, etc.) Thirty years ago, perhaps, you were diagnosed with autism if you showed symptoms X, Y, or Z; now you may be diagnosed on the autism spectrum if you show two or more out of the symptoms X, Y, Z, U, V, or W.


There is one additional twist to the story: Because the diagnostic criteria have changed (and because diagnosing disorders like autism is a lot trickier than, say, bacterial diseases where a pathogen is or is not present in a pretty concrete way), it may be impossible to figure out if the prevalence of autism really has changed at all. This is unfortunate because it makes it that much harder to study the condition and figure out what the causes really are; and while some high-functioning people with autistic spectrum disorders are fine just the way they are, low-functioning autism can be a pretty awful thing. It’s bad enough that researchers are sidetracked and distracted by claims to study these spurious vaccine danger claims (certainly a worthwhile topic to study! —but it’s been done again, and again, and again…).

(Leave a comment)

Thursday, May 28th, 2009

Information market misinformation marketing

I came across an article the other day (originally found via Pharyngula), Where is Everyone?, which aims to track the trend in how information is found and accessed over the past 200 years. This is a very ambitious, and very interesting project. The article starts with a colourful graph, and proceeds to analyse the implications of the graph:

Information flow

Of course, somebody asked the author (a Thomas Baekdal) what data the chart was based on, and he freely answered:

The graph was based on combination of a lot of things, a number of interviews, general study, general trend movements, my experience etc. I cannot give you a specific source though, because I used none specifically.

The graphs before 1990 are all based on interviews, and a large number of Google searches to learn about the history of Newspaper, TV and Radio - and more specifically, what people uses in the past. The graphs from 1998 and up to today, is based on all the things that have happened in the past 11 years, of which I have probably seen 1000 surveys (it is what I do for a living). And the graph from 2009 and forward is based on what I, and many other people predict will happen in the years to come.

One very important thing though, this is not a reflection of my opinion. This is the result a careful analysis. There are always variations, and different types of people. But I believe that this graph accurately reflects consumer focus.

…Have you ever seen such tripe in your life? Merely using different sources and methods for different time periods would introduce uncertainty to the results, but at least this would be inevitable. But, given this guy’s process, that doesn’t even register as a problem. He cannot give you a specific source though, because [he] used none specifically. He has probably seen 1000 surveys of the trends of the past 11 years, but can’t be bothered to cite even one. The x-axis has an arbitrarily compressed scale, skewing the shape and the speed of the trends. And, most damning of all, he gives no indication of how he measures the ‘information’ metric! (The only truly objective measure of information that I know of is Shannon’s ‘bits of entropy’, which is certainly very concrete—but there’s no indication that this is what’s meant, nor can I think of any way in which a person’s total information input can be objectively measured in these units.) How on Earth can anyone analyse the graph critically without knowing what the numbers measure?

The answer to the last question, of course, is that it’s impossible, except in the sense that I am analysing it critically here: Calling it bunk. By presenting a graph, he gives himself a veneer of scientific responsibility (Look, I have data!), but since the graph doesn’t actually objectively represent anything (so far as the reader can tell), it’s really just a distraction, an attempt to gain enough credibility in the reader’s mind that the purported analysis that follows is swallowed whole.

And he has the gall to claim that

this is not a reflection of my opinion. This is the result a careful analysis.

If he hadn’t pretended this (id est, if he had said up front that this is a mashup of various analyses of a practically unquantifiable commodity, but that he hopes that his argument, once followed through, will persuasively show a genuine trend), I might have given him some respect, but given what he actually did, he is either a fool or a liar. Either option should persuade you not to take him seriously.


To address the article as though it weren’t total bunk, his extrapolation into the future is on shaky grounds for reasons that should be painfully obvious even to someone who does buy into the graph: By extrapolating current trends into the future, he seems to be ignoring the fact that the big new things of recent years—social networks, social news, etc.—came out of nowhere and took internet culture by storm. What the internet does—the most important thing it does—is enable distribution of information to vast numbers of people with virtually no marginal cost. Logistically, I can reach a thousand people as easily as one; a million almost as easily as a thousand; a billion with only a little more difficulty with a million. When someone does come up with the next killer idea—the next Facebook or Twitter or Google, or whatever it may be—it can explode at an incredible rate. On the internet, where no one is limited by broadcast range, print batch size, or radio band constraints, the primary limiting factor is user interest. The Next Great Thing may grow slowly and incrementally, or it may explode geometrically, as fast as server capability can handle (and how fast that is depends on what the Next Great Thing is, which of course we don’t know).

In other words, even if I try to buy into the general idea, I think that his predictions are about as reliable as any ever are in futurology, and if I view the whole thing critically, it’s bunk. Either way, I can’t say I am impressed.

(3 comments | Leave a comment)

Wednesday, April 29th, 2009

Alt-med company kills thousands, sues whistle-blower

Specifically, vitamin salesman Matthias Rath and his associates in and outside of South Africa have indirectly caused the deaths of untold thousands of people from AIDS by pushing and lobbying for notions that anti-retrovirals kill and vitamins should be taken instead.

Vitamins are good. If you must pick the one or the other, medicine to block viral transmission is a hell of a lot better.

And, of course, when exposed, these second-hand murderers have sued people all over the place (with a notable lack of success). In author Ben Goldacre’s own words:

This is the “missing chapter” about vitamin pill salesman Matthias Rath. Sadly I was unable to write about him at the time that book was initially published, as he was suing my ass in the High Court. The chapter is now available in the new paperback edition, and I’ve posted it here for free so that nobody loses out.

Although the publishers make a slightly melodramatic fuss about this in the promo material, it is a very serious story about the dangers of pseudoscience, as I hope you’ll see, and it was also a pretty unpleasant episode, not just for me, but also for the many other people he’s tried to sue, including Medecins Sans Frontieres and more. If you’re ever looking for a warning sign that you’re on the wrong side of an argument, suing Medecins Sans Frontieres is probably a pretty good clue.

The chapter may be found here.

(1 comment | Leave a comment)

Good bullet points responding to the anti-vaccination movement

Said bullet points found at AntiAntiVax.

(Leave a comment)

Monday, April 27th, 2009

Faith healing exposed!

So great was the confidence in him, that the blind fancied they saw the light which they did not see—the deaf imagined that they heard—the lame that they walked straight, and the paralytic that they had recovered the use of their limbs. An idea of health made the sick forget for a while their maladies; and imagination, which was not less active in those merely drawn by curiosity than in the sick, gave a false view to the one class, from the desire of seeing, as it operated a false cure on the other from the strong desire of being healed. Such was the power of the Irishman over the mind, and such was the influence of the mind upon the body. Nothing was spoken of in London but his prodigies; and these prodigies were supported by such great authorities, that the bewildered multitude believed them almost without examination, while more enlightened people did not dare to reject them from their own knowledge. The public opinion, timid and enslaved, respected this imperious and, apparently, well-authenticated error. Those who saw through the delusion kept their opinion to themselves, knowing how useless it was to declare their disbelief to a people filled with prejudice and admiration.

The faith healer? An Irishman called “Greatraks”. The bold skeptic? An Irish poet called “St. Evremond”, in The Miscellanies of St. Evremond (extracts can be found here). The year? 1665.

The religulous, of course, still buy it.

(Leave a comment)

Tuesday, April 14th, 2009

The “Big Pharma” Gambit

In areas like alternative medicine and the anti-vaccine movement, one argument that is frequently brought up is that the status quo looks the way it does because Big Pharma is suppressing the argumentor’s favoured research results—they suppress evidence that vaccines cause health problems because they are greedy and want to make money from vaccines even if harmful; they suppress evidence that homeopathy works because they are greedy and do not want to lose the drug market to homeopaths…

First, let me state quite baldly that I firmly believe that the large pharmaceutical companies fall pretty squarely in the Big Evil Corporation category and frequently engage in questionable or reprehensible behaviour. Certainly many of their executives are as motivated by greed, and as ruthless, as executives of other Big Evil Corporations, like oil companies. I do not dismiss out of hand claims that Big Pharma are doing questionable things. But obviously, that doesn’t mean that they are guilty of all the evils of which they are accused, and we have to look at the actual claims, and corroborating evidence, in order to figure out what’s what.

Frankly, I find the vaccine claim outright puzzling. I have taken vaccines for a number of different things. Every vaccine shot I have taken, even ones I had to pay for entirely out of my own pocket, have cost less than or around $50. Assuming that a vaccine requires one booster shot, that represent a sales potential of $100 for my entire lifetime, a statistical 80 years or so. That’s not a lot of profit.

On the other hand, if vaccines are not available for a disease, the disease has to be cured and controlled. Consider polio, a disease eradicated by vaccines:

There is no cure for polio. The focus of modern treatment has been on providing relief of symptoms, speeding recovery and preventing complications. Supportive measures include antibiotics to prevent infections in weakened muscles, analgesics for pain, moderate exercise and a nutritious diet. Treatment of polio often requires long-term rehabilitation, including physical therapy, braces, corrective shoes and, in some cases, orthopedic surgery.

Portable ventilators may be required to support breathing. Historically, a noninvasive negative-pressure ventilator, more commonly called an iron lung, was used to artificially maintain respiration during an acute polio infection until a person could breathe independently (generally about one to two weeks). Today many polio survivors with permanent respiratory paralysis use modern jacket-type negative-pressure ventilators that are worn over the chest and abdomen.

Suppose that some utterly ruthless Big Pharma executive sits down and does the math on this. We can either sell $100 worth of vaccines to quite a lot of people, but once only per patient lifetime…or, if we make no vaccine available (or allow it to be banned due to spurious health concerns) we can sell antibiotics, analgesics, braces, corrective shoes, surgical equipment, iron lungs…

I don’t claim to be an expert on the market, but I postulate that vaccines just aren’t big money makers compared to after-the-fact treatments, and obviously vaccines compete with curative and palliative drugs. As someone said in agreement with my opinion,

I had a friend working as an assistant on big pharm sponsored vaccine research project back in the early 90s. The pharm company eventually pulled funding for the research, and the researchers suspected that the motivation was that producing drugs to treat the illness in question was a better moneymaker than funding relatively expensive research to develop a vaccine. The vaccine would have essentially killed a bunch of the company's product lines.

Let me make this very explicit: This quote is pure anecdote and is not intended to be used as evidence, but presented as an example of why my argument is plausible. Nor do I have the market research and relative cost/profit analyses for vaccines versus conventional drugs. However, my point is that in order for the “Big Pharma” conspiracy theory to hold any water at all, this argument has to be addressed. In short, conspiracy theorists who view vaccines as poisoning for profit must believe that

  1. Big Pharma executives are so ruthless and greedy that they are willing to poison millions of children (including their own) for money;
  2. They do this, and get away with it with no sign of internal whistle-blowers (the critics are always outside critics, with no sign of leaked memos as is usually the case in attempts at corporate cover-ups); and
  3. Vaccine production is so profitable that even after R&D costs, it earns the company more money than selling curative and palliative treatments.
…And if they wish to be believed, they have to substantiate that.


Similar claims are often raised by supporters of “alternative” medical treatments like homeopathy and naturopathy. It’s not quite so sinister—they tend to accuse Big Pharma not of mass poisoning campaigns, but merely suppressing their own (surely superior) treatments for profit.

Once again, however, these economic accusations are very fast and loose and vague. Even if homeopathic remedies worked, would it really profit Big Pharma to suppress it? I would rather imagine that they would attempt to take over that market and drive the smaller players out. Simply by pushing for increased regulation (requiring similar standards of evidence of effectiveness and safety for “alternative” drugs as for conventional ones), they would kill a lot of companies that lack the R&D resources to run the necessary studies. (Why don’t they do it already? Well, since these treatments don’t work, the studies would never pass muster.)

Do I know that this is the way the finances would work out? Absolutely not! But the careful evasion of even raising the question makes me think that the alt-med advocates would rather no one think it through—it’s much easier, after all, to go with a knee-jerk Big corporate evil! reaction. There’s no reason to take the greed accusation seriously unless it can be shown to be logically coherent.

This argument has a second irony, of course: Alternative medicine is a huge industry. Billions upon billions of dollars are spent on “alternative” treatments every year—without all the R&D costs that real pharmaceutical companies have to battle with; freed of the expenses and vast time commitments of running large-scale, double-blind medical trials to show that the drugs work. Tu quoque is a logical fallacy, but when the argument amounts largely to character assault (Big Pharma is greedy and evil), it may be worth keeping in mind that “alternative medicine” is no more innocent of the character flaw at hand.

(Leave a comment)

Monday, April 13th, 2009

The Secret, or the Law of Attraction

Someone on a forum did me the distinct disfavour of posting the first 20 minutes of the trainwreck film, The Secret, where the “secret” refers to the “Law of Attraction”. Briefly, the idea is that thinking about things will cause them to come about—think about the bad things in life and bad things will happen to you; think about good things and they will happen instead. I’m not going to waste time and space talking about why this is preposterous. What motivates me to write this is rather my anger at this, and what I consider to be the harmful consequences.

Lots of people actually seem to believe in this crap. To some extent, that isn’t too surprising. The facile reasons are, first, that it certainly fits in with a lot of New Age magic; second, that the testimonials look good (the happy supporters they choose to speak out really are happy—of course they are, living in $4.5 million mansions…); and third, it is endorsed by highly visible and respected idiots, like Oprah.

More importantly, however, it ties in very neatly with things that are actually true.¹ Of course positive thinking tends to improve your life in many ways—it’s a well established psychological fact that acting happy tends to make you happier; happiness and confidence improve your interpersonal skills and relations; avoiding focusing on negative things frees you from brooding over misfortunes. None of this validates the “secret”. The fact that your mental attitude is connected to your mental state is painfully obvious, and a positive demeanour improving interpersonal relations (and through that avenue, your life) is only evidence that people respond better to happy, confident people than to sad or aggressive ones, and does not require the existence of some mysterious universal energy found by viciously abusing quantum physics.

All right, then, some hypothetical person might ask, what is the harm? It may be silly, but if it motivates people to engage in positive thinking, which you freely acknowledge is a good thing, then why should we discourage this stuff?

Apart from the fact that I am as dedicated as I am able to pursue truth, and consider it morally valuable in its own right, I do think that this silliness has a very sinister side.

The first and most obvious problem is that when people put their trust in anything that doesn’t actually work, there is a risk that they will eschew real, working solutions because they think they already have one. For instance, the 20-minute clip from The Secret has someone claiming: I’ve seen cancer dissolved.

Let me reiterate that. The Secret strongly implies that positive thinking can cure cancer.

That is when it ceases to be funny. People who swallow this whole are lead to believe that positive thinking suffices to cure cancer. This misinformation can kill. Nothing cures cancer like surgical steel (preferably with chemotherapy and/or radiation therapy as adjuvant therapies to prevent recurrence). Failing to seek proper help can kill you, painfully and horribly.

And, of course, we can extrapolate this to any other medical condition, or for that matter, any other problem.

The second repugnant consequence of this belief in the “Law of Attraction” is that while the film-makers focus strongly on the empowering effect, when positive thinking is believed to change your life, the explicit corollary is that negative thoughts lead to bad things happening. They make this very clear: These people assert not only that negative things will make bad things happen, but that whenever bad things keep happening to you, it is because you are thinking negative thoughts. It’s under your control, they say, and you have the power to change it—but if events are bad, you caused them to happen.

We know, of course, that this is bunk. However, those who believe it are also made to believe that all their misfortunes are their fault. If your house burned down, if you developed cancer, if you were raped—according to the makers of The Secret, this is your fault: You made it happen. This is not only nonsensical, it is also an extremely cruel thing to allege.


¹ There is a parallel here to the view of some bloggers, such as “Orac”, of “complementary and alternative medicine”, which are perceived to usurp some actually valid ideas, like nutrition and exercise: CAM practitioners prescribe good nutrition, exercise, and homeopathic remedies; good nutrition and exercise are clearly good for your health; therefore homeopathy must be good—stated so baldly, the intellectual bankrupcy of the notion is obvious.

(6 comments | Leave a comment)

Wednesday, April 1st, 2009

A correlation out of left field

Autism is a pretty mysterious condition. No one really knows what causes it (all we really know for sure, after all this testing, is that whatever else it does, the MMR vaccine definitely doesn’t cause it…), but it’s thought to be part genetic, part environmental. A Swedish study on indoor air pollutants has now suggested that, although the data are very tentative, vinyl flooring may increase the risk of autism!

The researchers found four environmental factors associated with autism: vinyl flooring, the mother's smoking, family economic problems and condensation on windows, which indicates poor ventilation.

Infants or toddlers who lived in bedrooms with vinyl, or PVC, floors were twice as likely to have autism five years later, in 2005, than those with wood or linoleum flooring.

Whether the link is real is, as the researchers very frankly point out, as yet unknown, and only further studies can reveal it. I find this interesting to consider, however, as a case study in how easy it is to get the wrong impression from results like these. There’s a number of interesting traps to fall into.

  1. It’s fairly likely that someone will report on this, or already has, under a headline like Research finds link between vinyl flooring and autism, giving the impression that it’s clear-cut, whereas the single most clear-cut message of this study is that it ain’t so.

  2. Correlation does not imply causation, and even when there’s causation, we have to make sure we get it the right way around. As one commenter to that article pointed out, autistic children tend to be extremely preoccupied with textures. Even if there’s a direct link between vinyl flooring and autism, that doesn’t mean that the former causes the latter. Maybe families with autistic children prefer vinyl flooring because it makes the children happier, and so in a sense, autism might cause vinyl flooring!

  3. Notice that they found four, that’s four environmental factors associated with autism: vinyl flooring, the mother's smoking, family economic problems and condensation on windows. However, these variables were not controlled for, and may not be independent.

    What does this mean? Well, it may be that any or all of these variables are connected: Maybe poorer people are more likely to smoke, less likely to afford good ventilation, and less likely to afford nice hardwood floors. If any of these things really does increase the risk of autism, the other variables will be associated with it: If, say, the mother’s smoking causes autism, and more poor mothers than wealthy mothers smoke, then vinyl floors and everything else associated with poor people shows up as associated with autism in the statistics. But while the correlation is there and is real, there is (in my example) no causative relationship at all.

    This sort of thing is always a problem with any studies, especially (I believe) when randomisation is poor or sample sizes are small. These are four known and named variables that may reasonably be correlated. What would we have thought of this article if they hadn’t mentioned smoking, wealth, or ventilation? It would have painted a very different picture. And it’s not necessarily dishonesty or editorial brevity that leaves variables out of the equation: Sometimes relevant data just aren’t measured—what if the study hadn’t asked about wealth or smoking?

    I’m reminded of the very poorly thought-out article I read a little while back that claimed that light pollution at night from all the street lights and so forth lead to—I don’t recall: Some health problem or other. However, light pollution goes with industrialisation, and the number of variables you introduce when you compare a more industrial to a more agricultural country is ridiculously large. The article made no mention of those at all, but spoke as though there had to be a direct causal link from light pollution to the health issue at hand (which is why I consider it such a poor article).

  4. The study was not designed to look for these data, which means that we must suspect data mining. Data mining refers to digging through a set of data looking for any relationships, whether the ones originally examined or not. The problem is that some relationship will always be found.

    Suppose, for instance, that a study is in some global sense 99% reliable. What does this mean? It means that we set out to discover whether X causes Y, and if the study says yes, we can be 99% certain that we’re right. On the flip side, there’s a 1% chance that we’re wrong. Now suppose that, since we have all these statistics anyway, we decide to check of X causes Z, or A causes B…and so on. For every single one of these, we may (very generously) be 99% certain that it’s correct, but if we look for 100 different relationships, we know that we’ve probably got at least one wrong!

    In fact, we’re 73% likely to have got at least one wrong, and that’s with a 99% confidence level and the very generous assumption that the data are as reliable in unknown areas. In reality, I expect that will often not be the case: Even if I design my study to control for a lot of variables surrounding the hypothesis I set out to explore, I can’t possibly do the same for a bunch of hypotheses someone constructs from my data after the fact.

    This is why data mining is frowned upon in scientific studies. We can look at data like that and find correlations that intrigue us, and use those correlations to inspire new studies—just as this Swedish study means that it might not be a bad idea to look at possible connections between vinyl flooring (and phthalates) and autism…but we shouldn’t be fooled into thinking that they necessarily mean anything, because we know that if we look hard enough at any set of statistics, we will be able to find some spurious connections.

(Leave a comment)

Children die from measles; Jenny McCarthy doesn’t mind

Normally, the anti-vaccine brigade tend to be either coy or in denial of the fact that eliminating vaccines—or in fact just lowering the vaccination prevalence in the population below the critical herd immunity point—will take a toll in the lives of thousands of children dying every year, suffering complications like encephalitis from measles, and so on. Jenny McCarthy, one of the more notorious proponents of this for of mass collateral child murder, however, spoke up a bit more straightforwardly in an interview with TIME (sic censorship, below):

I do believe sadly it's going to take some diseases coming back to realize that we need to change and develop vaccines that are safe. If the vaccine companies are not listening to us, it's their f___ing fault that the diseases are coming back. They're making a product that's s___. If you give us a safe vaccine, we'll use it. It shouldn't be polio versus autism.

It should be noted that there exists no good evidence that vaccines are in general unsafe, let alone that any cause autism—that particular scare was based on studies by a Dr. Wakefield, which were widely considered to be so poorly conducted as to be useless, even before his data were discovered to be, in all likelihood, fraudulent. It should also be noted and never forgotten that little-feared diseases like measles are actually pretty serious, as the interviewer (a Jeffrey Kluger) points out:

Measles is among the top five killers in the world of children under 5 years old, yet it kills virtually no one in the U.S. thanks to vaccines.

McCarthy replies that

If you ask a parent of an autistic child if they want the measles or the autism, we will stand in line for the f___ing measles.

You be her judge, keeping in mind that

  1. There’s no evidence that any vaccine causes autism.
  2. There’s no evidence that most vaccines (certainly the mainstream childhood vaccines!) are at all prone to causing any long-term complications. Certainly some children have reactions, but ours is not a perfect world: Choose X% chance of a reaction to a vaccine, or Y% chance of dying from any of a large number of diseases. Typically, X is much smaller than Y, and the consequences much less dire.
  3. Unvaccinated children put other children at risk. If vaccination isn’t common enough, pandemics can still spread, and because no vaccine is 100% effective, vaccines prevent pandemics by making it too difficult for diseases to spread. Additionally, some children can’t be vaccinated due to allergies or other, valid health reasons: They rely on other children’s being vaccinated for protection.
  4. Low vaccination rates have already re-introduced measles to the UK. The first death was in 2006. As vaccination rates sink, more deaths will follow.
  5. Wasting people’s time and energy testing and re-testing vaccines for safety with respect to the fictional autism risk takes resources away from useful research—like addressing the real issues with vaccines that have them, or researching the actual causes of autism.
(9 comments | Leave a comment)

Friday, March 27th, 2009

“Why Most Published Research Findings are False”

“Orac” over at Respectful Insolence has a writing style that’s fairly prone to offend—definitely pugnacious, and very fond of side swipes at those he dislikes (primarily alternative medicine quacks)—and I don’t blame him for his distaste, which in fact I share, but it does sometimes make his essays a bit harder to slog through. (He also has an inordinate fondness for beginning sentences with Indeed. This is one area where I can tentatively claim superiority: I can also be pugnacious and come off as offensive, but while I am no less prone than Orac to complicated sentence structure, I’ve never been accused of any such repetitive verbal tic.)

However, those foibles aside, he has written some very good stuff (he’s on my list of blogs I ready daily for a reason), and this article, summarising and explaining the work of a John Ioannidis, was very interesting indeed. The claim it looks at is a very interesting and puzzling one: Given a set of published clinical studies reporting positive outcomes, all with a confidence interval of 95%, we should expect more than 5% to give wrong results; and, furthermore, studies of phenomena with low prior probability are more likely to give false positives than studies where the prior probabilities are high. He has often cited this result as a reason why we should be even more skeptical of trials of quackery like homeopathy than the confidence intervals and study powers suggest, but I have to confess I never quite understood it.

I would suggest that you go read the article (or this take, referenced therein), but at the risk of being silly in summarising what is essentially a summary to begin with…here’s the issue, along with some prefatory matter for the non-statisticians:

A Type I error is a false positive: We seem to see an effect where there is no effect, simply due to random chance. This sort of thing does happen. Knowing how dice work, I may hypothesise that if you throw a pair of dice, you are not likely to throw two sixes, but one time out of every 36 (¹/₆×¹/₆), you will. I can confidently predict that you won’t roll double sixes twice in a row, but about one time in 1,296, you will. Any time we perform any experiment, we may get this sort of effect, so a statistical test, such as a medical trial, has a confidence level, where a confidence level of 95% means there’s a 5% chance of a Type I error.

There’s also a Type II error, or false negative, where the hypothesis is true but the results just aren’t borne out on this occasion. To the best of my knowledge, there is no equivalent of the confidence level for Type II errors.

This latter observation is a bit problematic, and leads into what Ioannidis observed:

Suppose there are 1000 possible hypotheses to be tested. There are an infinite number of false hypotheses about the world and only a finite number of true hypotheses so we should expect that most hypotheses are false. Let us assume that of every 1000 hypotheses 200 are true and 800 false.

It is inevitable in a statistical study that some false hypotheses are accepted as true. In fact, standard statistical practice [i.e. using a confidence level of 95%] guarantees that at least 5% of false hypotheses are accepted as true. Thus, out of the 800 false hypotheses 40 will be accepted as "true," i.e. statistically significant.

It is also inevitable in a statistical study that we will fail to accept some true hypotheses (Yes, I do know that a proper statistician would say "fail to reject the null when the null is in fact false," but that is ugly). It's hard to say what the probability is of not finding evidence for a true hypothesis because it depends on a variety of factors such as the sample size but let's say that of every 200 true hypotheses we will correctly identify 120 or 60%. Putting this together we find that of every 160 (120+40) hypotheses for which there is statistically significant evidence only 120 will in fact be true or a rate of 75% true.

Did you see that magic? Our confidence interval was 95%, no statistics were abused, no mistakes were made (beyond the ones falling into that 5% gap, which we accounted for), and yet we were only 75% correct.

The root of the problem is, of course, the ubiquitous problem of publication bias: Researchers like to publish, and people like to read so journals like to print, positive outcome studies rather than negative ones, because a journal detailing a long list of ideas that turned out to be wrong isn’t very exciting. The problem is, obviously, that published studies are therefore biased in favour of positive outcomes. (If not, all 800 studies of false hypotheses would have been published and the problem would disappear.)

Definition time again: A prior probability is essentially a plausibility measure before we run an experiment. Plausibility sounds very vague and subjective, but can be pretty concrete. If I know that it rains on (say) 50% of all winter days in Vancouver, I can get up in the morning and assign a prior probability of 50% to the hypothesis that it’s raining. (I can then run experiments, e.g. by looking out a window, and modify my assessment based on new evidence to come up with a posterior probability.)

Now we can go on to look at why Orac is so fond of holding hypotheses with low prior probabilities to higher standards. It’s pretty simple, really: Recall that the reason why we ended up with so many false positives above—the reason why false positives were such a large proportion of the published results—is because there were more false hypotheses than true hypotheses. The more conservative we are in generating hypotheses, the less outrageous we make them, the more likely we are to be correct, and the fewer false hypotheses we will have (in relation to true hypotheses). Put slightly differently, we’re more likely to be right in medical diagnoses if we go by current evidence and practice than if we make wild guesses.

Now we see that modalities with very low prior probability, such as ones with no plausible mechanism, should be regarded as more suspect. Recall that above, we started out with 800 false hypotheses (out of 1000 total hypotheses), ended up accepting 5% = 40 of them, and that

It's hard to say what the probability is of not finding evidence for a true hypothesis because it depends on a variety of factors such as the sample size but let's say that of every 200 true hypotheses we will correctly identify 120 or 60%. Putting this together we find that of every 160 (120+40) hypotheses for which there is statistically significant evidence only 120 will in fact be true or a rate of 75% true.

That is, the proportion of true hypotheses to false hypotheses affects the accuracy of our answer. This is very easy to see—let’s suppose that only half of the hypotheses were false; now we accept 5% of 500, that is 25 false studies, and keeping the same proportions,

…Let's say that of every 200 500 true hypotheses we will correctly identify 120 300 or 60%. Putting this together we find that of every 160 (120+40) 325 (300+25) hypotheses for which there is statistically significant evidence only 120 300 will in fact be true or a rate of 75% 92% true.

We’re still short of that 95% measure, but we’re way better than the original 75%, simply by making more plausible guesses (within each study, we were still equally likely to make either Type I or Type II errors). The less plausible an idea is, the higher the proportion of false hypotheses will be out of all the hypotheses the idea generates: A true/false ratio. Wild or vague ideas (homeopathy, reiki, …) are very likely to generate false hypotheses along with any true ones they might conceivably generate. More conventional ideas will tend to generate a higher proportion of true hypotheses—if we know from long experience that Aspirin relieves pain, it’s very likely that a similar drug does likewise.

This is not to say that no wild ideas are ever right. Of course they sometimes are (though of course they usually aren’t). What it does mean is that not only should we be skeptical and demand evidence for them, there are sound statistical reasons to set the bar of evidence even higher for implausible than for plausible modalities.

It is also a good argument for the move away from strict EBM (evidence-based medicine) to SBM (science-based medicine) where things like prior probability are taken into account. Accepting 95% double-blind trials at face value isn’t good enough.

(2 comments | Leave a comment)

Tuesday, March 24th, 2009

Guest post on Ziztur!

Skeptical blogger Ziztur has a little project called Ray a Day wherein she (with the assistance of her boyfriend) go through one question a day from Ray Comfort’s book, You Can Lead an Atheist to Evidence, But You Can’t Make Him Think: Answers to Questions from Angry Skeptics. The other week, she circulated an email among her awesome commenters, in which number I have been granted the honour of inclusion.

Here’s my contribution, wherein I address Ray’s “answer” to a question challenging the existence of Hell, and muse a bit on the motivation underlying such unapologetic apologetics. Go read it, then read the rest of Ziztur’s blog.

(3 comments | Leave a comment)

Sunday, March 15th, 2009

Edward Current clears up misunderstandings


link

(1 comment | Leave a comment)
Previous 20