## 0882: "Significant"

This forum is for the individual discussion thread that goes with each new comic.

Moderators: Moderators General, Prelates, Magistrates

Dason
Posts: 1311
Joined: Wed Dec 02, 2009 7:06 am UTC
Location: ~/

### Re: 0882: "Significant"

If you're just going to lower your alpha level according to the p-value what is the point of even reporting the alpha level? To me it becomes meaningless at that point. Just report the p-value if that's all you're going to do.

To go to your example (which I admit is a better analogy than what I gave) - just because you got lucky and had a better crop of students this year doesn't mean that that's your standard. Let's reduce it down to the case of only considering accepting a single student. Your requirements say that they need to be in the top 5%. You would gladly accept anybody in the top 5%. You get lucky and the student you pick is in the top 1%. Does that mean that you should report that you only accept students in the top 1%? No - because you're not that strict and you're just lying to the public if you say you are. What's the point of even reporting alpha then?
double epsilon = -.0000001;

biodomino
Posts: 67
Joined: Thu Apr 07, 2011 7:18 am UTC

### Re: 0882: "Significant"

Well, my point was actually that often times .05 is used where .01 could have just as well been used. The results don't change, so what's wrong with indicating a lower probability of a type I error? If .01 could have been used but .05 was used as the indicator of significance, I see no problem in selecting a more favorable alpha considering p will be reduced accordingly.

i.e., if .01 CAN be used while maintaining statistical significance, but isn't, doesn't that reflect that 1/20 studies that report at a .05 level committing a type I error is actually a conservative estimate? If 20/20 studies report at .05 but find significance at .01, then the risk of a type I error is only 1/100 at the respectively lower p values. I don't think the lay person understands that it's the relationship between alpha and p that is telling. When you report only that p>.05 (as the comic did), but not the size of the effect, you conceal the fact that this applies only to the size of the effect, and not the standardized "validity" (for lack of a better word) of the study.

When we only report significance/insignificance according to a .05, we don't actually give any good indication of the robustness of the findings. Granted this may not have much practical significance, but I thought this is why many studies report the level at which p is significant in addition to an a priori alpha.

sora-ten
Posts: 1
Joined: Thu Jun 30, 2011 10:13 pm UTC

### Re: 0882: "Significant"

p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite. Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient). We can also state p<0.01 when we want to say it's REALLY good data, but we never write p=0.00 because that can't really be true even if the computer spits it out.

tl;dr: Please exchange > and < in this comic!

phlip
Restorer of Worlds
Posts: 7572
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

### Re: 0882: "Significant"

sora-ten wrote:p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite.

Not quite... it's the other way around: p<0.05 means that, if we were wrong, there's a less than 5% chance we'd get a false-positive result anyway. As we see in the comic: they do 20 tests, and get 19 negatives and one false positive; a 5% false-positive rate, which is what you expect setting p<0.05 as your significance threshold. Note that this doesn't translate directly directly into a "probability of being wrong"... to do that you need Bayes' theorem, and to use that you need a prior.

sora-ten wrote:Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient).

No, he has it the right way around - he says p>0.05 for the tests that have no link, and p<0.05 for the test that (by chance) does find a (false) link.

Code: Select all

`enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}`
[he/him/his]

Editer
Posts: 68
Joined: Mon Mar 31, 2008 4:17 am UTC

### Re: 0882: "Significant"

As usual, it takes a while, but science catches up with xkcd.

This article in the journal Psychological Science explicates the effects of running a lot of tests and only reporting the ones with the low p-values. (The scientists in the comic don't do this, the newspaper does, but the result is effectively the same.)

To help illustrate the problem, we conducted two experiments designed to demonstrate something false: that certain songs can change listeners’ age. Everything reported here actually happened.

I love the term the authors use, "researcher degrees of freedom".

Good commentary here, too.

A key part of the story is that, although such manipulations could be performed by a cheater, they could also seem like reasonable steps to a sincere researcher who thinks there’s an effect and wants to analyze the data a bit to understand it further.
These days, if you don't have ADD, you not paying close enough attention. -- J.P. Barlow

justinlbarker
Posts: 5
Joined: Wed Nov 05, 2014 8:31 pm UTC

### Re: 0882: "Significant"

This comic is featured in Useful Genetics from the University of British Colombia on Coursera https://www.coursera.org/ubc. When I first read it I got a bit of a chuckle from the minecraft reference and the obviously overblown newspaper headline. However I had no idea what p stood for and I had never heard of a null hypothesis. Now that I know a little more I appreciate the comic's depth from the seemingly backwards > in the second panel to the exactly 20 different colors tested. It's really amazing to me that Randall was able to entertain me in my ignorance and is able to entertain even more as I grow. I wonder what else I've missed.

Posts: 10256
Joined: Sun Sep 12, 2010 4:44 am UTC
Location: Oregon Coast: 97444

### Re: 0882: "Significant"

Peristarkawan wrote:This demonstrates the fallacy that scientists refer to as "jelly bean picking".

You may have a future in Journalism.
Life is, just, an exchange of electrons; It is up to us to give it meaning.

We are all in The Gutter.
Some of us see The Gutter.
Some of us see The Stars.
by mr. Oscar Wilde.

Those that want to Know; Know.
Those that do not Know; Don't tell them.
They do terrible things to people that Tell Them.