If you're just going to lower your alpha level according to the pvalue what is the point of even reporting the alpha level? To me it becomes meaningless at that point. Just report the pvalue if that's all you're going to do.
To go to your example (which I admit is a better analogy than what I gave)  just because you got lucky and had a better crop of students this year doesn't mean that that's your standard. Let's reduce it down to the case of only considering accepting a single student. Your requirements say that they need to be in the top 5%. You would gladly accept anybody in the top 5%. You get lucky and the student you pick is in the top 1%. Does that mean that you should report that you only accept students in the top 1%? No  because you're not that strict and you're just lying to the public if you say you are. What's the point of even reporting alpha then?
0882: "Significant"
Moderators: Moderators General, Prelates, Magistrates
Re: 0882: "Significant"
double epsilon = .0000001;
Re: 0882: "Significant"
Well, my point was actually that often times .05 is used where .01 could have just as well been used. The results don't change, so what's wrong with indicating a lower probability of a type I error? If .01 could have been used but .05 was used as the indicator of significance, I see no problem in selecting a more favorable alpha considering p will be reduced accordingly.
i.e., if .01 CAN be used while maintaining statistical significance, but isn't, doesn't that reflect that 1/20 studies that report at a .05 level committing a type I error is actually a conservative estimate? If 20/20 studies report at .05 but find significance at .01, then the risk of a type I error is only 1/100 at the respectively lower p values. I don't think the lay person understands that it's the relationship between alpha and p that is telling. When you report only that p>.05 (as the comic did), but not the size of the effect, you conceal the fact that this applies only to the size of the effect, and not the standardized "validity" (for lack of a better word) of the study.
When we only report significance/insignificance according to a .05, we don't actually give any good indication of the robustness of the findings. Granted this may not have much practical significance, but I thought this is why many studies report the level at which p is significant in addition to an a priori alpha.
i.e., if .01 CAN be used while maintaining statistical significance, but isn't, doesn't that reflect that 1/20 studies that report at a .05 level committing a type I error is actually a conservative estimate? If 20/20 studies report at .05 but find significance at .01, then the risk of a type I error is only 1/100 at the respectively lower p values. I don't think the lay person understands that it's the relationship between alpha and p that is telling. When you report only that p>.05 (as the comic did), but not the size of the effect, you conceal the fact that this applies only to the size of the effect, and not the standardized "validity" (for lack of a better word) of the study.
When we only report significance/insignificance according to a .05, we don't actually give any good indication of the robustness of the findings. Granted this may not have much practical significance, but I thought this is why many studies report the level at which p is significant in addition to an a priori alpha.
Re: 0882: "Significant"
p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite. Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient). We can also state p<0.01 when we want to say it's REALLY good data, but we never write p=0.00 because that can't really be true even if the computer spits it out.
tl;dr: Please exchange > and < in this comic!
tl;dr: Please exchange > and < in this comic!
 phlip
 Restorer of Worlds
 Posts: 7572
 Joined: Sat Sep 23, 2006 3:56 am UTC
 Location: Australia
 Contact:
Re: 0882: "Significant"
soraten wrote:p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite.
Not quite... it's the other way around: p<0.05 means that, if we were wrong, there's a less than 5% chance we'd get a falsepositive result anyway. As we see in the comic: they do 20 tests, and get 19 negatives and one false positive; a 5% falsepositive rate, which is what you expect setting p<0.05 as your significance threshold. Note that this doesn't translate directly directly into a "probability of being wrong"... to do that you need Bayes' theorem, and to use that you need a prior.
soraten wrote:Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient).
No, he has it the right way around  he says p>0.05 for the tests that have no link, and p<0.05 for the test that (by chance) does find a (false) link.
Code: Select all
enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵╰(ಠ_ಠ ⚠) {exit((int)⚠);}
Re: 0882: "Significant"
As usual, it takes a while, but science catches up with xkcd.
This article in the journal Psychological Science explicates the effects of running a lot of tests and only reporting the ones with the low pvalues. (The scientists in the comic don't do this, the newspaper does, but the result is effectively the same.)
I love the term the authors use, "researcher degrees of freedom".
Good commentary here, too.
This article in the journal Psychological Science explicates the effects of running a lot of tests and only reporting the ones with the low pvalues. (The scientists in the comic don't do this, the newspaper does, but the result is effectively the same.)
To help illustrate the problem, we conducted two experiments designed to demonstrate something false: that certain songs can change listeners’ age. Everything reported here actually happened.
I love the term the authors use, "researcher degrees of freedom".
Good commentary here, too.
A key part of the story is that, although such manipulations could be performed by a cheater, they could also seem like reasonable steps to a sincere researcher who thinks there’s an effect and wants to analyze the data a bit to understand it further.
These days, if you don't have ADD, you not paying close enough attention.  J.P. Barlow

 Posts: 5
 Joined: Wed Nov 05, 2014 8:31 pm UTC
Re: 0882: "Significant"
This comic is featured in Useful Genetics from the University of British Colombia on Coursera https://www.coursera.org/ubc. When I first read it I got a bit of a chuckle from the minecraft reference and the obviously overblown newspaper headline. However I had no idea what p stood for and I had never heard of a null hypothesis. Now that I know a little more I appreciate the comic's depth from the seemingly backwards > in the second panel to the exactly 20 different colors tested. It's really amazing to me that Randall was able to entertain me in my ignorance and is able to entertain even more as I grow. I wonder what else I've missed.
Re: 0882: "Significant"
Peristarkawan wrote:This demonstrates the fallacy that scientists refer to as "jelly bean picking".
You may have a future in Journalism.
Life is, just, an exchange of electrons; It is up to us to give it meaning.
We are all in The Gutter.
Some of us see The Gutter.
Some of us see The Stars.
by mr. Oscar Wilde.
Those that want to Know; Know.
Those that do not Know; Don't tell them.
They do terrible things to people that Tell Them.
We are all in The Gutter.
Some of us see The Gutter.
Some of us see The Stars.
by mr. Oscar Wilde.
Those that want to Know; Know.
Those that do not Know; Don't tell them.
They do terrible things to people that Tell Them.
Return to “Individual XKCD Comic Threads”
Who is online
Users browsing this forum: GlassHouses, Moose Anus and 110 guests