Bayesian vs Frequentist : the Final Showdown!!

For the discussion of math. Duh.

Moderators: gmalivuk, Moderators General, Prelates

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5967
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Bayesian vs Frequentist : the Final Showdown!!

Postby jestingrabbit » Fri Nov 14, 2008 5:26 am UTC

ATCG wrote:
Spoiler:
Assuming that the problem allows for some assignment of a probability, by symmetry that probability (of my pulling out a red marble) is no different than that of my pulling out a green marble which in turn is equal to that of my pulling out a blue marble. Since these are the only three possibilities, the three equal probabilities sum to one, giving me a probability of 1/3 for pulling out a red marble.


Principle of indifference for the win!!
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

User avatar
skeptical scientist
closed-minded spiritualist
Posts: 6142
Joined: Tue Nov 28, 2006 6:09 am UTC
Location: San Francisco

Re: Losing Your Marbles

Postby skeptical scientist » Fri Nov 14, 2008 5:58 am UTC

Principle of indifference? Who cares?

Seriously, do we really need a name for something so obvious?
I'm looking forward to the day when the SNES emulator on my computer works by emulating the elementary particles in an actual, physical box with Nintendo stamped on the side.

"With math, all things are possible." —Rebecca Watson

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5967
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Losing Your Marbles

Postby jestingrabbit » Fri Nov 14, 2008 6:24 am UTC

skeptical scientist wrote:Principle of indifference? Who cares?

Seriously, do we really need a name for something so obvious?


Given that its fundamental to the reasoning that we use in probability theory, I think its useful to have a name for it, though I actually only first heard about it when I was reading Jaynes.
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

++$_
Mo' Money
Posts: 2370
Joined: Thu Nov 01, 2007 4:06 am UTC

Re: Losing Your Marbles

Postby ++$_ » Fri Nov 14, 2008 7:35 pm UTC

Tass wrote:It doesn't matter if the machine is rigged, since you do not know if or how it is.
This is now turning into a philosophical discussion of probability, but it matters very much whether the machine could be rigged.

For example, let's say you are playing a dice-rolling game. You write your guess on a piece of paper and put it in a sealed envelope. The game operator rolls a six-sided die, and if the face bearing your number comes up, you win. Otherwise, you lose. Now, if the die is known to be fair, then clearly you have a 1/6 chance of winning the game. In the real world, however, you don't know that the die is fair, but your chance of winning (at least on your first play, if you're not allowed to observe the die) would still be 1/6 if you choose your guess randomly.

Now let's change the situation. You run into a similar game, except you don't get to make a guess. The game operator rolls a die. If he rolls a 6, you win. Otherwise, you lose. If the die is fair, you still have a 1/6 chance of winning. But in the real world, you don't know whether the die is fair. What is your probability of winning?

I'd argue: If the die is fair, your probability of winning is 1/6. If the die is rigged to always come up 5, your probability of winning is 0. If the die is rigged to come up 6 exactly 1/12 of the time, your probability of winning is 1/12. But you can't combine these all into a single "probability of winning" in any way. It just doesn't make sense to do that, because either the die is rigged in one of these ways, or it is fair. There's no probability involved here.

User avatar
Darth Eru
Posts: 77
Joined: Fri May 25, 2007 10:52 pm UTC

Re: Losing Your Marbles

Postby Darth Eru » Fri Nov 14, 2008 11:57 pm UTC

++$_ wrote:I'd argue: If the die is fair, your probability of winning is 1/6. If the die is rigged to always come up 5, your probability of winning is 0. If the die is rigged to come up 6 exactly 1/12 of the time, your probability of winning is 1/12. But you can't combine these all into a single "probability of winning" in any way. It just doesn't make sense to do that, because either the die is rigged in one of these ways, or it is fair. There's no probability involved here.

As you said, this is getting to be a philosophical discussion. The counterargument to your argument is the whole point of probability is that probabilities are relative to the knowledge you have. In your example, to you the person watching the die roll, the probability that you win is 1/6, because according to what you know, all the outcomes are equally likely. To the person rolling the die, who knows more, the probability may be different precisely because they know more.

One argument against this view is that it seems to lead to determinism, i.e. if you have all possible knowledge then all things that occur have a probability of 1. This can be seen as a counter-argument since quantum physics has proved that even absolute knowledge cannot provide absolute certainty. However, I believe that knowledge still affects probability. It simply doesn't ever allow a probability of 1. Basically, my philosophy is that the more knowledge you have, the more accurate your knowledge of the probabilities are.

So my basic point is that when talking about probability the knowledge given to your viewpoint plays a factor. The fact that every event does have an inherent probability (i.e. the probability of an event if you had all possible knowledge that could affect that probability) does not matter. If we do not have knowledge we cannot use it. When asked to provide a probability, we must simply use all knowledge available to us. The other way of looking at it is ultimately useless, since we would never be able to completely ascertain all pertinent information, and therefore never be able to come to a conclusion.

In a real-life example, when counting cards in 21 you must assume that every card that you haven't seen has an equal probability of being on the top of the deck, even though there are cards you haven't seen that were dealt to other players and are actually already out of the deck. From a statistical viewpoint, that doesn't matter. The 10 of hearts your opponent has hidden still has an equal probability (to you anyway) of being on the top of the deck as the 3 of clubs that actually is on the top of the deck.
Signature:
This is a block of text that can be added to posts you make. There is a 255 character limit

There are 2 types of people in the world: those who know hexadecimal, and those who don't.

Dvorak > QWERTY

User avatar
Tass
Posts: 1909
Joined: Tue Nov 11, 2008 2:21 pm UTC
Location: Niels Bohr Institute, Copenhagen.

Re: Losing Your Marbles

Postby Tass » Sun Nov 16, 2008 2:06 pm UTC

Darth Eru wrote:In a real-life example, when counting cards in 21 you must assume that every card that you haven't seen has an equal probability of being on the top of the deck, even though there are cards you haven't seen that were dealt to other players and are actually already out of the deck. From a statistical viewpoint, that doesn't matter. The 10 of hearts your opponent has hidden still has an equal probability (to you anyway) of being on the top of the deck as the 3 of clubs that actually is on the top of the deck.


Thank you. My point exactly.

In the example with a game that you can lose or win, and which the other player controls, there is a biased incitament for him to cheat. That is different, and impossible to calculate exactly.

Goplat
Posts: 490
Joined: Sun Mar 04, 2007 11:41 pm UTC

Re: Losing Your Marbles

Postby Goplat » Sun Nov 16, 2008 5:34 pm UTC

Tass wrote:It doesn't matter if the machine is rigged, since you do not know if or how it is.
Unless you have a probability distribution over the possible behaviors of the machine, you can't calculate the answer.

Spoiler:
If the machine has 100% probability of being fair, the answer is 1/3.
If the machine has 90% probability of being fair but 10% probability of being rigged so that the green balls all come out first, the answer is 9/30.
And if the machine has any nonzero probability of being affected by the order the balls were put in, then you need a probability distribution on that order too if you want an answer.

tricky77puzzle
Will take "No Tresspassing Signs" for 500
Posts: 488
Joined: Thu Feb 28, 2008 11:02 pm UTC

Re: Losing Your Marbles

Postby tricky77puzzle » Sun Nov 16, 2008 10:57 pm UTC

Goplat wrote:
Tass wrote:It doesn't matter if the machine is rigged, since you do not know if or how it is.
Unless you have a probability distribution over the possible behaviors of the machine, you can't calculate the answer.

Spoiler:
If the machine has 100% probability of being fair, the answer is 1/3.
If the machine has 90% probability of being fair but 10% probability of being rigged so that the green balls all come out first, the answer is 9/30.
And if the machine has any nonzero probability of being affected by the order the balls were put in, then you need a probability distribution on that order too if you want an answer.


Wow, this is the exact thing that ++$_ said in his first post (post 2 in this thread).

jjono
Posts: 71
Joined: Mon Oct 27, 2008 11:35 am UTC

Re: Losing Your Marbles

Postby jjono » Mon Nov 17, 2008 5:09 am UTC

Goplat wrote:
Tass wrote:It doesn't matter if the machine is rigged, since you do not know if or how it is.
Unless you have a probability distribution over the possible behaviors of the machine, you can't calculate the answer.


Depends what you mean by "calculate the answer". Think of it this way - would you bet money that the ball was a particular colour? Imagine if I told you I have a ball in my hand that is either red, green or blue. Then I offered you odds of 5:1 to guess the colour right. Would you take it? What about if I offered you odds of 1:1? If you would take the first bet but not the second, doesn't that say that you think the probability is between 1/6 and 1/2 that the ball is your chosen colour? But wait...the ball is one colour and one colour only, right?

I doesn't matter that I know what colour the marble is, or that the marble is definitely either red, blue or green. To the best of your knowledge, each outcome has equal probability.

As a side note, almost all probabilities are like this. Do you think that the outcome of something as large as a die roll is much affected by quantum indeterminism? I really doubt it. More likely, you just don't have enough information to predict it. Which puts you in a very similar situation to picking a marble from my hand.

Goplat
Posts: 490
Joined: Sun Mar 04, 2007 11:41 pm UTC

Re: Losing Your Marbles

Postby Goplat » Mon Nov 17, 2008 6:09 am UTC

I doesn't matter that I know what colour the marble is, or that the marble is definitely either red, blue or green. To the best of your knowledge, each outcome has equal probability.
No; I don't know how you chose it, so given my knowledge the three probabilities are not defined. Where would I get those probabilities from?

++$_
Mo' Money
Posts: 2370
Joined: Thu Nov 01, 2007 4:06 am UTC

Re: Losing Your Marbles

Postby ++$_ » Mon Nov 17, 2008 10:12 am UTC

jjono wrote:Depends what you mean by "calculate the answer". Think of it this way - would you bet money that the ball was a particular colour? Imagine if I told you I have a ball in my hand that is either red, green or blue. Then I offered you odds of 5:1 to guess the colour right. Would you take it? What about if I offered you odds of 1:1? If you would take the first bet but not the second, doesn't that say that you think the probability is between 1/6 and 1/2 that the ball is your chosen colour? But wait...the ball is one colour and one colour only, right?

I doesn't matter that I know what colour the marble is, or that the marble is definitely either red, blue or green. To the best of your knowledge, each outcome has equal probability.
Totally different scenario, because I get to guess what the color of the marble is. Therefore, I can win with a legitimate 1/3 probability by randomizing my guess.

In an analogous scenario, I would be forced to guess whether or not the marble is red, so that I couldn't randomize my guess. Also, to make the outcomes indifferent to you, the money that I would get paid if I won would come from Bill Gates. So what odds should I accept? You would argue that the odds should be exactly 2:1. However, if you were actually in this situation, I think you would start to reason based on the personality of the marble concealer. If he's offering 100:1 odds, is that because he wants to help you out, or because he wants to ensure that you take the bet? And so on. I agree that 2:1 is a good benchmark, because it's what the odds would be if the game were fair, but it's not necessarily the actual odds.

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5967
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Losing Your Marbles

Postby jestingrabbit » Mon Nov 17, 2008 5:21 pm UTC

I'm just going to point out that we are now arguing Bayesian and frequentist probability. A lot of people have had this argument, and are having this argument. Ultimately, its a philosophical argument imo, so it probably doesn't have a correct answer, for some values of correct.

I'm actually more persuaded by the Bayesian position myself (the probability is 1/3) than the frequentist position (the probability is inherently unknowable), but I can see that both positions have some merit.
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

GreedyAlgorithm
Posts: 286
Joined: Tue Aug 22, 2006 10:35 pm UTC
Contact:

Re: Losing Your Marbles

Postby GreedyAlgorithm » Mon Nov 17, 2008 7:26 pm UTC

jestingrabbit wrote:I'm actually more persuaded by the Bayesian position myself (the probability is 1/3) than the frequentist position (the probability is inherently unknowable), but I can see that both positions have some merit.

Awesome. I see the clear utility of using the Bayesian definition of probability and haven't ever seen a coherent explanation of the alleged frequentist definition. Can you explain it to me? I am currently under the impression that all "frequentist" definitions are merely special cases or more limited forms of the Bayesian definition, useful only for simplifying approximations. A Bayesian who understands the frequentist arguments sounds like just the person to turn me into a Bayesian who understands the frequentist arguments.
GENERATION 1-i: The first time you see this, copy it into your sig on any forum. Square it, and then add i to the generation.

User avatar
skeptical scientist
closed-minded spiritualist
Posts: 6142
Joined: Tue Nov 28, 2006 6:09 am UTC
Location: San Francisco

Re: Losing Your Marbles

Postby skeptical scientist » Mon Nov 17, 2008 8:00 pm UTC

jestingrabbit wrote:I'm just going to point out that we are now arguing Bayesian and frequentist probability. A lot of people have had this argument, and are having this argument. Ultimately, its a philosophical argument imo, so it probably doesn't have a correct answer, for some values of correct.

Yo jestingrabbit, is there someone who might be able to split off this discussion into its own thread so that we can continue it? Like a moderator or something? :)

I too don't really understand the frequentist position, and would be curious what you have to say about it.
I'm looking forward to the day when the SNES emulator on my computer works by emulating the elementary particles in an actual, physical box with Nintendo stamped on the side.

"With math, all things are possible." —Rebecca Watson

Goplat
Posts: 490
Joined: Sun Mar 04, 2007 11:41 pm UTC

Re: Losing Your Marbles

Postby Goplat » Mon Nov 17, 2008 8:48 pm UTC

There is no justification for assigning a uniform distribution on anything when the problem doesn't say so. Sometimes it'll give you something ridiculous: "Goplat's pet is either a cat or a rhinoceros. What is the probability that it's a rhino?" Uniform distribution says 1/2.

Sometimes it'll even give you different answers for equivalent phrasings of the same question! "I am holding either a shoe or a sock; what is P(sock)?" 1/2, right? "I am holding either a left shoe, a right shoe, or a sock; what is P(sock)?" Now the "Bayesian" answer is 1/3, despite the fact that the semantics have not changed.

Really, it's the "Bayesians" who have some 'splaining to do. I'm not the one who's pulling probabilities out of nowhere..

GreedyAlgorithm
Posts: 286
Joined: Tue Aug 22, 2006 10:35 pm UTC
Contact:

Re: Losing Your Marbles

Postby GreedyAlgorithm » Mon Nov 17, 2008 9:48 pm UTC

Goplat wrote:There is no justification for assigning a uniform distribution on anything when the problem doesn't say so. Sometimes it'll give you something ridiculous: "Goplat's pet is either a cat or a rhinoceros. What is the probability that it's a rhino?" Uniform distribution says 1/2.

Sometimes it'll even give you different answers for equivalent phrasings of the same question! "I am holding either a shoe or a sock; what is P(sock)?" 1/2, right? "I am holding either a left shoe, a right shoe, or a sock; what is P(sock)?" Now the "Bayesian" answer is 1/3, despite the fact that the semantics have not changed.

Really, it's the "Bayesians" who have some 'splaining to do. I'm not the one who's pulling probabilities out of nowhere..

Goplat, this is exactly why I'd like jestingrabbit to explain... because he apparently understands what it is "Bayesians" are actually claiming, and so won't get distracted making silly claims about why they're misguided. Here it is for you:

Probability is a measure of how confident someone(something,whatever) should be that a state is the case, given the knowledge that entity has. A known fair coin, then, does not have some intrinsic probability 0.5 of landing heads. Instead 0.5 is the probability you, knowing the coin is fair, should assign to the coin landing heads if you also know it's not a trick flip, etc. Barring infinite set corner cases, there's no such thing as an "undefined" probability. You can't say "oh but the unfair coin will actually land heads about 70% of the time, you just don't know it yet, and so if you don't know if it's fair or not the probability is undefined" because in the Bayesian definition of probability, probability is not in the coin, it's in the mind of the person assigning the probability. It's a measure of the knowledge of the assigner, not a measure of some ineffable quality of the trick coin. It just so happens that if you have perfect knowledge of something, you will assign the probabilities to it that match up with what its long-term behavior (or whatever other definition of probability you like) is.

And so priors, like assigning a uniform prior to discrete events you know absolutely nothing about, is simply the measure saying "I know nothing about these events". In practice uniform is sometimes a good approximation of an actual state of knowledge, but more often you know many things, through experience, through past experiments showing which types of things are more likely to be true than others, through whatever... so usually your prior is not uniform.

Let's take your example. "I am holding either the thing I've arbitrarily labeled A or the thing I've arbitrarily labeled B; what is P(B)?" Yes, I should say 1/2. Now if you give me more information and say A is a sock, and B is a shoe, then I have to take into account your proclivities, whether you're trying to trick me, etc. Similarly if you say "A, B, or C. P(C)?" then yes, the answer is 1/3. But again if you give me more information and say "left shoe, right shoe, or sock?" then I have to analyze what it means that you asked me, etc.

To be explicit: Suppose you ask "shoe or sock?". Call that X. Now suppose you ask "right shoe, left shoe, or sock?". Call that Y. We'll call it R that you're holding a right shoe, L that you're holding a left shoe, and S that you're holding a sock. Your beef with Bayesians (if I understand right) is that you think they say:

P(R or L) != P(R)+P(L) in general, which is obviously false since R and L are disjoint.

But that is not it at all. Instead, they say:

P(R or L | Y) != P(R | X) + P(L | X) in general.

Notice how what they're actually saying is obviously true. The Bayesians have now 'splained. Speak up if you're still confused about what they mean.
GENERATION 1-i: The first time you see this, copy it into your sig on any forum. Square it, and then add i to the generation.

User avatar
Lord Aurora
Posts: 566
Joined: Mon Mar 03, 2008 6:14 am UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Lord Aurora » Mon Nov 17, 2008 11:00 pm UTC

GreedyAlgorithm wrote:
P(R or L | Y) == P(YA RLY!)



MANY LULZ!

Also, this:

"Goplat's pet is either a cat or a rhinoceros. What is the probability that it's a rhino?"

Not the right question. Similarly, for example:

"I loaded one bullet in a six chamber gun and shot three times at my friend's head. What is the probability that he is dead?"

The question for things that have ALREADY happened (my friend is either dead or he isn't, whichever one is ACTUALLY TRUE, you either have a cat or a rhinoceros) is "How certain can we be that X is true?" "We can be 50% certain that Goplat owns a cat." Probability is only used for future outcomes.

Anyway, this bugs me. Carry on.
Decker wrote:Children! Children! There's no need to fight. You're ALL stupid.

tricky77puzzle
Will take "No Tresspassing Signs" for 500
Posts: 488
Joined: Thu Feb 28, 2008 11:02 pm UTC

wtf? ...

Postby tricky77puzzle » Mon Nov 17, 2008 11:01 pm UTC

...hold on! How did my Logic Puzzles thread about marbles get moved all the way over here without anyone telling me about it?

...besides that, I didn't know that my puzzle would generate that much discussion. I'm of the Bayesian opinion myself.

GreedyAlgorithm
Posts: 286
Joined: Tue Aug 22, 2006 10:35 pm UTC
Contact:

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby GreedyAlgorithm » Mon Nov 17, 2008 11:14 pm UTC

Lord Aurora wrote:The question for things that have ALREADY happened (my friend is either dead or he isn't, whichever one is ACTUALLY TRUE, you either have a cat or a rhinoceros) is "How certain can we be that X is true?" "We can be 50% certain that Goplat owns a cat."

And the "Bayesians" use this as the definition of probability.

Lord Aurora wrote:Probability is only used for future outcomes.

Probability is used to describe your state of knowledge about outcomes. If you don't know what happened in the past, probability describes that. If you don't know what will happen in the future, probability describes that. Why would you constrain it to only talk about what you know about the future? It gains you nothing to add such a constraint. So far as I know... maybe once I learn WTH "frequentists" are actually talking about I will see why they make a distinction.
GENERATION 1-i: The first time you see this, copy it into your sig on any forum. Square it, and then add i to the generation.

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5967
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby jestingrabbit » Tue Nov 18, 2008 12:18 am UTC

The way I see it two schools can be distinguished by their description of how probabilities relate to reality (duh!).

On the one hand you have Bayesians. Bayesians think that an assignment of probabilities is a statement about beliefs. So, for instance, consider the tossing of a coin. When a Bayesian says that the probability of getting a head is 1/2, all they are saying is that they half believe the proposition "the coin will come up heads". Jaynes, whose book you can download a fair bit of on the interweb, goes further and describes the assignment of probabilities as the (essentially) unique extension of the standard True and False valued Boolean logic that satisfies certain very reasonable conditions imo.

On the other hand, you have the frequentists. They seem to take the position that there is some actual probability inherent to a situation, and that successive trials will eventually give you a good approximation of these actual probabilities.

That's it basically.

tricky77puzzle wrote:...hold on! How did my Logic Puzzles thread about marbles get moved all the way over here without anyone telling me about it?

...besides that, I didn't know that my puzzle would generate that much discussion. I'm of the Bayesian opinion myself.


I split the pertinent part off. Someone suggested it, it made sense, so I did it. Its not getting undone unless you've got a good reason.

fin
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

GoldenYears
Posts: 12
Joined: Thu Sep 04, 2008 6:12 pm UTC

Re: Losing Your Marbles

Postby GoldenYears » Tue Nov 18, 2008 9:00 pm UTC

Darth Eru wrote:One argument against this view is that it seems to lead to determinism, i.e. if you have all possible knowledge then all things that occur have a probability of 1. This can be seen as a counter-argument since quantum physics has proved that even absolute knowledge cannot provide absolute certainty. However, I believe that knowledge still affects probability. It simply doesn't ever allow a probability of 1. Basically, my philosophy is that the more knowledge you have, the more accurate your knowledge of the probabilities are.


Quantum Physics hasn't disproved squat. It has just found a case where the obvious statement "If you alter an object by your means of observing it you cannot truly know the object by those means" applies. So stop saying quantum physics refutes determinism, it does not. What it does is put determinism in the category of epistemological empty content, simply because determinism can't verified.

tricky77puzzle
Will take "No Tresspassing Signs" for 500
Posts: 488
Joined: Thu Feb 28, 2008 11:02 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby tricky77puzzle » Tue Nov 18, 2008 10:55 pm UTC

jestingrabbit wrote:I split the pertinent part off. Someone suggested it, it made sense, so I did it. Its not getting undone unless you've got a good reason.


It's not that I want it undone, it's because it just suddenly happened like that. I'm perfectly cool with your decision.

Cauchy
Posts: 602
Joined: Wed Mar 28, 2007 1:43 pm UTC

Re: Losing Your Marbles

Postby Cauchy » Wed Nov 19, 2008 8:55 am UTC

GoldenYears wrote:Quantum Physics hasn't disproved squat. It has just found a case where the obvious statement "If you alter an object by your means of observing it you cannot truly know the object by those means" applies. So stop saying quantum physics refutes determinism, it does not. What it does is put determinism in the category of epistemological empty content, simply because determinism can't verified.


Quantum Physics has done a pretty good job of refuting determinism, albeit not a perfect job.

http://en.wikipedia.org/wiki/Hidden_variable_theories
(∫|p|2)(∫|q|2) ≥ (∫|pq|)2
Thanks, skeptical scientist, for knowing symbols and giving them to me.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11129
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Yakk » Wed Nov 19, 2008 6:23 pm UTC

jestingrabbit wrote:Bayesians think that an assignment of probabilities is a statement about beliefs.

Ah, that makes sense. The problem is that some Bayesian mathematics is predicated on bad beliefs -- like "if we know nothing, we should assume uniform probabilities"?

Suppose we have a basket of balls. We know there are red, green and blue balls in it. We do not know the distribution of these balls. At the start, we have no knowledge about the distribution of balls.

A Bayesian would presume there is an equal probability of a fair and randomly selected ball being any one of the 3 colors.

If someone who possibly has more knowledge than us makes an offer predicated on that knowledge that isn't symmetrical, this would break the symmetry, and the assumption that the probabilities are 1/3 each would be thrown out. The result would probably be too complex to calculate?

Similarly, if someone does a selection through an unknown process (we don't know if it is fair) after any symmetry breaking, we would also not know the resulting distributions, as they would be too difficult to calculate?

But now we have a situation where we do not know what the probabilities are, because we cannot calculate them: so saying that probabilities are descriptions of our knowledge, does it make sense to talk about 'we do not know what our knowledge is'? If probabilities are descriptions of our knowledge or expectations, they seem to be a projection of our knowledge or expectations into one dimension -- ie, far from complete. And that one dimension they are a projection into, isn't that the frequentest ... dimension? (To our current knowledge, if you found a way to repeat this trial X times with the same initial knowledge, we'd expect event K to happen P(K)*X times on average) Or, a projection consistent with that, in the cases where frequentests say that probability is a valid thing to be talking about?

Meh. Too much nonsense makes break go ow.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

GreedyAlgorithm
Posts: 286
Joined: Tue Aug 22, 2006 10:35 pm UTC
Contact:

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby GreedyAlgorithm » Wed Nov 19, 2008 7:35 pm UTC

Yakk wrote:
jestingrabbit wrote:Bayesians think that an assignment of probabilities is a statement about beliefs.

Ah, that makes sense. The problem is that some Bayesian mathematics is predicated on bad beliefs -- like "if we know nothing, we should assume uniform probabilities"?

Suppose we have a basket of balls. We know there are red, green and blue balls in it. We do not know the distribution of these balls. At the start, we have no knowledge about the distribution of balls.

A Bayesian would presume there is an equal probability of a fair and randomly selected ball being any one of the 3 colors.

If someone who possibly has more knowledge than us makes an offer predicated on that knowledge that isn't symmetrical, this would break the symmetry, and the assumption that the probabilities are 1/3 each would be thrown out. The result would probably be too complex to calculate?

Precisely. Knowledge of the fact that someone is offering a bet is extremely relevant knowledge. Calculating the correct probability becomes very hard, because now it requires an intimate understanding of human psychology. This result is quite intuitive to me - you are no longer trying to make a statement about your knowledge of the balls, you are trying to make a statement about your knowledge of the balls and a Sicilian.

Yakk wrote:Similarly, if someone does a selection through an unknown process (we don't know if it is fair) after any symmetry breaking, we would also not know the resulting distributions, as they would be too difficult to calculate?

Technically yes. Humans might have predilections for picking Red or the most prevalent something. But practically, unless you actually know these predilections, your probability of which is picked remains almost unchanged. There is no symmetry breaking here. To be precise, here are the two situations in which I claim the correct probability should change by such a small amount it is not worth thinking about:

A) "There is an urn with an unknown number of red, blue, and green balls in it. An unknown process chooses a ball from the urn. This is all you know. What is the probability the chosen ball is red, blue, or green?"

B) "There is an urn with an unknown number of red, blue, and green balls in it. A ball is chosen from the urn uniformly at random. This is all you know. What is the probability the chosen ball is red, blue, or green?"

Remember that this is a statement about your knowledge, not the unknown process'. Someone who knows how the ball is chosen would certain assign different probabilities than you, and that's perfectly alright, since they have more knowledge.

Yakk wrote:But now we have a situation where we do not know what the probabilities are, because we cannot calculate them: so saying that probabilities are descriptions of our knowledge, does it make sense to talk about 'we do not know what our knowledge is'? If probabilities are descriptions of our knowledge or expectations, they seem to be a projection of our knowledge or expectations into one dimension -- ie, far from complete. And that one dimension they are a projection into, isn't that the frequentest ... dimension? (To our current knowledge, if you found a way to repeat this trial X times with the same initial knowledge, we'd expect event K to happen P(K)*X times on average) Or, a projection consistent with that, in the cases where frequentests say that probability is a valid thing to be talking about?

Here's the thing: we could calculate the probabilities given lots of time and computing power. It's just math, but it's lots of math. Probability theory gives us an exact answer assuming we can get down to a symmetrical position anywhere, like at the atomic level, or even below. Since it's too hard, though, we need approximations. That's where all these statistical techniques come in - they are all approximating an underlying mathematically precise probability calculation. This precise calculation is guaranteed to tell us the most information we can get out of our current knowledge, and the approximation should come close to that if it's a good one. And so when you say that probability is the number P(K) where if you found a way to repeat the experiment X times with the same initial knowledge, you'd expect K to happen P(K)*X times, to my mind that's merely a standard result of probability theory. Why it should put a constraint on what things can be said to have a probability based on whether you can conceive of a way to "repeat the experiment X times with the same initial knowledge" is beyond me.

Stay tuned for a discussion of quantifying lack of knowledge about the results of too-difficult calculations. :D
GENERATION 1-i: The first time you see this, copy it into your sig on any forum. Square it, and then add i to the generation.

User avatar
skeptical scientist
closed-minded spiritualist
Posts: 6142
Joined: Tue Nov 28, 2006 6:09 am UTC
Location: San Francisco

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby skeptical scientist » Wed Nov 19, 2008 9:48 pm UTC

It seems to me that a sensible philosophy is a hybrid of the two schools: when we say that a die rolled has a 1/6 chance of being a six, that's a statement of our knowledge in the sense that about 1/6 of the time when we have the knowledge "a die is to be rolled", we find that the roll comes up a six. So when we say the chance of X given Y is p, we mean that if we find ourselves in situation Y many times, we expect the proportion of times X occurs to be roughly p. Or is this just the standard Bayesian school?
I'm looking forward to the day when the SNES emulator on my computer works by emulating the elementary particles in an actual, physical box with Nintendo stamped on the side.

"With math, all things are possible." —Rebecca Watson

Tac-Tics
Posts: 536
Joined: Thu Sep 13, 2007 7:58 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Tac-Tics » Wed Nov 19, 2008 10:12 pm UTC

Philosophy of statistics is amusing.

Q: If you have a bag of colored marbles, and you pick them out one at a time, noting their color before replacing them, can you figure out what the distribution of the marbles is?

A: Nope.

Being so absolute about things isn't always practical of course, and statistics allows insurance companies and skilled poker players to make shittons of money. But just because you're right most of the time doesn't mean you're always right. Nor does it give you any clue at all as to when you'll be right.

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5967
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby jestingrabbit » Thu Nov 20, 2008 12:41 am UTC

Yakk wrote:If someone who possibly has more knowledge than us makes an offer predicated on that knowledge that isn't symmetrical, this would break the symmetry, and the assumption that the probabilities are 1/3 each would be thrown out. The result would probably be too complex to calculate?

Similarly, if someone does a selection through an unknown process (we don't know if it is fair) after any symmetry breaking, we would also not know the resulting distributions, as they would be too difficult to calculate?


If the 'symmetry breaking' can be meaningfully interpretted, ie someone we know wants to make money offers a wager, and they know more about the process, then we don't know the disribution but we can still reasonably change our priors by making reasonable assumptions about how much money they want to make, how much risk they would want to accept etc.

Absent the gambling fairy turning up and creating arbitrary wagers, most information can be used in a meaningful way to change our priors, or else ignored.

GreedyAlgorithm wrote:Knowledge of the fact that someone is offering a bet is extremely relevant knowledge. Calculating the correct probability becomes very hard, because now it requires an intimate understanding of human psychology. This result is quite intuitive to me - you are no longer trying to make a statement about your knowledge of the balls, you are trying to make a statement about your knowledge of the balls and a Sicilian.


Its not as hard to calculate as all that.

GreedyAlgorithm wrote:A) "There is an urn with an unknown number of red, blue, and green balls in it. An unknown process chooses a ball from the urn. This is all you know. What is the probability the chosen ball is red, blue, or green?"

B) "There is an urn with an unknown number of red, blue, and green balls in it. A ball is chosen from the urn uniformly at random. This is all you know. What is the probability the chosen ball is red, blue, or green?"


I would argue that our priors should be identical for these two situations, not just 'so slightly different that we don't care'.

Yakk wrote:To our current knowledge, if you found a way to repeat this trial X times with the same initial knowledge, we'd expect event K to happen P(K)*X times on average.


Every time a trial is performed, a Bayesian updates their probabilities. So, doing it X times with the same knowledge makes no sense to me. After a degree of experimentation, we could start to predict that our believed distribution will change relatively little.

GreedyAlgorithm wrote:Here's the thing: we could calculate the probabilities given lots of time and computing power. It's just math, but it's lots of math. Probability theory gives us an exact answer assuming we can get down to a symmetrical position anywhere, like at the atomic level, or even below. Since it's too hard, though, we need approximations. That's where all these statistical techniques come in - they are all approximating an underlying mathematically precise probability calculation. This precise calculation is guaranteed to tell us the most information we can get out of our current knowledge, and the approximation should come close to that if it's a good one. And so when you say that probability is the number P(K) where if you found a way to repeat the experiment X times with the same initial knowledge, you'd expect K to happen P(K)*X times, to my mind that's merely a standard result of probability theory. Why it should put a constraint on what things can be said to have a probability based on whether you can conceive of a way to "repeat the experiment X times with the same initial knowledge" is beyond me.


This is frequentist balderdash.

GreedyAlgorithm wrote:Stay tuned for a discussion of quantifying lack of knowledge about the results of too-difficult calculations. :D


Although this seems to be an attractive avenue, Jaynes and I agree that its not as useful as it seems.

skeptical scientist wrote:It seems to me that a sensible philosophy is a hybrid of the two schools: when we say that a die rolled has a 1/6 chance of being a six, that's a statement of our knowledge in the sense that about 1/6 of the time when we have the knowledge "a die is to be rolled", we find that the roll comes up a six. So when we say the chance of X given Y is p, we mean that if we find ourselves in situation Y many times, we expect the proportion of times X occurs to be roughly p. Or is this just the standard Bayesian school?


I think that this is actually pretty frequentist.

A lot of the things that we think of as being random are actually not really random at all. And in fact, the more we interrogate the idea of randomness, the more we find it to be completely hollow.

Here is a little thing about coin tossing.

http://en.wikipedia.org/wiki/Coin_toss# ... n_flipping

I've also seen some people rolling dice in a way that is far from random. This is ultimately why Backgammon is played with a dice cup instead of rolling them out of the hand. If you give a person more control over the dice they can use that to control the outcome. The dice outcome isn't random, but our lack of control over it creates ignorance which is, to someone who isn't paying attention, indistinguishable from what we often think of as randomness.

I remember reading a perfectly ghastly book that is freely available online by that guy who writes Dilbert, and after I was done with it I wanted that day of my life back with a burning passion. It positted that randomness was a fundamental substance of the universe and it made me want to scream because its just bullshit. It was called 'God's Debris' and I wanted to write a book called 'God's Defecation' as a witty rejoinder...

Tac-Tics wrote:Q: If you have a bag of colored marbles, and you pick them out one at a time, noting their color before replacing them, can you figure out what the distribution of the marbles is?

A: Nope.


You're right, but you can end up knowing as much as you'd like to about the marble selection process.
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

GreedyAlgorithm
Posts: 286
Joined: Tue Aug 22, 2006 10:35 pm UTC
Contact:

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby GreedyAlgorithm » Thu Nov 20, 2008 3:08 am UTC

jestingrabbit wrote:Although this seems to be an attractive avenue, Jaynes and I agree that its not as useful as it seems.

Spoiler:
Yeah but he doesn't get to that 'til like chapter 18 or something!
GENERATION 1-i: The first time you see this, copy it into your sig on any forum. Square it, and then add i to the generation.

User avatar
Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Velifer » Thu Nov 20, 2008 4:37 pm UTC

Tac-Tics wrote:Q: If you have a bag of colored marbles, and you pick them out one at a time, noting their color before replacing them, can you figure out what the distribution of the marbles is?

A: Nope.


Calculate the maximum likelihood function of the distribution of colors based on repeated sample sets. Return the most likely distribution from the set of possible distributions. Y'know, just get all Bayesian on it.
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx

User avatar
Yakk
Poster with most posts but no title.
Posts: 11129
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Yakk » Thu Nov 20, 2008 7:03 pm UTC

Instead of returning the maximium likelyhood distribution...

Why not return the set of distributions that, if they where what was in the bag, what you pulled out wouldn't be all that unlikely?

In this case, each time you pull a marble out, you tend to chip away at the space of reasonably likely marble distributions. And you can quantify an "unexpected" event as one that substantially extended your possible-marble space in a particular sub region.

Because if you did what the previous poster said, after drawing 37 white marbles, 52 black marbles, 10 grey marbles, and 1 orange marble, you'd say "37% white, 52% black, 10% grey, and 1% orange". :)
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Velifer » Thu Nov 20, 2008 7:11 pm UTC

Yakk wrote:Because if you did what the previous poster said, after drawing 37 white marbles, 52 black marbles, 10 grey marbles, and 1 orange marble, you'd say "37% white, 52% black, 10% grey, and 1% orange". :)


Yeah, I see my error. The probability that the above distribution is correct (or any discrete distribution given) is 0. The probability that a range of likely distributions is correct could be calculated, given some bound on "likely distributions," and would provide a degree of belief in the possible contents of the bag.
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx

Goplat
Posts: 490
Joined: Sun Mar 04, 2007 11:41 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Goplat » Fri Nov 21, 2008 3:23 am UTC

Velifer wrote:
Tac-Tics wrote:Q: If you have a bag of colored marbles, and you pick them out one at a time, noting their color before replacing them, can you figure out what the distribution of the marbles is?

A: Nope.


Calculate the maximum likelihood function of the distribution of colors based on repeated sample sets. Return the most likely distribution from the set of possible distributions. Y'know, just get all Bayesian on it.

The set of possible marble distributions here is the set of functions mapping colors to non-negative integers, [Color → N]. According to Bayesianism one should start out by assigning probability uniformly over this space, but it's infinitely large so that's not possible.

User avatar
Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Velifer » Fri Nov 21, 2008 7:00 pm UTC

A frequentist would say there's an absolute distribution, but no way to know it. Agnostics.
A Bayesian would pick a marble, expecting red but getting white, and assure you that all marbles are pink. Deists.
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx

Tac-Tics
Posts: 536
Joined: Thu Sep 13, 2007 7:58 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Tac-Tics » Fri Nov 21, 2008 8:49 pm UTC

Goplat wrote:The set of possible marble distributions here is the set of functions mapping colors to non-negative integers, [Color → N]. According to Bayesianism one should start out by assigning probability uniformly over this space, but it's infinitely large so that's not possible.


Theorem: All bags of colored marbles have a uniform distribution.

Proof: If you know the distribution already, surf the Internet until you forget what it was. Then, by the above lemma by Bayes, assume the probability is uniform. QED!

User avatar
Charlie!
Posts: 2035
Joined: Sat Jan 12, 2008 8:20 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Charlie! » Sat Nov 22, 2008 9:05 am UTC

Tac-Tics wrote:
Goplat wrote:The set of possible marble distributions here is the set of functions mapping colors to non-negative integers, [Color → N]. According to Bayesianism one should start out by assigning probability uniformly over this space, but it's infinitely large so that's not possible.


Theorem: All bags of colored marbles have a uniform distribution.

Proof: If you know the distribution already, surf the Internet until you forget what it was. Then, by the above lemma by Bayes, assume the probability is uniform. QED!


Perhaps it's something more like "the sum of all possible bags of colored marbles has a uniform distribution," meaning that if your bag, at the moment, might be any possible bag o' marbles with equal probability, acting as if it's uniform is the best bet until you get more information.
Some people tell me I laugh too much. To them I say, "ha ha ha!"

tricky77puzzle
Will take "No Tresspassing Signs" for 500
Posts: 488
Joined: Thu Feb 28, 2008 11:02 pm UTC

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby tricky77puzzle » Sat Nov 22, 2008 3:16 pm UTC

Tac-Tics wrote:Then, by the above lemma by Bayes, assume the probability is uniform. QED!


Your QED usage fails.

User avatar
Wildcard
Candlestick!
Posts: 253
Joined: Wed Jul 02, 2008 12:42 am UTC
Location: Outside of the box

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Wildcard » Thu Dec 11, 2008 10:32 pm UTC

I don't think anyone has made this particular point here at all, so here goes.

My own view is that neither of the two views of probability has independant truth. Rather, I don't personally operate on the idea of Absolute Truth, I operate off of what is workable. It doesn't make sense to operate any other way, at least not to me (solipsists may disagree).

That's moving slightly off topic, or at least it would be if I clarified it fully. The point is, which viewpoint, if any, of probability can actually be used to attain some result? Let's take gambling, because knowledge of probability has an observable relevance to the field of gambling. I like the example given above:

jjono wrote:Depends what you mean by "calculate the answer". Think of it this way - would you bet money that the ball was a particular colour? Imagine if I told you I have a ball in my hand that is either red, green or blue. Then I offered you odds of 5:1 to guess the colour right. Would you take it? What about if I offered you odds of 1:1? If you would take the first bet but not the second, doesn't that say that you think the probability is between 1/6 and 1/2 that the ball is your chosen colour? But wait...the ball is one colour and one colour only, right?

I doesn't matter that I know what colour the marble is, or that the marble is definitely either red, blue or green. To the best of your knowledge, each outcome has equal probability.


I like this example. Look at it this way: either I am telling the truth or I am not. If I am not, and the marble turns out to be purple, you would be justified in renegeing on your bet, as I'm sure anyone would agree. So let's assume I'm telling the truth. If the marble turns out to be either red, green or blue, some money will change hands if you accept the bet. If I give (to exaggerate and make the point easier to see) 1000:1 odds, then you are putting down one dollar to 1000 of mine on the probability that you can guess the color of the marble. I'd call that a good bet, no matter how you view probability.

IF, however, I give you the same odds, tell you that the marble is either red, green or blue, and offer you 1000:1 odds only if you want to bet that the marble is RED--you'd be a fool to take the bet, unless you knew from other means that I was very seriously trying to get rid of a lot of money. This is where motive comes into play, and when motive comes into play, "all bets are off" as the probabilities go to hell.

These are both fairly obvious in terms of the choices involved. The thing that may be missed is, probabilities in any real life circumstance do NOT necessarily sum up to one, nor are they absolute. We normally only look at a specialized cross section of probabilities. I like what an earlier poster said (that I am too lazy to look for and quote) in another forum: if you choose a point randomly on a line with even distribution, the probability of you choosing a point OTHER than the exact center is 1. "Or, to put it another way, events with probability 0 happen all the time."

Goplat wrote:There is no justification for assigning a uniform distribution on anything when the problem doesn't say so. Sometimes it'll give you something ridiculous: "Goplat's pet is either a cat or a rhinoceros. What is the probability that it's a rhino?" Uniform distribution says 1/2.

Sometimes it'll even give you different answers for equivalent phrasings of the same question! "I am holding either a shoe or a sock; what is P(sock)?" 1/2, right? "I am holding either a left shoe, a right shoe, or a sock; what is P(sock)?" Now the "Bayesian" answer is 1/3, despite the fact that the semantics have not changed.

Really, it's the "Bayesians" who have some 'splaining to do. I'm not the one who's pulling probabilities out of nowhere..


It will indeed give different answers for different phrasings of the same actual scenario. So what if I say, "It's a sock?" You can then know that it is a sock, with probability 1, according to your logic. What you're forgetting is that all of these situations only deal in the specialized cross section of situations wherein I am telling the truth. :P

The simplistic view is, P(heads) + P(tails) = 1/2 + 1/2 = 1. In actual practice that works, so we use it. But if you're going for Absolute Truth, you have to ask, what coin? Where? You wind up factoring in P(the coin spontaneously combusting) + P(the coin never landing due to gravity failure) + P(the coin landing on its edge) + P(the coin falling through a hitherto unnoticed crack in the floor and winding up unobservable in its final position) + ... + P(heads) + P(tails) = 1. This is all quibble, though, because P(heads) + P(tails) = 1/2 + 1/2 = 1 is a good, workable truth, as far as it goes.

What are the odds of a quarter landing heads when I flip it? Most people would say 1/2. What about when I add the fact that we are on an orbiting space station? :wink:
There's no such thing as a funny sig.

User avatar
Xanthir
My HERO!!!
Posts: 5426
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Xanthir » Fri Dec 12, 2008 1:33 am UTC

Wildcard wrote:IF, however, I give you the same odds, tell you that the marble is either red, green or blue, and offer you 1000:1 odds only if you want to bet that the marble is RED--you'd be a fool to take the bet, unless you knew from other means that I was very seriously trying to get rid of a lot of money. This is where motive comes into play, and when motive comes into play, "all bets are off" as the probabilities go to hell.

This is incorrect, though. Probability is just as relevant in this situation as any other, whether you're a frequentist or a Bayesian. It's just that you have to pull in ancillary knowledge of human psychology and such to properly formulate the probabilities.

The problem is difficult, not impossible.
Last edited by Xanthir on Fri Dec 12, 2008 7:17 pm UTC, edited 1 time in total.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

Re: Bayesian vs Frequentist : the Final Showdown!!

Postby Velifer » Fri Dec 12, 2008 6:22 pm UTC

Great, now we're gonna have ourselves a subjectivist vs. objectivist smackdown...
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx


Return to “Mathematics”

Who is online

Users browsing this forum: No registered users and 11 guests