Fair multiple choice test
Moderators: gmalivuk, Moderators General, Prelates
Fair multiple choice test
Heya guys,
I was wondering if you could help me with a problem... I'm not that good at maths and certainly not about probability stuff like this.
First of all, where I live tests are graded a mark out of 10, where 0.0 (or sometimes 1.0, depending on the type of education) is the worst and 10.0 is the best mark you can get.
So, my problem
Let's say you're a teacher and you want your pupils to do a multiple choice test. You create a number of questions, all of them with 4 possible answers, of which only one option is right. The kids do the test and then you have to grade it. Here's the point:
The easiest thing to do would be to give a 0.0 mark if 0% of the questions are right, a 2.5 if 25% of the questions are right and a 10.0 if 100% of the questions are correct.
But this way you ignore the fact that someone answering all the questions by randomly guessing would get 25% right on average. To correct for this I can think of another way. Let's say 25% right gives a 0.0 mark, anything less than 25% also gives a 0.0 mark and anything between 25% and 100% can be calculated by a simple linear function where 100% once again gives a 10.0 mark.
But now there's another problem. What if someone does not guess any questions but simply is very bad at the test, does his very best, and only manages to get a score of 20%? He got some stuff right so he deserves a mark higher than a 0.0.
I think there must be a way to find a completely fair way of grading and also minimizing the 'random guessing'bias.
So I know this now:
0% correct > mark = 0.0
25% correct > 0.0 < mark < 2.5
100% correct mark = 10.0
So I would like to know how to find out (1) what mark should be given if someone gets 25% right and (2) should the grades be given in a linear function like in the examples or should the change in the mark somehow be dependent on the number of correct answers?
Also (for da bonus points) I would like to know if any important stuff changes if the minimum mark is 1.0 instead of 0.0.
Thanks in advance for your time
I was wondering if you could help me with a problem... I'm not that good at maths and certainly not about probability stuff like this.
First of all, where I live tests are graded a mark out of 10, where 0.0 (or sometimes 1.0, depending on the type of education) is the worst and 10.0 is the best mark you can get.
So, my problem
Let's say you're a teacher and you want your pupils to do a multiple choice test. You create a number of questions, all of them with 4 possible answers, of which only one option is right. The kids do the test and then you have to grade it. Here's the point:
The easiest thing to do would be to give a 0.0 mark if 0% of the questions are right, a 2.5 if 25% of the questions are right and a 10.0 if 100% of the questions are correct.
But this way you ignore the fact that someone answering all the questions by randomly guessing would get 25% right on average. To correct for this I can think of another way. Let's say 25% right gives a 0.0 mark, anything less than 25% also gives a 0.0 mark and anything between 25% and 100% can be calculated by a simple linear function where 100% once again gives a 10.0 mark.
But now there's another problem. What if someone does not guess any questions but simply is very bad at the test, does his very best, and only manages to get a score of 20%? He got some stuff right so he deserves a mark higher than a 0.0.
I think there must be a way to find a completely fair way of grading and also minimizing the 'random guessing'bias.
So I know this now:
0% correct > mark = 0.0
25% correct > 0.0 < mark < 2.5
100% correct mark = 10.0
So I would like to know how to find out (1) what mark should be given if someone gets 25% right and (2) should the grades be given in a linear function like in the examples or should the change in the mark somehow be dependent on the number of correct answers?
Also (for da bonus points) I would like to know if any important stuff changes if the minimum mark is 1.0 instead of 0.0.
Thanks in advance for your time
Last edited by Soapy on Wed Mar 11, 2009 9:47 pm UTC, edited 1 time in total.
Re: Fair multiple choice test
Why are you assuming that if somebody guesses they get 25%, that's only statistics. You should also check how long they needed for the test. Just my 0.02
 Xanthir
 My HERO!!!
 Posts: 5400
 Joined: Tue Feb 20, 2007 12:49 am UTC
 Location: The Googleplex
 Contact:
Re: Fair multiple choice test
The standard way to handle this is to make a correct answer be worth 1 point and a wrong answer worth 1/3 point (scale as you wish). This way the randomanswer chooser on average gets a zero. It is then a linear increase to full credit as they get more correct.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))
Re: Fair multiple choice test
If you have to do a multiple choice test, I think Xanthir's solution is the most fair. If a student isn't sure about the question, they're better off not answering rather than guessing.
On the other hand, the BEST solution would be not to use a multiple choice test. Multiple choice is basically useless for testing any sort of higher thought, for that matter, they're barely passable at testing even the lowest levels of understandingthey're just convenient for teachers who are too lazy to put together a real exam.
[edit] I guess this being the math forum, I should pretend my answer is mathematical... The value of guessing is inversely proportional to the number of answers. Thus, we can decrease the value of guessing and increase the rewards of not guessing by increasing the number of possible answers. Take the limit as # of answers goes to infinity, and you converge to an ideal solution.
On the other hand, the BEST solution would be not to use a multiple choice test. Multiple choice is basically useless for testing any sort of higher thought, for that matter, they're barely passable at testing even the lowest levels of understandingthey're just convenient for teachers who are too lazy to put together a real exam.
[edit] I guess this being the math forum, I should pretend my answer is mathematical... The value of guessing is inversely proportional to the number of answers. Thus, we can decrease the value of guessing and increase the rewards of not guessing by increasing the number of possible answers. Take the limit as # of answers goes to infinity, and you converge to an ideal solution.
 Cosmologicon
 Posts: 1806
 Joined: Sat Nov 25, 2006 9:47 am UTC
 Location: Cambridge MA USA
 Contact:
Re: Fair multiple choice test
LaserGuy wrote:On the other hand, the BEST solution would be not to use a multiple choice test. Multiple choice is basically useless for testing any sort of higher thought, for that matter, they're barely passable at testing even the lowest levels of understandingthey're just convenient for teachers who are too lazy to put together a real exam.
I strongly disagree with this. You can test higherlevel thinking perfectly well with multiplechoice tests. Have you ever taken an AMC test?
One disadvantage of multiplechoice tests, however, is that a single question by itself tells you nothing. The finer you want to distinguish between your students, the more questions you need. If someone scores a 4/10, it's possible that they really knew those 4 questions well, and it's also quite possible that they just guessed. You can't tell. On the other hand, if they score 400/1000, you can be pretty sure they're not just guessing. My advice is if you're only giving a 10question test, use Xanthir's advice and accept that you can't tell between poorbutbetterthannothing performance and luck.
Re: Fair multiple choice test
LaserGuy wrote:If you have to do a multiple choice test, I think Xanthir's solution is the most fair. If a student isn't sure about the question, they're no better off guessing than not answering.
Fix'd.
Of course, this assumes guesses are actually random, which they seldom are. Good students can do much better than random guessing on multiple choice tests without knowing a single answer, by ruling out incorrect ones and guessing between the remainder. Poor students can do much poorer than random guessing by knowing just enough to convince themselves of the plausible alternatives.
addams wrote:This forum has some very well educated people typing away in loops with Sourmilk. He is a lucky Sourmilk.
Re: Fair multiple choice test
mikel wrote:Of course, this assumes guesses are actually random, which they seldom are. Good students can do much better than random guessing on multiple choice tests without knowing a single answer, by ruling out incorrect ones and guessing between the remainder. Poor students can do much poorer than random guessing by knowing just enough to convince themselves of the plausible alternatives.
Which is why the "1/(k1) point penalty for incorrect guesses among k choices" system is good, because it specifically rewards having enough knowledge to do better than random chance.
That said, I agree with LaserGuy about the relative uselessness of multiple choice tests. This is not to say that they cannot be used to measure ability. Certainly they can. But as a teaching aid, by which students can learn where they made mistakes and get some useful feedback, they are definitely inferior to openresponse tests.
This is especially true on tests like the SATs where there are "almostcorrect" answers put in place. This way, when a student seriously misapplies a rule and finds the answer he got as one of the choices, he thinks, "Yes! My understanding is correct." The positive reinforcement from this moment negates any learning that happens when (if) he finally learns that this answer was actually wrong. A former professor of mine compared it to training a dog as follows: when he pees on the carpet, tell him "Good boy!" and pat him on the head. Then, three weeks later, say, "Remember that time you peed on the carpet? Bad dog!"
Re: Fair multiple choice test
multiple choice is bad.
Re: Fair multiple choice test
Buttons wrote:...there are "almostcorrect" answers put in place.
You can always design your answers with the "almost correct" idea in mind, then give partial credit for those answers. If you use a type of problem that you have tested with before you should have a good idea as to what the "common" mistakes are AND how much partial credit you would give for those mistakes.
For example if you supply one answer which is wrong because of a sign error on a step that may be worth 70% of the points. Another answer where the error was more grievous but to get to that spot required some correct steps you could give 30% or the points. A correct answer for 100% and a total screwball answer for 0% rounds things out.
On this scale a random guesser averages 50% of the points, so fails. A person that can only eliminate the worst of answers and guess between the remainder averages about 66% of the points, so fails but is still in a position to recover on later tests. A person that can eliminate the two poorest answers averages an 85% and of course the person that knows it all has a 100%.
Maybe the 70 and the 30 should be scaled down a bit and maybe the 0 should change to a negative number but I like the idea of partial credit for almost right answers.
 Cosmologicon
 Posts: 1806
 Joined: Sat Nov 25, 2006 9:47 am UTC
 Location: Cambridge MA USA
 Contact:
Re: Fair multiple choice test
Buttons wrote:as a teaching aid, by which students can learn where they made mistakes and get some useful feedback, they are definitely inferior to openresponse tests.
Well, a single openresponse question definitely gives more useful feedback than a single multiplechoice question. But of course that's not how it works out: one openresponse question might replace 10 multiplechoice questions. If you want to, you can definitely form useful feedback based on a pattern of responses over a large number of multiplechoice questions. I think the only reason it's not often done is that tests (especially finals and the SAT) are intended primarily to evaluate, not to be teaching aids.
Re: Fair multiple choice test
I taught multivariable calculus last semester using an online multiple choice system for homework. It did have some advantages over open response (and of course the disadvantages people have already mentioned). For one, students get immediate feedback on if their answer is correct, and have the opportunity to rework the problem and try again (for fewer marks, but still worth it). For two the questions automatically have many many variants programmed in, so students always get different numbers from each other, which means less cheating possibilities, although the versions are similar enough that people can talk about them with each other.

 Posts: 83
 Joined: Tue Mar 27, 2007 9:18 pm UTC
Re: Fair multiple choice test
Soapy wrote:I think there must be a way to find a completely fair way of grading and also minimizing the 'random guessing'bias.
CollegeBoard (graders of the AP and SAT tests) already has a system like this: you start the test with 0 points, you get 1 point for every correct answer, you lose a quarter of a point for every incorrect answer, and your score doesn't change if you leave a question blank. This seems to me like it would do a pretty good job of eliminating the guessingbonus; if you guess five questions and get one right, you break even.
Who is online
Users browsing this forum: No registered users and 10 guests