Fair multiple choice test

For the discussion of math. Duh.

Moderators: gmalivuk, Moderators General, Prelates

Soapy
Posts: 4
Joined: Fri May 30, 2008 7:57 pm UTC

Fair multiple choice test

Postby Soapy » Wed Mar 11, 2009 9:37 pm UTC

Heya guys,

I was wondering if you could help me with a problem... I'm not that good at maths and certainly not about probability stuff like this.
First of all, where I live tests are graded a mark out of 10, where 0.0 (or sometimes 1.0, depending on the type of education) is the worst and 10.0 is the best mark you can get.

So, my problem
Let's say you're a teacher and you want your pupils to do a multiple choice test. You create a number of questions, all of them with 4 possible answers, of which only one option is right. The kids do the test and then you have to grade it. Here's the point:

The easiest thing to do would be to give a 0.0 mark if 0% of the questions are right, a 2.5 if 25% of the questions are right and a 10.0 if 100% of the questions are correct.
But this way you ignore the fact that someone answering all the questions by randomly guessing would get 25% right on average. To correct for this I can think of another way. Let's say 25% right gives a 0.0 mark, anything less than 25% also gives a 0.0 mark and anything between 25% and 100% can be calculated by a simple linear function where 100% once again gives a 10.0 mark.

But now there's another problem. What if someone does not guess any questions but simply is very bad at the test, does his very best, and only manages to get a score of 20%? He got some stuff right so he deserves a mark higher than a 0.0.
I think there must be a way to find a completely fair way of grading and also minimizing the 'random guessing'-bias.
So I know this now:
0% correct --> mark = 0.0
25% correct --> 0.0 < mark < 2.5
100% correct mark = 10.0
So I would like to know how to find out (1) what mark should be given if someone gets 25% right and (2) should the grades be given in a linear function like in the examples or should the change in the mark somehow be dependent on the number of correct answers?
Also (for da bonus points) I would like to know if any important stuff changes if the minimum mark is 1.0 instead of 0.0.

Thanks in advance for your time :)
Last edited by Soapy on Wed Mar 11, 2009 9:47 pm UTC, edited 1 time in total.

User avatar
BotoBoto
Posts: 191
Joined: Mon Mar 09, 2009 9:31 pm UTC
Contact:

Re: Fair multiple choice test

Postby BotoBoto » Wed Mar 11, 2009 9:40 pm UTC

Why are you assuming that if somebody guesses they get 25%, that's only statistics. You should also check how long they needed for the test. Just my 0.02

User avatar
Xanthir
My HERO!!!
Posts: 5400
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Fair multiple choice test

Postby Xanthir » Wed Mar 11, 2009 9:49 pm UTC

The standard way to handle this is to make a correct answer be worth 1 point and a wrong answer worth -1/3 point (scale as you wish). This way the random-answer chooser on average gets a zero. It is then a linear increase to full credit as they get more correct.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
LaserGuy
Posts: 4581
Joined: Thu Jan 15, 2009 5:33 pm UTC

Re: Fair multiple choice test

Postby LaserGuy » Wed Mar 11, 2009 10:35 pm UTC

If you have to do a multiple choice test, I think Xanthir's solution is the most fair. If a student isn't sure about the question, they're better off not answering rather than guessing.

On the other hand, the BEST solution would be not to use a multiple choice test. Multiple choice is basically useless for testing any sort of higher thought, for that matter, they're barely passable at testing even the lowest levels of understanding--they're just convenient for teachers who are too lazy to put together a real exam.

[edit] I guess this being the math forum, I should pretend my answer is mathematical... The value of guessing is inversely proportional to the number of answers. Thus, we can decrease the value of guessing and increase the rewards of not guessing by increasing the number of possible answers. Take the limit as # of answers goes to infinity, and you converge to an ideal solution.

User avatar
Cosmologicon
Posts: 1806
Joined: Sat Nov 25, 2006 9:47 am UTC
Location: Cambridge MA USA
Contact:

Re: Fair multiple choice test

Postby Cosmologicon » Wed Mar 11, 2009 10:56 pm UTC

LaserGuy wrote:On the other hand, the BEST solution would be not to use a multiple choice test. Multiple choice is basically useless for testing any sort of higher thought, for that matter, they're barely passable at testing even the lowest levels of understanding--they're just convenient for teachers who are too lazy to put together a real exam.

I strongly disagree with this. You can test higher-level thinking perfectly well with multiple-choice tests. Have you ever taken an AMC test?

One disadvantage of multiple-choice tests, however, is that a single question by itself tells you nothing. The finer you want to distinguish between your students, the more questions you need. If someone scores a 4/10, it's possible that they really knew those 4 questions well, and it's also quite possible that they just guessed. You can't tell. On the other hand, if they score 400/1000, you can be pretty sure they're not just guessing. My advice is if you're only giving a 10-question test, use Xanthir's advice and accept that you can't tell between poor-but-better-than-nothing performance and luck.

mike-l
Posts: 2758
Joined: Tue Sep 04, 2007 2:16 am UTC

Re: Fair multiple choice test

Postby mike-l » Thu Mar 12, 2009 12:49 am UTC

LaserGuy wrote:If you have to do a multiple choice test, I think Xanthir's solution is the most fair. If a student isn't sure about the question, they're no better off guessing than not answering.


Fix'd.

Of course, this assumes guesses are actually random, which they seldom are. Good students can do much better than random guessing on multiple choice tests without knowing a single answer, by ruling out incorrect ones and guessing between the remainder. Poor students can do much poorer than random guessing by knowing just enough to convince themselves of the plausible alternatives.
addams wrote:This forum has some very well educated people typing away in loops with Sourmilk. He is a lucky Sourmilk.

Buttons
Posts: 858
Joined: Wed May 02, 2007 3:27 pm UTC
Location: Somerville

Re: Fair multiple choice test

Postby Buttons » Thu Mar 12, 2009 12:59 am UTC

mike-l wrote:Of course, this assumes guesses are actually random, which they seldom are. Good students can do much better than random guessing on multiple choice tests without knowing a single answer, by ruling out incorrect ones and guessing between the remainder. Poor students can do much poorer than random guessing by knowing just enough to convince themselves of the plausible alternatives.

Which is why the "1/(k-1) point penalty for incorrect guesses among k choices" system is good, because it specifically rewards having enough knowledge to do better than random chance.

That said, I agree with LaserGuy about the relative uselessness of multiple choice tests. This is not to say that they cannot be used to measure ability. Certainly they can. But as a teaching aid, by which students can learn where they made mistakes and get some useful feedback, they are definitely inferior to open-response tests.

This is especially true on tests like the SATs where there are "almost-correct" answers put in place. This way, when a student seriously misapplies a rule and finds the answer he got as one of the choices, he thinks, "Yes! My understanding is correct." The positive reinforcement from this moment negates any learning that happens when (if) he finally learns that this answer was actually wrong. A former professor of mine compared it to training a dog as follows: when he pees on the carpet, tell him "Good boy!" and pat him on the head. Then, three weeks later, say, "Remember that time you peed on the carpet? Bad dog!"

User avatar
Sygnon
Posts: 32
Joined: Tue Jan 23, 2007 1:47 am UTC
Contact:

Re: Fair multiple choice test

Postby Sygnon » Thu Mar 12, 2009 1:48 am UTC

multiple choice is bad.

Yesila
Posts: 221
Joined: Sun Dec 16, 2007 11:38 am UTC

Re: Fair multiple choice test

Postby Yesila » Thu Mar 12, 2009 3:25 am UTC

Buttons wrote:...there are "almost-correct" answers put in place.


You can always design your answers with the "almost correct" idea in mind, then give partial credit for those answers. If you use a type of problem that you have tested with before you should have a good idea as to what the "common" mistakes are AND how much partial credit you would give for those mistakes.

For example if you supply one answer which is wrong because of a sign error on a step that may be worth 70% of the points. Another answer where the error was more grievous but to get to that spot required some correct steps you could give 30% or the points. A correct answer for 100% and a total screwball answer for 0% rounds things out.

On this scale a random guesser averages 50% of the points, so fails. A person that can only eliminate the worst of answers and guess between the remainder averages about 66% of the points, so fails but is still in a position to recover on later tests. A person that can eliminate the two poorest answers averages an 85% and of course the person that knows it all has a 100%.

Maybe the 70 and the 30 should be scaled down a bit and maybe the 0 should change to a negative number but I like the idea of partial credit for almost right answers.

User avatar
Cosmologicon
Posts: 1806
Joined: Sat Nov 25, 2006 9:47 am UTC
Location: Cambridge MA USA
Contact:

Re: Fair multiple choice test

Postby Cosmologicon » Thu Mar 12, 2009 3:41 am UTC

Buttons wrote:as a teaching aid, by which students can learn where they made mistakes and get some useful feedback, they are definitely inferior to open-response tests.

Well, a single open-response question definitely gives more useful feedback than a single multiple-choice question. But of course that's not how it works out: one open-response question might replace 10 multiple-choice questions. If you want to, you can definitely form useful feedback based on a pattern of responses over a large number of multiple-choice questions. I think the only reason it's not often done is that tests (especially finals and the SAT) are intended primarily to evaluate, not to be teaching aids.

User avatar
HenryS
Posts: 199
Joined: Mon Nov 27, 2006 9:16 am UTC
Location: Melbourne
Contact:

Re: Fair multiple choice test

Postby HenryS » Sun Mar 15, 2009 6:34 am UTC

I taught multivariable calculus last semester using an online multiple choice system for homework. It did have some advantages over open response (and of course the disadvantages people have already mentioned). For one, students get immediate feedback on if their answer is correct, and have the opportunity to rework the problem and try again (for fewer marks, but still worth it). For two the questions automatically have many many variants programmed in, so students always get different numbers from each other, which means less cheating possibilities, although the versions are similar enough that people can talk about them with each other.

Captain_Thunder
Posts: 83
Joined: Tue Mar 27, 2007 9:18 pm UTC

Re: Fair multiple choice test

Postby Captain_Thunder » Sun Mar 15, 2009 11:50 am UTC

Soapy wrote:I think there must be a way to find a completely fair way of grading and also minimizing the 'random guessing'-bias.


CollegeBoard (graders of the AP and SAT tests) already has a system like this: you start the test with 0 points, you get 1 point for every correct answer, you lose a quarter of a point for every incorrect answer, and your score doesn't change if you leave a question blank. This seems to me like it would do a pretty good job of eliminating the guessing-bonus; if you guess five questions and get one right, you break even.


Return to “Mathematics”

Who is online

Users browsing this forum: No registered users and 10 guests