## Are "logical thinking" tests reliable/meaningful?

For the discussion of math. Duh.

Moderators: gmalivuk, Moderators General, Prelates

el matematico
Posts: 114
Joined: Mon Oct 19, 2009 12:20 pm UTC

### Are "logical thinking" tests reliable/meaningful?

Today I supervised a general knowledge test. One of the subjects was akin to critical thinking/logical thinking and featured those typical logic sequence puzzles with shapes and orderings and stuff like that, where you have to guess the next element. Some questions kept me thinking, which I'll copy here.

1) Find the sum of the digits of the next number of the sequence: 1, 5, 19, 49, 101, ?
Choices are: 7, 8, 10, 12

2) Find the next element of the sequence: 2/4; 3/12; 5/30; 7/56; ?
Choices are: 9/81, 9/90, 10/100, 11/110.

3) Find the nex letter in the sequence: B, C, E, H, J, K, ?
Choices are: T, N, Ñ, V.

Ñ comes after N and before O in the Spanish alphabet, if anyone was wondering.

I have solved the first and second ones, but I still think the reasoning to get the answers is bad. I have no idea of the third, similar methods to the other ones give nothing useful.
Spoiler:
In the first one, make a sequence a_n with the difference of consecutive numbers, 4, 14, 30, 52. Repeat and you get a sequence b_n: 10, 16, 22. The numbers are in a linear progression (with just 3 elements, yay), so b_4 is 28, a_5 is 80 and the sixth number in the original sequence is 181. The sum of the digits is 10.

For the second one, if you take those elements as fractions and simplify, you get 1/2, 1/4, 1/6, 1/8. If you consider the numerators, they're 2, 3, 5, 7, apparently a list of prime numbers. So the next one has 11 as numerator and simplifies to 1/10, the answer is 11/110.

The third one, no idea. The position of the letters are 2, 3, 5, 8, 10, 11. Doing the thing with the substraction gives 1, 2, 3, 2, 1. The pattern is complete, and continuing with 0 would give K as an answer, not valid. Assuming the patten is just 5 long and repeat, the next number is 2, which would give an M, another invalid choice. There's also no pattern that I could recognize with the names of the letters, and that would be cheap anyway.

The problem is that with the given information (which is just the elements in order) I can't justify the validity of my methods. The first one only works if the numbers are produced by evaluating consecutive numbers in a polynomial of low enough degree (three in this case). The second method works because I could recognize a particular sequence. If I had attended that class I'd probably be able to reproduce a method for the third problem. But then, what is the test measuring? Is it really logical thinking, or just the ability to solve pattern recognition tests with a specific mindset? None of those methods could do anything if the sequence was exponential, or obtained via modular arithmetics or a high degree polynomial. The visual sequences are the same, you have to assume they're formed by following very simple patterns (rotations, cyclic permutations of elements...) and in general, you have to use very little data to extrapolate.

So, in short: Are these tests useful for measuring logical skills? Do they make good predictions for intelligente/grades? Is it even possible to get a single, logical answer in the third problem?
This is my blog (in Spanish). It's not perfect, but it's mine. http://falaci.blogspot.com/

doogly
Dr. The Juggernaut of Touching Himself
Posts: 5538
Joined: Mon Oct 23, 2006 2:31 am UTC
Location: Lexington, MA
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

It's bullshit. People don't want to test things with actual mathematical meaning, because then it would be math, and this is "just pure logic," so it's just nothing.

Or, it's "guess what the test writer was thinking." It certainly wouldn't work without being multiple choice.

They are very frustratingly inane.
LE4dGOLEM: What's a Doug?
Noc: A larval Doogly. They grow the tail and stinger upon reaching adulthood.

Keep waggling your butt brows Brothers.
Or; Is that your eye butthairs?

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

### Re: Are "logical thinking" tests reliable/meaningful?

I share your concerns. There's certainly a logic in the sequence, but the way to figure it out is not logical thinking but random guessing until something seems plausible. As far as I know, the only thing these tests can measure is whether you've been subjected to the whims of similar "logic" tests before, but I cannot cite any relevant studies on that matter.

Luckily, I managed to dig up something relevant I once read: https://blog.codinghorror.com/separatin ... ing-goats/
One interesting point is that their test is not scored by correctness, but by consistency. If subjects developed a mental model that they could consistently apply, they still belong to the successful group, even if the model they were using was actually the wrong one.
A bunch of sequences, where every single sequence seems to follow its own little rules and quirks, is unsuitable for testing whether the subject can apply rules consistently.

About your sequences: For all real-world-sequence guessing, there's always oeis. For the first problem, it suggests only one sequence:, a(n) = n^3 - n^2 + 1, which would give the same next number you got.

You know the next letter in the sequence J F M A M J J A? It's an S of course. S for September, if you were wondering. Do you know any common list of words in spanish that start with your letters? I certainly hope it's not a number sequence in disguise, because the oeis results for that are scary.

Soupspoon
You have done something you shouldn't. Or are about to.
Posts: 4060
Joined: Thu Jan 28, 2016 7:00 pm UTC
Location: 53-1

### Re: Are "logical thinking" tests reliable/meaningful?

"Which is the odd one out: House, Kennel, Tent, Cave?"

That was an (alleged) MENSA test question. Even better than the fact that every one of those has a perfectly good reason to be the odd one out, it was hidden behind a layer of puzzle in which each word was given in a jumbled form. Which you may think is simple enough, in this instance, but shouldn't be there in a test for intelligence, unless you don't care about biasing in favour of education (specifically in the English language) thus probably all in favour of perpetuating some debunked attitudes towards racial intelligence.

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

It's very frustrating.

el matematico
Posts: 114
Joined: Mon Oct 19, 2009 12:20 pm UTC

### Re: Are "logical thinking" tests reliable/meaningful?

For a long time I've thought these things are meaningless, but I asked because 1) they're still used a lot in intelligence tests and admission exams from what I can tell and 2) I think pattern recognition could maybe tie to logical thinking. What bothers me the most is the idea that logical thinking involves deducting rules and extrapolating from very small samples with little extra info. Like for example, the first problem would have been fine if it was stated that the numbers come from evaluating consecutive integers in a polynomial, since then my method is justified.

You know the next letter in the sequence J F M A M J J A? It's an S of course. S for September, if you were wondering. Do you know any common list of words in spanish that start with your letters? I certainly hope it's not a number sequence in disguise, because the oeis results for that are scary.

The K kills this kind of method, almost no Spanish word starts with K, with the most common usage being kilo, but it's not a list of SI prefixes.

Anyway, thanks for the replies.
This is my blog (in Spanish). It's not perfect, but it's mine. http://falaci.blogspot.com/

doogly
Dr. The Juggernaut of Touching Himself
Posts: 5538
Joined: Mon Oct 23, 2006 2:31 am UTC
Location: Lexington, MA
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

Yeah the tests get written this way for some really well intentioned reasons - they want to just do logical power or "aptitude" without measuring specific knowledge, because that would privilege people with certain educational backgrounds and you're not getting at the raw stuff of their reasoning power. But you're still not getting the raw stuff of reasoning power, now you're just getting "did you study for this specific test?" rather than "did you study trigonometry?" or "have you read Tolstoy?" It is very much a negative adaptation.
LE4dGOLEM: What's a Doug?
Noc: A larval Doogly. They grow the tail and stinger upon reaching adulthood.

Keep waggling your butt brows Brothers.
Or; Is that your eye butthairs?

nash1429
Posts: 190
Joined: Tue Nov 17, 2009 3:06 am UTC
Location: Flatland
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

Even if you could create defensible logical thinking questions, I don't think presenting them in exam form will ever be reliable or meaningful for anything except measuring how well people perform on logical thinking exams.
Last edited by nash1429 on Tue Sep 13, 2016 12:42 am UTC, edited 1 time in total.

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

Well, they are used in IQ tests because "how well people perform on logical thinking exams" is correlated with most other measures of intelligence. So the idea is to measure intelligence by proxy by measuring performance in many areas established to be positively correlated to try to get at the underlying psychometric g. There is nothing wrong with that, but it's obviously imperfect and some tests will be better than others.

nash1429
Posts: 190
Joined: Tue Nov 17, 2009 3:06 am UTC
Location: Flatland
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

Eebster the Great wrote:Well, they are used in IQ tests because "how well people perform on logical thinking exams" is correlated with most other measures of intelligence. So the idea is to measure intelligence by proxy by measuring performance in many areas established to be positively correlated to try to get at the underlying psychometric g. There is nothing wrong with that, but it's obviously imperfect and some tests will be better than others.

Aren't those "other measures of intelligence" just variations on logical thinking exams? All the research I've ever seen only shows a correlation between intelligence scores and career outcomes within one standard deviation of the mean (everyone 1+ SD below is pretty much equally SOL; for people 1+ SD above other factors seem to be at play).

I suppose you could argue that the tests are still valuable for the 70% in the middle, but that still leaves the question of whether scores are meaningful at an individual level or only a population level.

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

nash1429 wrote:
Eebster the Great wrote:Well, they are used in IQ tests because "how well people perform on logical thinking exams" is correlated with most other measures of intelligence. So the idea is to measure intelligence by proxy by measuring performance in many areas established to be positively correlated to try to get at the underlying psychometric g. There is nothing wrong with that, but it's obviously imperfect and some tests will be better than others.

Aren't those "other measures of intelligence" just variations on logical thinking exams? All the research I've ever seen only shows a correlation between intelligence scores and career outcomes within one standard deviation of the mean (everyone 1+ SD below is pretty much equally SOL; for people 1+ SD above other factors seem to be at play).

No. Typically they involve vocab, reading comprehension, analogies, coding, mental arithmetic, vector pattern recognition, and lots of other components. In well-researched IQ tests, every subscore has a strong positive correlation with every other one and strong test-retest validity, among other statistical measures, in multiple systematic reviews, and even every single test used for each subscore has at least some research backing. They aren't just pulling random ideas for testing out of thin air.

I mean, it's not perfect. It's not even really that good. But it's far from bullshit. Do you really want to say the whole idea of intelligence is flawed? Or that you don't know people you would consider smarter than others? Or that it was impossible to determine such a thing? (If it was impossible to determine, on what basis your impressions?)

I suppose you could argue that the tests are still valuable for the 70% in the middle, but that still leaves the question of whether scores are meaningful at an individual level or only a population level.

Even in the best case scenario, scores can only be meaningful for that portion of the population over which they are calibrated. For instance, scores between 40 and 160 (4 SDs on either side of the median) have at least some statistical verification for the WAIS, but beyond that they aren't very useful. In reality, even that exaggerates their accuracy. But they are not completely useless either. The fact that smart people score highly while stupid people score lowly should not surprise you.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

### Re: Are "logical thinking" tests reliable/meaningful?

Eebster the Great wrote:Do you really want to say the whole idea of intelligence is flawed?

Yes it is. At least the idea that intelligence is a one-dimensional variable and every human having some ranking on this scale that puts them clearly above/below others with higher/lower score. It does have it's uses as you are describing but there are massive caveats. It's a flawed concept with only limited use and it's only saving grace is that we simply don't have anything better to answer the questions it's supposed to answer.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

ahammel
My Little Cabbage
Posts: 2135
Joined: Mon Jan 30, 2012 12:46 am UTC
Location: Vancouver BC
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

Eebster the Great wrote:[Do you really want to say the whole idea of intelligence is flawed? Or that you don't know people you would consider smarter than others? Or that it was impossible to determine such a thing?

I'd be perfectly happy to say that the idea of "intelligence" refers to a loose grab-bag of skills, some of which are unrelated and some of which are intercorrelated in complicated ways. You can certainly measure somebody's acumen at any of those skills, but it would be a bit silly to try to totally-order humanity by some measure of the lot of them lumped together, and equation silly to pick the one that's related to the largest number of the others and claim that that's what intelligence really is.

I'm no expert, though. Maybe the people who study such things would contradict me.
He/Him/His/Alex
God damn these electric sex pants!

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

The whole point of intelligence is that the subscores are strongly positively correlated. In other words, someone who is good at mental math is probably also good at coding, memory, reading, spotting patterns, and so on. There are some people who are good at some of these and bad at others, but they are in the extreme minority. This positive correlation is not random; there must be some factors in common among the tests, and those are what we consider to be "general intelligence." It is not just a random equation applied to arbitrarily chosen questions.

ahammel
My Little Cabbage
Posts: 2135
Joined: Mon Jan 30, 2012 12:46 am UTC
Location: Vancouver BC
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

Eebster the Great wrote:The whole point of intelligence is that the subscores are strongly positively correlated. In other words, someone who is good at mental math is probably also good at coding, memory, reading, spotting patterns, and so on. There are some people who are good at some of these and bad at others, but they are in the extreme minority.
That may be true, but I have to admit that I have a hard time believing it. If I was asked, I'd have to say that "good at some of those things and bad at others" describes both me and just about everybody I know well enough to make any judgement.
He/Him/His/Alex
God damn these electric sex pants!

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

I am not a psychometrician, but the data seem pretty clear and uncontroversial among researchers in the field. So far, no theory claiming multiple orthogonal dimensions of intelligence has statistical support. In fact, when subjected to testing, all have found the supposedly independent factors to be strongly correlated with each other and with g. Wikipedia has a decent article describing this.

Note that the g-factor obviously doesn't account for all of the variability in subtest scores between people, only around 50 percent or so. But that is still a lot. It's why you don't see a lot of idiots solving differential equations or a lot of geniuses struggling to multiply fractions. You see some, but not many.

curtis95112
Posts: 639
Joined: Thu Jan 27, 2011 5:23 pm UTC

### Re: Are "logical thinking" tests reliable/meaningful?

Note that one factor accounting for ~50% of variability is ridiculously good outside of the hard sciences. The popular media downplay IQ and g so much that most people have no idea how amazing this is.
Mighty Jalapeno wrote:
Tyndmyr wrote:
Роберт wrote:Sure, but at least they hit the intended target that time.

Well, if you shoot enough people, you're bound to get the right one eventually.

Thats the best description of the USA ever.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

### Re: Are "logical thinking" tests reliable/meaningful?

curtis95112 wrote:Note that one factor accounting for ~50% of variability is ridiculously good outside of the hard sciences.

It's accounting for ~50% of the variability between individuals in cognitive tests. If a factor accounts for only 50% of the variability in test scores on a test that is supposed to measure that factor, you have a very weak test. And than there is still the question what that factor actually measures besides how well someone does on IQ-/cognitive tests. Not saying that it's all bullshit, but the whole concept is clearly flawed, and the research community agrees. It's just really really hard to come up with something that is less flawed and that's why everyone is using it.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

doogly
Dr. The Juggernaut of Touching Himself
Posts: 5538
Joined: Mon Oct 23, 2006 2:31 am UTC
Location: Lexington, MA
Contact:

### Re: Are "logical thinking" tests reliable/meaningful?

curtis95112 wrote:Note that one factor accounting for ~50% of variability is ridiculously good outside of the hard sciences. The popular media downplay IQ and g so much that most people have no idea how amazing this is.

Yeah, and people who work in Education rather than Psychology are very not attuned to this research. They continue to use "Multiple Intelligences" as their model because it seems friendlier or something? Or, more charitably, because "Multiple Pedagogicals Methods" is an effective way to teach, and they take this to mean that their students have Multiple Intelligences. Which is silly.
LE4dGOLEM: What's a Doug?
Noc: A larval Doogly. They grow the tail and stinger upon reaching adulthood.

Keep waggling your butt brows Brothers.
Or; Is that your eye butthairs?

Eebster the Great
Posts: 3485
Joined: Mon Nov 10, 2008 12:58 am UTC
Location: Cleveland, Ohio

### Re: Are "logical thinking" tests reliable/meaningful?

lorb wrote:
curtis95112 wrote:Note that one factor accounting for ~50% of variability is ridiculously good outside of the hard sciences.

It's accounting for ~50% of the variability between individuals in cognitive tests. If a factor accounts for only 50% of the variability in test scores on a test that is supposed to measure that factor, you have a very weak test. And than there is still the question what that factor actually measures besides how well someone does on IQ-/cognitive tests. Not saying that it's all bullshit, but the whole concept is clearly flawed, and the research community agrees. It's just really really hard to come up with something that is less flawed and that's why everyone is using it.

It accounts for 50% of variability between subscores of individuals. So if your, say, working memory index was better than mine, and I knew absolutely nothing else about us, I could predict that around half the difference was due to you having a higher IQ and around half was due to you having better cognitive traits specific to memory tests. However, if you scored better in every subtest, the odds that your IQ was higher increase substantially. It is more likely that your g was higher than that every individual factor cooperated to make every subtest higher.

The WAIS-IV test is actually pretty accurate and fine-grained near the median (100), but its accuracy falls off after you get more than around 2.5 SD (37.5 IQ) away. So if you score 110, you can be reasonably confident you are more intelligent than someone who scored 100, but if you score 170, it's hard to know if you are really any smarter than someone who scored 160, even though there are twenty times as many people who score 160 as 170 and only twice as many people who score 100 as 110. This is due to several factors but most of all the difficulty of calibrating a test for people comprising just a hundredth of a percent of the population.