Wednesday
Jun082011
Do Standardized Tests Measure Intelligence or Socialization?
Wednesday, June 8, 2011 
This is what the baby drinks!What does the baby drink? Can you point?
That is the question that my friend @AureliaCotta's three year old son was asked during an IQ test known as the Wechsler Preschool and Primary Scale of Intelligence. His choices:
- A giant bottle filled with bright red juice that looked like Kool-Aid
- A pacifier
- A big jug of red juice
- A toy truck
How about "none of the above"? My friend's son was breastfed and graduated from the breast to a sippy cup (not a bottle). She is also a stay-at-home mom, and while they go to play groups and other activities, he wouldn't have been exposed daily to numerous babies being given bottles. Even if he was, I doubt they would have been drinking bright red juice...from the bottle or from the jug.
This is not the first time I've heard about problems with tests like these. As a child, I was unable to associate an iron with an ironing board in a similar test. My mom didn't iron in front of the kids out of fear that one of us would pull the hot and heavy iron down off the ironing table (a fear based on experience). My kids wouldn't be able to either, not because they inherited my psychological issues (although that is probably true too), but because I don't buy clothing that needs to be ironed or, if I do, it rarely gets worn because I'm too lazy to iron.
These types of tests, according to Dr. Brian MacDonald, an Ottawa-based child psychologist and one of the hosts of the Family Anatomy podcast, "are designed to be as 'culture-free' as possible." However, he explains, "it's impossible to be totally culture-free. Australian children might not be familiar with all of the same vocabulary that Canadian kids are, even though they both speak English." He further explained that if kids have trouble with a few items that it wouldn't have a significant impact on their overall test scores. He said: "This is why training is required to conduct assessments, and why we don't have computer software that can do assessments for us. Psychologists take into account background information and other factors (such as test anxiety, to name a common issue) when we interpret the results."
I understand the need to have standardized tests for children and I understand that most children probably have sufficient mainstream socialization to do well on the test. However, I still worry about the conclusions that may be drawn from them for two reasons.
First, I worry that the tests are out of date. Breastfeeding is normal. Certainly a lot of babies also get bottles, but in Canada most mothers do choose to breastfeed and many of them also opt to take the one year of maternity and parental leave that is available to them. That means that there is a significant cohort of babies who may not ever drink from a bottle and may not see a lot of other babies drinking from bottles since most daycare age children would be using sippy cups. The tests may include other out of date items too, antiques like alarm clocks and radios and calculators, which are quickly being replaced by iphones.
Second, I worry that the tests reflect mainstream choices and may penalize a child whose family has made different choices. While I don't have a copy of the tests in front of me, I could see a variety of things like gender roles, food choices, parenting styles and lifestyle choices playing into the relevance of the questions and supposedly correct answers to any given child. The less mainstream a family is, the lower the child's score is likely to be.
Where does this leave the children being tested? My guess is that an intelligent and well socialized child from a mainstream family will do very well. An extremely intelligent child from a non-mainstream family with less socialization may have some trouble, but it may be evident that this is based on "other factors". However, I worry that a child who is struggling a bit with some of the questions and who also doesn't recognize certain items because they are out of date or because the family hasn't made mainstream choices, could end up getting a lower score than appropriate.
This highlights the need for the psychologist who is administering the test to get to know the family and to ask questions about their background that may impact the relevance of the test. However, that won't solve everything without some serious revisions and updates to the tests. A bottle with red juice? I don't know anyone whose baby was drinking anything like that (except the Germans who give their babies a reddish tea). It almost seems like it is a trick question to identify the parents who nourished their babies with Kool-Aid.
Image credit: M J M on flickr
Reader Comments (27)
You raise some very interesting points. I think that in doing these tests it's important to allow the children to explain why they chose their answers. My mom likes to tell a story about when I was tested at around age 5 in a new school. The teacher was telling her that I was too young to move ahead and I wasn't understanding certain concepts. The example she gave was from a standardized test. The question asked which person would win a race- then showed a baby and a young child. I had circled the baby. This demonstrated to the teacher that I couldn't tell the difference. My mom asked me why I circled the baby. I explained that we always let the baby at daycare win the races, as it's not nice to beat those younger than you.
Needless to say my Mom's point was proven and I was moved ahead :)
Both my daughters are on the autism spectrum and were given many developmental tests. Most recently my 2 year old was asked to identify some pictures and few of them were outdated like the way the TV had an antenna. We explain to the psychologist performing the eval that just didn't have a few of the items shown like a lamp or a telephone (we only have cell phones), she made a note of it.
Are these types of tests done routinely? I have never heard of this sort of thing done except when delays are suspected.
Smooches:
No, I don't think the tests are done routinely.
In sociology, it is widely known that these tests are not only culturally/racially biased (usually in favour of caucasians), but it is also class biased (for the middle-class). You have basically touched on that regarding the more mainstream a family is, the better they will do. It is very true. Wish i knew off hand the sociological studies that have been done to back this up. It has just been too long since I've read them. Great post.
If parents want kids to do well on the tests then train them in advance. If they do a few at home (you can order most of those tests from the research group that developed them or find similar ones in the library sometimes ... might have to be a university library) then they'll become better test-takers and score higher.
You're definitely right that there are biases in those tests. Sometimes the questions are nonsensical, as Alison saw explained, if you have a slightly different twist or "overthink" compared to the psychologist and test subjects the test was tuned on. Numerical type questions are more reliable, but those tests also test knowledge, which is totally hit or miss depending on the background of the child.
By the way, nice job Alison: Great story of a five year old outsmarting a psychologist. :)
I think that most parents who are having their children tested are looking for an honest assessment. These tests are not usually conducted in order to get into the Harvard of preschools. They are conducted in order to identify whether a child has a developmental delay and, if so, what the nature of that delay is.
Just to reply to Smooches, my son has a speech delay, and we aren't sure if it means that he is just like the rest of us, (quirky,ADHD, speech issues) or if he has something else, after his high risk birth.
Many of the speech therapists we've seen won't focus on treating his speech and endlessly go on about making him behave because they are sure he has autism or brain damage or (insert speculative thing here) and they only like treating kids who behave properly.
I'm hoping this report will say he needs them to treat his speech and his IQ + language are normal, so stop driving his mother crazy. Or something like that. :)
One of my kids had to name objects in pictures for a speech and hearing assessment at the age of 5. There were outdated drawings of phones with dials, but he managed those. It was when he used Thomas the Tank engine inspired British terms for a few items or "fancy" vocab for things like Chrysalis for cocoon that he started to score "low." Luckily, the assessing therapist had enough common sense to "fix' the score. It was such a strange way of testing someone- you had to guess the "correct" term
Oh and about Dr.MacDonald, yes, a skilled clinician will interpret tests taking that into account, but not all clinicians ask parents these things, or allow them to speak during testing, because they are afraid of screwing up the tests.
It's not that big a deal for most kids to miss a few, but like Annie, I worry some kids, who are on a margin, or are non-verbal, like mine, will get a misdiagnosis.
I think there's real reason for concern. Not only are the questions skewed but results are unreliable. If the preschooler is tired, cranky or wants to run around, if the clinician and setting are not optimal, findings are impacted. You can go to three different practitioners and get three different labels or diagnoses, depending on the combination of tests used and the expertise/background of the doc or psychologist.
Three-year-olds don't always behave, especially in an unfamiliar setting with strangers. If the outcome helps him get supports he needs and you're happy with it, great. If not, keep going mom.
What about a consult with a speech pathologist, who can assess for receptive and expressive language? Ultimately, you want to connect with people who can really help your wee guy and you -- whoever they may be.
That sounds extremely frustrating! We've been really lucky that all the therapists we've seen have just evaluated based on their area of expertise. Some referred us for further testing, but they able to start treatment (speech, specifically) even though my daughter also had behavior concerns. Also, all of the specialists we'd seen included our feedback in their assessments. After each eval we were asked if that was typical of the behavior we usually see at home and we were repeatedly told that we know our child best so our feedback was the most valuable. We were present during all their testing and some testing even included a "parent interview" portion. It's unfortunate that not all evaluators are as impartial.
We have seen multiple speech therapists. And they are sort of doing some speech therapy, but it's really mostly behavioural evals, or trying to force him to sit still, it's a mess actually. Which is why we are getting this overall eval done.
This evaluator at least, makes rules about never bringing a kid when he's tired, hungry, sick so it doesn't skew results. And they have been very good about following his lead, etc. I do still worry that not every kid gets this kind of evaluation, or interpretation of results.
This reminds me of the GATE tests, or whatever they were called in my day...I had a similar experience to one of the commenters. I was asked "if you are going to be late for school, what do you do?" I said I'd go home and get my mom to drive me. "Correct" answer as RUN. We are in a spanish-immersion school and they say the tests are not language or culturally biased, but if you look at who is in the after school GATE Club/who scored highest on the test I'd have hard time believing it.
I took a sociology or something class in college, and they used to have worse questions than a bottle with red juice - like who was prettier? (showing a blond haired blue eyed woman with light smooth skin, vs a darker frizzy haired woman with darker more splotchy skin) and one where they asked what you should do if you found a wallet. Going to the police got you the most points, going to your mom the 2nd, keeping it the third (the problem was certain races knew they'd likely get in trouble if they told the police they'd found a wallet).
Our Parents as Teachers parent educator was wise to this one, so she would put the items (and the bottle was tiny and opaque white) on the table, hand the child a doll, and say, "Oh, my, the baby is hungry! What will you do?" She gave full points to a child who lifted his/her shirt and nursed the baby. I was always pleased with the testing PAT did, because the P.E. was a regular visitor to our home, someone i considered a friend, and who could interpret test results in context, which 'i think is key to useful testing.
Here's a possibility I could see on a test, where my children may have trouble:
Where does a baby sleep?
The mainstream answer is "a crib," of course, but my children have all co-slept in bed with me. We do not have a crib in our house. This question would also impact children from other cultures where co-sleeping is simply normal.
That would be true in our house too.
As a school psychologist, I was afraid to even read this post for fear my blood pressure would rise. :-) I was pleasantly surprised that the tone was not "Standardized tests and all evaluators are evil!" I will agree that the tests are always going to be biased; it's impossible to make a test that is not. I think the real problem is under-trained or lazy evaluators. The mantra of my grad school program was, "A test is a measure of behavior at a specific point in time." A test should NEVER be the sole determinant of a child's placement or services.
The other thing to consider is what the test is actually measuring. IQ tests in American public schools are used to see if a child's academic performance is approximately equal to her potential. "Potential" includes socialization. Maybe it shouldn't, but how well a child can function in a mainstream school setting is tied to what socialization she receives.
Finally, never ever ever believe that an IQ test is a measure of one's value as a person! Any professional that even suggests that should be fired. :-P
(I'll be following this conversation thread, so if anyone wants to know more from 'the other side of the tracks', I'd be happy to answer questions!)
Great post! I'm sad to see that these issues are still coming up; we've known about the bias in the tests for years!
I remember doing some testing like these standardized tests in the kindergarten-grade 1 age; they involved both reading comprehension as well as fine motor skills. According to the tests, I was "slow": I couldn't read an analog clock, do phonics, tie my shoes (we spent a whole lot of time barefoot or in slip-on shoes and boots as wee ones), or tell my left from my right, even though I did okay on other parts of the test. Later on, it turned out that I was just dyslexic and not "slow." The tests, however, required me to learn and develop in the way considered "best" at the time; since I didn't follow that model, I was *obviously* "slow"....
My family jokes about it now, seeing as I'm about a year away from getting a Ph.D.!
My favorite example: I grew up in a town very near Olympic National Park, which contains a temperate rain forest. In the first grade we took a field trip there, and not long after did standardized testing. One of the questions was "Which picture shows a rain forest?" One of the pictures showed a tropical rain forest. Another showed a forest of conifers. Every student at the school got that question wrong!
I think this is a big problem with standardised tests. When my brother had started school they assumed he had a delay of some kind because he was making zero progress in Maths. Turns out he was just bored and Maths ended up being his best subject.
The lack of standardized testing (and grades) is a big reason our children attend a Montessori school.
I believe it is a disservice to most children to be rated and classified according to biased tests and scales. Children need to be allowed to develop and grow and learn according to what is interesting to them, what captures their attention, what they need at the time, at their own pace, not what a test or grade or age tells their parents or teachers they "should" be learning or knowing at the time.
And harumph to red juice. The least they could have done was made it white liquid in the bottle picture.
Lol! My kids would be so confused with that one, baby sleeps in our bed, in the sling, in the pram, on the lounge floor, on my shoulder, in her bouncy chair... Almost anywhere except in her getting dusty crib. :D
Typical standard of assessment in speech pathology (my field) is that standardized tests only take up 1/3rd of the assessment at most. We also use dynamic assessment (test-teach-test- etc), observations of behavior (free play, structured play, language samples, etc), checklists of developmentally appropriate behavior, work samples from the classroom, and many other things. The problem in the speech world is that some insurance uses standardized scores to cover outpatient treatment, but this less true these days. Also, parents will often complain if assessment looks too unstructured (like when we spend time getting a free play sample).
Now, I can't speak for other fields (like psyc/neuropsyc that would be doing IQ & cognitive testing) or even for everyone in my own field that is overworked with oversized caseloads enough to just make it through the required standardized testing. And yes, the tests are definitely outdated. The work that goes into standardizing the test is enormous too; it really prohibits updating them frequently. Now, I've seen tests that haven't been standardized since the 40s and are still in common use... which is a problem, lol. But again, I'm not too concerned because of how much variety is (supposed to be) included in these assessments.
The problem you're really getting at is that standardized tests are somehow a part of 'normal' kids' permanent record if I'm understanding you, which I'm not sure about. It helps make the decision about gifted enrollment, but that's also up to teachers. But, if they're 'well socialized', like you're saying, they're likely the ones noticed by teachers as well as doing well on the tests! I think testing as a general course of action may be more harmful than good since it does categorize people. I was going to say that no one cares about those scores after school, but by then, many opportunities have gone by...
I had a long conversation with a woman one time about how it's not fair that 'they' still get hired or admitted to schools despite poor performance on tests like the SAT. I explained about cultural differences and other things, but her response was - well, why can't they just keep more books in the home, go to playgroups, etc etc. Besides just sort of being in awe of the ignorance, I asked her, 'why don't we just hand out a manual on how to be a _good_ white middle class citizen?' Like a manual on cultural membership. Luckily, she didn't think I was serious and actually realized the absurdity.
/epic comment
Thanks for your thought provoking posts!
Also, I think boobies should totally be in standardized testing pictures:
A. Bottle (partial credit)
B. Shot glass (negative credit)
C. Boobs (extra credit)
I usually reserve a separate time to talk to the parents since it really does hurt my observations when a parent jumps in a lot. Parental reporting is sometimes the only clear data I get, though!
Just recently read a book…can’t recall name…but there was one whole section talking about a study that tested children intelligence at age 3-5…nursery entering tests…and later after 10 years they tested them again along with the group of children who didn’t do well as toddlers…well not sure of results in numbers but the fact was that a very small percentage of those who did well at age 3 had same intelligence performance after 10 years – it was less while a very big percentage of those who failed actually was doing better then those who were thought to be smart as toddlers.
Personally I would NEVER expose my child to that kind of treatment no matter how good a nursery represent them self to be. It’s so unfair towards children.
whatever happened to being a mother and care for their children until they are 5 without worrying how smart they are!!!!
That book is Nurture Shock & is fascinating. The authors get to the point that most parents suspect & a few of us intuit outright: testing very young kids rarely gives a true assessment of them & isn't an accurate predictor of future performance. They go on to say that intelligence doesn't develop in a true linear fashion, but bounces around (like physical growth). My little one is only 11 mos old & I'm already wary of tests, scores, charts, assessments & all the rest.
Is it just me, or have we gotten a little obsessed with quantifying our kids? When did it become not enough simply to have a happy, healthy, normal kid?