How you phrase a question influences test answers

Justin Thomas looks at the state of International English Language Testing in the UAE

When the outcomes of a test can have such profound consequences, we have to try to ensure that the test is as fair as possible. Jeff Topping / The National

Many important things hinge on test results. We test people to discriminate and categorise. Tests are scary, perhaps because they directly challenge the fuzzy feel-good notion that we are all equal. Access to opportunities isn’t equal. In fact, very often it’s determined by test performance.

The International English Language Testing System, better known as the IELTS, is a good example. This test is widely used as a gatekeeper to higher education in the UAE. However, a recent article, published in Learning and Teaching in Higher Education: Gulf Perspectives, calls into question the cultural fairness of this widely-used test.

Cultural bias on any test essentially means that some questions are worded or presented in such a way that they systematically favour individuals from one cultural group over another. The IELTS scores are important, because they can mean the difference between access to, or exclusion from, particular jobs or academic programmes.

When the outcomes of a test can have such profound consequences, we have to try to ensure that the test is as fair as possible. In an academic context, tests ought to measure the aptitude in question as cleanly as possible. For example, I don’t want my mathematics exam to be unduly influenced by your knowledge of casinos and how a roulette wheel works. While the roulette wheel might be a convenient reference for framing questions of probability, it disadvantages those with no knowledge of gambling or those from cultures where gambling is prohibited.

We know for sure that the way we phrase questions can greatly influence the answers we get. For example, asking people if they would like beef that is 80 per cent lean is more likely to elicit a "yes", than asking them if they would like beef that is 20 per cent fat. Consider also what Pew Research Center asked in 2010: “Do you happen to know what Barack Obama’s religion is?” Eighteen per cent of respondents answered “Muslim”. How many more might have answered incorrectly if the question had been worded: “Do you happen to know what Barack Hussein Obama’s religion is?”

Psychologists have studied the effects of how questions have been framed for decades. It is well established that wording, question order and even the number and sequence of response options can systematically influence the ways in which people respond.

On top of this, if we include culturally alien concepts (the Holy Grail) and phrases (canary in a mine), we will bias the test in favour of those familiar with the particular originating culture. If it is a test of cultural awareness then that is fine, but if we are claiming to assess the candidates’ knowledge of psychology, history or language, then this can be hugely problematic.

This is not to say that anyone who gets a low score on the IELTS can blame it on cultural bias. Cultural bias might just be one minor factor.

Issues of preparedness, motivation, hard work and even exam anxiety may all play roles in the outcome.

However, if it is possible to further reduce culture and gender bias from these important assessments, why wouldn’t we? This is not about political correctness – I hate that phrase – but about the pursuit of excellence.

The IELTS website makes the following claim: “The IELTS approach is recognised as being fair, reliable and valid to all candidates, whatever their nationality, cultural background, gender or special needs.”

I would argue that this claim would be better stated as an aspiration. Those involved in the development and validation of assessments know that there is always room for improvement. Given that the stakes are high in relation to the IELTS in the UAE, this is certainly a topic worthy of greater attention.

Justin Thomas is an associate professor of psychology at Zayed University and author of Psychological Well-Being in the Gulf States

On Twitter: @DrJustinThomas