Several years ago a video went viral. It featured a man and a woman talking on a sofa. The woman is complaining about pain and pressure she’s experiencing. The camera is zoomed in tight to heighten dramatic effect, which limits you to seeing no more than just one side of her lower face. The man is sympathetically nodding but there’s something mysterious and conflicted about his expression. As he stumbles toward a suggestion, the camera pans out and you’re able to see the woman’s full profile: protruding from her forehead is a nail. This is what the man has tried to communicate but she’ll have none of it. It’s not about the nail.
Jumping to solutions knows no gender. We all do it—and sometimes it is about the nail, but more often, it isn’t. Patience is essential to developing effective solutions. “Understand then act” are pretty good words to live and work by. Recently, I would have gotten it wrong. Thankfully, our company culture insists on seeing the truth before providing a solution.
At Emmersion, we provide remote language testing solutions. However prior to COVID-19, many clients chose to use an on-site testing protocol where the tests were administered under more traditional proctored conditions. Just writing it makes me nostalgic. To go back to the time when we met together to get things done. Wasn’t that nice?! We’ll get there again soon I hope. In the meantime, though, our flexible platform can help initiate the digital transformation for language testing and many organizations.
One of the most acute pain points to transition from in-person operations to remote transactions is fraud or, more precisely, assumed-fraud. Test administrators confronted with moving their testing from in-person to remote testing raised that fear loudly, particularly embodied in two issues: identity verification and cheating prevention. We talked about it enough that we had to develop an acronym—IVCP.
This shorthand only made it seem more manageable. It is not. Identity verification and cheating prevention are tall mountains to climb with no easy routes. We are certainly not timid, though. Timid test developers don’t tackle automated adaptive speaking assessments. However, it is in part because of that culture of ambition that we needed to take the right next step. A small step in the right direction is much better than a large step in the wrong direction.
It’s about confidence
Where inspiration really peaked is when we realized that we were not trying to stop something; we were trying to protect something. Confidence. Administrators need confidence that they are getting good data—data that is informative and reliable. Concerns about identity verification and cheating prevention are as real as a runny nose. However, it is a symptom of an immune response to a different underlying problem
In this case, we needed to provide more than just a tissue. A tissue solution may make a person more comfortable because it takes care of an acute secondary problem, but it is not going to make the real problem any better.
Just as having had a cold or watching someone sneeze does not make you an epidemiologist, we respected that while we were experienced in this domain, there were likely experts that would have insights we could profit from. There are two who wrote books that were very helpful. It should be no surprise that we came across Dr. Gregory Cizek’s book Cheating on Tests: How to Do it, Detect it and Prevent It. It’s a great resource. This bold assertion stood out to us:
“The most effective strategy to prevent cheating is simply to define, discuss and encourage academic integrity…[Test-takers] should be clearly informed as to what actions are expected.”
On my honor
This seemed to resonate with one of the solutions that came up fairly early and frequently in our discovery: an integrity pledge or integrity agreement. It is deceptive in its simplicity. Before beginning an assessment, a test-taker signs or commits to conditions of behavior during the test under the banner of ‘honor.’
Perhaps we can blame it on living with constant suspicion that every stranger in our vicinity is spewing microbes that could get you very sick, but you would not be alone if you had doubts that something so small would actually make a difference against what is feared to be widespread and inescapable.
However, our second expert brought immense confidence that this small step was the right place to start. Dan Ariely is an acclaimed cognitive behavioral scientist. You may have read one of his many books. He is a professor out of Duke University who has collaborated with top researchers across the country. No one has put the human problems of dishonesty and cheating under a microscope like Dr. Ariely.
He lays out his findings in a very approachable and engaging way in The Honest Truth about Dishonesty. With his research team, he examined cheating from almost every angle. He found effective ways to elicit predictable and patterned cheating and, in doing so, he was able to then convincingly identify what does and doesn’t make a difference in stopping it.
Cheater Cheater Pumpkin Eater
First, it’s not just cheaters that will cheat. Under the right conditions, many people will cheat. Dr. Ariely is convincing that while cheating can be pervasive, in whole, the extent to which any one person cheats is actually less egregious than most people would expect.
When cheating happens, it is usually much smaller in consequence than our efforts to stop it would indicate it should be. This holds true even as the stakes for cheating rise. In fact, Dr. Ariely found that the greater the potential return on cheating, the more averse participants were to cheat. As the incentive to cheat increased, people cheated less, not more.
What does not reduce cheating like you would expect? Letting people see you are trying to catch people cheating. This was frankly really shocking to me. He looked at the fear of being caught under a couple of different conditions. First, would people cheat more if they destroyed all evidence that they had cheated? What if they destroyed only half of the evidence? Surely, if they were going to leave behind something that could incriminate them, they would cheat less. Turns out, they did not.
Cheating levels were consistent with previous research (i.e. less than expected) and equal across both conditions. Maybe destroying the evidence was not enough to make people believe they would or would not be caught.
What about proctoring? To test the effect of a proctor on cheating levels, Dr. Ariely replicated the prior experiments with two groups. Both groups had a proctor in the room to do what proctors do—observe what is happening in the room before, during, and after the test was given. The difference? One of the proctors was noticeably blind. Would this change the test-takers level of cheating? It definitely would wouldn’t it? It did not. Fear of getting caught by observation does not limit cheating like you think it would.
At this point, you are probably thinking, “well what does work?” Trust me, I’ve been there.
Fortunately, these two thoughtful authors and researchers were far from the extreme that increasing integrity is a lost cause. Both offered specifics that can be considered depending on the type of threats to integrity a situation presents. Both affirmed that we were headed in the right direction with our plans to implement an integrity agreement.
It may be easiest at this point for me to show you what we did and then explain how it connects to the research that we found. After the instructions to the test had been presented, we added two features to our test flow: an identity check and an integrity agreement.
Make integrity personal
Most people do not cheat if cheating threatens their ego. According to Dr. Ariely, the stronger the sense of self is during an opportunity to cheat, the less likely a person is to cheat.
We all want to see ourselves as honest, honorable people. The implications of this for us was the need to re-invoke that sense of self in the testing experience. We did this subtly through an identity check just before the scored sections of the assessment begin. The test presents the test-taker’s first and last name and asks them to confirm that they are the person the test has been assigned to. We also ask the test-taker to confirm that they can present photo identification to confirm their identity, if requested.
In both seeing their name and visualizing their photo ID, we have raised the test-taker’s ego. Seeing oneself in a mirror or on camera has been shown to drastically reduce theft and encourage other pro-social integrity behaviors. Sure, this reflection evokes the phenomenon of being watched, but the proctor research suggests that that may not be as much of a deterrent.
The more powerful prevention is someone being confronted with the internal question of “what kind of person do I expect myself to be?” Now that the test-taker’s sense of self is supercharged, the identity check hands off to the integrity agreement.
Make integrity visual
Revisiting the ideals raised by Dr. Cizek, the most important prevention of cheating is to define, discuss, and encourage integrity. In this case, we needed to articulate what integrity would look like. What should the test-taker do? Not, “what should they not do?” Again, this is a bold move.
It was the right move for two reasons. First, we are trying to tap into that sense of best-self and personal honor. Evoking suspicion seemed off-key. Second, linguistically it’s clearer; negation is more likely to be misunderstood particularly when you are navigating the agreement in your second language.
Building on that point of clarity, we expect that, in this early stage where the identity check and integrity commitment is presented only in English, we want to give the test taker every reasonable assistance to understand. The pictures communicate the ‘done’ state of the four terms of the agreement.
Make it clear what is to be done. Give a test taker a vision of what they will see when the action is met helps to make that vision a reality. We looked at ways in which our tests could be cheated. Someone who accepts and follows these actions will eliminate 99% of the threats.
Pro-integrity interventions work
Now that I have explained the integrity agreement, you are likely to ask, “well, does it work?”We heard from clients that they have seen similar features to be effective in other testing. Dr. Ariely did a research study that certainly seems to suggest that it will make a difference.
He divided research participants from Yale or MIT into several groups. One group signed an integrity pledge or agreement before the test. A second signed it after and the third did not sign it at all. What was the difference? Those that did not sign the pledge, or signed it after the test, demonstrated levels of cheating similar to previous research. Those that signed a pledge to the Yale or MIT honor code before they started the test did not cheat at all.
Fun fact: Yale and MIT do not have an honor code.
We love what we do at Emmersion. We do not design and develop to just finish. We design and develop so that we can get started. This next step of adding an integrity pledge is not our last effort to increase score confidence of administrators and test-takers. We are excited to roll it out across our products and then observe its effect on the experiences and outcomes for those that choose to adopt it so that we can figure out how it works, when it doesn’t, and how to make it better.
We have embedded some really great metrics in our products that should provide some powerful insights. We will certainly keep you posted, but don’t wait if you’d like to connect. Clearly, we love to talk about our work (and our dogs…go ahead ask me about my dog–I double dog dare you).
Emmersion certifies language ability for organizations around the world using a fully automated and adaptive language assessment engine. It’s revolutionizing the language testing process with instant, accurate scoring for speaking, grammar, and writing ability in 9 global languages and counting. With a scalable assessment solution they can count on, hundreds of global businesses are building successful teams, reducing turnover, and improving their customer satisfaction scores. Learn more at www.Emmersion.ai.