Robin Pocornie brought a lamp with her to court. It’s nothing special, just a basic Ikea floor lamp. But for the masters student, the lamp was a useful prop to help explain how she believes her university’s exam supervision software discriminated against her based on the color of her skin.
Pocornie and her lamp stood in front of the Netherlands Institute of Human Rights, a court focused on discrimination claims, in October 2022. But the first time she encountered remote-monitoring software was two years earlier, during the pandemic, when her course at Dutch university VU Amsterdam was holding mandatory online exams. To prevent students from cheating, the university had bought software from the tech firm Proctorio, which uses face detection to verify the identity of the person taking the exam. But when Pocornie, who is Black, tried to scan her face, the software kept saying it couldn’t recognize her: stating “no face found.” That’s where the Ikea lamp came in.
For that first exam in September 2020, and the nine others that followed, the only way Pocornie could get Proctorio’s software to recognize her was if she shone the lamp uncomfortably close to her face—flooding her features with white light during the middle of the day. She imagined herself as a writer in a cartoon, huddled over her desk in a bright spotlight. Having a harsh light shining in her face as she tried to concentrate was uncomfortable, she says, but she persevered. “I was afraid to be kicked out of the exam if I turned off the lamp.”
On Google, Pocornie soon found other Black students sharing similar experiences—saying that they had to use lamps or flashlights or sit near windows to get exam software to recognize them. All over the world, students of color were being forced to create complicated light arrangements just to take their exams.
Pocornie’s exams are over, but she decided to challenge the university’s use of software that she says is discriminatory by taking her university to the Dutch human rights court. Exam software of this kind has already outlived the pandemic lockdowns that made it mainstream, and Pocornie says that all universities using face detection software need to reckon with the way this technology discriminates against people with darker skin tones.
“If education isn’t working according to everybody’s needs,” she says, “then it’s just not working.”
VU Amsterdam and Proctorio declined to comment.
There have been legal challenges to the use of anti-cheating software in the US and in the Netherlands, but so far they’ve mostly focused on privacy, not race. An August case in Cleveland, Ohio, found that the way Proctorio scans students’ rooms during remote tests is unconstitutional. Another Dutch case in 2020 tried and failed to prove that exam-monitoring software violated the European Union’s privacy rules. Pocornie’s case is different because it focuses on what she describes as bias embedded in the software. “If [these systems] do not function as well for Black people in comparison to white people, that feels to us discriminatory,” says Naomi Appelman, a cofounder at the volunteer-run Racism and Technology Center who has helped Pocornie with her case.
Porcornie’s legal case is still ongoing. In December, the Dutch Institute of Human Rights issued an interim ruling saying it strongly suspected that the software used by VU Amsterdam was discriminatory and giving the university 10 weeks to file its defense. That defense has not yet been made public, but VU Amsterdam has previously argued that Pocornie’s log data—showing how long she took to log into her exam and how many times she had to restart the software—imply her problems were due to an unstable internet connection, as opposed to issues with the face detection technology. A ruling is expected later this year.
Producers of anti-cheating software like Proctorio’s were boosted by the pandemic, as exam halls were replaced by students’ own homes. Digital monitoring was meant to help schools and universities maintain business as usual throughout lockdown—without creating an opportunity for unsupervised students to cheat. But the pandemic is over and the software is still being used, even as students around the world return to in-person teaching. “We don’t believe it is going away,” said Jason Kelly, who focuses on student surveillance at the US-based Electronic Frontier Foundation, in a 2022 review of the state of student privacy in December.
In the US, Amaya Ross says her college in Ohio still uses anti-cheating software. But every time she logs in, she feels anxious that her experience during the pandemic will repeat itself. Ross, who is Black, also says she couldn’t access her test when she first encountered the software back in 2021. “It just kept saying: We can’t recogize your face,” says Ross, who was 20 at the time. After receiving that message three or four times, she started playing around with nearby lamps and the window blinds. She even tried taking a test standing up, directly underneath her ceiling light.
Eventually she discovered that if she balanced an LED flashlight on a shelf near her desk and directed it straight at her face, she was able to take her science test—even though the light was almost blinding. She compares the experience to driving at night with a car approaching from the other direction with its headlights on full-beam. “You just had to power through until it was done,” she says.
Ross declines to name the company that made the software she still uses (Proctorio has sued at least one of its critics). But after her mother, Janice Wyatt-Ross, posted about what happened on Twitter, Ross says a representative from the business reached out, advising her to stop taking tests in front of white walls. Now she takes tests with a multi-colored wall-hanging behind her, which so far seems to work. When Ross asked some of her Black or darker-skinned friends about the software, a lot of them had experienced similar problems. “But then I asked my white friends and they’re like, ‘I’m taking tests in the dark,’” she says.
Typically, face-recognition and detection technology fails to recognize people with darker skin when companies use models that were not trained on diverse data sets, says Deborah Raji, a fellow with the Mozilla Foundation. In 2019, Raji copublished an audit of commercially deployed face-recognition products, which found that some of them were up to 30 percent worse at recognizing darker-skinned women than they were white men. “A lot of the data sets that were in mainstream use in the facial recognition space before [2019] contained 90-plus percent lighter skin subjects, 70-plus percent male subjects,” she says, adding progress has been made since then, but this is not a problem that has been “solved.”
“Before we can get to an acceptable degree of performance across these different demographics … it doesn’t make sense for us to deploy that technology in high-stakes scenarios such as proctoring,” Raji says, especially when students are given no way to opt out or appeal.
After spending more than a year searching for a university official who would take her concerns about Proctorio seriously, Pocornie hopes this case will force VU Amsterdam to create a process that students can use to complain about discriminatory software.
For her, Pocornie says this legal case is about making sure other students don’t suffer the same experience or have to deploy Ikea lamps just to take a test. She doesn’t want to retake her exams, even though she believes the software impacted her grades. Because she was taking two full-time masters courses at the same time, she says she was able to compare the grades she received in both (only one course replaced in-person exams with proctoring software during the pandemic; the other used coursework). The stress of online proctoring, Pocornie says, means the grades she received for her VU Amsterdam course were significantly lower than the grades she received from the University of Maastricht. To protect her privacy, she asked WIRED not to share the exact difference.
“So that’s the quantitative effect,” she says. “But on top of that, there is obviously also the distress, and I think that should count for something.”
