SimCheck AI Writing Detection Tool – Why the UW has Opted Out

Last updated: September 26, 2023

In spring quarter, SimCheck added a tool that purported to estimate how much of a student’s submission is generated by AI. Unlike most digital learning tools that UW adopts, Turnitin did not provide the tool in advance of its release for evaluation, piloting, or support planning. This was concerning to UW and many of our peer institutions (Knox, 2023).

UW adopted SimCheck as a plagiarism detection tool with the belief that it could complement an instructor’s own judgment, with high scores creating an opportunity for intervention rather than punishment. The key difference between SimCheck’s plagiarism detection and the AI writing detection is that reports generated by SimCheck’s plagiarism tool are transparent about the sources that were used for scoring – instructors can see for themselves whether the sources flagged in the report are valid and substantive. The report generated by the AI tool, on the other hand, provides no evidence for how the score was reached.

There have also been inconsistencies in SimCheck’s reporting on what are known as “false positives” (the incorrect flagging of text as being AI-generated). The company initially reported “a <1% false positive rate,” but has since confirmed that the false positive rate is higher. Turnitin/SimCheck has not disclosed the new rate, but the sentence-level positive rate is approximately 4% (D’Agostino, 2023).

This false positive rate has the potential to inaccurately call into question the submissions of thousands of students. Additionally, a study from Stanford found that AI detection tools more often identify the writing of non-native English speakers as AI generated (Mathewson, 2023). Experiences at Boston University have shown that international students seem to be particularly at risk for being flagged by the SimCheck AI Detection tool (Rubenstein, 2023).

Due to these concerns, UW requested that SimCheck/Turnitin turn off the AI Writing Detection tool in UW’s Simcheck integration.

References