How to prepare for and discuss the possibility of false positives

Last week, Turnitin’s Chief Product Officer, Annie Chechitelli, discussed the hot topic of false positives within AI writing detection tools and offered a few high-level tips on how to address them. This week, we are going to take a much deeper dive into those situations, from both the instructor and the student vantage points.

Before we go into that, we want to offer some reassurance: our detector currently has a less than 1% false positive rate, and we acknowledge that even this low rate of false positives may create tension and anxiety.

With this in mind, we want to stress our commitment to educators AND students; our approach to AI writing detection is one of continual improvement. Our AI scientists are hard at work evolving the AI detection tool generally, with attention on the rate of false positives. That work is in progress now and will continue into the future. When we at Turnitin say that we intend to act as your partners in this new era, we mean it.

What can educators do to handle the possibility of false positives?

While it may not feel like it right now, educators have a lot of power, and the resources that we share are designed to help you leverage them in a way that both honors a culture of academic integrity AND protects students from any undue harm. Efforts to protect students must rest on the strength of educator relationships with students, regardless of AI considerations. Should a concern about false positives present, educators and students will need to engage in an open and honest dialogue, which can only happen if strong relationships are already established.

First, let’s look at what the educator can do before and/or during the assignment.

1. Clearly articulate what is/is not permissible, even on a particular assignment. If possible, reference any institutional and/or departmental guidance. Set the expectation that if students do use AI writing tools, they must provide a citation.
2. Collect a diagnostic writing sample in order to have a baseline for comparison. *Pro Tip: Do this as an on-demand, in-class assignment to further minimize the possibility of misconduct.
3. Use our AI misuse rubric to make sure the prompt/assignment is less vulnerable to misconduct. Use our AI Misuse checklist to structure the writing process in ways that reduce the threat of any unauthorized use of AI writing tools.
4. Consider using tools such as Turnitin’s Draft Coach or the option for re-submissions inside Feedback Studio to collect artifacts of the work throughout the writing process.
5. Plan ahead for the process to be used if a question of AI misuse arises and set the expectation for students so that they aren’t caught unaware. Will educators ask the student challenge questions that might show how well they understand the content of the writing? Will educators ask the student reflection questions designed to reveal the writing choices made in the piece? Will educators ask the student probing process questions that might reveal gaps or issues? Know the answers beforehand and make sure students know too.

Now, let’s consider what educators can do after the final draft of the writing assignment is submitted.

1. Use Turnitin’s AI writing indicator and make sure all educators understand how it works. Check out Turnitin’s main AI landing page for resources on the functionality once it’s released!
2. Remember that the indicator is not a replacement for educator judgment. Turnitin’s goal is to provide educators with the insights needed to make good decisions. However, educators know their students and their work, and that is equally important.
3. Revisit the assignment and process to determine any gaps that may have been left and evaluate the extent of the risk for AI tool misuse.
4. TALK TO THE STUDENT. This kind of judgment should never be made without a respectful dialogue.
5. Compare the writing in question to the diagnostic sample previously collected. Does the flagged writing align to the style, complexity, and sophistication of the student’s previous writing? One suggestion: look for distinctive phrases - idioms or expressions that seem out of place; discussing these should be part of the conversation with the student.
6. Adopt an attitude that assumes positive intent. If the evidence isn’t clear after the review, give the student the benefit of the doubt. Educators will have had all the right conversations, asked the right questions, and if they’re still not sure - students must not be penalized. The good news is that the experience should act as a powerful deterrent for any potential future misuse!

NOTE: We’ve created a standalone document with this information so that you can use it yourself, share it with colleagues, or even make it available as part of a professional learning activity.

What can students do to be proactive, given the availability of AI writing tools?

The power dynamic between student and instructor can feel unbalanced from the student’s perspective; with that in mind, there are steps that students themselves can take to be proactive.

To get ahead of the issue, let’s first look at what students can do before and/or during the assignment:

1. Recommend that they use the writing process! Not only can it improve the quality of the writing, each stage produces an artifact of the development of original work. Let students know that when instructors can see brainstorming, drafting, revisions, and even the incorporation of feedback from a peer, tutor, or other trusted source, it shows thinking coming to life. *Pro Tip: If it’s available, educators can make this easier by advising students to use Turnitin’s Draft Coach or enabling the option of multiple submissions inside Turnitin’s Feedback Studio. Either of these places will allow students to create a running record of writing taking shape, making their own original work visible.
2. Guide students to ask each instructor what, if any, use of AI writing tools is acceptable and be sure they understand expectations. The answers may vary!
3. Explain to students the importance of establishing their writing style and voice, which will mean putting in real effort when they are asked for a writing sample or within every assignment.
4. Make sure students know to cite any use of an AI writing tool. That way, it is clear that they are not representing AI work as their own. In fact, make sure they understand the importance of providing clear sources and citations for all work, where relevant to the assignment.

Now, let’s take a look at what students can do after the final submission:

1. Advise students to remain calm if questions come up. This is a new frontier for everyone, and students should consider that educators are learning along the way too. Just because they are asked a question, students should try not to become defensive. Instead, prompt students to have a genuine discussion and state their case.
2. Remind students that instructors’ ultimate goal is to help them on their learning journey. Starting from a place of mutual respect is going to help everyone in the long run. It may be helpful to refer back to the original discussion of what is/is not acceptable use of AI writing tools.
3. Prepare students to discuss their own writing process: where/when did they work on the assignment? Who else read it along the way and maybe even provided feedback? What changes did they make and why did they make them? How did they go about their research? Are there specific language choices they made? Is any of their personal experience or ideas clearly articulated in the writing? Hearing students discuss their process will help the instructor understand what went into the work and lends credibility to the student’s assertion that the writing is their own, original work.
4. Suggest that students be prepared to share evidence: share any notes, outlines, version histories, or drafts created during the writing process, as well as any feedback they might have received from a peer or trusted reviewer. If the student has used Turnitin’s Draft Coach or the instructor has enabled multiple submissions inside Turnitin’s Feedback Studio, either of these can serve as artifacts of their work taking shape.

NOTE: We’ve created a standalone document with helpful information for students so that you can use it instructionally, send students here for guidance, or even print and distribute. We made sure it was written in student-facing language for ease-of-use.

Everyone, on all sides, is learning right now - this a new, evolving intersection of technology and education. That means we have to approach these situations knowing that we may not have all the answers. Avoiding assumptions, being explicit, and creating trusting relationships are the only ways to get to a place of shared understanding.