Rehan N's Senior Project Blog

Project Title: Adapting AI Detection Tools to Mitigate Biases for ESL Writers. BASIS Advisor: Ms. Ainslie Internship Location: Remote Onsite Mentor: N/A

Project Abstract

As AI-generated text from large language models becomes ever more prevalent, professionals in fields ranging from education to journalism have begun to rely on detection tools to assess content authenticity. Yet, a growing body of research highlights potential biases in these systems, which is particularly concerning as there is a lack of transparency on performance metrics of commercial tools like GPTZero or Turnitin. These systems—often trained on limited linguistic profiles—may disproportionately misclassify writing from ESL (English as a Second Language) speakers as AI generated—raising ethical concerns around the equity of AI-assisted content evaluation. My project seeks to develop a solution that addresses these biases by analyzing text samples from native English speakers, non-native English speakers from diverse geographic locations and ethnic backgrounds, and AI-generated outputs. I begin by feeding these samples into a customizable open source AI detector and examine key metrics like accuracy and false positive rate in regards to their impact on non-native English speakers. The resulting data quantifies the existing disparities and showcases systemic blindspots in current detection algorithms. Building on these findings, I modify the text-classification model by retraining it with a broader spectrum of linguistic styles to optimize for inclusive performance. Then, I rerun the tests and perform a comparative statistical analysis to evaluate improvements. The post-training results underscore the potential for thoughtful data curation mitigating biases. Ultimately, my research advocates for greater awareness of how detection tools are developed and deployed and charts a path towards systems that more accurately and more fairly distinguish between human-authored and machine-generated text.