Rethinking Homework in the Age of AI

Overview

The emergence of large language models has led to the development of powerful tools such as ChatGPT that can produce text indistinguishable from human-generated work. With the increasing accessibility of such technology, students across the globe may utilize it to help with their school work—a possibility that has sparked ample discussion on the integrity of student evaluation processes in the age of artificial intelligence (AI). To date, it is unclear how such tools perform compared to students on university-level courses across various disciplines. Further, students’ perspectives regarding the use of such tools in school work, and educators’ perspectives on treating their use as plagiarism, remain unknown. Here, we compare the performance of the state-of-the-art tool, ChatGPT, against that of students on 32 university-level courses. We also assess the degree to which its use can be detected by two classifiers designed specifically for this purpose. Additionally, we conduct a global survey across five countries, as well as a more in-depth survey at the authors’ institution, to discern students’ and educators’ perceptions of ChatGPT’s use in school work. We find that ChatGPT’s performance is comparable, if not superior, to that of students in a multitude of courses. Moreover, current AI-text classifiers cannot reliably detect ChatGPT’s use in school work, due to both their propensity to classify human-written answers as AI-generated, as well as the relative ease with which AI-generated text can be edited to evade detection. Finally, there seems to be an emerging consensus among students to use the tool, and among educators to treat its use as plagiarism. Our findings offer insights that could guide policy discussions addressing the integration of artificial intelligence into educational frameworks.

Homework AI

Key Capabilities

  • Benchmarked ChatGPT against students in 32 university courses
  • Evaluated AI-detection tools like GPTZero across paraphrased and original outputs
  • Surveyed 1,600+ students and faculty across five countries
  • Measured norm conflict between institutional expectations and student intentions
  • Highlighted gaps in academic policy regarding AI-assisted work
Homework AI

Results

  • ChatGPT matched or outperformed students in 12 of 32 courses (38%)
  • AI-text detectors misclassified both AI and human text frequently
  • Paraphrased ChatGPT outputs bypassed detectors in 95% of cases
  • 74% of students expressed intent to use ChatGPT next semester
  • Faculty underestimated student AI use and strongly favored treating it as plagiarism

Paper

Read the full paper: Perception, performance, and detectability of conversational AI across 32 university courses

Collaborators

  • Hazem Ibrahim (NYUAD)
  • Fareed Zaffar (LUMS)
  • Talal Rahwan (NYUAD)
  • Fengyuan Liu (NYUAD)
  • Yasir Zaki (Advisor, NYU Abu Dhabi)