Grading “assistants” that use generative artificial intelligence offer teachers a tantalizing promise: You can spend less time poring over hundreds of student essays while still giving kids the thoughtful feedback they need to make their writing better.
But how do these tools actually stack up against experienced educators? Education Week wanted to find out.
We asked two teachers to grade the same piece of writing—a short persuasive essay from a 7th grader in Missouri—against a set of criteria for organization, clarity, and mechanics. Then, we gave the same instructions to ChatGPT, the generative AI chatbot developed by OpenAI.
The teachers—Heather Van Otterloo, a middle school English teacher in Joplin, Mo., and Chad Hemmelgarn, a high school English teacher in Bexley, Ohio—have both used AI tools to help them give feedback on student work in their classes. Even so, both say that they’re always involved in the process, reviewing AI’s comments or making the final decision about how to score a piece of writing.
This student writing sample, from one of Van Otterloo’s middle schoolers, is from an assignment she might run through her AI grading platform, she said.
The prompt—“Should people ever wear pajamas in public? Why?”—asks students to defend their position on this issue as a formative measure of their ability to take a stance and support it with evidence. Students were directed to make a claim, provide reasons and evidence, offer a counterexample, and include a conclusion. They received the set of criteria that Van Otterloo would use to evaluate their papers and some general guidance on writing for organization and clarity.
Education Week provided all this information to Hemmelgarn, and to ChatGPT. All three respondents’ answers are below. Take the quiz to see if you can tell whose response is whose—and whether AI can match teachers’ insights.
Student writing sample
Pajamas should never be worn in public. First of all, they are not a good representation that you would want to be seen as in public. Pajamas state a message that you didn't put in effort to make yourself presentable or simply just don't care. Pajamas are seen by many as something that should only be worn to sleep in, so wearing them in public may have a negative affect on your reputation.
Second of all, pajamas are not made to be worn out in public anyways. They are made to be slept in and not to be worn out of the house. If they are not used for their original purpose then there isn't a point in having them. If all you have is pajamas in your wardrobe then you will not have anything to wear to particularly serious events, like a nice dinner or a program.
Some may say that pajamas are acceptable in public because they are comfortable and relaxing, however there are brands that make comfortable outfits that are still more presentable to be worn out of the house, such as Lululemon and Vuori.
In conclusion, pajamas should never be worn in public because they do not give off a positive representation of yourself and are not made for that purpose but to specifically be worn to bed.
Scroll down to see Rubrics A, B, and C. Click the tabs to view each graded rubric.
Meets Expectations
Approaching Expectations
Needs Improvement
Unsatisfactory
Claim (Thesis Statement)
X
Reasons (Explanation of Opinion)
X
Support with Examples
X
Counter-Argument
X
Conclusion
X
Organization and Clarity
X
Grammar and Mechanics
X
Meets Expectations
Approaching Expectations
Needs Improvement
Unsatisfactory
Claim (Thesis Statement)
X
Reasons (Explanation of Opinion)
X
Support with Examples
X
Counter-Argument
X
Conclusion
X
Organization and Clarity
X
Grammar and Mechanics
X
Meets Expectations
Approaching Expectations
Needs Improvement
Unsatisfactory
Claim (Thesis Statement)
X
Reasons (Explanation of Opinion)
X
Support with Examples
X
Counter-Argument
X
Conclusion
X
Organization and Clarity
X
Grammar and Mechanics
X
Meets Expectations
Approaching Expectations
Needs Improvement
Unsatisfactory
Claim (Thesis Statement)
The claim is clearly stated at the beginning of the response and is easy to identify. The stance is strong and unambiguous.
The claim is stated, but may not be as clear or obvious. The stance is present but might need further emphasis.
The claim is somewhat unclear, or the stance is not fully defined.
The claim is missing or unclear, and the stance is unidentifiable.
Reasons (Explanation of Opinion)
Provides multiple well-developed reasons supporting the claim. The reasoning is clear, thoughtful, and strongly related to the topic.
Provides at least one solid reason, but might lack depth or fully developed explanation.
Provides a reason that is somewhat vague or underdeveloped.
Lacks sufficient reasoning or explanation.
Support with Examples
Includes at least two specific examples that directly support the reasons given, such as personal experiences, facts, or cultural references. Examples are strong.
Includes one strong example or two examples that are somewhat relevant but could be expanded upon.
Includes vague or generic examples that don't strongly support the reasons.
Lacks examples or uses examples that don't support the reasons.
Counter-Argument
Acknowledges the opposing view thoughtfully, with a clear explanation of why the writer's stance is stronger.
Acknowledges the opposing view but does not fully explain why the writer's stance is stronger.
Mentions the opposing view, but the response is weak or unclear.
Does not acknowledge the opposing view.
Conclusion
Provides a clear, thoughtful summary that reinforces the claim and reasons. Includes a final reflection that ties the argument together.
Summarizes the claim and reasons, but the conclusion may lack depth or new insight.
Includes a weak or generic conclusion. The summary of reasons is unclear.
Conclusion is missing or does not adequately summarize the points made.
Organization and Clarity
The essay is well-organized, with clear paragraphs that each support a specific point. Transitions between ideas are smooth, and the essay is easy to follow.
Organization is clear, but some paragraphs may lack clear focus or transitions may be awkward.
The essay lacks organization or clear transitions, making it harder to follow.
The essay is disorganized, making it difficult to understand the argument.
Grammar and Mechanics
Few or no errors in spelling, grammar, punctuation, and sentence structure.
A few minor errors in grammar or punctuation, but they do not distract from the content.
Some errors that may make the essay harder to read or understand.
Frequent errors in gramma or mechanics that interfere with readability.
Coverage of education technology is supported in part by a grant from the Siegel Family Endowment, at www.siegelendowment.org. Education Week retains sole editorial control over the content of this coverage.
All content on Education Week's websites is protected by copyright. No part of this publication shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic or otherwise, without the written permission of the copyright holder. Readers may make up to 5 print copies of this publication at no cost for personal, non-commercial use, provided that each includes a full citation of the source. For additional print copies, or for permission for other uses of the content, visit www.edweek.org/help/reprints-photocopies-and-licensing-of-content or email reprints@educationweek.org and include information on how you would like to use the content. Want to seamlessly share more EdWeek content with your colleagues? Contact us today at pages.edweek.org/ew-for-districts-learn-more.html to learn about how group online subscriptions can complement professional learning in your district or organization.
