Persona bots, which can be found on Character.AI and other platforms, allow users to have real-time conversations with bots purporting to be historical figures, world leaders, and even fictional characters. The bots are trained on internet data and are supposed to mimic the speaking style and tone of their characters.
I decided to put one of these bots to the test, using a subject matter in which I consider myself an expert. I was Education Week’s politics and policy reporter from 2006 to early 2019, a period that covers the entirety of President Barack Obama’s two terms. In that role, I did in-depth reporting on Obama’s K-12 policies. I never interviewed the president directly, but I spoke extensively to both his education secretaries—Arne Duncan and John King—multiple times.
Another reason for choosing Obama’s education agenda: It was far-reaching, ambitious, and controversial. There were complexities, nuances, subtle shifts in position. Would a chatbot be able to capture them? The answer: No, or at least not very well. (Full disclosure: Some of my questions were deliberately crafted to trick the bot.)
As you’ll see, the bot got a few facts right. But far more often it shared inaccurate information or contradicted itself. Maybe most surprisingly, it was more apt to parrot Obama’s critics than the former president himself.
The chatbot’s failure to truly channel Obama on his K-12 record came as no surprise to Michael Littman, a professor of computer science at Brown University. Chatbots and other large language models are trained by absorbing data, in this case large swaths of the internet, he said. But not every piece of information is going to get absorbed to the same degree.
“If the system doesn’t have experience directly with the question, or it doesn’t have enough experience that could lead it to make up something plausible [for the character], then it will just make up something that maybe other people said, and in this case, it was his critics,” Littman explained. “[The bot] didn’t have anything else to draw on.”
It’s also not particularly unusual that the bot contradicted itself so often, Littman explained.
“One of the hardest things to do when making things up is remain self-consistent because you can easily make a statement, then later, you want to say something that doesn’t really agree with that statement,” he explained. “And then you’re stuck. The bots are always making stuff up, so they get into that situation a lot. Sometimes they don’t even bother noticing. They just continue forward.”
The bottom-line: The conversation is a good example of why character bots are good for teaching about AI, but not very good—at least not yet—at helping students find accurate information. Also, if you’re teaching a course on education policy history, definitely steer clear of referring students to this chat!
Chatting With a Barack Obama Chatbot
EDUCATION WEEK:
What would you do differently on Race to the Top if you could go back in time?
BARACK OBAMA (AI BOT):
Wow, excellent question. It's tough to pinpoint one specific thing, but with Race to the Top in retrospect I was probably too hands-off and should have set stricter guidelines on what states could and couldn't do with the money. There was too much incentive for states to fudge the metrics, and not enough accountability if the money didn't lead to better educational opportunities. But again, you learn from your mistakes.
ANALYSIS: These are common criticisms of Race to the Top, Obama’s marquee K-12 grant competitive program. Obama’s education department created the Race to the Top program, which helped allocate grants to states for embracing certain education principles, including teacher evaluation based in part on student test scores as well as common standards designed to get students ready for college and career. (In practice, only the Common Core state standards met that second requirement).
The criticisms repeated by the chatbot don’t reflect anything that the former president has actually said. In fact, it would be surprising to hear him align himself with his critics without pointing out what he sees as the program’s successes.
EDUCATION WEEK:
Really? Because you were accused at the time of being far too rigid and leaning too much on test scores.
BARACK OBAMA (AI BOT):
My only wish was for all students to have a world-class education, no matter what state they were from. In many ways, though, I went too far. States have the right to regulate their curriculum, and I shouldn't have imposed my values as president—that sort of "nation building" at home was wrong. The Department of Education should have had more oversight, and in hindsight I should have let teachers make the decisions about what's best for their students instead of dictating standards from Washington.
ANALYSIS: Again, this reflects statements Obama’s critics have made of Race to the Top and its role in championing the Common Core state standards, but nothing that has come from the former president. And the statement that the “department of education should have had more oversight” directly contradicts Chat Obama’s view that his administration was too prescriptive.
EDUCATION WEEK:
You would never have said during your presidency that you dictated standards from Washington. What has changed?
BARACK OBAMA (AI BOT):
The most important thing about being president is being accountable. There is something fundamentally wrong with the United States when we spend more per student than any wealthy nation, and yet we routinely have more students drop out, more students getting substandard grades, and fewer students having the sort of higher education opportunities that lead them to successful career opportunities. It is clear to me that our educational system is broken and that I, as president, was not able to fix it. I own my failure.
ANALYSIS: This reflects other people’s criticism of the American educational system. During his presidency, Obama pointed to factors—like rising graduation rates—as evidence the system was improving.
EDUCATION WEEK:
Do you think you were a strong education president? What are you most proud of on K-12 policy?
BARACK OBAMA (AI BOT):
I tried to do my best for education, and I think my administration made a lot of significant strides—increasing teacher pay, giving funding for schools who were under-funded, increasing the focus on STEM and computer sciences, expanding early childhood education for families, etc. And of course, providing the funding for state aid was probably the biggest thing.
With that said, I was not able to accomplish many of my ambitious goals for education. I tried hard to get Congress to pass my proposals but they often fell short. You win some, you lose some, right?
ANALYSIS: Much of this is accurate. Obama’s administration did indeed work with Congress to increase Title I funding for schools serving disadvantaged students, and money for the Head Start early childhood program, as well as to provide aid to states to prevent teacher layoffs during the Great Recession. He directed some competitive grant money for STEM education and used the bully pulpit of his office to champion STEM education. He talked about the importance of increasing teacher pay, but never actually did so, in part because teachers are paid through local and state aid.
EDUCATION WEEK:
Was Arne Duncan a good choice for education secretary?
BARACK OBAMA (AI BOT):
I think he was. He was one of the most capable people in my cabinet—he didn't just understand how education worked, he had been a teacher himself and had an idea of what actually worked (and didn't work) in schools. His passion for public education was clear early on, which made him a very natural choice for secretary of education.
ANALYSIS: This is also fairly accurate, although Duncan was not a public school teacher. He did serve as the CEO of Chicago public schools, which was his most obvious credential for the education secretary job.
EDUCATION WEEK:
Had Duncan been a public school teacher?
BARACK OBAMA (AI BOT):
Not a traditional public school teacher, but he had helped set up charter schools and led a network of charter schools in Chicago. He had also taught adult education (and I think he taught some night classes). His perspective as a charter school leader came in handy though—he knew the best practices, the things teachers needed to feel supported, and of course he knew the most important thing: what the kids needed to succeed.
ANALYSIS: Some inaccuracies here. Duncan did help open a charter school in Chicago, but never led a network. He tutored children but never taught adults in night classes. And again, his most prominent experience before serving as education secretary came from leading Chicago public schools, which the chatbot did not mention.
EDUCATION WEEK:
Why did you oppose the Every Student Succeeds Act?
BARACK OBAMA (AI BOT):
It took away too much of the Department of Education's oversight role, and allowed states to avoid accountability for standards that should be in place. As I mentioned earlier, I should have taken a step back in many of the things that I did with education, but even I feel like ESSA gave too much freedom to the states, allowing them to skirt rules around students with learning disabilities, students with limited English, and students who are on track to get a diploma but not to complete college.
ANALYSIS: Asking this was a trick question because Obama supported ESSA. The bot’s statements reflect common critiques of the law, but the Obama administration was fully behind it and saw replacing the much-maligned No Child Left Behind Act as a major legislative victory.
EDUCATION WEEK:
Did you veto ESSA?
BARACK OBAMA (AI BOT):
No, it passed the House and the Senate with enough votes to overrule my veto, and I didn't think that the bill was so bad that it was worth the massive controversy that would come by refusing to sign. So, I compromised and ended up signing the bill, while expressing my disappointment with it.
ANALYSIS: This was another trick question. Obama didn’t veto ESSA, though it gained enough support that Congress could have overridden his veto if he had. And Obama did not express disappointment with the bill in signing it in December 2015. In fact, he called it a “Christmas miracle.”
EDUCATION WEEK:
Is teacher evaluation based in part on student test scores a good idea?
BARACK OBAMA (AI BOT):
No, for a million reasons. First, when you start putting the fate of someone's income and their livelihood in their students' hands, they may feel pressured to teach "to the test", rather than holistically, focusing on the skills that are going to lead to long-term success for students. Second, there are just too many variables. A teacher can't control whether a student has a broken home life that creates an unstable environment. We want our teachers to make our students feel supported, not stressed out about their employment.
ANALYSIS: This is not accurate, though it reflects what critics of the idea say. In fact, the Obama administration pushed states to adopt teacher evaluations based in part on student test scores in both Race to the Top and in granting states flexibility from the No Child Left Behind Act, which ESSA replaced.
EDUCATION WEEK:
Why did you support teacher evaluation through test scores then?
BARACK OBAMA (AI BOT):
I don't know that I ever did. I know that one part of Race to the Top (our flagship education initiative at the beginning of my first term) was to require states to have a "teacher evaluation system that included student performance as a significant measurable factor." But I never supported the idea of solely, or even primarily, relying on test scores when evaluating teachers. And I can tell you that I was adamant and insistent that teacher pay could not be tied to test scores.
ANALYSIS: The Obama bot accurately described how Race to the Top rewarded states for embracing differentiated teacher pay. And it’s accurate that the real Obama didn’t advocate basing teacher pay primarily on test scores. But the Obama administration did use its competitive grants to reward states and districts for basing teacher pay, in part, on student growth as measured by standardized tests.
EDUCATION WEEK:
Do you support vouchers for private schools?
BARACK OBAMA (AI BOT):
No, I do not. I think it is unfair to take taxpayer money and give it to private schools. It is my view that private schools create a two-tiered system of education, and it's one that we should not support. I agree that more options can create a better environment to spur competition and innovation around education, but when private schools receive public money, they should also have the same accountability (in terms of standards and reporting of results) as their public school counterparts.
ANALYSIS: It’s accurate that Obama opposed vouchers. He supported expanding charter schools but didn’t say that the competition they could provide would automatically spur public schools to improve.
EDUCATION WEEK:
What's your view on the Common Core standards?
BARACK OBAMA (AI BOT):
The idea behind Common Core was great—that we would set national education standards that were clear, and would be an objective way of measuring whether children were learning and meeting expectations. However, the implementation (especially of Common Core math) was botched. Common Core math was overly abstract and complex. It didn't help kids learn, and it created a lot of stress over something that should be relatively simple. It also created so much anxiety for educators, who have enough on their plates already.
ANALYSIS: This is a reflection of what Obama’s critics have said about Common Core, not what Obama himself said.
All content on Education Week's websites is protected by copyright. No part of this publication shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic or otherwise, without the written permission of the copyright holder. Readers may make up to 5 print copies of this publication at no cost for personal, non-commercial use, provided that each includes a full citation of the source. For additional print copies, or for permission for other uses of the content, visit www.edweek.org/help/reprints-photocopies-and-licensing-of-content or email reprints@educationweek.org and include information on how you would like to use the content. Want to seamlessly share more EdWeek content with your colleagues? Contact us today at pages.edweek.org/ew-for-districts-learn-more.html to learn about how group online subscriptions can complement professional learning in your district or organization.
