Dozens of schools across the city and state were flagged in a study of 2009 state standardized test scores that sought to use statistical analysis to ferret out possible examples of cheating on the PSSA exam.
The analysis, prepared for the Pennsylvania Department of Education in July 2009, highlights roughly 60 schools with suspicious results due to multiple statistical irregularities, including 22 Philadelphia district schools and seven Philadelphia charters.
Among the Philadelphia district schools referenced in the report is Roosevelt Middle School, which has been at the center of a controversy this year involving alleged cheating on the PSSA. In 2009, the analysis reveals, results of both the reading and math PSSA exams taken by Roosevelt’s 7th and 8th graders showed a highly unlikely number of wrong answers that were erased and changed to the correct answer. The results also showed highly improbable increases over the previous year in the percentage of students who scored proficient or advanced.
For example, the odds that the wrong-to-right erasure patterns that showed up on Roosevelt’s 7th grade reading response sheets occurred purely by chance were slightly less than 1 in 100 trillion.
The “data forensics technical report,” prepared by the Data Recognition Corporation, was made available to the Notebook by the state. At the Notebook‘s request, Andrew Porter, dean of the Graduate School of Education at the University of Pennsylvania and an expert on testing, reviewed the report and some of its data.
Porter stressed that that statistical analysis alone, without witnesses or confessions, cannot definitively prove that there was cheating. But he added that the report “describes a reasonable approach to identifying schools where there may have been cheating.”
Nevertheless, it appears that the state never followed up with any further investigations. The forensic data analysis was discontinued in 2010, although state Department of Education spokesperson Timothy Eller said it would be reinstated this year.
“We’re going to reinstitute those reports this year and they are actually going to mean something,” said Eller. He criticized the 2009 report as “convoluted.”
Earlier this year, teachers from Roosevelt and FitzSimons High told the Inquirer that they had witnessed numerous instances of cheating at their schools in 2010 and 2011. The Roosevelt teachers also spoke to the Notebook. FitzSimons was not one of the schools flagged in the state report for suspicious results in 2009.
Eller acknowledged that the state Department of Education has received two complaints from Philadelphia about possible cheating and said the department had ordered the district to investigate. Earlier this week, district officials said they found no wrongdoing at Roosevelt or FitzSimons. On Friday, district spokesperson Shana Kemp told the Inquirer that the district is investigating 15 schools and would wait until all are complete before submitting the reports to the state.
Eller sought to minimize the relevance of the 2009 report, pointing out that it was done under the previous administration of Gov. Ed Rendell and saying that “it is difficult to glean anything from that information.”
Eller stopped short, however, of saying it was wrong, emphasizing rather that it had not been presented in comprehensible form.
“It’s not information the average person would understand,” he said.
The report and accompanying analysis cost the state $113,000.
A DRC spokespersonsaid that the company could not answer any questions about the significance or context of report’s findings without state Department of Education’s permission, which was not granted.
Prominent in the DRC report is an analysis of erasures on the state tests. The study says that schools flagged for irregular erasure patterns are “of particular interest” because “these results may strongly suggest that a testing irregularity occurred in the school.”
At some Philadelphia schools, erasure patterns were flagged across numerous tested grades and subjects. For example, Olney Elementary, a K-8 school, was flagged for its erasure patterns in both reading and math in every tested grade at the school.
But the study looked at other possible evidence of test cheating as well.
A Notebook review showed roughly 60 schools across Pennsylvania received three or more “flags” in a single grade for improbable jumps in students’ performance levels and unlikely increases in schools’ scale scores on reading and math tests, as well as unusual erasure patterns. Nearly half of those schools singled out for having particularly suspicious results were in Philadelphia: 22 district schools and six seven Philadelphia charter schools had “flags” on three or more measures in a single grade.
“Multiple flags mean that the school data were highly unusual on more than one indicator,” said Penn’s Porter. “There are many ways to cheat, so it’s not wise to look at the data in only one way.”
Six of the 22 district schools had more than three “flags” for more than one grade.
Wagner Middle School in West Oak Lane, for example, was flagged in the report six times for its 7th grade results and three times for its 8th grade results. Overall, the likelihood of the erasure patterns on the school’s 8th grade response sheets in reading and math occurred purely by chance was more than 1 in 100 trillion.
From 2007-08 to 2008-09, the percentage of Wagner 8th graders scoring proficient on the PSSA exam jumped almost 14 percentage points, from 56.5 percent to 69.3 percent.
Statewide, 22 high schools had three or more flags. Three of those were Philadelphia district high schools, including Northeast High, which was flagged on every analyzed dimension of its 11th grade math results. Three local charter high schools were also flagged, including Imhotep Institute Charter High School, which was flagged seven of a possible eight times, including for a wrong-to-right erasure rate on its 11th grade math results that had a one in 10,000,000,000,000,000,000,000,000,000,000,000,000,000 chance of occurring purely by chance.
Questions about test score cheating in Philadelphia come on the heels of the district’s recent announcement that Philadelphia public schools improved their test scores for the ninth straight year in 2010-11. Over the past several months there have been several cheating scandals across the country.
Earlier this year, USA Today published a major investigation analyzing several years of test scores in six states and the District of Columbia that found statistical evidence of potential cheating. The series prompted a probe in Washington, D.C., where unusually high erasure rates were found in more than half of the schools.
In Atlanta, the state ordered an investigation into cheating after years of reporting by the Atlanta Journal-Constitution on similar unusual statistical patterns and other testing irregularities. The results, released this week, confirmed cheating in 44 of the 56 schools state investigators examined. Based on more than 2000 interviews, it named 178 educators as engaging in unethical behavior, said 80 of them had confessed, and concluded that the administration of former superintendent Beverly Hall had ignored or covered up evidence. Hall issued a statement Friday apologizing for any “shortcomings.”
A focus in both of those scandals was on suspicious erasure patterns detected by testing companies on student response sheets.
Eller, of the state Department of Education, said that new Pennsylvania Secretary of Education Ron Tomalis, appointed by Republican Gov. Tom Corbett earlier this year, has directed that forensic analysis and security audits be reinstated for the current year.
Eller gave no timetable for the completion of these reports. The PSSAs were administered in March.
To some, the need for more scrutiny of test score results grows in tandem with the intense pressure for schools to do well on standardized tests.
“As a general rule, the higher the stakes for high test performance, the more incentive to cheat, [and] the more cheating will be done,” Porter said.
