In the competition for $4 billion in Race to the Top grants, states have made their best pitches, a secret jury has debated and scored their applications—and now U.S. Secretary of Education Arne Duncan must decide who’s good enough to make it to the final round.
Mr. Duncan’s announcement of the Race to the Top Fund finalists, which is expected as early as this week, caps a dash by the Department of Education to recruit, vet, and train peer reviewers who wield tremendous power in determining who will win this high-stakes education reform competition.
In this first round of competition, all but 10 states applied for awards financed by the economic-stimulus package that will range from an estimated $350 million to $700 million—badly needed money that would help cash-strapped states.
Once the first-round awards are made in April, the department will publicize the scores and comments for each applicant from the 49 peer reviewers who have been given responsibility for judging 30 criteria that measure a state’s commitment to improving teacher quality, low-performing schools, data systems, and academic standards.
Despite all that influence, the ultimate power rests with Mr. Duncan, who will have the final say in awarding those cash prizes. But if the secretary deviates from the highest scores in awarding the Race to the Top grants, department officials have said he pledges to justify his reasons.
Each of 49 peer reviewers evaluated four or five state applications, which were randomly assigned. Comments and scores were due February 8, 2010.
IMAGES: iStockphoto.com/auras
SOURCES: U.S. Department of Education; Education Week
Mr. Duncan also has sole discretion in choosing how many finalists will come to Washington the week of March 15 to make presentations to the peer reviewers.
The purpose of the presentations, said Joanne Weiss, the department’s Race to the Top director, is not only to verify and clarify what’s in the applications, but also to ensure that those who must implement the plan know and understand what’s been promised.
“The real issue is, we’re giving away hundreds of millions of dollars, and we don’t want to just rely on what you wrote on a piece of paper,” Ms. Weiss said. “We want to look you in the eye.”
Looking for Conflicts
In August, the Education Department put out a call for peer reviewers to judge the applications and drew 1,500 applicants. They will pull from this pool for the second round of the competition, which starts in June when applications are due and ends in September, when the final dollar is awarded.
Vetting the list for potential conflicts of interest was no easy task. Any nominee who works for a state or district was automatically eliminated. So was anyone who helped out in any way on a state’s application, or who has a financial interest in the outcome of the competition.
Lawyers and ethics officers from the department did most of that work, although four career and four political staff members also reviewed the applicants to make sure they had expertise in the four school improvement areas emphasized in Race to the Top.
The department also hired a contractor to do a public-records search—even a Google search—into nominees to uncover information not previously disclosed. The department was also on the lookout for anyone who might have “significant identification with a specific pedagogical or philosophical viewpoint that might give an unfair advantage or disadvantage to an applicant,” according to department documents.
“It’s wasn’t like, ‘Do you have a belief?’ ” Ms. Weiss said. “It was more about, ‘Do you have an agenda?’ ”
Of the 49 peer reviewers, the department granted 12 waivers for “indirect” conflicts of interest, most commonly that a reviewer had worked for a state or school district within the past 12 months.
The reviewers receive $5,000 for what amounts to hundreds of hours of work that will span at least two months and involve tedious reading of applications that can exceed 1,000 pages each, counting appendices.
Though the department released basic demographic information on the reviewers—most, for example, have doctoral degrees and at least a dozen served on a state board of education or as a local or state school superintendent—federal officials have caught criticism for keeping their identities secret.
One reason the reviewers’ names should be made public is because of Mr. Duncan’s pledge to run a transparent competition, said Frederick M. Hess, the director of education policy studies for the American Enterprise Institute and a blogger hosted by Education Week. “We’re talking about an extraordinary sum of cash in play, and that’s why the identities of the judges are so important.”
In fact, the peer reviewers had to sign confidentiality agreements declaring they won’t disclose their identities or talk to the press—in effect, a gag order that applies until the department lifts it. Department officials have said they will make the identities public once the winners are announced, although it does not intend to reveal which reviewers scored which application.
Process So Far
In the course of the judging process, each peer reviewer received four or five applications to score; they were not allowed to judge applications from the states where they live. Then, the reviewers came together in Washington to form five-member panels to discuss and debate each application.
Since the reviewers rotate among panels, forming a unique one for each state, the department hopes to smooth out any scoring variances that might occur with different groups reviewing different applications.
Peer reviewers could refine their scores and comments based on their panel discussions, and the five scores were averaged to arrive at a final score. Ms. Weiss said any outliers whose grades were particularly low or high compared with the other panelists were not pressured to come to consensus.
Throughout the process, peer reviewers were instructed to do no independent research and to rely entirely on what’s written in the application.
Checks and balances were built into the system to fact-check the states’ applications. Ms. Weiss said, for example, that making the applications public, including the scores, comments, and video from the finalists’ presentations in Washington, will help ensure accountability.
Although the department has established a detailed, 500-point scoring guide, peer reviewers’ judgment will still ultimately come into play. For example, a peer reviewer will have to decide how to weigh an overly ambitious application, with a price tag that far exceeds the department’s nonbinding award estimates, against a less ambitious plan that stays within those estimates. “It’s entirely the peer reviewers’ judgment,” Ms. Weiss said. “You will see the reviewers made comments such as, ‘This is underfunded. Or this is overfunded.’ ”
Wild Card
Such judgment is the wild card in the competition, said Michael J. Petrilli, a vice president of the Thomas B. Fordham Institute, a think tank in Washington.
“What this comes down to is people. And invariably there are going to be judgment calls being made even as prescriptive as the criteria are,” said Mr. Petrilli, who worked with some peer-reviewed grant programs while at the department under President George W. Bush. “That’s when policy leanings are going to come into play.”
Still, the peer-review system was probably the best choice for the department in judging Race to the Top, said Mark Schneider, the vice president for education, human development, and the workforce program at the American Institutes for Research.
“Peer review is probably the best system we’ve come up with for judging products on merit,” said Mr. Schneider, a former commissioner of the National Center for Education Statistics. “If it’s done right, many people, especially losers, are unhappy but it does have a justification and a long history of support.”
There’s been less clarity to how Secretary Duncan and his staff will decide how many finalists will be selected—and how many awards will be given out.
The department has said that the applications will be ranked in order by score, with the top scorers getting money, until the money runs out. And Mr. Duncan has promised to be tough in setting a high bar for states to win.
Ms. Weiss said she couldn’t talk specifically about the process the department would use in determining the number of finalists or awards. But factors will include whether there are “natural breaks” that separate the superior applications from the not-so-good—and how much money is involved in awarding the grants.
One example: If a few big states with large grants are top scorers, there might be fewer finalists and fewer winners in the first round because the department has promised to leave a sizable pot of money for Phase 2.
And what will the losers get?
The same feedback as the winners, an average of seven to 10 pages of comments from each reviewer, which will serve as a roadmap for Phase 2.
