Researchers condemn shoddy methods, analysis of test data investigation



UPDATE (3/27/12): The Atlanta Journal-Constitution has removed any references to Two Rivers Middle School from the original story.



UPDATE (3/26/12): The Washington Post agrees with our objections

The Atlanta Journal Constitution (AJC) has hired a marketing firm to tout its investigative story on testing data from several large school systems despite being informed of numerous flaws in its methodology and analysis. The story is scheduled to appear in this Sunday’s edition and, while Metro Schools has not seen the story, the paper shared its data and methodology earlier this week in its first contact with the district. The information provided is questionable and shows a complete lack of knowledge or understanding of the system and students. Other school systems contacted have voiced similar protests of the data.



“We looked at the data provided, along with the methodology for their analysis, and immediately identified numerous flaws, as did an independent statistician from Vanderbilt,” said Dr. Paul Changas, executive director for Research, Assessment and Evaluation for Metro Schools. Dr. Changas was the first statistician hired by the State of Tennessee to assess student achievement results.



The AJC has based its analysis of school testing data from the nation’s largest school districts by “flagging” any movement in student achievement that is extremely high or low. Metro Schools’ has provided detailed information to the AJC outlining the serious flaws in the information provided to the district and in the methods they used to calculate their data. Some of those include:

• The AJC analysis assumes that students in one grade level at a school one year are the same students that were there the previous year in the previous grade.
  
  
  
  • Metro Schools has mobility rates between 35%-40%.
  
  
  • Zoning changes have impacted student enrollment.
  
  
  • A significant number of ALCs (Alternative Learning Centers) and special schools (including homebound) are flagged in the data; these schools often have very fluid populations - ALC populations have 100% annual turnover.
  
  
  
  


• This AJC information flags schools with unusually high numbers of discrepancies from predicted scores – both high and low.
  
  
  
  • 77% of the cases flagged in MNPS were due to low performance (significantly below predictions).
  
  
  • The negative scores we saw were not part of a pattern of high gain one year followed by significant loss the next (as would be expected if there were improper testing procedures).
  
  
  • There was not a single case of unusually high gains flagged for 2011.
  
  
  
  


• The AJC data we received had obvious errors, including children who were absent from testing being assigned a “zero” rather than being excluded from the analysis - resulting in average scores that were below the minimum score possible.


• Tennessee adopted much more difficult academic standards in 2010, resulting in a significant decrease in student test scores.


• Metro Schools has the highest population of English Learners in the state, with approximately 1 in 4 of our students coming from non-English Language Backgrounds. National data show EL students typically make among the largest gains from year to year as they learn the English language; several of the schools flagged for positive gains have high EL populations.


• The Tennessee Value Added Assessment System (TVAAS) uses much more sophisticated analytical techniques (Henderson’s mixed model methodology) and follows actual cohorts of students to analyze gains of teachers, schools, and districts and TVAAS results have typically been comparable to statewide trends for several years.

Read a detailed list of issues with AJC's methods

Other information provided to the AJC included information from Dr. Dale Ballou, associate professor of Public Policy and Education with Vanderbilt University’s Peabody College of Education, who has reviewed Metro Schools’ testing results for a number of years. Dr. Ballou has reviewed the data provided and found some statistically impossible anomalies. In addition, Dr. Ballou has been involved with an independent analysis of test gains of MNPS middle school math teachers over three years (2006-07 through 2008-09). This review, conducted by Dr. Brian Jacob of the University of Chicago on behalf of Vanderbilt University, did not show any unusual or suspicious patterns.

• The analysis was conducted as part of the Project on Incentives in Teaching (POINT) study, to determine whether tying significant financial incentives for teachers (up to $15,000 per teacher per year) to student achievement made an impact.


• The analysis conducted by Dr. Jacob was specifically to determine if evidence existed of invalid test scores. None was found.

“We are shocked and disappointed the Atlanta Journal Constitution has included our district in their investigative report to be released Sunday, despite these and a number of additional flaws in their analysis brought to their attention prior to its publication,” said Director of Schools Dr. Jesse Register. “Their lack of understanding of our schools and district is reflected in the data we have seen.”



Michael Casserly, executive director for the Council of the Great City Schools, a national organization representing large school districts, has also criticized the Atlanta Journal-Constitution’s approach, saying, “These are serious charges that should be supported by the strongest evidence. That strong evidence has not been provided.”

Read the full statement from the Council of Great City Schools