Drowning in Data; Thirsty for Information

A group of teachers are in the middle of the desert, dying of thirst.  What they need more than anything else is a drink of water.  It’s a simple request – just a drink of water.  On the horizon, they see three large caravans, each containing large numbers of people and animals, a certain sign that help is on the way.  Surely at least one of the people in one of the caravans can offer them a drink.

The first caravan looks odd – people in suits and wire-rimmed glasses riding on camels, bulging briefcases labeled “water” at their side.  As they pass by, they empty the contents of their suitcases – pages and pages of spreadsheets and PowerPoint diagrams about water.

The second caravan appears to be from the circus, with lions, elephants, and clowns.  At last the ringmaster appears, delivering an inspirational speech and heart-rending anecdotes about the value of water.

Finally, the third caravan arrives – surely this one must include a life-saving drink.– But hope is quickly displaced by despair as the dour-looking crew announces in stentorian tones new regulations about the professional responsibility of teachers to use water more wisely, and threatening them with consequences should they fail to comply with these regulations that the crew is setting forth.

I don’t know how this story ends, but based on what I’ve been hearing from schools around the world, I wouldn’t recommend that the thirsty teachers in our story wait around for the fourth caravan.  Schools, teachers, students, leaders, and policymakers remain thirsty for information, even as an increasing tidal wave of data washes over them.  Here are three ideas to help find order among the chaos.

First, start with questions, not theoretical capabilities.  Data salesmen are fond of claiming “This system can do everything!”  But teachers and school leaders are not interested in “everything”. Rather, we need the answers to very specific questions.  For example:

1. What is last year’s profile of a student who received a D or an F in 9th grade math or English this year?
2. Which 2nd grade teachers last year produced the highest proportion of students who are proficient readers in 3rd grade this year?
3. What is the profile of students who received honor roll grades last year, but who were not proficient on external exams this year?
4. What were the specific curriculum, assessment, and instructional strategies used by teachers whose students had the best achievement gains?

That’s a start.  The answers to these questions provide information that teachers and administrators can really use.  Capabilities of data systems are not relevant unless their salespeople and advocates can address these real-world questions.  The fourth question in particular is of interest, because virtually no data systems, despite their multi-million dollar expense, even attempt to answer it.  We focus as a nation on effects, not on causes.  It is as if we diverted the entire budget of the National Institutes of Health on weighing the student population, but never inquired as to the diet and exercise habits of those children.  What are the big questions that you want to ask?  Don’t spend money – yours or the taxpayers’ – until you consider those questions.

Second, focus on the classroom.  Many data systems are designed to work splendidly on the desk of the Director of Technology, a very capable person who also has the latest, best, and fastest computer in the school system.– But most of the time the Data Utility Point is Ms. Washington’s 3rd grade class – not the Director of Technology.  If Ms. Washington is endure yet another lecture about the value of data-driven decisions, then she should exchange desktop computers with the Director of Technology.  As this is unlikely to happen, vendors should be required to demonstrate the value of their wares in the classroom, not just in a school’s central office.

Third, ask “So what?” In our studies of thousands of schools that claim to use data, collaboration, and learning communities, we have noticed that it is far more common for schools to view data than to use data.  While colorful charts and graphs adorn the walls of schools and board rooms, the unanswered question remains, “So what?”  The next time you encounter claims of data use, ask these questions:

1. How will next year’s schedule be different?  We have abundant data showing that some kids need more time than others.  So what?
2. How will next year's instruction be different?  We have abundant data showing that some instructional practices are more effective than others.  So what?
3. How will next year’s leadership be different?  The data show unequivocally that some leadership practices are more likely to improve educational equity and excellence than others.  So what?

While we ponder these questions, the teachers in the desert are still thirsty, still waiting for a caravan that offers more than data, platitudes, and rules.