Your Data Dream. My Data Nightmare.

One of my greatest frustrations as a teacher is listening to those who work beyond the classroom dream up brilliant plans for reforming schools that are simply impossible to implement because they’re dependent on new resources that no one is willing to provide. 

Take the constant drumbeat around “data informed decision-making” as a case in point. 

Poke your nose into any conversation about teaching and learning and you’re bound to hear “how easy it would be to improve education” in our country if schools were just willing to “confront the brutal facts” about student achievement.

The best teachers and learning teams are “hungry for facts,” prognosticators will argue.  They take an “action orientation” towards numbers, looking for “trends and patterns” that can inform their instruction. 

And of course, someone is BOUND to throw out the ever-popular argument that our schools need to take a more “business-like approach to data.”

Now don’t get me wrong:  I’m down with the idea that using data to identify and amplify effective instructional practices can change our schools.  Need proof? 

Check out the work my team has done to break down our curriculum into measurable pieces or the work that I’ve done with spreadsheets and pivot tables.  Heck, I even wrote a chapter on using data to drive decisions for both teachers and principals.

My beef is a simple one, though:  Teachers and teams in most schools are asked to do this complex work with nothing more than the most rudimentary paper-based tools.

Data notebooks have become THE fashionable accessory in professional learning team meetings.  Otherwise known as three-ring binders, data notebooks require teachers to record student learning results—quiz scores, classroom observations, project performance, assessment results—by hand and then to look for patterns in the piles of papers that they collect.

When was the last time that you carefully flipped through a three-ring binder trying to learn?  Right.  High School.  That’s where the binder died.  You probably haven’t touched one since.

There’s a reason for that:  They don’t work.

Exit slips are almost as popular as data notebooks.  Otherwise known as sticky notes, students answer review questions on small slips of paper left with the teacher at the end of each class period. Teachers then sort through stacks of responses during their planning periods, often writing down results on the class rosters stored in their data notebooks!

Think about how useless data notebooks and exit slips really are:  Neither tool allows student responses to be recorded automatically, creating yet another overwhelming clerical task for teachers. 

Neither allow data to be sorted or studied systematically.  Data queries can’t be developed to test teacher hypotheses about student learning on their teams.  Sophisticated manipulation of information depends on determined teachers willing to work through data no matter how inefficient the process is. 

And neither allow data to be easily reported to anyone, making it difficult at best to get parents and other professionals—guidance counselors, instructional resource teachers, administrators at the school or district level—involved in making those data-informed decisions that everyone thinks so highly of. 

Imagine the consequences if we translated these data management and evaluation procedures to large corporations like Wal-Mart, whose data expertise has been well-documented in the last decade.

• Instead of automatically tracking purchases by demographic category and geographic region, cashiers would keep a clipboard next to their registers and make tallies after each sale.
• Instead of monitoring inventory through shared databases accessed by suppliers, warehouse managers would place new orders after counting each item, going through cashier tally sheets, and calling dozens of distributors directly.
• Instead of using GPS technology to maximize the efficiency of the routes taken by delivery trucks, drivers would be given road atlases and sent on their way.
• And instead of using radio frequency identification microchips to digitally monitor the expiration dates and storage conditions of grocery products, stock boys would carry notepads, manually recording lists of items in need of quick sale or replacement.

Digital tools HAVE transformed data management and decision-making in corporate America—and reformers ALWAYS argue that schools should function more like businesses—yet teachers are expected to analyze the results of their work using nothing more than binders and sticky notes.

How’s that for an epic fail?

Comments

[Joe]

Not to mention the implied logic of schools being more like corporate America. How's that working out for us these days?

I'm so far past all this "data-driven" nonsense. Nowhere in this whole post is there a conversation about what we're actually having the kids learn and what they're actually doing with this knowledge to improve their own lives and that of their community?

Increasing the worship of data and technology and blah blah blah just serves to increase the busy-work of teachers and to take away from authentic learning experiences. Funny how that's often left out of data-driven decision-making.

I'm all for assessment, but what are we assessing here? Whether we can make kids into widgets?

[Jose]

As someone who's been given the onus of helping people interpret and organize data at my school, I believe you're 100% correct. Your assertions about data binders and sticky notes say lots about how everyone views the profession we're in. For instance, let's say in the business world, they have this "exit slip" business. I can imagine that when they do, they have someone working full-time to aggregate all those nots and making them into a substantive report. Teachers are going to work full-time grading papers, writing the lesson plans, going to each and every meeting, AND adding another thing to their plate? I'm not saying it's not a valid strategy for feedback, but priorities priorities.

And another thing about those assessment binders is that, as with most reports like that, in the classroom, these documents are static by their very nature. In the middle of class, you're not gonna pull it out, look at a kid, and say, "Oh well, because you're a level 3, then you need to be here." Rather, teachers use a variety of factors, and the best ones can evaluate in an instant (of course, with a little pre-planning but always with some acute thought).

I strongly believe in data, like you. I just don't think it's the end-all-be-all.

[Matt Townsley]

Epic fail continued....
Teachers continue to use exit slips and other derivatives of paper data collection methods because they WANT to collect the data (rather than being told to do so) but DON'T HAVE ACCESS to the digital tools necessary to automate the process.

#schoolfundingfail

[Brian Crosby]

Wow, this post really hits a sore spot with me. You nailed it. I will add that we are told what data to collect, much of which is poor, has little to nothing to do with making good decisions about instruction and so we supplement with our own data pieces that adds to the avalanche that we keep in our binder. It has gotten so bad where I teach that we were asked in a meeting to come up with a way to use the data from our benchmark tests that are useless ... so we took valuable time and tried to think of a use for really poor data that we are required to collect. Major fail.

[Bill Ferriter]

Hey Brian,

In some ways, I hate to hit a sore spot with you...but in other ways, I'm jazzed. It means that I'm not the only one tired of dancing to the beat of the ignorant data drums.

Our learning team has struggled with district wide benchmark tests. We like using them because they're already created and they're automatically scored.

The problem is that we're not totally convinced that the tests align with the state-wide assessments and we're certain that they don't ask questions that are as difficult as those that our kids see on the state tests.

So back to your conundrum, right?

We can use really poor data tools efficiently to collect numbers, but those numbers don't make us any more efficient as teachers.

Arrgghhh.
Bill

[Zinnia]

Another great post, Bill. You seem to be able to take a step back and explicate very clearly how schools are evolving into a Kafkaesque nightmare for educators.

At my school, we have advanced to using Excel spreadsheets. Whoopee. But sticky notes are too expensive to buy for student use, so we use index cards for something like exit cards.

[Dave]

Walmart has the benefit that they can build and program software for an overall system and implement it at thousands of stores. Obviously, solo schools have a much harder time when they can't enjoy such economies of scale.

If the US dept. of education would step up and build high-quality tools for free use by schools, it would increase productivity, reduce clerical overhead time, and free up millions of dollars for salaries and classroom expenses.

Why is every school in the country paying for grade-keeping software, when a standardized program could work for 99% of us?

[Matt]

" . . . yet teachers are expected to analyze the results of their work using nothing more than binders and sticky notes."

One issue I'm seeing is that too many educators are intimidated by learning how to use existing tools. What good is it to develop new programs or other data management technology when a teacher is afraid to learn how to use Excel in place of a 3-ring binder?

[Bill Ferriter]

Matt asked:

What good is it to develop new programs or other data management technology when a teacher is afraid to learn how to use Excel in place of a 3-ring binder?

My answer: We're not afraid to use Excel. Look:

http://teacherleaders.typepad.com/the_tempered_radical/2008/12/using-spreadsheets-and-pivot-tables-to-track-data-.html

The hitch is this: I spent the better part of 3 full days trying to pull this all together. And that was only because I was motivated to do so.

No one provided me with professional development to figure this out---In fact, if I would have asked to go to an Excel training, I would have been denied because we have no money for professional development.

And that's the problem: Until we work systematically to provide professional development in the use of tools like Excel, their application will remain isolated to the classrooms of teachers who have the time to explore on their own.

Bill

[Matt]

Sorry - I didn't mean to imply you were afraid to learn Excel (or other existing tools). Only that too many are and until they can get past that - through training or funding or personal motivation - then they'll use the tools they are most comfortable with. Adding more technology or tools (without addressing the need to learn how to use what they do have) doesn't make much sense to me.

[Renee]

Heh. Get this. As my school was preparing to build our Data Folders (not even a binder, but a folder with prongs), we were TOLD to hand write everything on pre-printed forms, even though the program we use will export our data from the site to an excel file.

Everyone was so impressed when I showed up with a spreadsheet. How did you DO that?

Business has been doing this kind of data consolidation and aggregation by computer for decades. It often isn't the teachers who can't use the software, it is the upper management at the district level who can't or won't upgrade.

[Bill Ferriter]

Matt wrote:

Adding more technology or tools (without addressing the need to learn how to use what they do have) doesn't make much sense to me.

This is where our thoughts align, Matt. Long and short of it: There are teachers---and school leaders---who aren't comfortable with data no matter what tool they've got to interpret it with.

Until that attitude---and aptitude (or lack thereof)---changes, we won't get very far.

I'm still wondering, though, whether it's possible for schools to completely skip over the Excel-type-data-tools and leap directly to more advanced applications that exist today. Kind of like Africans skipping landlines completely.

Anyway...
Bill

[Joanne Lockwood White]

Let's see .....from 3:00 yesterday until 8:00 this morning, I washed 120 test tubes, created a focused current events article question-sheet, created 3 hands-on learning centers, maintained the terrariums, weeded the school garden, read all the pbwikis, graded papers, shopped for soil for the garden, and made a phone call for an upcoming field trip. I could, instead, analyze data. I KNOW where weaknesses are and I know how to inspire kids and get the job done. I cant do it when I am trapped under mountains of onerous (sp?)
, absurd calculations, tally marks and anecdotes.

[coach]

I have to second Joanne. Metrics are a big buzzword at my org, but we use them mostly as a rear-view mirror. It's a feather in management's cap, but what does it all mean w/out a strategic direction? Time I could spend researching & developing content is spent in faux analysis of mountains of data.

[John]

It is always unwise to speak in generalizations. In my district we have an expensive data warehouse where State Assessments as well as district/school/classroom data can be analyzed very easily. After 4 years of providing PD to secondary teachers in my district less than 20% are willing to look at the data generated by their students from current or previous years. Many teachers are assessing, grading, and then forgetting.

[Supervidoqo.blogspot.com]

The thing to remember too is that the whole impetus driving this is concentrations of failure; i.e. poor schools. So in schools where student improvement is most crucial, it is going to be the most difficult to implement; there are just so many more pulls on the teacher's time. Whether it is dealing with discipline issues, calling home, doing more fundraising or working with limited resources - everything just consumes more time.

The greatest irony to me, in sitting through mandated data-collection meetings that sometimes stretched on for hours, was that it was actively taking up time that could have been spent actually looking at *real* data! Troubled schools simply need more resources - smaller classes, more aids, more resources period.

If we are really going to try and effect change based on targeted populations of low achievement, we need to provide the resources to actually do it. Half-measures like these that sound good on paper, but aren't given the support required, end up having a net negative effect because they are impossible to adequately implement.

[Sandy]

I just retired from a progressive district that had online benchmark tests where data could be aggregated in different ways and used for reinstruction. The problen is that data entry is just one of the tasks that teachers must add to their busy days. The data entry and analysis must always be done by teachers on their own time. I was recently listening to a HCR debate where someone commented that doctors wouldn't want to spend their time learning online systems and entering the patient data. Teachers have always been expected to do this, without additional help or compensation.

[Rachel Norton]

Thanks for this post. As a school board member, I appreciate the reminder that irrational exuberance about data-driven teaching should be accompanied by appropriate investments in technology for teachers!

[KJ]

My beef with this push for data collection is that, having graduated from college slightly over a year ago, i had less than one full course of data aggregation/analysis. I learned to work with database tools and Excel, however, the data collection and analysis end was totally nonexistent. In the future, we will be required to collect high-quality data and know what to do with it, especially if teaching is to finish its journey from blue-collar career to organized, fully-respected (and well paid?) profession. I am hungry for good data on what my students know and need to know- but I just don't have the skills (and tools!) to collect and organize good data. Also, in a large business, it is usually up to an intern or temp to do data-entry, however, in a school the teachers are the ones entering data. It would be so much easier if i could produce the data and then give it to someone else to enter into a computer system to be analyzed!

[Catina]

Now, I'm faced with data, data, data, data--constant required common testing, every core subject, every 4.5 weeks! Constant testing is not the answer--assessment need not always be a test! Furthermore, at some point, some decisions for USING the data must happen! I'm sure you agree. Read my rantings here: http://cchapmandcps.edublogs.org/2010/05/11/data-data-data/

[Deirdre]

Interesting argument. I agree that our current methods of data collection are a bit outdated and it would be nice to have some digital tools to assist in this process. Not to mention the time it would save scoring exit slips, marking data, the graphing the data. If we had tools to do this electronically, we could be quicker to see where are students are struggling. Thus, we would be faster when it comes to taking action for these students. I love the idea of using digital technology for this process but, in my school, we would probably purchase tools like this and never be trained properly on how to use them. That has been a struggle for me in this first year of teaching. We have some data tools that generate graphs and charts for individuals, classes, or even grade levels. Problem is, I was given a brief, 30min, optional seminar on how to use the program. For me, that was enough to get me started and I learned more about it as I continued to use it. But, much of the staff in my district is older and not as computer literate as my generation. Therefore, many of them would have a hard time using such tools given a brief "optional" introduction. So, although I think we should have better ways to view and use our data, I feel many schools would fail to properly use such a system in the school unless they are lucky enough to have a person designated to entering data into the system and calculating results for meetings.