A Practical and Profound Look at Testing and Assessment

Instruction That Measures Up: Successful Teaching in the Age of Accountability
by W. James Popham
(ASCD, 2009)

Reviewed by Kenneth J. Bernstein, NBCT
High School Government & Social Studies (MD)



Teacher Leaders Network

This is a book by one of America’s acknowledged experts on assessment: now emeritus from UCLA, Popham has been a leading figure in research (having served as President of the American Educational Research Association), in publication (of his many books and articles, and as editor of a major journal on evaluation), and as a person whose opinions on matters educational are always worth considering. For those interested, you may read a professional bio here.

Popham has in recent years been critical of how our educational policies have approached the matter of tests, assessment and evaluation. This book therefore may catch some off-base, because Popham now moves beyond criticism to try to help those in the classroom deal with the reality of test-based accountability, something to which the current national administration has made clear its commitment.

The purpose of Instruction That Measures Up can be clearly understood from one paragraph in the preface, which appears on p. 2:

I believe the best way for teachers to deal with test-based accountability pressures — the way that benefits students — is to accept those pressures as a given, then plan and carry out instruction knowing that it will take place on an accountability-spotlighted stage. What teachers must do is focus on providing instruction that measures up: to the expectations of administrators, parents, and taxpayers; to their own professional standards; and, most essentially, to the needs of their students.

The book is divided into 7 chapters:

1. Teaching Through an Assessment Lens
2. A Quick Dip in the Assessment Pool
3. Curriculum Determination
4. Instructional Design
5. Monitoring Instruction and Learning
6. Evaluating Instruction
7. Playing the New Game

There is also a two page list of resources, an index, and some background on the author, with a total of 174 pages.

For those who are not all that knowledgeable about matters of assessment — which not only includes many of those in the classroom, but far too great a percentage of those involved with making educational policy — the second chapter by itself justifies the book. Popham divides the “Assessment Pool” into four broad categories: Testing as Score-Based Inference Making; The Core Concepts of Assessment; The Categories of Educational Tests; and The Summative and Formative Functions of Educational Assessments.

He provides clear explanations of the meaning of terms. Where necessary, he offers a great deal of detail with easy-to-comprehend explanations. A reader who pays attention will begin to grasp the importance of how psychometricians understand key terms.

The four core concepts Popham addresses are the key ideas of Reliability, Validity, Assessment Bias, and Instructional Sensitivity. Any assessment that fails to take into account these core concepts, whether it is designed by a classroom teacher for instructional purposes or by outside organizations and imposed from above for purposes of accountability, will by definition be at risk of being unable to provide sufficiently accurate information to allow one to draw appropriate inferences from the data.

Popham offers a number of blunt statements about problems with our current schemes of assessment, and a number of key warnings, of which one on p. 29 caught my attention: “It serves no one to ascribe unwarranted precision to educational tests.” Unfortunately, our national obsession with numbers and our desire to rank and compare means that it is precisely ascribing too much precision to the data we obtain from tests that has been distorting much of our educational policy for the past several decades.

Each of the chapters has some important material at the end. We have a “Chapter Check-Back” in which key concepts are repeated in summary form, as well as a list of several “Suggestions for Further Reading” on the topic of the chapter. The chapter on Monitoring Instruction and Learning has four key concepts, among which is this:

Feedback to students is most effective when it is task focused, directive, timely and simple; it is least effective when it comes in the form of grades. (p. 125)

Among the suggested readings are works by notable names such as Rick Stiggins, Robert Marzano, Grant Wiggins and Jay McTighe, and Popham himself, as well as some valuable works by lesser known lights. Each suggestion is accompanied by a brief explanation by Popham as to why the work is included: for example, about the address offered by D. A. Frisbee as outgoing president of the National Council of Educational Measurement, Popham tells us

Frisbee lays out a set of basics in educational assessment – concepts that he feels have been distorted in recent years. The article gives teachers a list of important misconceptions to avoid. (P. 51, italics in original).

Many of the chapters also contain political cartoons. Through these Popham pokes fun at a lot of rhetoric commonly encountered in current discussion on educational policy. The final cartoon on p. 160 portrays a pre-test pep rally, with a sign in the background reading:

Tomorrow’s Accountability Test
-- Cost teachers their jobs.
-- Close our school.
-- Destroy your future!

The speaker, apparently the principal, is urging the assembled to “Try harder, and harder, and harder!” while one teacher in the audience comments: I can see why Confucius said, “High stakes are for string-beans.”

Popham is for PROPER use of assessment. He quotes three sentences from Dylan Williams of Britain. The middle sentence reads:

 It is only through assessment that we can find out whether what has been taught has been learned.

That, Popham says, is “one I’d like to see in neon lights above the entrance to every school in the world.” (p. 101)

This is a book Popham intends to be of practical use to teachers. One may not agree with all his formulations -- this reader had some questions about the approach Popham offers as the structure of an effective lesson. Nevertheless there is a great deal of insight and practical advice. If nothing else, readers should come away with a deepened understanding of the terminology, and of the appropriate uses and inappropriate misuses of assessment of various kinds.

Before some final remarks, allow me to share a number of very brief selections which will give you a real sense of Popham and his approach:

...rarely can anyone look at a planned instructional activity and say for certain that it’s going to be effective. (p. 18)

Still, few educators, though seemingly awash in an ocean of test-based accountability, currently recognize how few accountability tests are even mildly sensitive to the quality of a teacher’s instruction. (p. 38)

It is far better for students to master a modest number of truly potent, large-grain curricular aims than it is for them to superficially touch on a galaxy of smaller-grain curricular aims. (p. 61)

Teachers must always concern themselves with what’s best for their students. (p. 70)


Remember, instructionally insensitive accountability tests are essentially insensitive to instruction, meaning what a teacher emphasizes in class is probably not going to make a substantial difference in students’ scores on an instructionally insensitive accountability test. (p. 71)

Self-reflection is a teachers’ ally. (p. 114)

...it is fundamentally wrongheaded to try to use a test to help students monitor their own learning while, at the same time, using the results of that test to grade or rank those students. (p. 119)

...formative assessment’s focus is on getting students to learn, not outperform other students. (p. 120)

...although many of those earlier researchers set out on a quest for a silver bullet that would permit the definitive appraisal of a teacher’s competence, such a bullet was never found. IT still hasn’t been.

The insuperable obstacle to the creation of a sure-fire, cookie-cutter approach to teacher evaluation is teaching’s profound particularism. (p. 145)

And one final quotation, that may help summarize Popham’s thinking:

  So, as someone who’s been dipping in and out of the teacher evaluation research literature for more than 50 years, I’ve come to a conclusion about the only truly defensible way to evaluate a teacher’s skill. Because of the inherent particularism enveloping a teacher’s endeavors, I believe the evaluation of teaching must fundamentally rest on the professional judgment of well-trained colleagues. (p. 146)

Ultimately this is a book about teaching. It is presented through the lens of a deep understanding of what assessment and evaluation can contribute to the improvement of teaching practice, as well as some serious cautions, offered throughout the work, about the dangers of pushing the instruments we have beyond the limits of the valid information they can provide us. Or rather, the reliable information from which we are able to draw valid -- even if often limited -- inferences.

I come away from the book agreeing with Popham that teachers should insist on getting a better grounding in assessment and evaluation — which is about far more than testing but which should thoroughly cover matters of testing -- as part of their professional development. It’s best if it is part of teacher preparation, but not too late as a part of continuation for those already in the classroom.

This is a valuable book. It is for and about teachers, but can be profitably read by anyone interested in improving teaching by the proper understanding and application of assessment. That should mean everyone, for we are all affected by educational policy, even if only through decisions made on how to spend the taxes we all pay.

I highly recommend this book.

Kenneth J. Bernstein is a National Board-certified teacher of social studies at Eleanor Roosevelt High School Eleanor Roosevelt High School in Greenbelt, Md., and a member of the Teacher Leaders Network. He is nationally known as a blogger on education and other issues under his online name of teacherken. Bernstein is also a 2010 recipient of The Washington Post’s Agnes Meyer Outstanding Teacher Award.