Well.

I’m not one to take a book like Doug Lemov’s on faith.  I loved the ideas, and I loved the techniques, but I also wasn’t completely convinced that his work was truly useful to everybody.  And Paul Bambrick-Santoyo asserted even more radically in his book that the only way to close the achievement gap was to create and then dig deep into the data about individual students and their responses to individually crafted questions.

So I did what any good scientist would do.  I conducted a test, in two parts. The first part was that I taught for two weeks, very consciously using Doug Lemov’s techniques, in the first part of May.  I used the techniques with two classes of ninth graders, who were graduating anyway, and thus avoided the risk of accidentally ‘contaminating’ a class with bad ideas.  I then spent the last two weeks of school “doing what I usually do” and being much less strict about using Lemov’s techniques.

The second part was that I spent all of May constructing a rubric for the essay on the final examination, based on the Epic of Gilgamesh.  I also wrote 50 multiple choice questions, designing each question with a best answer, a second-best answer, an incomplete answer, and a wrong answer.  This was incredibly difficult, and I have a lot of kudos for the test designers who do this well. I also thought carefully about each question, and how it related to the skills I was trying to teach: some represented inference, some tested reading comprehension.  Others tested whether students were doing their nightly reading, and whether they retained information they read outside of homework, or only material that they reviewed in class.  And finally, I sorted the questions into two categories — material taught while using the Lemov techniques (admittedly as a beginner) and material taught while not using the Lemov techniques.

The Lemov work suffered from two different disadvantages.  First, I was teaching using these techniques for the first time, after doing nothing more serious than reading the book and composing a list of the 10 techniques that seemed I would get the most milage out of them.  That list rode in my pocket the first two weeks of May.  Then I threw it away, and just did things my way.  The second disadavantage that the Lemov material operated under was that I did the Lemov way first, and then returned to my usual teaching style, and never returned to that material again — so students had two weeks to forget that material, and no chance to review it except independently.

The group was pretty small: Two classes with a total of 18 kids, one group labeled as ‘learning challenged’ in a variety of ways, the other high-functioning but reluctant readers and writers.  Of course the 11 kids in the less-supported group did better; of course the 7 kids in the more-supported section had more difficulty.

After the examination, I plugged the answer to every child’s multiple choice sheet into a spreadsheet that listed ‘best’, ‘second-best’, ‘inadequate’ and ‘wrong’ answers to every question.  So I could SEE which kids were answering which questions, and get complete data on what the kids were doing right, what they were doing almost-right, and what they were doing outright-wrong.

The results were staggeringly obvious.  I almost fell over in shock.

On almost all of the questions in the first half of the multiple choice examination, every student picked the best or second best answer.  No one picked the wrong answer, and there were only four ‘inadequate’ answers.   Doing things my usual way, though, resulted in incredibly low scores across the board.  The number of ‘inadequate’ and ‘second-best’ answers surged.  All of the no-response-givens were in the second half of the test, and most of them in the last quarter of the test — the material most recently studied.

The best retention of material was in the earliest material given (but also the material most carefully delivered).  Moreover, I learned that most of my students didn’t do the reading. They retained some of what they learned from class review of material given in homework, but showed (in aggregate — some exceptional individuals ‘got it’) no deep understanding of text they had read for homework but not reviewed in class.

The essays were even more revealing, once I had a clear set of guidelines about what I was looking for.  (I didn’t share this rubric with the students; I wanted to understand my own biases here, not penalize them… those who wrote the minimum five paragraphs got full credit, regardless of what they wrote or how well). I learned that some kids bought into the online research model, and some didn’t.  Some bought into the idea that Gilgamesh and Enkidu’s slaughter of Humbaba is representative of humanity’s quest for raw materials from Nature.  Some didn’t.

All of them wrote clearly and consciously and competently about those portions of Gilgamesh which they had learned while I used Lemov’s techniques in the classroom and in assessing work.  They had grasped and absorbed that material thoroughly.  That material dealing with the current events of the Deepwater Horizon oil spill in the Gulf — they got that material too, if I taught it during the first two weeks of class. If it was stuff I presented to them in my usual way, about half of them got it, a third more sort-of got it, and the rest of them didn’t.

The irony here is that without Bambrick-Santoyo’s insistence on a carefully designed test or assessment, I wouldn’t have known any of this.  I could have taught Lemov’s way, or dumped most of it, or kept only the interesting parts.  I’d have philosophically agreed with him, or not, and moved on.  But by deciding to test both of them together, and ‘comparing notes’ so to speak, I learned things about them, and about my students, and about myself, that thirteen years of teaching hadn’t taught me.

Needless to say, it made more work for me.  My end-of-term narratives are usually a short paragraph or two long.  This time, I wrote nearly two pages on every kid, delving into what it appeared they knew, what it appeared they didn’t know, and how to close the gap between the two.  They were probably the most effective comments about students I’ve ever written — because a teacher who read one would have an idea what to do next with each student.

This is sort of long at this point (over 1100 words!), so I’ll cut it off here, and write more tomorrow. In the meantime, think on this: Lemov and Bambrick-Santoyo are right about this — if you have an achievement gap, the way to close the gap is to use the right set of teaching techniques, and the right set of assessments, and then USE THE DATA.

I can’t use the data to remediate the students. I can only pass the data on to those who will use it.  Even if I were continuing at this school, the students are graduated.  But now I know better how to teach in the future, and I have a better sense of how to make it work for everyone, not just the 50% who get what I’m saying or asking them to do on any given day.

The study was pretty conclusive, though.  I’m convinced Lemov’s observed techniques work.  But do your own study. Prove it to yourself.  Go on, I dare you.

Liked it? Take a second to support Andrew on Patreon!