Inter-Rater Reliability Session Reflection

Over the course of two days, Courtney and I hosted about 60 of our areas 80 principals. The focus of the work was to do a round of calibration for inter-rater reliability when rating teachers. Many APPR plans require the leaders to come to this type of training on a somewhat regular basis while some districts are doing this work in-house. It is part of the regs, so districts have to find a way to do it.

The session was structured around scoring all four domains of Danielson. So we had pre-conference notes and a lesson plan for them to work through for Domain One. One thing they struggled with immediately was the feeling that these artifacts weren't enough. Many said that they would ask more or leading questions to try to illicit more information than was provided by the artifacts. One principal said he runs his pre-ob conferences with a focus on the Effective score level. Essentially asking, "are you here?" or "how do we get you here?" I find this to be a fair method. It recalls good rubric use. I.e.- here is the performance criteria, this is what proficiency looks like, let's work towards that.

So, doing the module in this format, as TLS does when calibrating in Albany, the artifacts exist as is. No leading questions, no other help. What you see is what you get. Generally, our principals scored this teacher too high in Domain One. There was discussion of the students with different needs, there were standards clearly enumerated, there was a scheduled lesson plan. For the audience, they found this performance to be proficient. However, the calibrated score is developing. While the standards are listed, there wasn't much connection to actual learning. And while there was a scheduled plan, it generally revolved around what students would be doing, not learning.

Then we watched about 20 minutes of the teacher in the classroom. It is a 4/5 literacy classroom, with ELLs, and she is a second year teacher. I was struck by how many of the leaders in the room flat out laughed at the teacher. We've shown a developing teacher to an audience of teachers before. And, teachers being more raw, it is somewhat expected. But it was off-putting that building leaders first reaction was to ridicule her. That their coaching instincts didn't kick in. I opted to remind them of the evidence collection cycle where the collection happens first, the alignment second, and only later, the value judgements. Suffice to say, they underscored her performance in Domains Two and Three. Easiest to say, the calibrators of this video are very strict to the language of the rubric. And our attendees generally disagreed. To digress, I just spent a day with math teachers who were disagreeing about the anchor papers for the math 3-8 assessments. Our facilitator just kept reminding the crowd, that these are the calibrated scored. You may not agree, but compliance is necessary for standardization across the state.

One other thing worth noting, we frequently show a teacher who exemplifies student cognitive engagement, constructivist learning, and 21st century learning. We almost always get push back that this isn't real life- show us a regular teacher. So, ha ha ha, we show a developing, regular ol' teacher, and we get push back- show us something exemplary.

Finally, the group was right on in scoring Domain 4, teacher reflection. They generally agreed there was hope for this teacher if she could translate her accurate reflection into future planning and practice.

- Posted using BlogPress from my iPad

Equity and Access

This post is a part of the ASCD Forum conversation “how do we define and measure teacher and principal effectiveness?” To learn more about the ASCD Forum, go to

How do we give students across the globe equitable access to effective teachers and principals?

Thinking about this structurally, there seems to be three ways which we could give students this access. We could "clean house," we could go "grass roots," or districts could focus on developing internal capacity. This third option seems to hold the most promise in my eyes.

"Cleaning house" to me first assumes there are a wide range of teachers and leaders out there of varying abilities. If we could only find an objective method- apply a rubric say- to weed out those that don't meet the standard, we could rid education of all those low performers. We need to apply objective standards, ISLLC 2008 for principals for example, but cleaning house in isolation won't be very effective. To start, take a look at NY for the challenges of implementing just such a system. And we'd have to have superintendents and principals take the drastic step of dismissing friends and colleagues. And the field would be left with plenty of vacancies to fill.

"Grass roots" would be a reinvention of our teacher and leadership training programs. In NY, this is just starting to happen. In the SUNY system, conversations have just begun to leverage Race to the Top funds to redesign the teacher prep programs to make them more rigorous and more experiential. There has even been talk of teachers being required to pass a bar-like exam to be initially certified. For leaders, I know of one SUNY school who is in the process of redesigning the Ed LEadership program from the ground up, complete with real life capstone projects and a two-year cohort model. This is all good work, and the right work, but it is going to take years to reap the benefits of these changes. 

To get our students across the globe equitable access, the best way forward is to focus immediately on building internal capacity of our teachers as leaders. Without removing them from the classroom, we should create teacher leaders who can coach, mentor, evaluate, and help improve those colleagues who need the extra help. And without removing them from the buildings, perhaps there is a way we can find a similar "leader of principals" path. Admittedly, that's a taller order. By focusing our efforts on building internal capacity, we gain the advantage of working with the troops on the ground. These peer leaders would also come with a greater level of trust than "the new guy" or "the outside expert." Perhaps, most importantly, if we want to give all students access to effective teachers and principals, if this is something we believe in, shouldn't we do this as soon as possible.

I think we can work within our current system fastest. I'm aware that we need some level of all three of these approaches. We need to weed out these who show consistently poor performers. We also need to prepare our future teachers and leaders differently. And some of the leadership work needs to come from within. But how many different initiatives can we complete effectively? A triple intensity may be too much. 

Your thoughts? Is there a fourth path I haven't considered?


This post was originally posted on ASCD Edge.


Let's revisit one of the recommendations from the Ed Reform Commission. You remember, the commission that said, "Pre-K for at-risk students works."

Recommendation number three reads:

"Transform and extend the school day and year to expand quality learning time for students, especially in underserved communities"
Great. I'm all for it. I especially agree that a longer school year could work.  Really, there's no reason students need to have two full months off.  I'd even compromise. North Carolina has an option for students to come for nine weeks, followed by three weeks off- but they come year round.  It works out to a similar amount of school days as your traditional year. The benefit though is that you never lose your students for more than three weeks at a stretch. No summer slide?

I think the realities of extending the school day are a little harder. Younger students I think are already tapped by three p.m.  Older students have jobs, baby sitting, and athletic events.  Yes, I agree, athletics are extra-curricular. But here at 43 degrees north latitude, daylight is an issue. If your October soccer game doesn't start by 3:30 or so, you're playing into the gloom.

Lastly on this issue: NY is still has a unionized education work force.  A work force which has demonstrated over the last couple of years a commitment more to maintaining traditional union values- work day, salary steps, retirement benefits- over making meaningful change.  The point is: show me the money.  This is not the type of change our unions will buy into without being bought.


About a year ago, Governor Cuomo created a commission to look at the issues facing education in New York State.  Just the other day, the commission released its initial report. Remember, this took a year of work with some of our "top-level" folks.  Here's a brief overview of their recommendations- their preliminary action plan.

  1. Provide high quality full-day pre-kindergarten for our most at-risk students;
  2. Create statewide models for “Community Schools” that use schools as a community hub to improve access to public, non-profit, and private services/resources, like health and social services, for students and their families;
  3. Transform and extend the school day and year to expand quality learning time for students, especially in underserved communities;
  4. Improve the teacher and principal pipeline to recruit and retain the most effective educators;
  5. Build better bridges from high school to college and careers with early college high schools and career technical education;
  6. Utilize all available classroom technologies to empower educators to meet the needs of a diverse student population and engage students as active participants in their own learning;
  7. Pursue efficiencies such as district consolidation, high school regionalization and shared services to increase student access to educational opportunities; and
  8. Increase transparency and accountability of district leadership by creating a performance management system.
    Really? Those are the recommendations? There isn't a faculty room or BoE that couldn't come up with the same list.  A colleague of mine described it as "a lit review of the past 10-20 years of ed research."