Measuring Teacher Effectiveness

Something I've been thinking a lot about lately is the idea of linking test scores to teacher evaluation. It's a topic that's everywhere this summer:

• The EFF school report card makes the point.
  
• It's also the heart of Arne Duncan's remarks at the NEA Representative Assembly.
  
• The ASCD put out a newsletter recently on union/management collaboration; tying test scores to evaluation was a big part of it.
  
• And of course, there's the state-level hijinx going on vis-a-vis HB2261, the State Board of Education, and the PESB. Wheeee!

One of the notions that you often hear during these discussions is, "The good teachers have nothing to be afraid of." Let's talk about that for a bit.

Last year, for one of my Master's classes, I dug into testing data I had on hand for the first grade team in my building. These are real numbers and real averages with real kids behind them; the test in question is the Measures of Academic Progress, from the Northwest Evaluation Association.

Teacher A: In the fall, her class had an average score of 162.5 on the MAP. In the spring the class average rose to 184.3, an average gain of 21.8 points.

Teacher B: Her fall average was 164.7; her spring average, 183.85, for an increase of 19.15 points.

Teacher C: 169.05 in the fall, 189.35 in the spring, so an average gain of 20.3 points.

Teacher D: An average score of 155.30 points in the fall and 174.85 in the spring. Her fall-to-spring gain, then, was 19.55 points.

With this data, then, you could argue the case for two different teachers as the "winners" in the group. If you look at the average gain, Teacher A is your champion:

1. Teacher A: 21.8 points
   
2. Teacher C: 20.3 points
   
3. Teacher D: 19.55 points
   
4. Teacher B: 19.15 points

But, if you look at the overall class average at the end of the year, Teacher C is far and away your winner:

1. Teacher C: 189.35
   
2. Teacher A: 184.3
   
3. Teacher B: 183.85
   
4. Teacher D: 174.85

If we went strictly by these numbers from this year, then, you can see who your quality teachers are. If you were judging solely by the numbers, you might also think that you have a problem with Teacher D--her class average trails the class average of everybody else by almost 10 points, which on the MAP is very nearly an entire year's worth of growth.

But we have to dig even deeper before making a statement about teacher quality, because here the raw numbers aren't telling the whole story.

In the fall, the average score for this test is 164 points. In the spring, the average score is 178. Knowing that, here's some new data to chew on.

In Teacher A's room in the fall, 10 kids scored in the below average range. In the spring, 6 kids scored below average.

In Teacher B's room, 7 kids were below average in the fall, while 3 were below average in the spring.

In Teacher C's room, 6 kids were below average in the fall, and 3 in the spring.

In Teacher D's room, 16 kids were below average in the fall, and 6 tested below average in the spring.

With this new information, you can make two new arguments. First, Teacher B is your best teacher because she had more of her kids cross the finish line (the goal score, 178) than the other teachers did. You could also argue that Teacher D is your best teacher because she lowered her percentage of kids who were below standard more than any of the other teachers did.

So, who is your Most Valuable Teacher?

Is it Teacher A, who added the most value to her class over the course of the year?
Is it Teacher B, who had more of her kids meet the year-end goal?
Is it Teacher C, whose class scored the highest in the spring?
Is it Teacher D, who turned around more failing kids than any of the others?

"Value" is a homophone; there's the value signified by the numbers, but there's also the values of the school, the district, and the state which have to be superimposed atop any effort to link the data to the teacher. If the incentive pay/merit pay/whatever pay in this case goes to only one of the four teachers, you're making a statement about the value of the work the other three did, and it's a pretty lousy thing to say to the other three who also made progress that their success didn't matter as much.

Similarly, can we countenance a system where every one of these teachers is given the bonus money, indicating that they all did a good job? In the eyes of some reformers I could see that being too close to what we do now, where every teacher is assumed to be a good teacher. If a merit pay system is intended to have winners and losers, and to inspire the "less-capable" teachers to emulate the "better" teachers, can we really have a 4-way tie?

These are the questions that have to be answered going forward.

If you'd like to see the raw scores presented in a spreadsheet, you can find them here.

Read more here, if any.

Where the Economics of Education Really Falls Apart....

....is in the ASCD Catalog.

In their most recent Education Update they trumpet the great sessions that went on at this year's ASCD Annual Conference in New Orleans. Looks like a neat event, would love to go someday if they ever get close to the left coast. I see a little blurb in the flyer about audio recordings of sessions from the conference and I think to myself, "Neat! I'd not mind having those for summer listening pleasure, what since I'm a professional and all!" and merrily make my way over to the website to download them, a la podcasts.

The catch: if you want a package of podcasts, you're going to have to shell out $150 first. Screw that.

I guess that's about what I should have expected, though, since the ASCD video series are almost all priced in the mid three-figure range, putting them out of the reach of most. Even single videos of 15 minutes can cost $89.

I guess when you're the biggest name in town you can get away with that, but golly Ned those prices seem out of line.

Read more here, if any.

STEM Week Day 1: Engineering: A Title IX Issue?

The December issue of the ASCD’s Education Update focuses on the 35th anniversary of Title IX, the landmark federal law that opened up the doors for women to have equal opportunity in athletics and education. It’s a good law that needed to happen, but I’m curious about where the ASCD sees things going. From the newsletter (p. 3):

The Society of Women Engineers (SWE) and other organizations are employing Title IX to ensure that women are equally represented in the science, technology, engineering, and mathematics (STEM) fields. In February 2007, Michelle Tortolani, then SWE president-elect, asked at the National Association of Multicultural Engineering national conference, “Why, while girls comprise 55 percent of undergraduate students, do they account for only 20 percent of engineering majors, and boys remain four times more likely to enroll in undergraduate engineering programs?”

Two Congressional briefings have focused on strengthening the STEM workforce by asking this question: Are more women and members of diverse groups needed in the STEM workforce? Resoundingly, SWE says “Yes” and offers the following recommendations:

*Policymakers should step up enforcement of Title IX with regard to STEM disciplines and fund programs that will help educate students, their parents, and STEM faculty of their rights under the law.

*Educational institutions should fulfill their obligations under the law; examine their institutional policies, procedures, or practices for gender bias; provide suggestions for areas to examine when evaluating programs for gender bias; and make this information accessible to the public.

*Federal funding agencies should fulfill their monitoring and enforcement obligations under the law, and make this information available to the public.

I know that as a male I’m probably better off running as fast as I can and as far as I can from anything having to do with Title IX, but on the face of it there’s a lot of problems that I see here.

The first is the difference between a bias of design and a bias of choice. If a program is structured in such a way that people are excluded on the basis of their gender or race, that’s immoral and should be addressed. It’s a different matter, though, when those of one gender tend to gravitate towards a career more than those of the other as a matter of choice. If more men are drawn to the engineering life, with it’s nerdy stereotype and long hours, is that a problem of the system, of the sexes, or of the society?

We have an example in our own field—elementary teachers are almost 90% women. With its family friendly policies and the chance to work with kids, that’s not a surprise. What it’s also not is discrimination, and anyone arguing that it is would be wrong.

Programs that expose girls to the STEM fields, like those that come across the WSTA listserv? Excellent! This nascent suggestion that we should have proportionality in the college programs, because equal must be inherently more fair? That’s an idea that takes us down a weird, exclusionary road that we’re better off not traveling.

And let’s also acknowledge that there’s a bias on the pro-girl side as well. In a section on the ongoing challenges associated with Title IX we find:

Young women represent the vast majority of students enrolled in high school cosmetology, child care, and health assistant courses. Child care workers earn a median salary of $7.43 per hour, while cosmetologists earn a median salary of $8.49 per hour.

We need good child care workers every bit as much as we need engineers. That the ASCD would tie the worth of the job to the pay is a sad statement of the priorities that we’ve established as a society, and shame on them for taking the bait. That someone could make more in engineering than hairdressing is a given, but why imply that there’s something wrong with wanting to be a hairdresser?

For anyone interested in gender equity in the mathematics and sciences, this is an article well worth finding.

Is there a problem with males being over represented in the STEM fields? How would you solve it?

Read more here, if any.