Goodhart's Law

Sometimes, students in jiu jitsu want to be promoted faster.

This is understandable: it takes a long time to progress in BJJ (Brazilian Jiu Jitsu), sometimes taking several years of dedicated training for even a single belt promotion. Perks can come with promotion, too, like being able to attend more advanced classes, and being able to compete at a higher level.

As a gym owner, there’s also an element of prestige that goes along with having a certain number of “upper belts” in your school.¹ If you have, say, ten black belts teaching and training at your academy and the other schools around have one or two, that can seem very impressive.

Oh hey, that’s me

There is, therefore, a ton of pressure on an instructor to promote their students, and a good instructor needs to be able to ignore this noise and make the decisions they need to make. I’ve felt this pressure for decades now, and I’ve enjoyed watching our instructors rise to the occasion, pushing back against this pressure.

Belts are not trinkets. In BJJ, a belt signifies a significant leap upward in skill, and we mean to keep it that way.

Some jiu jitsu schools out there will do belt tests, but these are much less common with BJJ than with most other martial arts, including judo, which makes a big deal out of demonstrating techniques in front of other people.

For the schools that do belt tests, many are thorough tests given by very reputable folks. I’m not talking about that type of school right now, though. I’m talking about schools that say things like, wouldn’t it be nice if we had just as many black belts as that other school over there?, and so they lower their internal standards so they can get there faster.

This is called teaching to the test, and it’s a great example of a larger phenomenon I want to call attention to today. Even if there’s not a physical test in my example above, the gym owner in question is presumably willing to modify their standards in order to get more black belts at their school.

Both of these phenomena are examples of Goodhart’s Law, which broadly states:

When a measure becomes a target, it ceases to be a good measure.

Indeed, as soon as you create a specific target, all sorts of perverse incentives can crop up. You can frenetically do things in order to reach goals, while ignoring much more important things that don’t have a specific goal tied to them.

An example from my own childhood should serve as an excellent cautionary tale involving this principle.

As you can probably guess, I liked to read a lot as a kid. While in elementary school, there was a contest whereby we got free pizza if we read a certain number of books.

Now, books were amazing, and I don’t think I would have traded the sense of wonder and adventure I got for anything! I mean, except maybe pizza. So, that’s what I did: I technically ran my eyes across a bunch of books and “read” them in that sense, getting enough points to dine like a prince.

I skimmed and forgot like twenty books during my fourth or fifth grade year, but you can bet I got that pizza.

This is a bit like the paperclip maximizer problem, where you ask an AI to create as many paperclips as possible, so (spoiler!) it uses up all of Earth’s resources to meet its metrics. Asking someone to track a metric or accomplish a goal instantly changes their outlook on everything else, and if there’s no incentive to keep everything else running properly, things can become catastrophic.

Ever visit your doctor to be seen immediately by the receptionist, then sent back to a room to wait some interminable time for your doctor to see you? This is almost certainly an internal effort to reduce wait times, but Goodhart’s Law points out that making the patient wait in a different room is still making them wait.

Similarly, you might run a social media campaign for a small business, with the goal being to get as many likes as possible. The goal of the small business owner probably should be something like get more customers in the door or sell more things, but it seems like they’ve gotten bogged down in a single metric to chase. What could possibly go wrong with just trying to get likes on social media?

You get the idea. Goodhart’s Law is pervasive and dangerous. Measuring things and accomplishing goals is important, but you have to zoom out and consider the big picture whenever you’re making any kind of targets.

Consider those perverse incentives carefully. Is there any way this goal can cause other areas of your business (or life) to get out of whack? Remembering that this tendency exists may be enough to improve outcomes significantly.

Have you noticed Goodhart’s Law in your own life? If you have any work stories related to this, please share them in the comments today!

1. Most folks seem to mean purple, brown, and black belts in jiu jitsu when they use this term, but since purple belt is right there in the middle (white, blue, purple, brown, and black), I don’t really think of it as all that advanced… although a purple belt in jiu jitsu is often vastly more experienced than a new black belt in most other martial arts.