File Drawer Problem

Human knowledge has continued to increase over time, with the last few decades of new discoveries picking up intensity and taking on a life all their own. I mean a life of their own somewhat literally—it is the very tools we are creating that are helping us to build even better tools.

We’ve gotten smarter and smarter about certain things, but we’ve also had to choose between depth and width in many circumstances. All of this knowledge needs to be carefully considered, then classified so it becomes useful.

Humans have historically managed this process internally, all by ourselves. We call this process of careful classification of knowledge wisdom. We decide which things to pay more or less attention to, and we ultimately determine what’s important to us.

Here is how I’ve described these character attributes we all have:

Intelligence is the attribute with the most baggage. I will concede that there is such a thing as absolute intelligence in a person: there are some folks who can simply hold a great deal more in their minds than I can, and they can process that information unbelievably quickly. At the same time, someone can know a vast amount about one thing and be a complete dummy in another area (trust me—for everything I’m good at, I’m equally inept at several others).

That means I can’t really talk about intelligence without simultaneously thinking about wisdom. Wisdom is the way you apply knowledge, and if this publication is about any one thing, it is surely wisdom.

This human problem has a decidedly non-human corollary in every field we study. The issue isn’t that we’re unable to collect loads of useful, meaningful data. It’s just that useful and meaningful don’t come already included with data.

This is called the File Drawer Problem, or sometimes publication bias.

People are ultimately the ones who decide whether or not a research finding is significant, and people have all sorts of biases. Publication bias perfectly captures the essence of this phenomenon, since publication is where the rubber really hits the road.

Are these results significant enough to share with the general public? That might depend on whether the results support your point, or whether or not you saw some connection that would matter to the scientific public. In short, we have our own issues with making this sort of decision—at the level of wisdom, per my earlier analogy.

Just how great the extent of the File Drawer Problem really is? That’s something we’re going to scratch our heads over for a long time. We don’t know what we don’t know among all the buried research out there. And, even worse, existing biases are baked right into the cake with the very tools we might use to solve the issue, AI.

Both wisdom and intelligence can be improved over time. When you’ve acquired some wisdom, you can create a sort of virtuous cycle, whereby you appreciate knowledge more. This hunger for knowledge drives you toward a better understanding of where to find new knowledge in the first place, and how to differentiate more important from less important knowledge.

Wisdom takes time, but you can also learn from the mistakes of others. The more you look around and take your time in thinking things through, the more your intelligence and wisdom will increase.

We’re essentially asking today’s AI tools to help us solve this problem, but it really just kicks the responsibility can down the road. Ultimately, we are the ones creating these tools that will decide what types of knowledge are important.

If you work in IT, I bet this is already on your radar. What instances of the File Drawer Problem have you noticed in your daily activities?