jarednielsen.com

Illusions of Competence & Humility in Software Development

May 04, 2020

jarednielsen solution illusions humility

It’s easy to overestimate our abilities. Not even a 10X developer can know everything. To succeed in software development, it’s important to maintain your humility and avoid illusions of competence.

This article originally published in my newsletter, The Solution

Illusions of Competence & Humility in Software Development

“The way you focus your intelligence is more important than how much intelligence you have.”

-Steve McConnell, Code Complete

Do you know what you think you know?

Are you sure?

In Make It Stick, the authors describe the The Dunning-Kruger effect, a phenomenon in which “incompetent people overestimate their own competence and, failing to sense a mismatch between their performance and what is desirable, see no need to try to improve.”

I’m not suggesting you are incompetent.

But.

It’s easy to fall into the trap of overestimating our abilities. As we’ll see, we’re hardwired to do it.

Humility in Software Development

In Code Complete, Steve McConnell humorously outlines the importance of humility in software development:

The people who are best at programming are the people who realize how small their brains are. They are humble. The people who are the worst at programming are the people who refuse to accept the fact that their brains aren’t equal to the task. Their egos keep them from being great programmers. The more you learn to compensate for your small brain, the better a programmer you’ll be. The more humble you are, the faster you’ll improve.

Programming and problem solving are both metacognitive activities.

To excel, we need self-awareness.

We need to think about our thinking.

This is true for learning as well.

Illusions of Competence and Knowing

In Make It Stick, the authors write:

Each of us is an astounding bundle of perceptual and cognitive abilities, coexisting with the seeds of our own undoing. When it comes to learning, what we choose to do is guided by our judgments of what works and what doesn’t, and we are easily misled.

Why are we easily misled? The authors outline several common illusions of competence:

  • Hunger for Narrative: When surprising things happen, we search for an explanation. The urge to resolve ambiguity can be surprisingly potent, even when the subject is inconsequential.
  • Distortions of memory: We cannot remember every aspect of an event, so we remember those elements that have greatest emotional significance for us, and we fill in the gaps with details of our own that are consistent with our narrative but may be wrong.
  • Imagination inflation: When asked to imagine an event vividly, people will sometimes begin to believe, when asked about it later, that the event actually occurred.
  • Suggestion: We are susceptible to the influence of others, intentional or not.
  • Interference: Outside events can distort memory.
  • Curse of knowledge: Our tendency to underestimate how long it will take another person to learn something new or perform a task that we have already mastered.
  • Hindsight bias: What is often called the knew-it-all-along effect, in which we view events after the fact as having been more predictable than they were before they occurred.
  • The feeling of knowing: Something you once heard that you hear again later carries a warmth of familiarity that can be mistaken for memory, a shred of something you once knew and cannot quite place but are inclined to believe. In the world of propaganda, this is called “the big lie” technique—even a big lie told repeatedly can come to be accepted as truth. 🤔
  • Fluency illusions: Our tendency to mistake fluency with a text for mastery of its content.
  • Social influence: Our memories tend to align with the memories of the people around us.
  • False consensus effect: We generally fail to recognize the idiosyncratic nature of our personal understanding of the world and interpretation of events and that ours differ from others’.

That’s a lot to think about!

How to Avoid Illusions of Competence

How do we avoid illusions of competence?

Practice.

“If it’s important, it needs to be practiced, and practiced again.”

In A Mind for Numbers, Barbara Oakley cautions us:

Beware—a common illusion of competence is to continue practicing a technique you know, simply because it’s easy and it feels good to successfully solve problems.

Okay, so what is the best way to practice?

Testing.

I know.

The last word you want to hear, especially if you’re a developer.

Testing forces us to retrieve information from memory and situate it in new contexts.

Don’t worry, though, the authors of Make It Stick outline several other tools and habits for calibrating your judgment, the first being closely related to testing:

  • Cues: Create a mental model of the material that integrates the various ideas across a text, connects them to what you already know, and enables you to draw inferences. How ably you can explain a text is an excellent cue for judging comprehension, because you must recall the salient points from memory, put them into your own words, and explain why they are significant—how they relate to the larger subject.
  • Peer instruction: Work through a problem with a partner. You will both learn more through the process of teaching one another than working in isolation.
  • Feedback: Ask for it. As the authors of Make It Stick write, “The path to self-insight leads through other people.”
  • Teams: “In many settings, your judgment and learning are calibrated by working alongside a more experienced partner.”
  • Simulation: “Training that simulates the kinds of demands and changeable conditions that can be expected in real-world settings helps learners and trainers assess mastery and focus on areas where understanding or competency need to be raised.”

In A Mind for Numbers, Barbara Oakley reinforces the importance of simulation:

You want your brain to become used to the idea that just knowing how to use a particular problem-solving technique isn’t enough—you also need to know when to use it.

How to Maintain Intellectual Honesty in Software Development

We can supplement the above list with several ways for maintaining intellectual honesty in software development, as outlined by Steve McConnell in Code Complete:

  • Refusing to pretend you’re an expert when you’re not
  • Readily admitting your mistakes
  • Trying to understand a compile warning rather than suppressing the message
  • Clearly understanding your program—not compiling it to see if it works
  • Providing realistic status reports
  • Providing realistic schedule estimates and holding your ground when management asks you to adjust them

In many ways, the items on this list all relate to the first, which is also closely related to the Dunning-Kruger effect and (sometimes willful) illusions of competence and knowing. McConnell offers some sage advice:

How can you learn anything new if you pretend that you know everything already? You’d be better off pretending that you don’t know anything. Listen to people’s explanations, learn something new from them, and assess whether they know what they are talking about.

In other words: be humble.


Jared Nielsen

Want to level up your problem solving skills? I write a weekly newsletter about programming, problem solving and lifelong learning. Join now