Johann   page 16

There was a wonderfully illustrative story which I thought I had bookmarked, but couldn't re-find: it was the story of a man whose know-it-all neighbor had once claimed in passing that the best way to remove a chimney from your house was to knock out the fireplace, wait for the bricks to drop down one level, knock out those bricks, and repeat until the chimney was gone. Years later, when the man wanted to remove his own chimney, this cached thought was lurking, waiting to pounce...

As the man noted afterward—you can guess it didn't go well—his neighbor was not particularly knowledgeable in these matters, not a trusted source. If he'd questioned the idea, he probably would have realized it was a poor one. Some cache hits we'd be better off recomputing. But the brain completes the pattern automatically—and if you don't consciously realize the pattern needs correction, you'll be left with a completed pattern.

I suspect that if the thought had occurred to the man himself—if he'd personally had this bright idea for how to remove a chimney—he would have examined the idea more critically. But if someone else has already thought an idea through, you can save on computing power by caching their conclusion—right?

In modern civilization particularly, no one can think fast enough to think their own thoughts. If I'd been abandoned in the woods as an infant, raised by wolves or silent robots, I would scarcely be recognizable as human. No one can think fast enough to recapitulate the wisdom of a hunter-gatherer tribe in one lifetime, starting from scratch. As for the wisdom of a literate civilization, forget it.

But the flip side of this is that I continually see people who aspire to critical thinking, repeating back cached thoughts which were not invented by critical thinkers.

Eliezer Yudkowsky

Cached Thoughts