Goodnight Wiki / Rationality and Decision Making

Rationality and Decision Making

This is the wiki's most self-reflective section — the one that asks how we think, why we think badly, and what (if anything) can be done about it. The articles cluster around a central tension: our reasoning tools are powerful but miscalibrated, and the act of learning about miscalibration doesn't automatically fix it. Knowing the name of a bias is much easier than correctly identifying when it applies.

The Bias Landscape

Cognitive Biases surveys the territory through Munger's 25 tendencies, the revisionist case for sunk costs, cultural transmission as bias correction (Henrich), reason as debating hardware rather than truth-seeking machinery (Mercier and Sperber), and prevalence-induced concept change (as problems decrease, we expand our definition of the problem). Replication Crisis shows what happens when the machinery of science is optimized for publishable results rather than true ones — ego depletion collapsed, the marshmallow test deflated, and most published research findings may be false.

Calibration And Measurement offers the constructive response: anything can be measured if you define it in terms of observables, determine what you already know, and compute the value of additional information. The Bayesian surprise framework formalizes what "interesting" means — not Shannon information but model updates, which is why random noise is boring despite being maximally entropic.

The Bayesian Core

Bayesian Epistemology provides the mathematical backbone: probabilism, conditionalization, and the extraordinary history of a framework that was discovered, abandoned, classified, persecuted, and eventually vindicated. The clinical application is the most striking — delusions as broken Bayesian updating where the prior-enforcement module (RDPC) is damaged, producing patients who select hypotheses solely on explanatory adequacy while ignoring base rates. The neurotransmitter mapping (glutamate as evidence, dopamine as precision, NMDA as priors) connects directly to Predictive Processing.

Decision Theoretic Paradoxes stress-tests the Bayesian framework: St. Petersburg reveals that expected value assumes ergodicity, Sleeping Beauty shows probability assumes a unique epistemic position, Dutch Books show coherence assumes a specific relationship between belief and action. Each paradox exposes a hidden assumption that, when it holds, makes the formalism work beautifully.

Maps, Territories, and Frameworks

Maps And Territories is the section's philosophical anchor — the algorithm-feels-from-inside problem (categories feel like reality, not like categories), the noncentral fallacy (importing emotional freight from a category's archetype to a non-central member), and the stunning amount of detail hiding in even simple activities like boiling water.

Fake Frameworks extends this: deliberately using models you know are wrong, in a mental sandbox, because a wrong model that highlights real patterns is more useful than no model at all. Simulacra Levels traces the degradation from "there's a lion" (fact) through strategic lying, tribal signaling, and pure game — with COVID as the case study showing how level 3 and 4 behavior prevented level 1 communication about an actual threat.

Pragmatic Ethics applies the same anti-foundationalism to morality: moral criteria are tools, not foundations; morality is habit, not principle; means and ends are the same series viewed at different stages. The Hume-Buddhism connection — that foundations don't matter, and life goes on exactly as before once you stop looking for them — links to Sunyata.

Coordination and Failure

Moloch personifies coordination failure as a secular demon: fourteen examples of individually rational choices producing collectively catastrophic outcomes. Inadequate Equilibria provides the taxonomy: no exploitable correction, expertise can't trickle down, bad Nash equilibria. Common Knowledge explains why coordination is so hard — the gap between everyone knowing something and everyone knowing that everyone knows it. Coordination Problems gives the concrete examples: Japan's 85% error reduction from pointing-and-calling that nobody else adopts because it feels silly.

Slack And Civilizational Fitness is Moloch's complement: perfect competition prevents innovation, and the deepest fitness pits are separated from you by hills that only slack lets you climb. Goodharts Law explains why metrics corrupt the thing they measure, connecting to Specification Gaming in the AI section.

Social Cognition

Tribal Epistemology shows that unconscious bias against political outgroups is 50% stronger than racial bias, and that the "dark matter" of political segregation produces 10^45-level separation between communities sharing the same physical space. Dominance And Prestige distinguishes two ways of being high-status, connecting to why rationalist communities can't cooperate (they reward only disagreement). The Toxoplasma Of Rage explains why the most controversial cases go viral while the clearest ones are ignored — the information ecology rewards controversy regardless of who benefits.

Adult Developmental Stages frames much of this through Kegan: only a third of adults have reached stage 4 (systematic thinking), and the mental operations rationalists prize — distinguishing map from territory, modeling alien minds, thinking probabilistically — are developmental milestones that some adults haven't reached. Double Crux and Sunk Cost Sophistication offer practical tools: finding the real disagreement, and recognizing that "ignoring sunk costs" is itself more complicated than the textbooks admit.

Game Theory And Cooperation connects the theoretical side: Edgeworth cycles as visible game theory, program equilibria where cooperation becomes a provable property, and the diagonal argument linking Cantor, Turing, and the prisoner's dilemma through the shared structure of self-referential limitation.

The Henrich Tension

There's a disagreement running through this section that the wiki doesn't resolve — and shouldn't, because it's genuinely open. The rationalist project, broadly construed, says: think carefully, update on evidence, avoid biases, and you'll do better than intuition and tradition. Cultural Evolution says: tradition beats reason when the causal structure is opaque, individual intelligence is overrated, and the most dangerous thing you can do is think for yourself about problems whose solutions were culturally evolved over millennia.

These aren't fully compatible. The rationalist who learns about the nixtamalization problem — where stripping away the "irrational" manioc processing killed people for centuries — should update toward respecting tradition. But the cultural evolutionist who learns about the replication crisis — where entire fields of psychology were built on P-hacked nonsense that nobody questioned because it was published — should update toward questioning received wisdom.

Legibility And Folk Knowledge sharpens the tension to a point: gri-gri bullet-proofing works as a coordination mechanism but only because people believe the wrong explanation. Understanding the mechanism destroys it. If some practices require active misunderstanding to function, the rationalist project of making everything legible is, in those specific cases, actively destructive.

The honest position is probably that both insights are true and the boundary between them is context-dependent. When the causal structure is legible (physics, programming, simple games), reasoning from first principles dominates tradition. When the causal structure is opaque (social norms, ecosystem management, cultural practices with thousand-year histories), tradition may encode solutions that reasoning can't reconstruct. Fake Frameworks offers a practical bridge: use the frameworks that work, flag them as provisional, and don't mistake any single framework for the territory.

What Connects Outward

The rationality section is the wiki's connective tissue. It bridges to AI through specification gaming and alignment. To philosophy of mind through introspection and predictive processing. To economics through coordination failures and market dysfunction. To fiction through thought experiments and the unreliable narrator. To linguistics through Sapir-Whorf and simulacra levels. The section's deepest insight: rationality isn't a fixed toolkit but an ongoing calibration — knowing which tools work where, and maintaining the humility to notice when they don't.

Open in stacked reader →