## Snippets about: Statistics

Scroll left and right !

## Sports on the Continuum

- Basketball statistics show high year-to-year correlations, a dispersed distribution of success, and strong predictive power - indicating a strong role for skill
- Hockey statistics show lower consistency, a more concentrated success distribution, and less predictive power - indicating a larger role for luck These statistical tools give us an objective way to assess the skill-luck balance in different domains and make better predictions.

Section: 1, Chapter: 3

Book: The Success Equation

Author: Michael Mauboussin

## Regression To The Mean

The phenomenon of regression to the mean explains why we often misinterpret the effects of rewards and punishments. When we observe an exceptional performance, which is likely due to a combination of skill and luck, we tend to reward the individual. However, on subsequent attempts, their performance is likely to regress back towards their average level, making it appear as if the reward has backfired. Conversely, when we observe an unusually poor performance, we tend to punish the individual, and their subsequent improvement due to regression to the mean makes it seem like the punishment was effective.

Section: 2, Chapter: 17

Book: Thinking, Fast and Slow

Author: Daniel Kahneman

## The High Cost Of Selective Attention

Imagine you're considering two slot machines to play at a casino. For machine A, you watch a dozen people play and 4 of them win. For machine B, you have no information. Which should you choose?

Most people avoid uncertainty and pick machine A. But rationally, machine B is more likely to pay off. Even though you have no data, the chances it pays off more than 33% are better than the chances it pays off less.

This is a direct implication of Laplace's Law: if you have no prior information, the probability of a positive outcome is (k+1)/(n+2) where k is the number of positive outcomes observed out of n attempts. For machine A, that's (4+1)/(12+2) = 36%. For machine B it's (0+1)/(0+2) = 50%.

Human attention, and media coverage, is drawn to the vivid and available, not the representative. This selective attention means that more media coverage of an event indicates that it's more rare, not necessarily more common. To be a good Bayesian, look for the silent evidence as much as the headline grabbing stories. Often what you haven't seen is as informative as what you have.

Section: 1, Chapter: 6

Book: Algorithms to Live By

Author: Brian Christian

## The Three Measures Of A Useful Statistic

A good metric should have three key properties:

- Consistency - Does the metric reliably measure the same thing across time and contexts?
- Predictive power - Does the metric actually predict the outcome we care about?
- Noisiness - How much random variability is there in the metric relative to the signal?

The best metrics are consistent, predictive, and have a high signal-to-noise ratio. Examples of useful metrics include:

- On-base percentage in baseball (consistent, predictive of scoring, less noisy than batting average)
- Customer retention rate in business (consistent, predictive of profits, less noisy than raw sales numbers)
- Sharpe ratio in investing (consistent way to measure risk-adjusted returns, predictive of fund quality)

Section: 1, Chapter: 7

Book: The Success Equation

Author: Michael Mauboussin

## Insights for Quantifying Uncertainty

Actionable insights for AI developers:

- Make AI systems' confidence scores actually reflect statistical uncertainty, not just relative ranking
- Build pipelines for "uncertainty handoff" to human oversight in high-stakes applications
- Extensively test AI systems on out-of-distribution and adversarial inputs to probe overconfidence
- Favor objectives and learning procedures that are robust to uncertainty over brittle "point estimates"

The upshot is that well-calibrated uncertainty is a feature, not a bug, for AI systems operating in the open world. We should invest heavily in uncertainty estimation techniques and make them a core component of AI system design.

Section: 3, Chapter: 9

Book: The Alignment Problem

Author: Brian Christian

## Surprising Virtue Of Persistence And Ignorance

Suppose you arrive in a new city and see a long line outside a restaurant. Knowing nothing else, what does that tell you about the restaurant's quality?

According to Bayes's Rule, long lines are more likely to form for good restaurants than bad ones. Even with no other information, the line is evidence of quality. The longer the line, the better the restaurant is likely to be.

But now imagine you go into the restaurant and have a terrible meal. What should you believe now - your own experience or the line of people waiting? Again, Bayes's Rule says to weigh the evidence based on sample size. A single bad meal is a small sample. The long line reflects dozens of people's opinions. It should still dominate your judgment.

The same idea applies to product reviews, survey results, or any wisdom of the crowd. If your own experience contradicts the majority, trust the majority. Your own sample size is too small to outweigh the group. First impressions can lead you astray.

Section: 1, Chapter: 6

Book: Algorithms to Live By

Author: Brian Christian

## Why Base Rates Matter

Base rates refer to the underlying probability of an event or category. For example, if 1% of the population has a particular disease, then the base rate of having the disease is 1%. When making predictions or judgements about individual cases, we often neglect base rates and instead focus on specific information about the case, even if that information is unreliable or uninformative. This can lead to significant errors, as we fail to consider the underlying probability of the event.

Section: 2, Chapter: 14

Book: Thinking, Fast and Slow

Author: Daniel Kahneman

## The Power Laws Of Performance

Mauboussin examines the surprising statistical regularities that show up across many domains of human performance. The key finding is that many performance metrics follow a power law distribution, wherein:

- A small number of top performers account for a disproportionate share of the total output
- The gap between the best and the rest is much larger than a normal distribution would predict.

For Example:

- City sizes (a few mega-cities, many small towns)
- Wealth distribution (a few billionaires, many people of modest means)
- Bestseller lists (a few blockbuster hits, many low-selling titles)
- Scientific citations (a few papers with massive impact, many rarely-cited papers)

The power law pattern arises from the complex social dynamics that shape success in these domains - in particular, the rich-get-richer effects of cumulative advantage.

Section: 1, Chapter: 6

Book: The Success Equation

Author: Michael Mauboussin

## Pareto Principle And The Vital Few

The Pareto Principle states that 80% of results come from 20% of efforts. A few key activities contribute most of the value in any given situation:

- In business, 80% of profits come from 20% of customers
- In society, 20% of criminals commit 80% of crimes
- In personal lives, 20% of activities produce 80% of satisfaction

Essentialists invest their time and energy only in the vital few choices and activities that matter most, with the understanding that most things are unimportant distractions.

Section: 1, Chapter: 3

Book: Essentialism

Author: Greg McKeown

## How Many Trucks Does It Take?

During World War II, the Allies wanted to estimate the number of German tanks being produced each month. Intelligence officers simply looked at serial numbers printed on captured German tanks. If tank #1,242 was built in May and tank #1,867 was built in June, then they assumed 625 tanks were built in June.

After the war, data from German factories showed that the traditional intelligence estimates were off by a factor of 2.5. But the serial number analysis was off by only 1%.

Imagine a jar filled with marbles labeled from 1 to N. You draw a random sample of marbles, noting their numbers. How can you estimate the highest number N - the total number of marbles in the jar?

Statisticians showed that the best estimator is to take the highest drawn number and multiply it by (k+1)/k, where k is the number of draws. So if you drew 4 marbles with a highest number of 78, you'd estimate there were 97.5 marbles total.

The key is that missing marbles provide information, since they indicate the jar contains numbers you haven't seen yet. The estimator performs optimally by considering both what was drawn and what likely wasn't drawn yet, given the sample. A Bayesian approach outperforms one using the observed data alone.

Section: 1, Chapter: 6

Book: Algorithms to Live By

Author: Brian Christian

## Statistical vs Causal Base Rates

There are two types of base rate information, and they are treated differently by our minds:

Statistical Base Rates: These are facts about the overall population or category, such as the percentage of people who have a college degree. We tend to underweight or ignore statistical base rates when specific information about an individual case is available.

Causal Base Rates: These are facts that have a direct causal bearing on the individual case. For example, if you learn that a student attends a highly selective university, this information has a causal influence on your judgement of the student's academic abilities. Causal base rates are treated as information about the individual case and are readily combined with other case-specific information

Section: 2, Chapter: 16

Book: Thinking, Fast and Slow

Author: Daniel Kahneman

## Using Statistics To Place Activities On The Continuum

Chapter 3 presents a method for empirically placing activities on the skill-luck continuum, using three statistics:

- Demonstrated consistency of outcomes across repeated trials (e.g. year-to-year correlation of batting averages)
- The distribution of success among the population of participants (e.g. number of players hitting above .300)
- The ability of a measure to predict future performance (e.g. correlation of on-base percentage to future runs scored) Using these measures, we can estimate the relative contribution of skill and luck in different domains.

Section: 1, Chapter: 3

Book: The Success Equation

Author: Michael Mauboussin

## Blending Skill And Luck To Match Real-World Results

Mauboussin illustrates the skill-luck continuum using the example of the NBA draft lottery. In the lottery, the worst teams get the highest probability of winning the top draft picks, in an effort to maintain competitive balance.

Mauboussin and his colleagues simulated the draft lottery under two scenarios: one with pure luck (a weighted coin flip) and one with pure skill (the worst team always gets the #1 pick). They then compared the actual results to these two extremes.

The results showed that the real NBA draft lottery results fell almost exactly halfway between pure luck and pure skill, suggesting that the lottery system does a good job of blending luck and skill to achieve its intended purpose. This type of simulation - comparing real-world results to luck and skill extremes - can be a useful tool for placing activities on the skill-luck continuum.

Section: 1, Chapter: 3

Book: The Success Equation

Author: Michael Mauboussin

## Reversion To The Mean: The Cruel Equalizer

Chapter 1 also introduces the concept of reversion to the mean - the idea that an outcome that is far from the average will be followed by an outcome that is closer to the average. Reversion to the mean occurs whenever two measures are imperfectly correlated. The lower the correlation, the more extreme the reversion. In activities with a significant element of luck, reversion to the mean can be a powerful force. A few key implications:

- Don't overreact to extreme performances, good or bad. They are likely to revert toward the average.
- Be wary of paying for past performance. The "hot hand" may be nothing more than luck.
- Anticipate reversion in your own performance. If you have a great year, don't expect to maintain that level. If you have a terrible year, better times are likely ahead.

Section: 1, Chapter: 1

Book: The Success Equation

Author: Michael Mauboussin