Overcoming Biases in Sampling | CFA Level I Quantitative Methods

Welcome back! In this lesson, we’ll explore various biases in sampling that can prevent a sample from being truly random. Let’s dive in!

1. The Dilemma of Large Sample Sizes

According to the Central Limit Theorem, larger samples reduce the standard error of the sampling distribution, resulting in narrower confidence intervals. However, larger samples are not always better due to a few reasons:

• Mixing populations: Larger samples may contain observations from another population with different characteristics, reducing the precision of the population parameter we’re trying to estimate.
• Cost: Obtaining extra observations can be costly, and the marginal benefit has to be weighed against the extra costs needed to obtain those data.

2. Sampling Bias

Beyond sample size, other issues arise in the form of sampling bias. We’ll discuss a few common types:

2.1 Data Snooping Bias

Data snooping bias occurs when researchers repeatedly overuse the same data to search for patterns, until they find one that works. It’s crucial to identify warning signs that point to data snooping bias:

• Many variables are tested until a pattern emerges.
• The wording suggests researchers were trying out variables based on their own observations and hunches.
• A missing or incongruent economic rationale for the strategy might also indicate data snooping.

2.2 Sample Selection Bias

Sample selection bias happens when some data is excluded, usually due to unavailability. This causes the observed sample to be non-random, making the conclusions flawed. Unavailable data might be fundamentally different from the available data.

EXAMPLE

A classic example is survivorship bias in mutual fund performance studies. Poor-performing or unpopular funds often close, causing the surviving funds’ performance to be biased to the upside. This gives an inaccurate picture of the overall market’s performance.

To solve this issue, use a sample where all funds started at the same time, and include the performance of discontinued funds.

2.3 Time-Period Bias

Time-period bias can occur if the data’s time period is too long or too short:

• A short time period may reflect a phenomenon specific to that period, and conclusions drawn may not represent longer-term trends.
• A long time period may cover periods with fundamental changes in relationships, causing conclusions to be inaccurate.

In such cases, divide the data into sub-samples and analyze them separately.

That’s it for our lesson on sampling and estimation! Join us in the next topic where we’ll apply these concepts further in hypothesis testing. See you there!