Mastering Hypothesis Testing | CFA Level I Quantitative Methods

In this lesson, we’ll dive into the world of hypothesis testing, an essential method in inferential statistics. By the end of this lesson, you’ll have a thorough understanding of the hypothesis testing process, complete with a real-world example to help you grasp the concepts. So, buckle up, and let’s get started!

Understanding Hypothesis Testing

While estimation focuses on building confidence intervals around point estimates, hypothesis testing revolves around assessing a statement or idea regarding a population. The process can be broken down into 7 steps:

1. State the hypothesis.
2. Identify the appropriate test statistic and its probability distribution.
3. Specify the significance level.
4. State the decision rule.
5. Collect the data and calculate the test statistic.
6. Make the statistical decision.
7. Make the economic decision.

Let’s explore each step in detail:

Step 1: State the Hypothesis

First and foremost, we need to state two hypotheses: the null hypothesis (H₀) and the alternative hypothesis (H_A). The null hypothesis is the one we’re testing, while the alternative hypothesis is accepted when the null hypothesis is rejected.

We can formulate the hypotheses in three ways, depending on the population parameter (θ) in relation to a possible value (θ₀):

• Null: θ = θ₀; Alternative: θ ≠ θ₀
• Null: θ ≤ θ₀; Alternative: θ > θ₀
• Null: θ ≥ θ₀; Alternative: θ < θ₀

Consider the alternative hypothesis as the one you suspect to be true or wish to prove. To prove it, you need to reject the null hypothesis.

Step 2: Identify the Test Statistic and Its Probability Distribution

The test statistic is the basis for rejecting the null hypothesis. Its general form is calculated as:

(Sample Statistic – H₀ Value) / Standard Error of the Sample Statistic

Now, we need to determine the test statistic’s probability distribution. In the curriculum, we’ll discuss four distributions: z, t, chi-square, and F-distribution. For now, let’s use the t-distribution. In our example, we collect 30 random observations of the mutual fund’s outperformance data. The degrees of freedom, in our example, are 30 – 1 = 29.

Step 3: Specify the Significance Level

Since hypothesis testing involves making inferences from sample statistics, there’s always a chance of error. There are two types of errors:

• Type 1 error: Rejecting the null hypothesis when it is actually true.
• Type 2 error: Failing to reject the null hypothesis when it is actually false.

The significance level (α) is the probability of making a Type 1 error, while the probability of making a Type 2 error is denoted by β. It’s challenging to quantify β, so we won’t cover it in this lesson. The power of a test is the probability of correctly rejecting a null hypothesis.

Ideally, we want to minimize both types of errors. However, there’s a trade-off between them. Decreasing α increases the probability of Type 2 error. The only way to decrease both errors is to increase the sample size. Due to the central limit theorem, the standard error of the sampling distribution decreases as the sample size increases.

In our example, we’ll set a moderate significance level of ⍺=5%.

Step 4: State the Decision Rule

The decision rule involves determining whether the test statistic is extreme enough to fall outside the critical values. If it does, the result is statistically significant, and we reject the null hypothesis.

For our example, we perform a one-tailed test on the right tail of the distribution. Based on the null hypothesis, we expect the sample mean to be near or less than zero. If the test statistic falls within this region, it’s statistically insignificant. However, if it falls in the right tail, it’s statistically significant, and we reject the null hypothesis.

Using a t-distribution with 29 degrees of freedom, we find a critical value of 1.699 for a one-tail level of 0.05. We reject the null hypothesis if the test statistic is greater than 1.699.

Step 5: Collect Data and Calculate Test Statistic

It’s crucial to ensure the quality of the sample to minimize the probability of flawed conclusions. Make sure the data is free from measurement errors and biases in sample selection. Be aware of time period bias and the effect of starting and ending dates on the conclusions drawn from the sample.

In our example, we collect 30 random observations of the mutual fund’s outperformance data. We find that the sample mean is 4.5% with a standard deviation of 12%. Thus, the test statistic is calculated to be 2.05.

Step 6: Make the Statistical Decision

In our example, the test statistic falls within the rejection region (t > 1.699), so we reject the null hypothesis. We conclude that the mutual fund does outperform the benchmark at a 5% significance level.

Step 7: Make the Economic Decision

Statistical significance doesn’t always imply economic significance. An analyst should consider other factors to determine if the strategy is economically advantageous. In the end, always remember to weigh the statistical findings with real-world context and implications before making a final decision.

Hypothesis Testing in Investment Decision Making

Earlier, we determined that statistically, the mutual fund does indeed outperform the benchmark. But does that mean an investor should buy the mutual fund instead of the index fund? Before we can answer this, we should consider other factors.

Key Factors to Consider

Let’s examine the following factors:

• Cost: Manager, advisory, transaction, and brokerage fees can add up. Sometimes, these fees can erode the outperformance of the mutual fund.
• Taxes: Factor in additional taxes that may be imposed, based on the investor’s tax regime.
• Risk: The mutual fund’s outperformance might be due to the manager taking on higher risk than the benchmark. Determine if this outperformance is worth pursuing, considering the investor’s risk profile.

Alternative Approach: The p-value Method

Before we end this lesson, let’s discuss an alternative approach to the critical value approach. We call this the p-value method.

The p-value is the smallest level of significance for which the null hypothesis can be rejected. In our example, we found our test statistic to be 2.05. From a t-distribution table, we find the corresponding p-value to be 2.5%.

The p-value method allows the reader to make their own conclusions about the significance of the results, rather than setting a specific rejection criteria.

Significance Levels and False Positives

The significance level (α) of a test represents the probability of a Type 1 error, or a false positive. If we perform a single test with a significance level of 5%, the likelihood of a false positive is 5%.

However, conducting multiple tests and comparing the number of significant results to the significance level can be misleading. Instead, we should adjust the significance level for each test using the following formula:

Adjusted Significance Level = (Rank of P-Value) × (Significance Level / Number of Tests)

For example, let’s say we test the hypothesis that the excess return of a mutual fund is significantly greater than zero for 50 different funds. We rank the p-values from lowest to highest and calculate the adjusted significance levels. If only one test has a p-value lower than its adjusted significance level, we fail to reject the null hypothesis at the 5% level.

And that’s all for this lesson! We’ll be going through more examples on hypothesis testing in the coming lessons. See you soon!