In the previous chapter we saw how to carr yout statistical inference for quantitative data by using t-tests and ANOVA. In this chapter, we will learn how to use hypothesis testing to analyze categorical data. We will begin by comparing two population proportions, then move into chi-square tests, which allow us to compare observed counts to expected counts and determine whether categorical variables appear to be related.
We have seen how we can carry out hypothesis testing for proportions when we are looking at a single group. But, sometimes it is helpful to determine if the population proportion is different between two or more groups (just like we saw if the mean was different between two or more groups). Therefore, we will be interested in the point estimate \(\hat{p_1} - \hat{p_2}\).
The process for carrying out a hypothesis test for one or two proportions will be exactly the same, the only difference will be in how we calculate the standard error. As a refresher, when we were dealing with one group, the test statistic was calculated as:
\[ \text{test statistic} = \frac{\hat{p} - p_0}{SE} = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} \]
After we calculate the test statistic then we can use a standard normal distribution to calculate the probability of that event or a more extreme event occurring.
If we only have two proportions then there are two scenarios that we should be aware of. The first is when we are interested in is if the testing difference equals a non-zero value and the second is if the testing difference equals zero (meaning \(p_1=p_2\)). The process will be the same for both of these problems, with the only difference being how we calculate the standard error.
If we are wanting to test for a difference of non-zero (meaning \(p_1-p_2=0.2\) or something), then our standard error will be as follows:
\[ SE_{\hat{p_1} - \hat{p_2}} = \sqrt{SE^2_{\hat{p_1}} + SE^2_{\hat{p_2}}} = \sqrt{\frac{\hat{p_1}(1-\hat{p_1})}{n_1} + \frac{\hat{p_2}(1-\hat{p_2})}{{n_2}}} \]
This is an unpooled standard error since we come into the problem assuming the proportions are different and thus cannot ``combine” them when calculating our standard error.
But, if we come into the problem assuming the difference equals 0, meaning we want to see if the proportions are different from each other (which is usually the case), then we can use a pooled standard error. This will look like the following:
\[ SE_{\hat{p_1} - \hat{p_2}} = \sqrt{\overline{p}(1-\overline{p})\left( \frac{1}{n_1} + \frac{1}{n_2}\right)} \qquad \text{where} \qquad \overline{p}=\frac{\hat{p_1}n_1 + \hat{p_2}n_2}{n_1 + n_2} \]
The same assumptions must also be validated before proceeding with the test for two proportions. For two-proportion tests, we want each group to have enough successes and failures. A common rule of thumb is that each group should have at least 5 successes and at least 5 failures. If these conditions are not met then we might have to use other means of testing the differences.
Let’s run through a problem: Working for on a political campaign, we are in-charge of choosing the online campaign ads. We randomly select the visitors who viewed an ad online and find that 50 of the 500 people who viewed ad A clicked it while 60 of the 450 people who viewed ad B clicked it. We want to see if one ad is more effective at getting a voters attention. Therefore, we are interested in testing the difference between the two ads.
\[ H_0: p_A = p_B \qquad \text{and} \qquad H_A: p_A \neq p_B \]
Since we are testing to see if there is no difference between the proportions we will use the pooled standard error:
\[ SE_{\hat{p_A} - \hat{p_B}} = \sqrt{ \frac{50 + 60}{500 + 450} \left( 1- \frac{50 + 60}{500 + 450} \right)\left(\frac{1}{500} + \frac{1}{450}\right)} = 0.02079 \]
We can then calculate the test statistic as:
\[ \frac{\hat{p_B} - \hat{p_A} - (p_B - p_A)}{SE_{\hat{p_A} - \hat{p_B}}} = \frac{\frac{60}{450}-\frac{50}{500}}{0.02079} = 1.60 \]
Now, since our test statistic is 1.6, we can calculate the probability using a standard normal distribution and find it to be 0.1096. This will mean we fail to reject the null hypothesis, meaning we do not have enough evidence to suggest that the two ads have different levels of effectiveness.
x1 <- 50
n1 <- 500
phat1 <- x1/n1
x2 <- 60
n2 <- 450
phat2 <- x2/n2
pbar <- (x1+x2)/(n1+n2)
se <- sqrt(pbar*(1-pbar)*(1/n1 + 1/n2))
se[1] 0.02079136test_stat <- (phat2 - phat1)/se
test_stat[1] 1.60323(1-pnorm(test_stat))*2 # Since it is a 2-sided test[1] 0.1088839We can do something similar in R, but it will give us slightly different results. I have set the continuity correction to be equal to false so that the p-value will be more similar.
prop.test(x=c(x1,x2), n=c(n1,n2), correct = FALSE)
2-sample test for equality of proportions without continuity correction
data: c(x1, x2) out of c(n1, n2)
X-squared = 2.5703, df = 1, p-value = 0.1089
alternative hypothesis: two.sided
95 percent confidence interval:
-0.074295649 0.007628982
sample estimates:
prop 1 prop 2
0.1000000 0.1333333 I just want to quickly mention that if we used the unpooled standard error, we would essentially get the same result. The pooled standard error will be more precise (when the equal proportion assumption holds) while the unpooled standard error will be more robust (more forgiveness).
If we have more than 2 levels then we have to take a different approach in calculating if the proportions are statistically different for each level. To do this, we will not be using a normal distribution, but rather a \(\chi ^2\) distribution (chi-squared). Within this section we will talk about two different tests. The first is the Goodness of fit test which will compare multiple levels within a single variable.
If we have data that falls into a finite number of categories, say \(k\) categories, then we might look into using the Chi-square test. Assume we have \(n\) independent observations, and that these observations fall into one of \(k\) categories. Then, \(E_i= \frac{n}{k}\) is the theoretical frequency for the observations that will fall into category \(i\), and \(O_i\) is the observed frequency for the number of observations that do fall into category \(i\). Then the test statistic will be:
\[ T= \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i}. \]
We can then consult tables or software to determine the \(p\)-value of the test statistic \(T\) and then make a conclusion on whether the sample data fit the desired frequency table well.
Let us look at an example. We are interested in if the proportion of people who like each of the primary colors is the same. To find this out, we sampled 50 people and got the following results (coded as 1=red, 2=blue, 3=yellow):
colors <- c(3, 1, 2, 2, 2, 3, 2, 3, 2, 3, 3, 3, 3, 1, 3, 1, 2, 1, 1,
2, 2, 2, 2, 2, 1, 2, 1, 1, 2, 3, 2, 2, 1, 2, 1, 3, 2, 2,
2, 2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1)
table(colors)colors
1 2 3
15 25 10 We are interested in seeing if the proportions are the same, so our hypothesis test is:
\[ H_0: p_r=p_b=p_y=\frac{1}{3} \qquad HA: \text{at least one is different} \]
We can then calculate the test statistic by calculating the following table with \(E_i=n/k = 50/3= 16.67\)
| color | code | \(O_i\) | \(E_i\) | \(\dfrac{(O_i - E_i)^2}{E_i}\) |
|---|---|---|---|---|
| red | 1 | 15 | 16.67 | 0.1667 |
| blue | 2 | 25 | 16.67 | 4.1667 |
| yellow | 3 | 10 | 16.67 | 2.6667 |
| total | 7.0001 |
Now that we have found our test-statistic we can evaluate the probability using a \(\chi^2\) distribution with the df= # levels - 1. We can also calculate the same thing in R using the chisq.test() function and passing it in a table. Both results will be the same and we can see that we would reject the null hypothesis and conclude that at least one color has a different proportion then the others.
1-pchisq(7.0001, df=2)[1] 0.03019587chisq.test(x=table(colors))
Chi-squared test for given probabilities
data: table(colors)
X-squared = 7, df = 2, p-value = 0.0302If we wanted to test to see if each level was equal to a specified proportion then we could do that as well, we would just need to alter the expected value \(E_i\).
Say we were interested in seeing if the proportions were actually 25% for red, 50% for blue, and 25% for yellow. Then we could make a table again and calculate the test statistic. We can also pass the probabilities into the chisq.test() function and will get the exact same results.
This will give us the following hypothesis test and table:
\[ H_0: p_r=p_y=\frac{1}{4} \text{ and } p_b=\frac{1}{2} \qquad HA: \text{at least one is different} \]
| color | code | \(O_i\) | \(E_i\) | \(\dfrac{(O_i - E_i)^2}{E_i}\) |
|---|---|---|---|---|
| red | 1 | 15 | 12.5 | 0.5 |
| blue | 2 | 25 | 25 | 0 |
| yellow | 3 | 10 | 12.5 | 0.5 |
| total | 1 |
The results below will indicate that we fail to reject the null hypothesis. This means that we do not have enough evidence to suggest that the proportions are different then the null hypothesis values.
1-pchisq(1, df=2)[1] 0.6065307chisq.test(x=table(colors), p=c(.25,.50,.25))
Chi-squared test for given probabilities
data: table(colors)
X-squared = 1, df = 2, p-value = 0.6065If we want to test whether there is a relationship between frequencies in two categorical variables then we can do a test of independence. The idea is the exact same thing as above, but not instead of having one variable we will have 2 variables. Calculating the expected value in each category will be slightly different in that it will be \(E_{ij}=\frac{n_iC_j}{N}\) where \(n_i\) is the number of values in each row and \(C_j\) is the number of values in each column. We will then calculate the test statistic by summing up the \((O_i - E_i)^2 / E_i\) for each cell and use a \(\chi^2\) distribution to calculate the probability. In this case the degrees of freedom will be \(df=(n_{\text{row}} - 1)(n_{\text{col}} - 1)\).
Let’s look at the following problem: Two carloads of manufactured items are sampled randomly to determine if the proportion of defective items is different for the 2 carloads. Assume the carloads are independent. Determine if the loads are independent. The contingency tables can be found below:
| Defective | Not Defective | Total | |
|---|---|---|---|
| Carload 1 | 13 | 73 | \(n_1 = 86\) |
| Carload 2 | 17 | 57 | \(n_2 = 74\) |
| Total | \(c_1 = 30\) | \(c_2 = 130\) | \(N = 160\) |
The expected table if the carloads and product quality were independent would be:
| Defective | Not Defective | Total | |
|---|---|---|---|
| Carload 1 | \(\dfrac{86(30)}{160} = 16.125\) | \(\dfrac{86(130)}{160} = 69.875\) | \(n_1 = 86\) |
| Carload 2 | \(\dfrac{74(30)}{160} = 13.875\) | \(\dfrac{74(130)}{160} = 60.125\) | \(n_2 = 74\) |
| Total | \(c_1 = 30\) | \(c_2 = 130\) | \(N = 160\) |
The null hypothesis would be that the loads are independent of the quality while the alternative hypothesis would be that they are not independent. We could then find the test statistic as:
\[ \text{Test statistic} = \frac{(13-16.125)^2}{16.125} + \frac{(73-69.875)^2}{69.875} + \frac{(17-13.875)^2}{13.875} + \frac{(57-60.125)^2}{60.125} = 1.61 \]
1-pchisq(1.61, df=1)[1] 0.2044918Therefore, we would fail to reject the null hypothesis. We do not have enough evidence to suggest that product quality and carload are associated. In other words, the data does not provide strong evidence that the proportion of defective units differs between the two carloads. We could do this in R and get the same result if we pass it in a contingency table:
cont.table <- matrix(c(13, 73, 17, 57), ncol=2, byrow=TRUE)
rownames(cont.table) <- c("carload 1", "carload 2")
colnames(cont.table) <- c("Defective", "Not Defective")
cont.table Defective Not Defective
carload 1 13 73
carload 2 17 57chisq.test(cont.table, correct = FALSE)
Pearson's Chi-squared test
data: cont.table
X-squared = 1.6116, df = 1, p-value = 0.2043This was a relatively simple example, but we could do it with more levels in each category if we wanted to. For instance: A researcher is investigating the relationship between educational attainment and voting preference in a certain region. They collect data from a random sample of 567 individuals and record both their highest level of education completed and their preferred political party. The data is summarized in the contingency table below:
| Education Level | Party A | Party B | Party C | Party D | Party E |
|---|---|---|---|---|---|
| High School | 50 | 20 | 20 | 30 | 5 |
| Associate’s Degree | 12 | 25 | 15 | 40 | 10 |
| Bachelor’s Degree | 60 | 35 | 25 | 20 | 15 |
| Master’s Degree | 40 | 70 | 30 | 25 | 20 |
Using a significance level of 0.05, conduct a test of independence to determine whether there is a significant association between the highest level of education completed and the preferred political party.
If degree attainment and party preference were independent we would expect the totals to be:
| Education Level | Party A | Party B | Party C | Party D | Party E |
|---|---|---|---|---|---|
| High School | 35.71 | 33.07 | 19.84 | 25.35 | 11.02 |
| Associate’s Degree | 29.14 | 26.98 | 16.19 | 20.69 | 8.99 |
| Bachelor’s Degree | 44.29 | 41.01 | 24.60 | 31.44 | 13.67 |
| Master’s Degree | 52.86 | 48.94 | 29.37 | 37.52 | 16.31 |
The test statistic is then calculated as the sum of \(\dfrac{(O_{ij}-E_{ij})^2}{E_{ij}} = 71.44589\) calculated by:
observed <- c(50, 20, 20, 30, 5, 12, 25, 15, 40, 10,
60, 35, 25, 20, 15, 40, 70, 30, 25, 20)
expected <- c(35.71, 33.07, 19.84, 25.35, 11.02, 29.14, 26.98, 16.19,
20.69, 8.99, 44.29, 41.01, 24.60, 31.44, 13.67, 52.86,
48.94, 29.37, 37.52, 16.31)
test_stat <- sum((observed-expected)^2 / expected)
test_stat[1] 71.445891-pchisq(test_stat, df=(5-1)*(4-1))[1] 1.716414e-10Therefore, we would reject the null hypothesis since the p-value is smaller then our level of significance. We would conclude that we have enough evidence to suggest that education level and party preference is not independent of each other.
We can do this in R if we make ourselves a table first. We will get the same result, just note that we have a slightly different test statistic and p-value due to rounding when calculating the expected values.
cont.table <- matrix(c(50,20,20,30,5,
12,25,15,40,10,
60,35,25,20,15,
40,70,30,25,20),
nrow=4, byrow=TRUE)
rownames(cont.table) <- c("High School", "Associate's",
"Bachelor's", "Master's")
colnames(cont.table) <- c("Party A", "Party B", "Party C",
"Party D", "Party E")
cont.table Party A Party B Party C Party D Party E
High School 50 20 20 30 5
Associate's 12 25 15 40 10
Bachelor's 60 35 25 20 15
Master's 40 70 30 25 20chisq.test(cont.table, correct = FALSE)
Pearson's Chi-squared test
data: cont.table
X-squared = 71.447, df = 12, p-value = 1.715e-10