Hypothesis Testing (t.test, chisq.test)

Published by Brayan K

Hypothesis testing is the statistical procedure for deciding whether a pattern in your data is real or could plausibly have arisen by chance, by comparing it against a null hypothesis of "no effect" and measuring the surprise with a p-value.

Part of the free R course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

In this lesson you'll learn the logic of a test and the meaning of a p-value, run one- and two-sample t.test(), switch to the robust wilcox.test(), test categorical counts with chisq.test(), and read every part of R's output — including the confidence interval.

What You'll Learn in This Lesson

1️⃣ The Logic & the t-test

Every test starts with a null hypothesis (H0) of "no effect," then asks how surprising your data would be if H0 were true. The t.test() compares means. A one-sample test checks a mean against a fixed target.

Worked example: one-sample t.test()

# A hypothesis test asks: is the pattern in my data REAL, or could it
# easily be chance? We state a NULL hypothesis (H0: "no effect") and
# ask how surprising our data would be IF H0 were true. That surprise
# is the p-value.

# One-sample t-test: is the mean weight different from a target of 50g?
weights <- c(48, 52, 49, 51, 47, 53, 50, 46, 54, 49)

t.test(weights, mu = 50)
# Read the printout, NOT just one number:
#   t          - the test statistic
#   df         - degrees of freedom
#   p-value    - chance of data this extreme if the true mean were 50
#   95% CI     - plausible range for the true mean
#   sample est - the observed mean
#
# If the p-value is below your threshold (commonly 0.05) you REJECT H0
# and conclude the mean differs from 50. Otherwise you don't.

Output


	One Sample t-test

data:  weights
t = -0.12157, df = 9, p-value = 0.9059
alternative hypothesis: true mean is not equal to 50
95 percent confidence interval:
 48.03916 51.76084
sample estimates:
mean of x
     49.9

The two-sample version asks whether two groups differ. R's default is Welch's t-test, which doesn't assume the groups share the same variance — the safer choice in practice.

Worked example: two-sample t.test()

# Two-sample t-test: do two GROUPS have different means? H0 is "the
# two population means are equal".
group_a <- c(20, 22, 19, 23, 21, 18, 24)
group_b <- c(25, 27, 24, 26, 28, 23, 29)

# Formula interface needs a tidy data frame (values + a group factor):
scores <- data.frame(
  value = c(group_a, group_b),
  group = rep(c("A", "B"), each = 7)
)

t.test(value ~ group, data = scores)
# By default R runs WELCH's t-test, which does NOT assume equal
# variances — the safe choice. Add var.equal = TRUE only if you have
# good reason to assume the spreads are the same.

# Same hypothesis, two equivalent calls:
t.test(group_a, group_b)            # supply the two vectors directly

Output


	Welch Two Sample t-test

data:  value by group
t = -4.3301, df = 12, p-value = 0.0009785
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
 -7.515876 -2.484124
sample estimates:
mean in group A mean in group B
             21              26


	Welch Two Sample t-test

data:  group_a and group_b
t = -4.3301, df = 12, p-value = 0.0009785
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -7.515876 -2.484124
sample estimates:
mean of x mean of y
       21        26

2️⃣ When Normality Fails: wilcox.test

The t-test leans on roughly normal data. With small, skewed, or outlier-ridden samples, switch to the non-parametric wilcox.test(), which compares distributions through ranks and is far more robust.

Worked example: wilcox.test()

# t.test assumes roughly normal data. When that's doubtful (small
# samples, skew, outliers, or ranks), use the NON-PARAMETRIC
# wilcox.test, which compares distributions via ranks instead of means.
group_a <- c(20, 22, 19, 23, 21, 18, 24)
group_b <- c(25, 27, 24, 26, 28, 23, 29)

wilcox.test(group_a, group_b)
# H0: the two distributions are the same (no shift). The output gives a
# W statistic and a p-value you interpret exactly like the t-test's.

# One-sample / paired versions exist too:
wilcox.test(group_a, mu = 21)             # one-sample
wilcox.test(group_a, group_b, paired = TRUE)  # paired observations

Output


	Wilcoxon rank sum test with continuity correction

data:  group_a and group_b
W = 2, p-value = 0.004844
alternative hypothesis: true location shift is not equal to 0


	Wilcoxon signed rank test with continuity correction

data:  group_a
V = 10.5, p-value = 1
alternative hypothesis: true location is not equal to 21


	Wilcoxon signed rank test with continuity correction

data:  group_a and group_b
V = 0, p-value = 0.01788
alternative hypothesis: true location shift is not equal to 0

You read its output the same way — a test statistic (W) and a p-value — but the underlying null is "the two distributions are the same," and no normality is assumed.

3️⃣ Categorical Data: chisq.test

When your data are counts in categories rather than numbers you average, the chi-squared test asks whether two categorical variables are associated. You feed it a contingency table of observed frequencies.

Worked example: chisq.test() on a table

# chisq.test works on COUNTS, not means. It asks whether two
# categorical variables are associated (independent or not).
# Build a contingency table of observed counts:
tbl <- matrix(c(30, 10, 15, 25), nrow = 2,
              dimnames = list(
                treatment = c("Drug", "Placebo"),
                outcome   = c("Recovered", "Not")
              ))
tbl

chisq.test(tbl)
# H0: treatment and outcome are INDEPENDENT (no association).
# The output gives X-squared, df, and a p-value. A small p-value means
# the variables ARE associated — the drug relates to recovery here.

# Goodness-of-fit (one variable vs expected proportions):
rolls <- c(8, 12, 9, 11, 10, 10)        # 60 die rolls, 6 faces
chisq.test(rolls)                       # H0: the die is fair (uniform)

Output

         outcome
treatment Recovered Not
  Drug           30  15
  Placebo        10  25

	Pearson's Chi-squared test with Yates' continuity correction

data:  tbl
X-squared = 9.9556, df = 1, p-value = 0.001604


	Chi-squared test for given probabilities

data:  rolls
X-squared = 1, df = 5, p-value = 0.9626

Finally, every test returns an object you can index. Pulling out the $p.value or $conf.int lets you use results in reports and pipelines.

Worked example: extracting results

# Pull a single number OUT of a test for use in a report or pipeline.
# Every test returns an object you can index with $.
weights <- c(48, 52, 49, 51, 47, 53, 50, 46, 54, 49)
res <- t.test(weights, mu = 50)

res$p.value        # just the p-value
res$conf.int       # the 95% confidence interval (two numbers)
res$estimate       # the sample mean
res$statistic      # the t value

# tidy() from the 'broom' package turns ANY test into a one-row tibble,
# which is ideal for tables and for testing many groups at once:
# library(broom); tidy(res)

Output

[1] 0.9059136
[1] 48.03916 51.76084
attr(,"conf.level")
[1] 0.95
mean of x
     49.9
         t
-0.1215661

Your turn. Fill in the ___ blank, run it, and read the p-values and intervals.

🎯 YOUR TURN: test a new teaching method

# 🎯 YOUR TURN — replace each ___ then run it.
# A teacher wants to know if a new method beats the old class average
# of 70, and whether two sections differ.

old_avg  <- 70
new_scores <- c(74, 78, 69, 81, 77, 72, 85, 79)

# 1) One-sample t-test: is the new mean different from 70?
test1 <- t.test(new_scores, mu = ___)   # 👉 replace ___ with old_avg
print(test1)        # look at the p-value and the 95% CI

# 2) Compare two sections with a two-sample (Welch) t-test.
section_x <- c(74, 78, 69, 81, 77)
section_y <- c(66, 71, 68, 73, 70)
test2 <- t.test(section_x, section_y)
print(test2)

# 3) Print JUST the p-value of test2 (hint: use $p.value).
cat("Section comparison p =", round(test2$p.value, 4), "\n")

Output


	One Sample t-test

data:  new_scores
t = 3.8044, df = 7, p-value = 0.006676
alternative hypothesis: true mean is not equal to 70
95 percent confidence interval:
 72.60188 81.14812
sample estimates:
mean of x
   76.875


	Welch Two Sample t-test

data:  section_x and section_y
t = 2.62, df = 6.5092, p-value = 0.03673
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
  0.5175985 11.8824015
sample estimates:
mean of x mean of y
     75.8      69.6

Section comparison p = 0.0367

Mini-Challenge: A/B Test the Right Way

The hardest part of testing is picking the RIGHT test. These are counts by category, so it's a chi-squared problem — not a t-test. Build the table, run it, and write a plain-English conclusion.

🎯 MINI-CHALLENGE: sign-ups by layout

# 🎯 MINI-CHALLENGE: choose and run the right test.
#
# You run an A/B test on a website. Of 200 visitors shown layout A,
# 48 signed up; of 200 shown layout B, 72 signed up.
#
# 1. These are COUNTS of a categorical outcome (signed up: yes/no) by
#    group (A/B) -> the right tool is chisq.test, not t.test.
# 2. Build a 2x2 contingency table with matrix() (rows = A/B,
#    columns = signed up / not).
# 3. Run chisq.test() on it. State H0 in a comment.
# 4. Read off the p-value and write a one-line plain-English conclusion
#    about whether layout affects sign-up rate.
# 5. Bonus: extract the p-value with $p.value and round it.
#
# Write it yourself, then run in RStudio or rdrr.io/snippets.

# your code here

Common Errors (and the fix)

Pro Tips

📋 Quick Reference — Hypothesis Tests

TaskR Syntax
Mean vs targett.test(x, mu = 50)
Two-group meanst.test(value ~ group, data = df)
Robust alternativewilcox.test(a, b)
Category associationchisq.test(table)
Paired testt.test(after, before, paired = TRUE)
Pull the p-valuet.test(x, mu = 50)$p.value

🎉 Lesson Complete!

Practice quiz

What does a p-value measure?

  • The probability of data this extreme if the null hypothesis were true
  • The probability the null is true
  • The size of the effect
  • The sample size needed

Answer: The probability of data this extreme if the null hypothesis were true. A p-value is the chance of data at least this extreme assuming the null (no effect).

Which test compares a numeric MEAN against a target or between two groups?

  • chisq.test()
  • t.test()
  • fisher.test()
  • prop.test()

Answer: t.test(). t.test() compares means (one-sample against mu, or two groups).

By default, R's two-sample t.test() runs which version?

  • Paired t-test
  • A one-sample test
  • Welch's t-test (does not assume equal variances)
  • A z-test

Answer: Welch's t-test (does not assume equal variances). R defaults to Welch's t-test, which does not assume equal variances.

Which non-parametric test replaces t.test when normality is doubtful?

  • chisq.test()
  • cor.test()
  • aov()
  • wilcox.test()

Answer: wilcox.test(). wilcox.test() compares distributions via ranks and is robust to non-normality.

Which test works on COUNTS to check association between categorical variables?

  • chisq.test()
  • t.test()
  • wilcox.test()
  • lm()

Answer: chisq.test(). chisq.test() tests association in a contingency table of counts.

In t.test(value ~ group, data = df), what does the formula say?

  • Multiply value by group
  • Sort value by group
  • Compare value across the levels of group
  • Add value and group

Answer: Compare value across the levels of group. value ~ group compares the numeric value across the groups; data = df supplies the columns.

Conventionally, what does p < 0.05 lead you to do?

  • Accept the null as proven
  • Reject the null hypothesis
  • Ignore the result
  • Increase the sample

Answer: Reject the null hypothesis. p < 0.05 is taken as evidence against the null, so you reject it (by convention).

If a two-sample 95% CI for the difference of means does NOT contain 0, this agrees with:

  • A non-significant result
  • An error in the data
  • No possible conclusion
  • A significant difference (p < 0.05)

Answer: A significant difference (p < 0.05). A CI excluding 0 is consistent with a significant difference at the 0.05 level.

How do you extract just the p-value from a test result res?

  • res$p.value
  • res$pval
  • pvalue(res)
  • res[p]

Answer: res$p.value. Tests return an object you index with $; res$p.value gives the p-value.

A non-significant result (p > 0.05) means:

  • The null is definitely true
  • There is no effect at all
  • You lack evidence against the null, not proof it is true
  • The test was run wrong

Answer: You lack evidence against the null, not proof it is true. Absence of evidence is not evidence of absence; you simply failed to reject the null.

Continue this course

Frequently asked questions

What exactly is a p-value, and what does p < 0.05 mean?

A p-value is the probability of seeing data at least as extreme as yours IF the null hypothesis (no effect) were true. A small p-value means your data would be surprising under 'no effect', which is evidence against the null. The 0.05 threshold is a convention, not a law of nature: p < 0.05 means you'd see a result this extreme less than 5% of the time by chance alone, so you 'reject the null'. Crucially, the p-value is NOT the probability that the null is true, and it says nothing about how BIG or important the effect is.

When should I use a t-test versus wilcox.test?

Use t.test when your data are roughly normal (or your samples are large enough that the sample mean is approximately normal by the central limit theorem) and you care about comparing means. Use wilcox.test (the Mann-Whitney / Wilcoxon rank test) when normality is doubtful — small samples, heavy skew, strong outliers, or ordinal data — because it compares distributions through ranks and is robust to those problems. The trade-off is a little less power than a t-test when the data really are normal.

What's the difference between chisq.test and t.test?

They answer different kinds of questions. t.test compares the MEANS of a numeric variable (e.g. average weight in two groups). chisq.test works on COUNTS of categorical variables and asks whether two categories are associated, using a contingency table of frequencies (e.g. is recovery related to treatment?). If your outcome is a number you average, reach for t.test; if your data are counts in categories, reach for chisq.test. Using the wrong one on the wrong data type gives meaningless results.

How do I read the confidence interval in the output?

A 95% confidence interval is a plausible range for the true value (the true mean, or the true difference between two means). If a two-sample interval for the difference does NOT contain 0, that agrees with a significant result (p < 0.05) — zero difference is implausible. If it straddles 0, the data are consistent with no difference. The CI is often more useful than the p-value because it shows both the direction AND the magnitude of the effect, with its uncertainty, rather than collapsing everything into a single yes/no.