Hypothesis testing is a structured way to assess how compatible observed data are with a stated statistical model and test hypothesis. Choosing a test should begin with the research question, design, unit of analysis, outcome type, group structure, data quality and assumptions—not with a software menu. A responsible report gives the question, design, descriptive statistics, sample size, estimate of difference or association, uncertainty such as a confidence interval where appropriate, test statistic, degrees of freedom if relevant, actual p value and a cautious contextual interpretation. A p value is not the probability that a hypothesis is true, nor does a threshold alone decide whether a finding is important (Greenland et al., 2016; USC Libraries, 2026).
What hypothesis testing can and cannot do
A statistical test compares the observed data with the pattern expected under a statistical model. In a familiar null-hypothesis setting, the null might specify no difference, no association or a particular parameter value. The test produces a statistic and a p value that summarise how unusual data at least as discrepant as those observed would be if the whole model—including the test hypothesis and other assumptions—were correct.
That is a narrower claim than many readers assume. A small p value does not identify which assumption may be wrong. It does not measure practical importance, prove a causal explanation, quantify the probability that a null hypothesis is true or tell a researcher whether to take an action. A large p value does not prove there is no difference or association. Greenland and colleagues (2016) emphasise that statistical tests, confidence intervals and power calculations demand close attention to model assumptions and should not be the sole input to scientific inference.
| Statement | Defensible interpretation | Interpretation to avoid |
| p = 0.03 | Under the stated model, data at least this discrepant would be relatively unusual. | “There is a 3% chance that the null hypothesis is true.” |
| p = 0.24 | The data are not especially unusual under the stated model. | “There is no effect” or “the null has been proven.” |
| A 95% confidence interval is narrow | The estimate is relatively precise under the method’s assumptions. | “There is a 95% probability that this fixed parameter lies in this interval.” |
| A result is statistically detectable | The data/model provide evidence against a specified hypothesis at the chosen threshold. | “The effect is large, meaningful or causal.” |
| A group average differs | The observed comparison should be reported with magnitude and uncertainty. | “Group membership caused the difference.” |
Start with the research design, not the test label
Test selection follows a sequence.
- First identify whether the goal is descriptive, comparative, association-focused, predictive or causal.
- Second, identify the unit of analysis and sampling or data-generating process.
- Third, define outcome and explanatory variables, including whether they are categorical, ordered, numerical, paired or repeated over time.
- Fourth, inspect distributions, group sizes, missingness and design features. Only then assess a procedure and its assumptions. The question-first design work is set out in Choosing a Statistics Research Question.
This order matters because the same variable types can arise in designs that demand different analysis. Two measurements from the same respondents are not independent groups. Monthly observations are ordered in time and can be autocorrelated.
Area-level data describe places, not individual persons. A large dataset does not remove the need for a coherent design. The research-design guidance from Texas A&M stresses alignment among problem, question, methods and sample, with analysis flowing from that design (Texas A&M University, 2026).
| Analytical purpose | Example research question | First design issue | Possible family of approach, subject to assumptions |
| Describe one numerical measure | What is the distribution of journey time in the available sample? | Does the sample/data source match the claim? | Descriptive table, plot and summary estimates. |
| Compare two independent groups | How do recorded waiting times differ between two service routes? | Are groups independent and measures comparable? | Difference estimate with suitable two-group comparison. |
| Compare paired measures | How did each participant’s score change from baseline to follow-up? | Are records matched correctly and time/order meaningful? | Paired comparison or repeated-measures approach. |
| Compare several groups | How does the outcome vary across defined categories? | Are group sizes, distributions and variance structure adequate? | Group comparison/model with planned contrasts where justified. |
| Associate two measures | Is expenditure associated with revenue in a monthly dataset? | Are units independent; is time/seasonality relevant? | Scatterplot plus association/regression approach. |
| Associate categories | Is travel mode associated with attendance category? | Are expected counts and coding appropriate? | Contingency-table association approach. |
The table identifies families of approach, not automatic choices. Institutional teaching, subject norms and qualified statistical guidance may be necessary for complex, small, clustered, weighted or sensitive data.
The question-to-test decision map
A useful planning map has seven questions.
1. What is the estimand or comparison of interest?
Write the quantity or relationship you want to learn about. It may be a difference in means, difference in proportions, median difference, association, rate ratio or model coefficient. If this cannot be stated clearly, test selection is premature.
2. What is the unit of analysis?
Each row may represent a person, transaction, area, household, month or event. The test’s independence assumptions must be evaluated at that level. Repeated observations from the same person or area cannot be treated as unrelated records.
3. What are the outcome and explanatory variable types?
Variable type shapes the analysis, but labels alone are not enough. A numerical score may be bounded or skewed. A category may be ordered. A count may have many zeros. Define the data rather than relying on a variable name.
4. What is the design and sampling route?
Random allocation, survey sampling, administrative records, convenience data and public aggregate data each impose different inferential limits. A non-random observational dataset normally needs association wording regardless of the p value.
5. Are assumptions plausible and checked?
Assumptions may include independence, distribution shape, variance structure, appropriate measurement, sufficient cell counts, model form and no unaddressed clustering. Checking does not mean looking for a single perfect diagnostic; it means understanding which deviations materially affect the intended inference.
6. What descriptive evidence accompanies the test?
Show group sizes, centre/spread or counts/proportions, and a clear plot or table. Descriptive Statistics is therefore part of test selection, not a separate task.
7. How will the result be reported and qualified?
Plan to state the estimate, uncertainty, p value, design limitation and practical context. Do not decide the conclusion from a binary threshold alone.
| Data/design feature | Question to resolve before testing | Why it changes the analysis |
| Same people measured twice | Are records genuinely paired? | Pairing affects the estimate and variability calculation. |
| Several observations per school, branch or person | Is clustering present? | Within-cluster observations may be more alike than independent records. |
| Very uneven group sizes | Are smaller groups adequate for the intended procedure? | Estimates and diagnostics can be unstable. |
| Extreme skew or strong outliers | Is mean-based inference suitable and robust in this design? | Extremes can dominate estimates and standard errors. |
| Many tests or outcomes | Was the analysis planned and how will multiplicity be addressed? | Selective reporting can overstate evidence. |
| Time-ordered data | Are trends, seasonality or serial dependence likely? | Standard independent-observation methods may be unsuitable. |
Null and alternative hypotheses
The null hypothesis is usually a precise statement used to generate an expected pattern under a statistical model, often no difference or no association. The alternative may be directional or non-directional depending on prior theory, ethical implications and the analysis plan. It should be set before examining outcome results, not adopted because it favours a preferred narrative.
A useful distinction is between the research hypothesis and the statistical hypothesis. A research hypothesis may state a substantive expectation about a population or process. A statistical null and alternative are formal model statements used in an analysis. They should connect logically, but neither is a substitute for a well-designed study. Barroga and Matanguihan (2022) note that hypotheses are testable predictions linked to research questions, variables and design.
Reporting results: estimate, uncertainty and context
Transparent reporting protects readers from treating a p value as the entire finding. USC Libraries recommends reporting descriptive statistics, confidence intervals, sample sizes, the test statistic, direction, degrees of freedom and actual p value where inferential statistics are used, along with rationale, assumptions and missing-data handling (USC Libraries, 2026).
| Reporting component | Illustrative form | Why it belongs in the result |
| Question and comparison | “We compared recorded wait time between Route A and Route B.” | Makes the target explicit. |
| Data and design | “The analysis included 114 independent weekday observations…” | States the unit and scope. |
| Descriptive evidence | “Median wait times were 8 and 11 minutes…” | Shows the observed data before a test. |
| Estimate | “The estimated difference was 2.4 minutes…” | Gives magnitude and direction. |
| Uncertainty | “95% CI 0.7 to 4.1 minutes…” | Shows precision under the method’s assumptions. |
| Test result | “t(112) = 2.78, p = 0.006…” | Documents model-based compatibility evidence. |
| Qualification | “The observational route comparison cannot establish why routes differed.” | Matches claim to design. |
Avoid writing only “the result was significant”. Significance has a technical, model-dependent meaning and is not a measure of real-world importance. A small difference may be statistically detectable in a very large dataset but unimportant in practice. A meaningful difference may remain uncertain in a small dataset. Discuss the estimate, its uncertainty, the quality of the data and the context together.
Fictional planning application
The following scenario is fictional. Elena is analysing an approved teaching dataset of 160 customers. Her question is whether average reported satisfaction differs between people who used an online booking route and people who booked by telephone. She initially selects a test because it appears first in a software menu.
Her tutor asks her to use the decision map. Elena checks that each row is one customer, the two booking-route groups are distinct, the satisfaction score is defined consistently and missing responses are recorded. She produces a frequency table and distribution plot for each group, records group sizes and checks whether there are unusual patterns. Her estimate is the difference in the two groups’ average score, accompanied by uncertainty.
Elena then reports the test statistic and actual p value in context. She writes that the available data show a difference in reported satisfaction between route groups, but the observational dataset does not establish that the booking route caused the difference. Customer type, wait time and reason for contact may also matter. Her analysis is stronger because the test supports a stated comparison rather than supplying a ready-made story.
Practical analysis sequence
First, state the question and intended estimand. Second, make a variable and design table: unit, outcome type, comparison/explanatory variable, group structure, source, period and sampling route. Third, clean and describe data transparently, checking missingness, distributions, group sizes and repeat observations.
Fourth, choose the minimally sufficient procedure with a documented rationale. Fifth, check relevant assumptions and record what was found. Sixth, report descriptive statistics, estimate, uncertainty and the test output together. Seventh, interpret at the level allowed by the design. Where SPSS is used, retain syntax or a reproducible record, but do not allow the availability of output to determine the research claim.
Critical limitations and safeguards
Do not run many procedures until one gives a preferred threshold, select a subgroup after observing results or omit outcomes that do not fit a narrative. Such practices distort interpretation. Pre-specify central questions and comparisons where possible, preserve data and syntax, and explain genuine analysis changes.
Do not claim individual, clinical, legal, financial or policy advice from a single test. Statistical methods involve assumptions and require subject-matter context. In sensitive or consequential work, seek appropriate qualified supervision. A test can quantify evidence relative to a model; it cannot replace ethical judgement, replication or transparent reporting.
FAQs
Q1. Does a p value below 0.05 prove the hypothesis?
No. It indicates that the data would be relatively unusual under the complete model used to calculate it. It does not give the probability that a hypothesis is true or establish practical importance or causation.
Q2. Should I choose a test before collecting data?
Plan the analysis before collection where possible, but base it on the question, design and expected variable structure. Confirm the final procedure after reviewing data quality and relevant assumptions.
Q3. Why report confidence intervals as well as p values?
An interval gives an estimate of magnitude and precision under the method’s assumptions. It helps readers assess values compatible with the data rather than relying only on a threshold.
Q4. Can SPSS choose the right statistical test for me?
SPSS can run procedures, but it cannot determine whether the research question, design, variables, sampling and assumptions justify them. That requires methodological reasoning.
References
American Statistical Association (2016) Guidelines for Assessment and Instruction in Statistics Education: College Report 2016 (Accessed: 25 August 2026).
Barroga, E. and Matanguihan, G.J. (2022) ‘A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles’, Journal of Korean Medical Science, 37(16), e121 (Accessed: 25 August 2026).
Greenland, S., Senn, S.J., Rothman, K.J., Carlin, J.B., Poole, C., Goodman, S.N. and Altman, D.G. (2016) ‘Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations’, European Journal of Epidemiology, 31, pp. 337–350 (Accessed: 25 August 2026).
NIST/SEMATECH (2025) e-Handbook of Statistical Methods (Accessed: 25 August 2026).
Texas A&M University (2026) Choosing the Right Research Design (Accessed: 25 August 2026).
USC Libraries (2026) Quantitative Methods (Accessed: 25 August 2026).