Games-Howell Post Hoc Test Guide

The Games-Howell post hoc test compares every pair of independent group means when a common population variance cannot reasonably be assumed. Unlike Tukey HSD, it does not rely on one pooled error variance: each pair receives its own Welch-style standard error and Welch-Satterthwaite degrees of freedom, while multiplicity is handled through the studentized range distribution.

This makes Games-Howell especially useful when heteroscedasticity and unequal group sizes occur together. It is not, however, a non-parametric or rank-based test, and it does not remove the need to think about independence, distribution shape, outliers, sample size, or whether all pairwise mean comparisons are actually the scientific question.

Quick answer: Use Games-Howell when you want all-pairs comparisons among independent group means and an equal-variance model is not defensible. Unequal sample sizes are allowed. If there are only two groups, use an appropriate two-sample method such as Welch's t-test instead. If only treatment-versus-control comparisons matter, an all-pairs procedure may be unnecessary.

What Is the Games-Howell Post Hoc Test?

Games-Howell is a simultaneous multiple-comparison procedure for testing differences between group means. With four groups, for example, it evaluates all six unique pairs; with five groups, it evaluates ten. The method is associated with Paul A. Games and John F. Howell, whose 1976 Monte Carlo study examined pairwise procedures under unequal sample sizes and heterogeneous variances. [1]

The key idea is simple. Tukey-style equal-variance methods estimate a common within-group variance and use that shared estimate across the comparison family. Games-Howell instead estimates uncertainty separately for each pair. A comparison involving two precise, low-variance groups can therefore have a much smaller standard error than a comparison involving a small, highly variable group.

Is Games-Howell parametric or non-parametric?

Games-Howell is parametric, not non-parametric. The standard procedure compares arithmetic means and is built from Welch-type variance calculations plus the studentized range distribution. PMCMRplus documentation explicitly describes the method as an all-pairs test for normally distributed data with unequal group variances and labels it a parametric method. [3]

Calling Games-Howell “non-parametric” usually comes from confusing robustness to unequal variances with distribution-free inference. A genuinely rank-based post hoc procedure, such as Dunn's test after a Kruskal-Wallis analysis, answers a different statistical question using ranks rather than ordinary sample means.

When Should You Use Games-Howell?

Games-Howell is a strong candidate when the outcome is quantitative, observations are independent across groups, every pair of group means is of interest, and the common-variance assumption required by pooled-error procedures is doubtful. Unequal sample sizes do not invalidate the method and often occur alongside unequal variances in observational and applied research.

A significant Levene or Brown-Forsythe variance test can support the conclusion that variances differ, but it should not be treated as an automatic switch. Variance heterogeneity is a matter of degree and pattern. Standard deviations of 10.0 and 10.2 are technically unequal, but they do not create the same problem as standard deviations of 2 and 20. Sample size also matters because the interaction between variance and group size can substantially change pairwise standard errors and error rates. Simulation evidence shows that the configuration in which small groups also have large variances can be particularly challenging. [2]

Unequal sample size alone is not a reason to choose Games-Howell. Tukey-Kramer is the unequal-sample-size extension of the Tukey procedure when a common variance model remains reasonable. The more important distinction is whether pooling the within-group variances is defensible.

Very small groups deserve caution. Games and Howell's original simulations found satisfactory error control with unequal group sizes of roughly six or more under the conditions they studied, while performance with groups around three observations was questionable. This is not a hard cutoff where n = 5 is invalid and n = 6 is automatically safe; it is a warning that variance estimates and Welch degrees of freedom become unstable when groups are tiny. [1]

Games-Howell and Welch ANOVA Answer Different Questions

Welch ANOVA and Games-Howell are commonly paired because they solve related but different problems. Welch ANOVA is an omnibus test: it asks whether the group means are all compatible with one common population mean while allowing unequal variances. Games-Howell is a pairwise procedure: it asks which specific group means differ while controlling multiplicity across the all-pairs family.

The conventional workflow is therefore Welch ANOVA followed by Games-Howell when the omnibus result is significant and pairwise follow-up is part of the analysis plan. That workflow is easy to explain and remains common in applied research.

There is an important nuance, however. A significant omnibus F test is not a mathematical prerequisite for every simultaneous multiple-comparison procedure. Methodological reviews classify Games-Howell among procedures that can control the comparison family without requiring a preliminary omnibus gatekeeper. [10] If the pairwise comparisons were the prespecified scientific targets, the analysis plan can focus directly on those simultaneous comparisons. If the study was designed around an omnibus hypothesis followed by exploratory post hoc testing, retaining the omnibus-first workflow is usually clearer.

The practical lesson is not “skip ANOVA.” It is that omnibus and pairwise hypotheses are different, so apparently inconsistent results do not automatically indicate an error.

What Assumptions Does Games-Howell Make?

Games-Howell removes the requirement that all groups share the same population variance. It also allows unequal group sizes. Those are its main advantages, not a license to ignore every other assumption.

The observations should be independent within the design being analyzed. Repeated measurements on the same participants, matched pairs, clustered observations, or longitudinal data require methods that represent that dependence. Standard Games-Howell is for independent groups.

The outcome should also be suitable for inference about means. Standard derivations are parametric and commonly framed for approximately normal data or residuals, especially when samples are small. Simulation research suggests ordinary mean-based methods can tolerate some non-normality, but severe skew, heavy tails, influential outliers, or extremely small groups can make the mean and variance estimates unreliable. Sauder and DeMars explicitly limited their recommendation to data that are close to normal and independent. [2]

Constant groups create another practical edge case. If every observation in a group is identical, its sample variance is zero. Current rstatix documentation notes that some Games-Howell comparisons can then be undefined and returned as NA, as can comparisons involving a group with only one observation. [4]

How the Games-Howell Test Works

For groups i and j, let the observed means be M_i and M_j, sample variances be s_i^2 and s_j^2, and sample sizes be n_i and n_j.

The pairwise mean difference is:

Difference_ij = M_i - M_j

The Welch standard error for that pair is:

SE_W = sqrt(s_i^2 / n_i + s_j^2 / n_j)

The corresponding Welch statistic is:

t_ij = (M_i - M_j) / SE_W

Games-Howell then converts this to the studentized-range scale:

q_ij = sqrt(2) x |t_ij|

The degrees of freedom are calculated separately for every pair using the Welch-Satterthwaite approximation:

df_ij = (s_i^2/n_i + s_j^2/n_j)^2 / [ (s_i^2/n_i)^2/(n_i-1) + (s_j^2/n_j)^2/(n_j-1) ]

The p-value is obtained from the studentized range distribution using the total number of groups and that pair's own degrees of freedom. This is why a Games-Howell output can show a different df for every row. PMCMRplus and rstatix document this Welch-plus-studentized-range construction. [3][4]

Confidence intervals use the same pair-specific uncertainty and an appropriate studentized-range critical value. Because the multiplicity is built into the procedure, an additional Bonferroni correction is normally unnecessary; rstatix explicitly states that no extra p-value adjustment is needed after its Games-Howell calculation. [4]

Games-Howell vs Tukey HSD: Which Should You Use?

The main difference is the variance model. Tukey HSD and Tukey-Kramer use a common pooled within-group variance. Games-Howell estimates the variance contribution separately for each pair and changes the degrees of freedom accordingly.

Situation	Usually more appropriate	Why
Similar variances and similar sample sizes	Tukey HSD	Efficient pooled-error all-pairs procedure
Similar variances but unequal sample sizes	Tukey-Kramer	Handles unequal n while retaining common variance
Meaningfully unequal variances	Games-Howell	Does not require a common variance
Unequal variances and unequal sample sizes	Games-Howell	Pair-specific SE and df address both problems
Two independent groups	Welch's t-test when equal variance is doubtful	No multiple-comparison family exists
Treatments compared only with one control	A Dunnett-type approach suited to the variance model	All-pairs testing answers more questions than required

It is also inaccurate to say that Tukey always inflates Type I error whenever variances differ. Heteroscedasticity can make a pooled-variance procedure too liberal or too conservative depending partly on how variance and sample size are associated. The problem is miscalibration, not a guaranteed one-direction error. Current statsmodels documentation warns that Tukey HSD can have rejection rates far from the nominal level when variances differ, especially when group sizes also differ. [7]

Simulation work comparing many procedures found Games-Howell, Dunnett C, Dunnett T3, and Tamhane T2 among the procedures that maintained Type I error best under unequal-size and unequal-variance conditions, with Games-Howell showing a small power advantage in the conditions studied. The same work warns that performance can deteriorate with very small groups. [2]

Games-Howell Compared With Other Post Hoc Tests

Bonferroni-adjusted Welch t-tests are a defensible alternative when you explicitly want to run selected pairwise Welch tests and control multiplicity with a general-purpose correction. They are not identical to Games-Howell because they use a different familywise adjustment and critical value.

Scheffé's procedure is designed for a broader family of contrasts, not only pairwise mean differences, but its classical form relies on a pooled error variance. If unequal variances are the central problem, that common-variance framework is exactly what Games-Howell avoids.

Standard Dunnett testing answers a different research question: treatment groups versus one designated control. IBM SPSS distinguishes that control-comparison procedure from unequal-variance all-pairs options such as Games-Howell, Tamhane T2, Dunnett T3, and Dunnett C. [5]

For small samples, there is no universally superior choice under every distributional condition. GraphPad Prism currently recommends Dunnett T3 rather than Games-Howell for smaller group sizes in its own workflow, while other simulation studies favor Games-Howell among unequal-variance procedures more generally. That difference is a useful reminder that software defaults and simulation conditions should not be turned into universal laws. [9][2]

Worked Example: Why Pair-Specific Variance Matters

Consider a synthetic four-group example generated for illustration with NumPy's random generator using seed 2. The generating distributions were A: n = 20, mean 50, SD 4; B: n = 16, mean 55, SD 5; C: n = 10, mean 60, SD 12; and D: n = 30, mean 54, SD 3. The observed sample summaries are shown below.

Group	n	Observed mean	Observed SD
A	20	49.80	3.61
B	16	55.62	5.34
C	10	59.70	11.69
D	30	54.10	3.29

A median-centered Levene/Brown-Forsythe variance test is significant, W = 9.38, p < .001, and Welch's ANOVA is also significant, F(3, 26.06) = 8.29, p < .001. The Games-Howell results are:

Comparison	Mean difference	Welch df	Adjusted p	95% CI	Decision
A - B	-5.82	25.32	.005	[-10.11, -1.53]	Significant
A - C	-9.90	9.87	.101	[-21.51, 1.70]	Not significant
A - D	-4.30	38.19	< .001	[-7.00, -1.59]	Significant
B - C	-4.08	11.38	.731	[-15.85, 7.68]	Not significant
B - D	1.52	21.25	.728	[-2.55, 5.60]	Not significant
C - D	5.61	9.48	.476	[-5.97, 17.18]	Not significant

The most instructive row is A versus C. Its raw mean difference, 9.90, is larger than the A-versus-B difference, 5.82, yet A versus C is not significant. Group C is small and highly variable, so the pair-specific standard error is much larger and the confidence interval is wide.

Now apply a pooled-variance Tukey test to the same synthetic data. Tukey flags A-C and C-D as statistically significant, whereas Games-Howell does not. The pooled variance lets the lower-variance groups contribute information to comparisons involving the highly variable Group C, producing narrower uncertainty than the Games-Howell model considers appropriate for those pairs. This is exactly why the variance model matters more than simply inspecting raw mean gaps.

How to Run Games-Howell in SPSS, R, and Python

SPSS

In IBM SPSS Statistics, open Analyze > Compare Means > One-Way ANOVA, place the outcome in the dependent list and the grouping variable in the factor box, then open Post Hoc and select Games-Howell under the procedures that do not assume equal variances. IBM also lists Tamhane T2, Dunnett T3, and Dunnett C in that unequal-variance group. [5]

R with rstatix

rstatix::games_howell_test() returns pairwise mean differences, confidence intervals, Welch degrees of freedom, and Tukey-adjusted p-values. Current versions can also add a Welch-oriented Cohen's d effect size. [4]

library(rstatix) games_howell_test( data = df, outcome ~ group, detailed = TRUE, effect.size = TRUE )

PMCMRplus provides another implementation through gamesHowellTest(), with documentation that explicitly shows the pairwise Welch-style statistic and the studentized-range p-value calculation. [3]

Python with statsmodels

As of statsmodels 0.15.0, released on August 27, 2026, Games-Howell is available directly through pairwise_tukeyhsd() by setting use_var="unequal". The 0.15 release notes identify Games-Howell support as a new enhancement, so older Python tutorials that say statsmodels cannot perform the test are now outdated. [6][7]

from statsmodels.stats.multicomp import pairwise_tukeyhsd result = pairwise_tukeyhsd( endog=df["outcome"], groups=df["group"], alpha=0.05, use_var="unequal" ) print(result.summary())

Pingouin remains another convenient Python option through pairwise_gameshowell(), particularly for pandas-based workflows. Software can differ slightly in rounding and numerical algorithms, so compare unrounded results and package versions before assuming small discrepancies indicate a statistical error.

Excel

Microsoft's built-in Analysis ToolPak provides one-way ANOVA and unequal-variance two-sample t-tests, but its documented analysis list does not include a named Games-Howell post hoc procedure. [11] A third-party add-in or manual implementation is therefore needed. For research that must be reproducible and easy to audit, R, SPSS, Python, or another package with a documented Games-Howell implementation is usually the cleaner choice.

How to Interpret Games-Howell Results

The mean difference tells you the direction and size of the observed difference. If software reports Group A minus Group B, a negative estimate means A's mean is lower than B's. Always confirm the subtraction order because packages differ in how they label contrasts.

The adjusted p-value answers whether the observed difference is statistically significant after accounting for the all-pairs comparison family. With alpha = .05, p < .05 is commonly treated as statistically significant, but the p-value should not be used as a measure of effect magnitude.

The 95% confidence interval shows a range of mean differences compatible with the procedure's model and multiplicity adjustment. If a two-sided interval contains zero, the corresponding comparison will ordinarily not be significant at the matching alpha level. A wide interval can be more informative than a non-significant p-value because it reveals poor precision.

The pair-specific degrees of freedom are a diagnostic signature of the method. Different df values across rows are expected because each pair has different sample sizes and variances.

Finally, report effect size and domain importance where possible. A difference can be statistically significant but trivial in practice, or practically important but estimated too imprecisely to cross a significance threshold. Current rstatix can calculate a Welch-oriented Cohen's d alongside Games-Howell output, although researchers should choose and interpret effect-size metrics in line with their discipline and study design. [4]

How to Report a Games-Howell Post Hoc Test

A useful report gives the reason for the unequal-variance procedure, the relevant omnibus result if one was used, and each important pairwise comparison with direction, mean difference, confidence interval, and adjusted p-value. Descriptive statistics and an effect-size measure add information that significance testing alone cannot provide.

Using the synthetic example above, a concise results paragraph could read:

A Welch one-way ANOVA indicated that mean scores differed across groups, F(3, 26.06) = 8.29, p < .001. Because the groups showed substantial variance heterogeneity, Games-Howell comparisons were used. Group A scored lower than Group B (mean difference = -5.82, 95% CI [-10.11, -1.53], p = .005) and Group D (mean difference = -4.30, 95% CI [-7.00, -1.59], p < .001). The A-C comparison was not statistically significant despite a larger raw mean difference (mean difference = -9.90, 95% CI [-21.51, 1.70], p = .101), reflecting the lower precision associated with Group C's smaller sample and greater variability.

Do not report only “Games-Howell was significant.” The procedure produces multiple hypotheses, so the reader needs to know which pairs differed and in what direction.

Common Games-Howell Mistakes and Practical Limitations

The most persistent mistake is calling Games-Howell non-parametric. It is a mean-based parametric procedure, not a rank transformation or rank-sum test. [3]

Another mistake is applying Bonferroni or false-discovery-rate adjustment to p-values that have already been produced by a standard Games-Howell all-pairs procedure. That changes the inferential procedure and is usually redundant unless the researcher is controlling an additional, broader family of hypotheses beyond the Games-Howell comparison set. [4]

A third mistake is letting Levene's p-value make the entire methodological decision. A variance test is useful evidence, but analysts should also examine group SDs, group sizes, plots, outliers, and the scientific comparison set. Large samples can detect small variance differences that have little practical impact; small samples can fail to detect important heterogeneity.

A fourth mistake is assuming Games-Howell fixes severe non-normality, dependence, or poor measurement. It does not. If observations are clustered or repeated, use a model that represents that dependence. If the outcome is extremely skewed, bounded, count-based, ordinal, or dominated by outliers, consider whether a different estimand or model is more appropriate than simply changing the post hoc test.

Finally, do not treat “n >= 6” as a universal validity rule. The original simulation result is best interpreted as a warning about tiny groups, not as a guarantee above a single threshold. [1]

Games-Howell Test FAQs

What is the Games-Howell test used for?

It is used for simultaneous pairwise comparisons of independent group means when equal variances cannot be assumed. It is especially useful when heteroscedasticity occurs together with unequal group sizes.

When should I use Games-Howell instead of Tukey?

Use Games-Howell when the all-pairs mean-comparison question is appropriate and a common within-group variance is not defensible. Use Tukey or Tukey-Kramer when the pooled-variance model is reasonable.

Is Games-Howell non-parametric?

No. Standard Games-Howell is a parametric mean-comparison procedure. It uses ordinary means and variances, Welch-Satterthwaite degrees of freedom, and the studentized range distribution. [3]

Does Games-Howell require equal sample sizes?

No. Unequal sample sizes are allowed. The method explicitly incorporates each group's own n and variance into every pairwise standard error.

Does Games-Howell require normality?

It is not distribution-free. Standard documentation and simulation recommendations generally frame it for normal or approximately normal independent data, with greater caution required when samples are very small or distributions are strongly non-normal. [3][2]

Can I use Games-Howell after Welch ANOVA?

Yes. Welch ANOVA followed by Games-Howell is a common and coherent workflow for a heteroscedastic one-way design because Welch handles the global mean comparison and Games-Howell handles the pairwise comparisons.

Can I use Games-Howell after a regular one-way ANOVA?

Technically, the pairwise procedure uses its own unequal-variance calculations, so it can be computed independently of the classical ANOVA error term. Methodologically, if heteroscedasticity is serious enough to motivate Games-Howell, Welch ANOVA is usually more coherent for the omnibus step than the equal-variance F test.

Do I need a significant omnibus ANOVA first?

Not as a universal mathematical requirement. Games-Howell is a simultaneous multiple-comparison procedure that can control the pairwise family itself. An omnibus-first rule is still a common and reasonable workflow when the study's inferential plan is explicitly “global test, then exploratory follow-up.” [10]

Does Games-Howell already adjust for multiple comparisons?

Yes, in its standard all-pairs form. It uses the studentized range distribution to account for the comparison family, so applying another routine Bonferroni adjustment to the same p-values is generally unnecessary. [4]

What is the minimum sample size for Games-Howell?

There is no universal hard minimum that guarantees valid inference. The original 1976 simulation found better Type I error control when unequal groups had about six or more observations and raised concern about groups around three, while later work also warns that very small groups can make the procedure liberal. Treat tiny groups as a precision and robustness problem rather than relying on a magical cutoff. [1][2]

Can Games-Howell be used with two groups?

The calculation is possible, but there is only one comparison, so a multiple-comparison procedure adds no practical benefit. An unequal-variance two-sample Welch t-test is the simpler choice.

What does a significant Games-Howell result mean?

It means the data provide evidence that the two population means in that specific comparison differ under the Games-Howell model and the chosen familywise error level. It does not tell you whether the difference is practically important, which is why the estimate, confidence interval, and effect size should also be considered.

Games-Howell Test: The Practical Decision

The Games-Howell post hoc test is best understood as a solution to a specific problem, not as a generic “safe” replacement for every post hoc procedure. Use it when you need all-pairs comparisons of independent group means and a single pooled variance is not credible, particularly when unequal variances and unequal group sizes occur together.

The highest-impact checks come before clicking the post hoc button: confirm that independence is defensible, decide whether means and all pairwise comparisons match the research question, inspect the size and pattern of the group variances, and identify very small or degenerate groups. Then use Welch ANOVA when an unequal-variance omnibus test is part of the analysis plan, run Games-Howell for the pairwise family, and interpret estimates and simultaneous confidence intervals alongside adjusted p-values and effect sizes.

The most important limitation is equally simple: Games-Howell solves the common-variance problem. It does not solve dependence, severe data-quality problems, inappropriate outcome modeling, or an analysis plan that asks the wrong comparisons.

You May Also Like Correlation and Rank Correlation in Statistics

References

Games, P. A., & Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal N’s and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113–125. https://doi.org/10.3102/10769986001002113

GraphPad Software. (n.d.). How the Dunnett T3, Games and Howell, and Tamhane T2 tests work. GraphPad Prism.

IBM. (n.d.). One-way ANOVA post hoc tests. IBM SPSS Statistics.

Microsoft. (n.d.). Use the Analysis ToolPak to perform complex data analysis. Microsoft Support.

Pingouin. (n.d.). Statistical API. Pingouin.

PMCMRplus. (n.d.). Games-Howell test. RDocumentation.

rstatix. (n.d.). Games Howell post-hoc tests. Datanovia.

Sauder, D. C., & DeMars, C. E. (2019). An updated recommendation for multiple comparisons. Advances in Methods and Practices in Psychological Science, 2(1), 26–44. https://doi.org/10.1177/2515245918808784

statsmodels. (2026). Release 0.15.0 notes. statsmodels.

statsmodels. (n.d.). statsmodels.stats.multicomp.pairwise_tukeyhsd. statsmodels.

A discussion and evaluation of statistical procedures used by JIMB authors when comparing means. (2024). Journal of Industrial Microbiology and Biotechnology.