Kruskal-Wallis test is not a panacea for unequal variances
Is the Kruskal-Wallis test a panacea?
What I often see in thesis drafts is the use of the Kruskal-Wallis test when an ANOVA or similar test should have been used, because
the two distributions did not have equal variances, so the Kruskal-Wallis test (hereinafter referred to as KW) was used.
The authors likely think they are performing a test similar to one for means. I have been seeing this for years and I am tired of it, so I will state it clearly here: KW is not a panacea for cases where variances are unequal. There are several references regarding this point, so please refer to them (e.g., Kasuya 2001). Note that the same essentially applies to the Mann-Whitney U test.
First, KW is a non-parametric test, and its null hypothesis is not about the "mean," nor is it about the "median" as is often mistakenly believed (Note 1). It is about the equivalence of means when data is replaced by ranks (in other words, it is like an ANOVA on ranks).
The null hypothesis of the Kruskal-Wallis test is that the mean ranks of the groups are the same.
https://www.sciencedirect.com/topics/medicine-and-dentistry/kruskal-wallis-test
Do you really want to test that?
Since it is difficult for me to verify the validity of this hypothesis, I will examine the problems that arise when applying KW to the means of two samples obtained from normal populations, using the distribution of P-values. Note that the mean and median of a normal population are equal.
First, let's establish the following statistical fact. This is a method often used to evaluate test procedures.
"When the null hypothesis is true (there is no difference), P-values are uniformly distributed from 0 to 1."*
In other words, if you repeat the test many times, the histogram of the obtained P-values will approach a flat shape with equal height (Note 1).
Also, the empirical cumulative distribution function (ECDF, Note 2) will be a 45-degree line extending from the origin to the upper right. If the actual data deviates significantly from this line, it is possible that the assumptions of the test or the nature of the data do not match the method.
R code: ECDF for t-test and KW test
Remove the "#" at the beginning of the line you want to execute among the three "pt[i] <-" formulas around line 17 (if you remove more than one, the later formula will take precedence).
The first formula is the t-test (not assuming equal variances; Welch's test), the second is the t-test (assuming equal variances), and the third is the KW test.
The sample size for each group is {100, 50}. Both follow a normal distribution, and the population mean is 0. I investigated the distribution of P-values under extreme conditions where the standard deviation (SD) is 50 times larger (1,000 iterations).
# P値の累積分布関数のシミュレーション
# 2026-01-25
rep <- 1000 # シミュレーションの反復数
n1 <- 100 # グループ1のデータ数
n2 <- 50 # グループ2のデータ数
pt <- numeric(rep) # 反復ごとの P-value を格納する変数
x <- factor(rep(c("A","B"), c(n1, n2))) # カテゴリー変数（2グループ x それぞれのデータ数）
for (i in 1:rep){
y1 <- rnorm(n1, sd = 1) # 正規分布（平均0, SD = 1）
y2 <- rnorm(n2, sd = 50) # 正規分布（平均0, SD = 50）
pt[i] <- t.test(c(y1, y2) ~ x, var.equal = FALSE)$p.value # t検定
# pt[i] <- t.test(c(y1, y2) ~ x, var.equal = TRUE)$p.value # t検定（等分散）
# pt[i] <- kruskal.test(c(y1, y2) ~ x)$p.value # KW検定
}
mean (pt < 0.05) # 危険率 alpha
# CDFのプロット
plot(ecdf(pt),
main="CDF of P-value",
xlab="P-value",
ylab="Cumulative Probability",
col="blue",
cex= 0.2
)
abline(0, 1, col="red", lty=2) # 理論上の一一様分布（赤線）The results are as follows.
The t-test not assuming equal variances (R's default) lies on the straight line, suggesting it is well-adjusted.
When equal variances are assumed, it deviates significantly from the straight line (the curve above the red line means that P-values tend to be smaller). Since the SDs differ by a factor of 50, it is natural that the formulation assuming equal variances is inappropriate.
Now, how about the KW test?
This is also not good at all (at least under these preconditions). When turned into a histogram,
The $${P}$$ values are clearly not uniformly distributed, with a spike below 0.1. In other words, there is a tendency for the values to be unfairly small.
Just to be sure, when I checked the number of iterations where P < 0.05 (i.e., the significance level) via simulation
As shown here, it is clear that it deviates significantly from the desired 5%. When the population follows a normal distribution, it is prone to becoming significant even if the population means (or medians) are equal.
Summary
The KW test reacts not only to differences in the location of distributions but also to the shape of the distributions. This time, it was an extreme simulation where the SD was 50 times larger. While I still need to study the strict details, at least when investigating differences between groups, it is clear thata naive attitude of 'The variances are different. No problem. Let's use the Kruskal-Wallis test!' is dangerous.
In most cases, what we are interested in is the mean of the groups or treatments. In many cases, if you want to know the 'difference in means', I believe thatWelch's test or ANOVA is sufficient.
Notes
However, if the shapes of the distributions are the same, it can be interpreted as a comparison of medians.
The proof that $${P}$$ values follow a uniform distribution involves using the distribution function to show that 'the probability of $${P < x}$$ is $${x}$$' (for example, the case where $${P < 0.5}$$ occurs in 50% of all iterations). I would like to experiment with this in R someday.
ECDF: A function representing the probability that a random variable $${X}$$ is less than or equal to a value $${x}$$, based on observed samples. As the number of samples increases, it approaches the cumulative distribution function (CDF) of the true distribution. Writing the CDF as a formula gives $${F(X) = P(X \leq x)}$$
Reference list
Kasuya E (2001) ANIMAL BEHAVIOUR, 61, 1247–1249
Update history
2026-03-18 Corrected several errors.