Since the variances are not equal, the Kruskal-Wallis test is no good (Part 2)

In the previous and the time before last, I demonstrated through experiments that the Kruskal-Wallis test does not seem to be a suitable alternative to the t-test. KW is neither a test for the difference in means nor a test for the difference in medians.

So, does that mean this test is completely useless? Well, it does have its uses... but they are quite limited. Ultimately, returning to the original point, it is effective when you want to show that "the mean values of the ranks are equal across multiple groups." However, in my research, I haven't encountered such a situation very often.

As usual, let's generate samples that follow this hypothesis in R and test how the distribution of P-values behaves.
We will create two groups with equal rank means in a pseudo-manner. This method is possible by creating a long vector and combining the maximum and minimum values in order (I asked Gemini).
For the two groups created in this way, we use a permutation test. In each simulation iteration, we store the P-value of the KW test and examine its empirical cumulative distribution function (ecdf). If it goes well, it will become a straight line. Also, the Type I error (the error of concluding there is a difference when there is none) should be kept to 5%.

# Kruskal-Wallis test 
# 並べ替えpermutationした場合のP値の分布
# 2026-01-29

# x, y サンプルサイズ
nx <- 100
ny <- 50
N <- nx + ny


# 母集団 x_ranks, y_ranks

all_ranks <- 1:N　# 1からNまでのランクを用意

x_ranks <- c(1:(nx/2), (N - nx/2 + 1):N)  # x: 大きい・小さいほうから(nx/2)個とる
　　　　　　　　　　　　　　　　　　　　　　　# xのランク平均はall_ranksの平均
mean_x <- mean(x_ranks)

y_ranks <- setdiff(all_ranks, x_ranks)　　# y: x以外の要素を抽出
mean_y <- mean(y_ranks)					　# ランク平均はやはりall_ranksの平均

mean_x; mean_y   # 確認: x, y とも (1+N)/2


# Simulation

nrep <- 1000
res <- rep (NA, nrep)		# P.valuesの格納用
res_mean_x <- numeric (nrep) # Xの平均ランク

for (i in 1:nrep){
	
	shuffled <- sample(all_ranks, replace = FALSE)	# ランクをランダムに並べ替え

	x <- shuffled[1:nx]
	y <- shuffled[(nx + 1):N]
	
	res_mean_x[i] <- mean(x)		
	res[i] <- kruskal.test(list(x, y))$p.value
}

mean(res < 0.05)		# Type-I error


# Plots

par(mfrow = c(2,1))

hist(res, freq = FALSE, breaks = 20, col = "orange",
		 main = "P-value Frequency Distribution", xlab = "P-value")
abline (h = 1)

# CDFのプロット

plot(ecdf(res), 
		 main="CDF of P-value", 
		 xlab="P-value", 
		 ylab="Cumulative Probability",
		 col="blue",
		 cex= 0.2
)
abline(0, 1, col="red", lty=2) # 理論上の一様分布（赤線）

par(mfrow = c(1,1))

The cumulative distribution of P-values seems to fit a straight line well.

The histogram of P-values appears to be a uniform distribution, as expected. (Refer to the time before last)

mean(res < 0.05)		# Type-I error

The Type I error was exactly 0.05 (results not shown).
Within the range I investigated, there was no problem even if the sample sizes of the two groups were different. For that reason, the Type I error is well-adjusted, and it can be said to be an excellent method.

Note that this code does not control for the variance of the ranks.
The variances of the ranks of x and y created at the beginning are
　　V(x) = 2735.6
　　V(y) = 212.5
which are completely different values. Nevertheless, the KW test seems to function well. Since this code uses a permutation test to shuffle the elements, that might be expected.

Oh my, it's difficult, isn't it? The KW test has its uses, but it seems one must correctly understand the background of the test. We must refrain from saying, "The variances are different, that's no good, let's use the KW test."