Accidental Hits and Truths in the Drawer: The Scientific Illusion Created by p-values

Intro

It was October 8, 2025.
While I was studying a book on linear regression with ChatGPT...
"I want to write a blog post, ChatGPT said, and it presented this manuscript.
Please take a look at ChatGPT's hard work.

Topic

1. Introduction: A hit! But is it real?

"Oh, it became significant!"
When doing research or data analysis, there are moments when you feel like pumping your fist in the air.

The p-value is less than 0.05, meaning "there is a difference that cannot be attributed to chance"—.
But can we really say it is "not by chance"?

In the world of statistics, there is a hidden structure where "accidental hits can look like the truth."
This time, let's take a relaxed look at two representative phenomena: the trap of multiple testing and the file drawer problem.

🤖🤖🤖

2. Multiple Testing: If you try until you hit, you'll hit

A "significance level of 5%" in a single test means that
5 out of 100 times, it will be significant even by chance.

But what if you test 20 hypotheses with the same data?
One of them will happen to be "p<0.05" by chance.
Even though it's actually random, you end up convinced that "there is a difference!"

🧮 Looking at it in a small table...

👉 If you try 20 times, you'll hit at least once.

Yet, if you only pick out that last one and report "It was significant!",
that is like bragging about a winning lottery ticket found by chance.

🎲 An analogy

Multiple testing is exactly that kind of phenomenon.

🤖🤖🤖

3. The File Drawer Problem: Where do the studies that didn't hit go?

Another major issue is the File Drawer Problem.

Imagine this:
100 researchers conduct experiments to test the effectiveness of the same drug.
If the drug has no real effect, at a 5% significance level, 5 people will accidentally get a 'significant' result.
The remaining 95 will find 'no effect'.

However—

• The 5 who got significant results submit them, thinking, 'This will be a paper!'

• The 95 who found no effect put them in a drawer, thinking, 'These are boring results...'

As a result, only the '5 significant cases' are released to the world.
Readers think:

But in reality, the remaining 95 quiet failures are hidden in drawers without ever being published.

🤖🤖🤖

4. 'Multiple Testing' and the 'File Drawer Problem' are related

In other words—
both are phenomena where if you try enough times, you'll hit the jackpot somewhere.
Multiple testing is a hit within one person, while the file drawer problem is a hit across the whole world.
The structure is identical.

🤖🤖🤖

5. When science becomes nothing but 'winning lottery tickets'...

This structure, where 'only significant results emerge,' became the root cause of the
replication crisis in the 2010s.

In fields like psychology, medicine, and economics,
there were successive reports that studies once considered 'established theories' could not be replicated.

The cause is this👇

• Changing conditions while looking at data to find 'significance' (multiple testing)

• Not publishing non-significant results (file drawer problem)

In short, science became filled with 'reports of accidental hits'.

🤖🤖🤖

6. What should we do? — Modern countermeasures

🤖🤖🤖

7. Summary: The p-value is a 'clue,' not the 'truth'

A small p-value is a sign that 'there is likely a difference that cannot be explained by chance.'
However, the p-value itself does not guarantee the truth.
This is because,

Because,

• if you try enough times, you will get an accidental hit, and

• most failed results disappear into the drawer.

🤖🤖🤖

8. Conclusion: The courage to look at quiet failures

What moves science forward is the courage to record 'misses' rather than 'hits.'
Results showing 'no effect' are the starting point for the next study.

Let's conclude with the motto of relaxed statistics.

🤖🤖🤖

📕 Influential literature

'Applied Linear Regression, 4th Edition,' Section 6.6 'Interpretation of Tests'
First edition, first printing, Kyoritsu Shuppan
By Sanford Weisberg, supervised by Etsuo Miyaoka, translated by Tomari Shimokawa and Takamasa Kurosawa

The End

Series Articles

Next Article

Table of Contents

Blog Introduction

I am writing an 8-part series of articles on note.
Please feel free to take a look!

1. Relaxed Statistics

This is a blog that roughly explores probability and statistics using the Statistical Test Grade 2 problem collection as a guide.
Feel free to read it as if it were casual conversation. Please take a look.
It corresponds to the official Statistical Test Grade 2 CBT-compatible problem collection.
Sample code for Python and EXCEL is also available.

2. Connecting with Statistics and Data Analysis

The series "Connecting with Statistics and Data Analysis" is a blog series that uncovers the "connections" to statistics and data analysis and presents them in a column format.
With the help of generative AI, I wrote these freely, imagining the entry points to statistics and data analysis.
For example...
・Connections between daily life and statistics
・Connections from Statistical Test Grade 2 to what lies beyond
I hope you enjoy reading them at your leisure 🍀

3. Experiment! Fun Bayesian Modeling 1 & 2 with PyMC Ver. 5

I will draw and analyze the Bayesian models used in psychological research from the books "Fun Bayesian Modeling" and "Fun Bayesian Modeling 2" using PyMC Ver. 5.
Like this book, many Bayesian models are written in R and Stan.
I will strive to explore the possibilities of PyMC and make Bayesian modeling easy to practice.
Since these are familiar and easy-to-visualize themes, please try running them with PyMC and let's enjoy it together!

4. Experiment! Bayesian Modeling from Iwanami Data Science Vol. 1 with PyMC Ver. 5

I will draw and analyze the Bayesian models by four Bayesians from the book "Experiment! Iwanami Data Science Vol. 1" using PyMC Ver. 5.
This book is a great resource for learning the basics of Bayesian programming.
I feel like I've become friends with Bayesian methods by happily running PyMC models.
Everyone, please try running them with PyMC and let's play and learn together!

5. Fun Copying: Bayesian, Python, etc.

I will blog about the results of my "book copying activities" for Bayesian, Python, and others.
I am mainly working on translations into Python.
I hope this serves as sample code for fellow learners who are also copying code 🍀

6. Introduction to Time Series Analysis for Psychology with R and Stan, using Python and PyMC Ver. 5

I will practice the time series analysis from the book "Introduction to Time Series Analysis for Psychology with R and Stan" using Python and PyMC Ver. 5.
This book is packed with themes on time series analysis!
I realized the depth of time series analysis.
I will enjoy learning time series analysis with my favorite language, Python.

7. Writing about Data Science-like Things

I write columns on statistics, data analysis, AI, machine learning, and Python on an irregular basis.
There are many articles related to statistics and data science books.
Series on "Statistics," "Python," "Mathematics and Python," and "R" have been born.

8. Python Machine Learning Programming Practice Journal

I wrote articles about my various thoughts while studying the book "Python Machine Learning Programming: PyTorch & scikit-learn Edition."
This book is a textbook for scikit-learn and PyTorch.
Please feel free to try it out if you like.

Thank you very much for reading until the end.