One point in a research paper to check before believing that 'supplements halve dementia risk'

Is it true that this supplement halves the risk of dementia?

Medical news seen on TV and social media is filled with claims that we want to believe. When you see headlines like 'Effective for 80% of women in their 50s' or 'Death risk halved in specific group,' it is a very natural reaction to feel that it might apply to you as well.

However, while working in the medical field, I have encountered many situations where I wanted to ask, 'Is that information really reliable?'

This is not about telling you to keep doubting information. It is about how knowing a little bit about how information works can help you make smarter choices.

Today, I will explain the concepts of 'primary analysis' (the main event) and 'subgroup analysis' (the bonus) in medical research. Just knowing this will completely change how you view health information.

Can you really trust that 'dramatic effect'?

Medical news often features 'dramatic effects.'

'New supplement halves dementia risk,' 'Symptoms improved in 80% of women in their 50s who followed this diet'—just reading these headlines can be exciting. You might even feel a sense of vague anxiety being eased.

But every time I encounter such information, I have started to check one thing first.

Is this result from the primary analysis or from a subgroup analysis?

Just knowing this difference will completely change how you read medical information.

Primary analysis vs. subgroup analysis—the decisive difference between the main event and the bonus

When researchers plan a clinical trial, they decide in advance what they most want to know from the study. This is the primary analysis.

For example, the entire study design is optimized for a single question, such as 'How much does blood pressure change after 6 months between the group that took this drug and the group that did not?' The required sample size, observation period, and measurement methods are all designed to answer that specific question.

The results derived from this are scientifically highly reliable. That is the strength of primary analysis.

On the other hand, subgroup analysis is, so to speak, a 'bonus' analysis. It is conducted based on the idea of 'what happens if we take out a specific group after getting the overall results?' For example, this applies to cases where 'there was no difference overall, but it seemed effective when narrowed down to women in their 50s.'

Subgroup analysis is not bad. It often provides important hints for forming hypotheses for the next study. However, the results of a subgroup analysis should not be read as 'certain facts,' but strictly as clues for considering the next move.

If you play rock-paper-scissors 20 times, a 'miracle' might happen once

Let me talk a little about statistics here.

In medical research, the figure 'p-value less than 0.05' is often used as a criterion to say that 'this difference is not a coincidence.' This means that the probability of this result occurring by chance is less than 5%.

However, there is a major pitfall here.

Let's say you perform 20 sub-analyses. Statistically, there is a possibility that one result will show 'effective (p<0.05)' purely by chance.This is mathematically unavoidable.

If you play rock-paper-scissors 20 times, it's bound to happen that you'll 'luck out and win' at least once. The exact same thing happens in data analysis.

And what the media reports is often 'that one' result. Even if 19 out of 20 analyses showed 'no difference,' they will prominently report only the one result that looked 'effective'—this is what is known ascherry-picking.

Even if the researchers are not acting dishonestly, this kind of thing happens structurally. That is precisely why the recipient also needs literacy.

Let's put it into practice—look at the 'Methods' of a paper for just 5 seconds

You might feel intimidated when told to 'read a research paper.' But there is only one term you need to check.

Open the 'Methods' section of the paper and look for the term 'Primary outcome'.

If the 'dramatic effect' reported in the news is something confirmed by that Primary outcome, the reliability is quite high. I think you can take it as such.

However, ifthe term 'Subgroup analysis' appears, I want you to pause for a moment. That is a 'candidate for a hypothesis,' not a 'proven fact'.

Of course, it is difficult to open a research paper every time. So, at the very least, try to get into the habit of asking yourself for just one second, 'Is this a primary analysis or a sub-analysis?' Just doing that will change how you view information significantly.

Why I want to convey this—what I see from the medical front lines

While working in the medical field, I often receive consultations from patients and their families asking, 'I heard that such-and-such is effective; what do you think?'

Much of that information 'looks like evidence' but is actually the result of a 'sub-analysis'.

It is not malicious information. But if you accept it as is, you might misjudge the options that are truly necessary. That has always concerned me.

From my experience as a certified public psychologist learning about human decision-making, I also feel that we are influenced by 'numbers' and 'authoritative headlines' more than we think. The phrase 'effective for 80% of people' is strongly etched into the brain, even if it is the result of a sub-analysis.

That is precisely why I believe it is important for the recipient to have a little bit of the 'power to question.' This is not about continuing to doubt information, but a power to 'protect your own judgment yourself'.

Summary

I will summarize what I wanted to convey in this article at the end.

There are two types of medical research: 'primary analysis (the main event)' and 'sub-analysis (the bonus),' and their reliability is completely different. What the media finds easy to pick up is the 'impressive result' of a sub-analysis, and it is somewhat dangerous to accept that as 'solid evidence.' If you perform 20 analyses, one will statistically appear 'effective' by chance—this is the problem of multiple testing. The habit of checking the term 'Primary outcome' in the 'Methods' section of a paper is the first step toward information literacy. And above all, the habit of asking yourself for just one second, 'Is this the main event or a bonus?' will be the power that protects your judgment.

Instead of being swayed by health information, become someone who can make good use of it. If I could provide that small spark, then writing this article was worth it.

Thank you for staying with me until the end, despite the length of this article. I hope to continue delivering information that is, in a good way, 'a little bit geeky'.

#HealthLiteracy #HealthInformation #FactCheck #Evidence #Statistics