How to Read a Supplement Study

I have learned that one of the easiest ways to misunderstand science is to start with the conclusion.

It is tempting.

A paper appears with an interesting title. The abstract contains a statistically significant result. A journalist reduces it to a headline. A supplement company reduces the headline to a claim.

Within days:

“Studies show…”

And almost nobody has asked the most important question:

What exactly did the study show?

When I read research for MindHeaven, I try to resist the conclusion for as long as possible.

I begin with the architecture.

Who was studied?

This is the first question I ask.

Twenty-five-year-old healthy students?

Older adults with cognitive impairment?

People with a nutritional deficiency?

Sleep-deprived soldiers?

Patients receiving medical treatment?

An effect observed in a deficient population may be completely legitimate while telling us very little about enhancement in healthy people.

The phrase “improved cognition” becomes meaningless if we remove the population from the sentence.

What did they actually receive?

Not simply:

“magnesium.”

Which magnesium?

At what dose?

For how long?

Was it taken alone?

Was it part of a mixture?

Was the formulation identical to the ingredient now appearing in a commercial product?

This matters more than people realise.

Evidence belongs most directly to the intervention that was actually tested.

A study on one extract is not automatically evidence for another extract.

A study on L-tyrosine is not automatically a study on N-acetyl-L-tyrosine.

And a trial of five ingredients together does not tell us which of the five produced the result.

Compared with what?

This question has changed the way I read supplementation research.

Suppose researchers find that:

L-theanine + caffeine performs better than placebo.

That is interesting.

But if the marketing claim is:

“L-theanine makes caffeine work better,”

then placebo is not the comparator I need.

I need:

L-theanine + caffeine versus caffeine alone.

A study can be perfectly well designed and still fail to answer the question someone later uses it to answer.

What was actually measured?

“Brain performance” is not an outcome.

Memory is an outcome.

Reaction time can be an outcome.

Accuracy can be an outcome.

Subjective fatigue can be an outcome.

A validated anxiety score can be an outcome.

A biochemical marker can be an outcome.

And these are not interchangeable.

A compound may change a biomarker without changing how a person thinks or performs.

It may improve one attention task and do nothing for memory.

The headline usually compresses all of this.

I try to expand it again.

What was the primary outcome?

This is one of the most important questions in the entire paper.

Modern trial-reporting standards distinguish the primary outcome - the prespecified outcome of greatest importance - from secondary outcomes. They also stress that reported outcomes should remain consistent with the protocol and trial registry, because switching or selectively reporting outcomes can favour statistically significant findings.

If the primary outcome fails, but one favourable secondary measure among fifteen succeeds, I do not describe the trial as straightforwardly positive.

The successful secondary result may be interesting.

But it is a different level of evidence.

How many people were involved?

Sample size does not tell me whether a study is good or bad.

But it tells me something about uncertainty.

A trial involving eighteen people can reveal a useful signal.

It can also produce an unstable estimate that changes dramatically when another eighteen people are studied.

Small experiments are often excellent places to begin.

They are much less comfortable places to stop.

Was randomisation actually protected?

“Randomised” on the first page does not end the methodological investigation.

How was the sequence generated?

Was allocation concealed?

Could participants infer what they received?

Could investigators?

Was blinding convincing?

Were the products different in taste, colour or sensation?

This is especially relevant for supplements.

Caffeine feels like caffeine.

Some botanical extracts taste unmistakable.

Some compounds produce immediate sensations.

A placebo is useful only when it behaves convincingly as a placebo.

I look beyond the p-value

A result can be statistically significant and practically irrelevant.

I want to know:

How large was the difference?

How uncertain is the estimate?

What does the confidence interval look like?

Was the outcome prespecified?

How many comparisons were performed?

Would I care about the size of this difference if I experienced it myself?

Science cannot be reduced to:

p < 0.05.

Then I look for what did not work

This is perhaps the most unusual part of my reading process.

Once I understand why the study appears interesting, I deliberately look for reasons to become less impressed.

Did another outcome fail?

Did another dose fail?

Did the effect disappear after adjustment?

Was there a dropout imbalance?

Did the authors acknowledge a limitation that the abstract barely mentions?

Did another research group fail to reproduce the effect?

This is not cynicism.

It is protection against falling in love with the result.

Finally, I ask the commercial question

Not:

Can we use this study?

But:

If MindHeaven did not sell anything related to this ingredient, would I describe the study in exactly the same way?

That question has become one of my most useful internal controls.

Because reading research is not primarily about finding supporting sentences.

It is about understanding boundaries.

A good study rarely tells us:

“This ingredient works.”

It tells us something much more precise:

In this population, at this dose, under these conditions, using this comparator, researchers observed this particular outcome with this degree of uncertainty.

That sentence is less exciting.

It is also much closer to science.