What Is Publication Bias?
A personal essay, not an evidence assessment. Our graded assessments of individual ingredients — written to a published standard, from full texts — are in the evidence section. Nothing here is a statement about what any product does.
Imagine performing ten studies.
Eight find nothing convincing.
Two produce exciting positive results.
Now imagine that only those two appear in journals.
If I later search the published literature, what do I see?
Two positive studies.
My conclusion might be:
The evidence is remarkably consistent.
Reality would be the opposite.
This is the basic intuition behind publication bias.
Science can be distorted by what we do not see
The scientific literature is not a perfect archive of every experiment ever performed.
Studies may remain unpublished.
Results may appear only selectively.
Certain outcomes may be omitted.
Statistically significant results may be easier to publish, promote or even write up.
That is a profound problem.
Because a review can analyse every published study perfectly and still reach the wrong conclusion if the published set is systematically incomplete.
Supplements are particularly vulnerable to this problem
Small studies are common.
Commercial interest exists.
Many endpoints may be measured.
There are numerous formulations.
Some trials may never progress beyond conference abstracts or internal reports.
This creates fertile conditions for a literature that looks more optimistic than the full research history.
Trial registries help
If a trial was registered before it began, we can sometimes see that it existed even if no conventional publication later appears.
We can also compare prespecified outcomes with published outcomes.
That does not solve everything.
But it creates a footprint.
This is one reason I consider prospective registration a positive transparency signal.
Funnel plots are useful, not magical
In meta-analysis, researchers sometimes examine whether smaller studies appear asymmetrically distributed around the pooled effect.
This can suggest small-study effects or missing evidence.
But interpretation is not simple.
Asymmetry can arise for reasons other than publication bias.
And with very few studies, the plot may tell us very little.
I dislike tools becoming rituals.
The point is not to produce a funnel plot.
The point is to ask:
What evidence might be missing?
Negative studies deserve infrastructure
This is why I want MindHeaven to actively search for them.
Not merely include them if we accidentally encounter them.
Search for:
null results, failed replications, registered trials without obvious publications, primary endpoints that failed,
corrections, retractions.
If we search only:
“Ingredient X benefits cognition”
we are programming our conclusion before opening the first paper.
Publication bias changes the meaning of “studies show”
Whenever I hear that phrase now, another question follows automatically:
Which studies do we not see?
We may never know the full answer.
But acknowledging the possibility is already better than pretending the visible literature is perfectly representative of reality.
Science is not only about analysing information.
Sometimes it is about recognising the shape of missing information.
Methodological sources
Reporting and appraisal standards referred to in this essay. They are not the evidence behind any product claim.