It sounds like what’s happening is an inevitable part of treating new, exciting findings that confound current thinking as a “new breakthrough”
The reality of the situation is that anything that’s interesting enough to be hailed as a brand new therapy must have a few things in common:
- It must be a novel result- therefore it must be newly studied, and have very little existing data about it.
- If it gets hailed as a brand new therapy, then it must be surprising- It must confound our current ideas and be sufficiently different to our current thinking so as not to be considered as a mere "improvement " to our current methods
- It must be communicated to the medical community and the general publis, and that communication gives rise to an understandable bias towards making your research stand out and be seen as important. This can lead to journalists and researchers over-emphasising the “breakthrough” element of the research (see below for a related article on why science communication is done so, so badly.)
With all these three elements in place, it’s easy to see why some of these exciting outliers fail to hold up when given more intense study. Under the rigorous gaze of empirical testing, many hopeful new breakthroughs turn out to be dead ends or just sink back into statistical noise, especially when dealing with a system as complex as the human brain. This is not a case of new treatments failing, it’s just the inevitable messiness of actual empirical study, which does not generally proceed by eye-catching headlines and heroic new ideas, but instead by gradual accumulation of evidence and the slow emergence of new and improved treatments and methodologies. Empiricism and the Scientific Method will not win many prizes for drama, but they work.
