"Most medical information is flawed, but the flaws vary from small, and probably unimportant, to fatal. A major goal of MedicineNet.com is to take medical information, subject it to critical review, and then present only the information that has withstood critical review by expert physicians. We do not just report this information. Our hope is that we also present the information in a manner that is understandable to people who do not have a scientific or medical background." Jay W. Marks, M.D.
I have a neighbor who gives me medical articles from the newspaper if she thinks they might be of interest to me. The other day she gave me an article reporting on a published study that suggested a new cause for irritable bowel syndrome. Because it was published in a newspaper, the article probably had a wide circulation and reached many individuals with irritable bowel syndrome. It is more common now to have patients discuss information they have read in the lay press with their physicians. Therefore, I thought I should be familiar with the article as well as the study itself. I went to the medical library and obtained a copy of the study from the medical journal in which it had been published.
The study was seriously flawed and it made me wonder why a newspaper would report on such a study. Perhaps just as important, the newspaper article did not provide either a critical evaluation of the study or enough information about the study so that the reader could critically evaluate it. Then, I began to wonder if people understood what is involved in a critical evaluation of a medical study. Careful physicians always critically review new information so that they can decide whether or not a study's findings should be used in caring for patients. I decided that it might be useful to describe how I review a medical study.
A published medical study usually has several parts. There is a title, which tells what the study is about and a list of the study's authors, the researchers who did the study. These are followed by a brief summary, or abstract, of the entire published study. The body of the published study includes an introduction, which presents background information to put the study into scientific perspective, and also provides a purpose or aim for the study. What follows next is a description of the methods used in the study, a presentation of the study's results, and finally a discussion of the results.
The first thing I do is look at the title of the article. If the title suggests that the study will be of interest to me, I will read the abstract. If not, I'll move on to the next study. If, after reading the abstract, I feel that the study may be a good one, I then look at the list of authors. If I recognize the authors as experts in the area of the study's research, it suggests to me that the study is likely to be well-done. Then I read the methods. If the methods used in the study are not good, there is no point in reading further. Bad methods always yield bad results. If the methods seem sound, I'll then read the introduction, results, and the discussion. You may notice, I don't get to the "meat" of the study until I have determined that the results are likely to be valid.
How did the study reported by the newspaper stack up? First of all, the study appeared in a second-tier journal, that is, a journal that does not usually publish the best studies. Despite the fact that many research studies are published, only a minority of them are well-done. The first-tier journals are very careful about the studies they publish. In these journals, studies are chosen for publication only after two or more scientists who know the area of research have critically reviewed the study. The reviewers can accept the study, reject it, or ask the authors to clarify or modify it. Only studies that hold up under critical, expert scrutiny and present new information are chosen. Second-tier journals also have expert reviewers for the studies they publish, but the first-tier journals usually receive the best studies first, and the standards of the second-tier journals therefore tend to be lower. Nevertheless, good studies do appear in second-tier journals. For example, they may present studies that hold up under critical review but do not report new information.
The title of the irritable bowel study I was reviewing certainly was of interest to me because of the importance of irritable bowel disease in a gastroenterology practice. The authors were known to me and reputable. However, they were reporting in an area that was not in their area of expertise. The abstract suggested that the study was a good one. I then turned to the section on methods, which is where I found the problem.
For the study, the authors tested patients at the beginning of the study and again at the end. There were 105 patients tested at the beginning, but many of them did not return for the second test. In fact, only 26 returned. It is true that during almost all studies you expect to see losses of patients. Some patients decide they just don't want to continue in the study or they develop side effects. Patient losses don't invalidate a study. For instance, it is well-accepted that losses of up to 20% of the patients do not affect the validity of a study. At a loss of 30%, however, validity becomes shaky. In the irritable bowel study I was reviewing, study losses were 75%. This necessarily invalidated the study's conclusions. Given this fatal flaw, there really was no reason to read further.
I think that the lay press does the public a disservice by reporting studies uncritically or at least not providing the information the reader needs to critically review the study. Sometimes, the press will seek an independent comment from an expert in the area of the study. Often enough, however, the comment is general and does not deal with the quality of the study. This principle of critical evaluation (peer review) is so important that the first-tier journals that have already accepted a study for publication often will ask an expert who has not seen the study before (i.e., a third reviewer, if you will) to comment on the study. The journal will then publish the comment along with the study as an editorial.
