Evaluating Medical Studies (cont.)
A published medical study usually has several parts. There is a title, which tells what the study is about and a list of the study's authors, the researchers who did the study. These are followed by a brief summary, or abstract, of the entire published study. The body of the published study includes an introduction, which presents background information to put the study into scientific perspective, and also provides a purpose or aim for the study. What follows next is a description of the methods used in the study, a presentation of the study's results, and finally a discussion of the results.
The first thing I do is look at the title of the article. If the title suggests that the study will be of interest to me, I will read the abstract. If not, I'll move on to the next study. If, after reading the abstract, I feel that the study may be a good one, I then look at the list of authors. If I recognize the authors as experts in the area of the study's research, it suggests to me that the study is likely to be well-done. Then I read the methods. If the methods used in the study are not good, there is no point in reading further. Bad methods always yield bad results. If the methods seem sound, I'll then read the introduction, results, and the discussion. You may notice, I don't get to the "meat" of the study until I have determined that the results are likely to be valid.
How did the study reported by the newspaper stack up? First of all, the study appeared in a second-tier journal, that is, a journal that does not usually publish the best studies. Despite the fact that many research studies are published, only a minority of them are well-done. The first-tier journals are very careful about the studies they publish. In these journals, studies are chosen for publication only after two or more scientists who know the area of research have critically reviewed the study. The reviewers can accept the study, reject it, or ask the authors to clarify or modify it. Only studies that hold up under critical, expert scrutiny and present new information are chosen. Second-tier journals also have expert reviewers for the studies they publish, but the first-tier journals usually receive the best studies first, and the standards of the second-tier journals therefore tend to be lower. Nevertheless, good studies do appear in second-tier journals. For example, they may present studies that hold up under critical review but do not report new information.
The title of the irritable bowel study I was reviewing certainly was of interest to me because of the importance of irritable bowel disease in a gastroenterology practice. The authors were known to me and reputable. However, they were reporting in an area that was not in their area of expertise. The abstract suggested that the study was a good one. I then turned to the section on methods, which is where I found the problem.
For the study, the authors tested patients at the beginning of the study and again at the end. There were 105 patients tested at the beginning, but many of them did not return for the second test. In fact, only 26 returned. It is true that during almost all studies you expect to see losses of patients. Some patients decide they just don't want to continue in the study or they develop side effects. Patient losses don't invalidate a study. For instance, it is well-accepted that losses of up to 20% of the patients do not affect the validity of a study. At a loss of 30%, however, validity becomes shaky. In the irritable bowel study I was reviewing, study losses were 75%. This necessarily invalidated the study's conclusions. Given this fatal flaw, there really was no reason to read further.