- Split View
-
Views
-
Cite
Cite
Dennis Drotar, Editorial: Thoughts on Establishing Research Significance and Preserving Scientific Integrity, Journal of Pediatric Psychology, Volume 33, Issue 1, January/February 2008, Pages 1–5, https://doi.org/10.1093/jpepsy/jsm092
- Share Icon Share
In this opening editorial, I would like to welcome our readers authors, and reviewers, to the first issue of the Journal Pediatric Psychology (JPP) that has been edited by myself and our associate editors. As the new editor of JPP, I am mindful of the extraordinary tradition of excellence in editorship that I am following. Together with our very talented associate editor team and editorial board, we will work very hard to continue this tradition and develop JPP as the premier outlet for the best and the brightest of scientific scholarship in our field. In the spirit of continuous quality improvement, we have implemented a number of changes in order to enhance the scientific contribution of JPP and utility for our readers (see JPP website http://jpepsy.oxfordjournals.org/): (a) new requirements for manuscripts with respect to effect sizes and confidence intervals; (b) a new program that provides a mechanism for junior reviewers to obtain mentoring concerning their reviews; (c) opportunities for our authors to provide feedback about their experience in the review process; (d) communication of JPP updates via the listserve and Division 54 newsletter; (e) promotion of commentaries on published work and new ideas; (f) special sections that emphasize the development of science in high need areas. These priority areas involve two general categories: (a) research methods and models [e.g., methodological issues, single subject designs, case studies and series; and review articles, including meta-analyses, and randomized controlled trials (RCTs) (note: the section on RCTs has already been in place)]; and (b) scientific content areas (e.g., diversity and health care disparities, application of technological innovations, prevention science, and family influences and adaptation). These sections are edited by associate editors who are distinguished scholars. In this issue, our associate editor team presents an outstanding set of editorial statements that describe the scientific need for and focus of the sections. Authors have a standing invitation to contribute to scientific knowledge in each of these priority areas. Please feel free to contact our associate editors prior to your submission have a dialogue with them about the suitability of the topic and method of your research.
It will be important that authors understand that the development of these sections in no way changes JPP's commitment to publishing the highest quality science in the diverse portfolio of research that is reflected in the mission statement of the Society of Pediatric Psychology (see website). Content areas included in the vision statement include the following: psychosocial, developmental and contextual factors contributing to the etiology, course and outcome of pediatric medical conditions, assessment and treatment of behavioral and emotional concomitants of illness, injury, and developmental disorders, promotion of health and health-related behaviors, prevention, education, training and mentoring of psychologists and providers of medical care, improvement of health care delivery systems and advocacy for public policy that serves the needs of children, adolescents, and their families.
Thoughts on Establishing Research Significance and Preserving Scientific Integrity
In my opening editorial, I wanted to share with you some thoughts on the tension between establishing scientific significance and preserving scientific integrity in authorship and peer review. The current relevance of this editorial stems from the extraordinary growth and development of the science of pediatric psychology and the need to sustain standards of scientific excellence and ethical principles in the peer review process (American Psychological Association, 1992).
As an author for 40 years, I am still learning through trial and error, revision, acceptance, and rejection, and am ever mindful of the need to convince reviewers and editors of the scientific significance of my scholarly work (Drotar, 2000). As all authors can attest, convincing reviewers and editors that one's research should be published is a tall order. As the editor of JPP, I am now entrusted with the difficult responsibility of being a gatekeeper of science in our field and deciding whether submitted research is significant. My new role has heightened my sensitivity to the potential conflict between editors’ and reviewers’ responsibilities as gatekeepers of science, on the one hand and authors’ strong interest in having their work published, on the other. In this sense, authors and editors may seem to be working at cross purposes. However, in the broadest, most ideal sense, we are not: We (authors and editors) are all part of a broader community of scholars who are the consumers of scientific knowledge in our field (Ziman, 1966). We also share ethical principles of concerning the conduct of research and reporting of results (American Psychological Association, 1992). Hence, we are all on the same team. But how can we best realize the potential of our team work?
Establishing Significance: Authors’ Responsibilities
Authors should be prepared for the fact that reviewers and editors may not necessarily find or appreciate the nugget of true scientific value (e.g., new knowledge, method, and data) that is contained within the amalgam of material presented in their manuscripts. Consequently, authors have both the burden and responsibility to convince the reviewers and editors that their work is significant. It is important to note that scientific significance is not the same as novelty (Sternberg, 1991). Research can be novel but not important. In order to be significant, research needs to exceed the threshold of current scientific work in a specific area of science. For example, does the research advance theory, research methods/data, and/or practice over and beyond what is already known? In order to establish the significance of their work, authors need to clearly describe the current benchmarks of research in a specific topic area (e.g., what is known and not known in a specific area of science). Then, they need to describe how their work transcends the threshold of current methods and available data. The impact or influence of an article involves the following areas of significance (Sternberg & Gordeeva, 1996): (a) empirical (e.g., do the data contribute to new scientific knowledge in the field of pediatric psychology?); (b) methodological (e.g., is the study question addressed by a more valid methodology that clarifies findings?); (c) theoretical (e.g., does the research develop and test an important theoretical question?); and (d) clinical or practical (e.g., does the research contribute to the development of assessment or intervention methods that enhance health and well-being of children and their families?). The impact of an article is also enhanced by the quality of its presentation, substantive and methodological interest, and value for future research (Sternberg & Gordeeva, 1996).
Because authors bear the burden of convincing reviewers and editors that their work is significant, they need to promote if not champion their work in no uncertain terms. However, the ethics of scholarly writing also require authors to temper promotion of their research by clear recognition of the scientific limitations of their work. That said, the formidable pressures on authors to publish their work and potential influences on their conduct as scientists also need to be appreciated. For example, academic success as defined by promotion and tenure depends in no small measure on the number of publications, especially first-authored publications, garnered by authors. Publication of work also facilitates authors’ abilities to compete for federal grants to fund their research, which is also very important in the development of scientific careers.
It should be recognized that the pressure on authors to establish the significance of their research, in order to enhance its eventual publication has the potential to disrupt the sound scientific presentation and interpretation of research findings. For example, many authors experience a potential conflict of interest between their roles as promoters of the significance of their work and their roles as scientists. In what ways can this happen? Apart from human nature (e.g., no one likes to have their cherished hypotheses disconfirmed), the mandate to publish or perish can tempt researchers to: (a) refrain from posing specific hypotheses, (b) conduct a large number of analyses, including post hoc analyses (data dredging) (Marshall, 1990; Vandenbroucke, 1990), and (c) selectively interpret their data to focus on positive (e.g., statistically significant) findings and neglect negative (e.g., nonsignificant) findings (DeMets, 1999).
Evaluating Significance: Reviewers’ and Editors’ Responsibilities
Reviewers and editors have an important ethical responsibility in rendering an unbiased judgment about the significance of research that they review (American Psychological Association, 1992). Making an unbiased editorial judgment of scientific and/or clinical significance is difficult and calls for experience, knowledge, and appraisal of the state of the art of an area of science. In order to make the best possible judgment about the significance of work, reviewers and editors face the daunting task of keeping up to date with current research and state of the art methods in our ever-expanding field. They are also charged with the important responsibility of putting their strong personal preferences (e.g., for their pet methods or content areas) aside, in order to provide a fair and balanced review of the work that is submitted.
In the course of their work, reviewers and editors also can fall prey to the bias of equating statistical significance with scientific significance, which persists despite its limitations (Bakan, 1966; Easterbrook, Berlin, Gopalan, & Matthews, 1991; Rosnow & Rosenthal, 1987).The new JPP policy of requiring authors to include effect sizes and confidence intervals was developed to address the above issue. Authors and reviewers should also appreciate that statistically nonsignificant findings can advance science if the study is well designed and executed, has sufficient power, and addresses an important question.
Finally, I believe it is important for reviewers and editors to appreciate the limitations imposed by absence of formal training in editorial responsibilities. (This was one impetus in developing the new mentoring program for reviewers for JPP.) Put bluntly, success and eminence as a scientist does not necessarily translate into expertise as a reviewer and editor. For this reason, we as reviewers and editors need to appreciate that we have a considerable “on the job” learning curve. Based on my experience, one important aspect of this learning curve involves the experience of dialogue with authors about their work and the opportunity to learn from other reviewers and editors. Reviewers and editors may approach manuscripts from very different angles and may appraise methodological issues differently (Fiske & Fogg, 1990). For this reason, it is very important that we learn from one another.
Some Safeguards to Establish Scientific Significance with Integrity
How can authors, reviewers, and editors best collaborate to publish scientifically significant work that meets the highest standards of research integrity? Here are some ideas about how to respond to this very difficult, yet important question. First and foremost, we need to recognize and appreciate our biases. For example, as authors we operate under significant pressures to have our work published that can impose potential conflicts of interest concerning the way in which we report our science. The primary responsibility to ensure that such potential conflicts of interest do not result in problematic science is ours as authors. In addition, reviewers and editors need to recognize that they are not immune to bias with respect to their own experiences, ideas, and preferences about ideal science and methods.
Here is a short “how to” list for authors to consider in reducing their conflict of interest in their manuscripts and for reviewers to consider in their evaluations of manuscripts: (a) Develop the theory and/or causal model that frames the research and include a priori hypotheses (Peyrot, 1996); Unfortunately, published science in the field of pediatric psychology has tended to minimize theory (Wallander, 1992); (b) Post hoc analyses should be clearly differentiated from a priori hypotheses, ideally based on theories that are sufficiently explicit to allow clear disconfirmation (Platt, 1964). If post hoc analyses are conducted, a clear rationale should be described. There are circumstances in which exploratory research is both needed and warranted. However, the need for and value-added significance of such exploration should be clearly articulated; (c) A third safeguard involves the presentation of data in common metrics that are independent of statistical significance, (i.e., effect sizes and confidence intervals) so that reviewers and readers can judge the power of and stability of data apart from sole reliance on p-values (Rosenthal, 1991; Schmidt, 1996); (d) A final safeguard involves the interpretation of one's data in a balanced, circumspect way that considers statistically significant and hypothesis-friendly findings along with those that negate one's hypotheses. Moreover, all of us may learn as much or more from negative findings as we do from those that affirm our expectations. In fact, the history of science is replete with examples of well-respected theories that did not stand the scrutiny of data and were eventually discarded. For this reason, the disconfirmation of theory, including the pitting of alternative theories in a strong test is a cornerstone of science (Platt, 1964; Popper, 1959). Although such strong tests of theory are all too rare in our field, they are very much needed.
Dialogue Among Authors, Reviewers, and Editors as a Safeguard
As an author and now as editor, I have a clear bias in favor of the peer review process as a critical safeguard in reconciling tensions between pressures on individual authors to promote their science versus our collective responsibilities to publish significant science. Peer review provides a system of checks, balances, and feedback that can help to identify and correct potentially misleading research methods and writing. Within this system, reviewers and editors are charged with the critical responsibility of rendering an evaluation of whether authors’ claims concerning the significance of their research are valid. At its heart, science is a social process that involves presentation and promotion of research, feedback, negotiation concerning the feedback, revision, more negotiation and final disposition (Ziman, 1966). At its best, peer review is an opportunity for a dialogue among a community of scholars.
No one's work is without potential methodological problems or threats to validity. Consequently, publication reflects the outcome of the collective dialogue with one's peers (the reviewers and editors) concerning the scientific merits of the research. For all these reasons, authors who submit their work to JPP should ideally regard their editorial critique an opportunity for dialogue with reviewers rather than as an attack on their work. As an editor in a position of power to render decisions concerning your work, I realize that this is easy for me to say. However, as a first author and as coauthor, I’ve received peer review for nearly 300 initial submissions to peer-reviewed journals and have lived (thus far) not only to tell about the process but affirm its benefits. Mercifully, I’ve lost count of all my resubmissions and repressed my frustrations with the process.
I admit I do not necessarily enjoy the prospect of editorial critique. However, I fully respect and understand the need for the process. For one thing, my work and that of my colleagues have benefited significantly from the critique of a large number of very dedicated reviewers and editors. This process has resulted in better science than we could have ever accomplished apart from peer review. It is no accident that some scientists and writers have had productive, though at times ambivalent relationships with their editors and reviewers. Such ambivalence is understandable, given our natural human tendency to think we are mostly, if not always “right.” However, in science the terms “right” and “wrong” are more properly understood as probabilities (e.g., more or less right, more or less better). The essence of peer review is a collaboration among reviewers, editors, and authors to help authors develop their work so that it becomes more “right” and valid. In this way, authors, reviewers, and editors all have critical roles to play in the consummate quality improvement initiative known as science.
The Role of Dialogue Among Authors, Reviewers, and Editors
I look forward to our collective dialogue and collaboration on behalf of the science of pediatric psychology and ultimately on behalf of the generations of children, adolescents, and families who stand to benefit from the research conducted by investigators in our field. There is no doubt that publishing in highly respected peer reviewed journals such as JPP, will benefit your (our) scientific careers. But together I would hope that we will keep our eyes on our opportunity and privilege to contribute to science and the public good that will hopefully result from our science (i.e., enhancing the quality of the lives of future children, adolescents, and families who may benefit from new scientific knowledge that is generated). As scientists, we serve our research participants and future generations of children and families who may benefit from our research.
As the editor of JPP and speaking on behalf of our associate editors, please understand that we look forward to our dialogues with you about your research and your experience with the editorial process. I can tell you that we may not always agree about the significance of your research. But that is to be expected in the process of peer review. On behalf of myself, our associate editors and reviewers, I will also promise you a timely, respectful, and fair editorial process and dialogue about your work. If we fail to achieve these key benchmarks of scientific review, please let us know. Your feedback to us concerning the review process is very important to the continued success of JPP as a premier outlet for science in our field.
Conflict of interest: None declared.