G Lyon & Elayne Esterline. Handbook on Communicating and Disseminating Behavioral Science. Editor: Melissa K Welch-Ross & Lauren G Fasig. Sage Publications, 2007.
The history of the profession has never been a particularly attractive subject in professional education, and one reason for this is that it is so unrelievedly deplorable a story. For century after century all the way into the remote millennia of its origins, the profession got along by sheer guesswork and the crudest sort empiricism. It is hard to conceive of a less scientific enterprise among human endeavors. Virtually anything that could be thought up for treatment was tried out at one time or another, and once tried, lasted decades or even centuries before giving it up. It was, in retrospect, the most frivolous and irresponsible kind of human experimentation, based on nothing but trial and error, and usually resulting in precisely that sequence.
While this quotation could pertain to the education profession, it actually refers to the field of medicine as it sought to gain its scientific footing over the past century, and the person who said it was the highly respected former president of the Memorial Sloan Kettering Cancer Center, Dr. Lewis Thomas (Thomas, 1983). Though Dr. Thomas’s observations on early medicine reflect some of the trial-and-error practices seen in education today, it is encouraging that medicine now is highly regarded in the scientific community for its allegiance to evidence-based practices. The journey from superstition, anecdote, and snake oil to science was a difficult one in medicine, and it was not until the turn of the 20th century that a series of scientific breakthroughs altered the value that both physicians and their patients placed in scientific research. Laboratory and clinical research had exposed ineffective and often harmful outcomes of common medical treatments such as blistering and bleeding and brought evidence-based practices, including antiseptic surgery, vaccination, and public sanitation to the forefront (Beck, 2004).
Despite the advances made in evidence-based medicine, no field can ensure that only the most valid scientific information is disseminated to the public or guarantee that it will be implemented (Cabana et al., 1999; Waddell, 2002). What is clear is that educational decisions must be guided by facts rather than by superstition, anecdote, intuition, and tradition. To achieve this goal, we must develop a clear understanding of the conditions that must be in place so that the field of education is defined by its reliance on evidence-based practices. The need to spread the most trustworthy scientific research findings into schools and classrooms to improve instructional practices is as relevant to education today as it was to medicine in the early 20th century (see Flexner, 1910).
This chapter begins by asking, Why is disseminating research findings from relevant sciences to the field of education essential for improving student learning and achievement? The answer has not always been apparent to educators or to scientists in the education field. Then, we will summarize what has been done both historically and more recently to translate and disseminate research findings in education, discuss the challenges that have been encountered and that remain before us, and suggest what more can be done to increase the spread of proven innovations, programs, and practices into schools and classrooms.
Specifically, we explain that, as educational historians and commentators have noted (e.g., Vinovskis, 2001), the quality of some education research is regarded by academics in other behavioral and social science disciplines as second rate, both conceptually and methodologically, a perception shared by policy makers. Though this conclusion is debated (see National Research Council, 2002), we see ample evidence to support it and so do not find it surprising that the dissemination of research findings has not had a discernable or productive effect on educational practices and policies. We go on to explain some of the philosophical underpinnings that appear to have contributed to the misuse of science in education. Because changing policy can be an effective way to encourage the use of science in practice, and because federal legislation has become a greater force in education in recent years, we discuss how developers of recent federal legislation attempted to infuse science into the education process to help improve educational outcomes for students.
Throughout the chapter, we refer to education as a field rather than a discipline because many disciplines contribute their particular traditions, methods, and values to the study of education. These include, but are not limited to, psychology, philosophy, history, anthropology, sociology, linguistics, and, more recently, cognitive and developmental neuroscience. As we will discuss, the contributions from each of these disciplines can be advantageous when knowledge is derived from systematic and well-coordinated multidisciplinary studies but can be less helpful when there is little planned interaction and collaboration among the scientists involved, as has been the case in education research. Likewise, scientists have been isolated from the culture and other realities of the school and classroom environment. This disconnection between scientists and educators has stymied the dissemination of research findings.
In the final section of the chapter, we suggest steps that will be critical for disseminating trustworthy and useful science that has greater potential to help students learn and achieve. Given the state of the science in education, most of the chapter focuses on the need to produce more sound and useful science in the first place. Before one can effectively disseminate research to guide educational practices, one must have something worthy of disseminating. However, in concluding, we also suggest how individuals and organizations in the education field might work to strengthen dissemination strategies and dissemination science.
Why Science Should Guide Education Practices
Keith Stanovich recently articulated a useful analogy that underscores the importance of scientific evidence in guiding educational practice:
When you go to your family physician with a medical complaint, you expect that the recommended treatment has proven to be effective with many other patients who have had the same symptoms. You may even ask why a particular medication is being recommended for you. The doctor may summarize the background knowledge that led to that recommendation and very likely will cite summary evidence from the drug’s many clinical trials and perhaps even give you an overview of the theory behind the drug’s success in treating symptoms like yours. (Stanovich & Stanovich, 2003, p. 3)
While Stanovich acknowledges the oversimplification of the analogy, he argues correctly that the doctor has provided you with data to support a theory about your complaint and its treatment. This is the essence of applying the scientific method to a clinical practice including education.
In this chapter, we take the position that something not done well is not worthy of dissemination. And, we argue that doing something well in education—improving teaching and student learning and achievement, for example—requires that curricular and instructional decisions are based on the most compelling and replicated scientific evidence available. We fully recognize that this position tends not to be accepted by many in the educational community for reasons we will explain. Unlike health care, welfare reform, defense, agriculture, engineering, or industry, education has undervalued and underused scientific research in developing, evaluating, and improving programs and policies (National Research Council, 2002). Educational programs and practices have historically been driven by values, traditions, philosophies, superstitions, and anecdotes rather than scientific evidence of effectiveness (Reyna, 2004). Until recently, the major policy input to both state and federal education departments has been provided by politicians and special interest groups rather than educational scientists (Lyon, 2005).
The limited utilization of scientific evidence in education is harmful to the field and to the public that it serves. Education, like other sectors that serve the public, must have reliable information about what works and why and how it works (see Reyna, 2005; Stanovich, 2000) for excellent discussions of this position). One could argue that scientific research and dissemination of reliable information to the educational community is nonnegotiable given that all sectors of a productive society depend on an educated workforce. Moreover, because of the rapidly changing demographics in schools and the heterogeneity of students in classrooms today, it is even more imperative to design, evaluate, disseminate, and implement programs that have a proven track record with a wide range of students from different cultural and socioeconomic backgrounds and academic preparation. In short, our view is that we have a responsibility to provide a free and effective education to all students in public schools. As Valerie Reyna (2004) so eloquently put it,
Research evidence is essential for identifying effective educational practice. Research— when it is based on sound scientific observations and analyses—provides reliable information about what works and how it works. This information is essential to designing effective instruction and to demonstrating that it is, in fact, effective. Responsible decisions about what is good for students, therefore, require scientific evidence. (p. 47)
What Do We Mean by Scientific Research?
Within the field of education, the conventional wisdom holds that applying the scientific method to questions of teaching and student learning is not productive. The argument frequently offered to support this perspective is that schools and classrooms are too complex to be examined empirically—the contextual variables and extraneous factors that influence student learning and achievement are simply too numerous and confounded to account for (see Lagemann & Shulman, 1999). In our view, this “conventional wisdom” is neither conventional nor wise. Indeed, the National Research Council (NRC) within the National Academy of Sciences recently concluded that the core principles of scientific inquiry are as relevant for education as they are for the physical sciences, the life sciences, the social sciences, and the behavioral sciences (NRC, 2002). But what qualities must research have to be sound and therefore potentially useful to education?
In its 2002 report titled Scientific Research in Education, the NRC stated that all scientific endeavors must (a) pose significant questions that can be investigated empirically, (b) link research to theory, (c) use methods that permit directly investigating the question, (d) provide a coherent and explicit chain of reasoning, (e) replicate and generalize across studies, and (f) disclose research data and methods to encourage professional scrutiny and critique. In essence, as Fletcher and Francis (2004) point out, scientific research is a process of reasoning based on the interaction between theories, methods, and results. These principles are not new and have been guiding scientific inquiry for centuries (Kuhn, 1962). However, they typically have not been applied in education, despite their successful application to the study of human learning (Reyna, 2005; Schwartz & Reisberg, 1991) and despite repeated appeals to educational researchers to enhance the credibility and impact of their work (Levin & O’Donnell, 1999; Reyna, 2005). If research is so critical to continually improving practice in other fields relevant to the health and welfare of children, then why do some of those on the front lines of education, such as teachers and administrators, frequently view research in education as trivial or irrelevant? And even when a scientific breakthrough has occurred (e.g., the discovery that phonemic awareness is essential to reading development), what is it about the field of education that makes translating, disseminating, and implementing the findings so difficult?
Challenges to Disseminating Education Research
Here we focus on what we see as three of the most significant impediments to disseminating education research: the antiscientific sentiment that has characterized education for well over a century, limitations in the quality of educational research, and the combined effect of an antiscientific culture and limited research quality on the demand in the educational marketplace for trustworthy research findings. (Space limitations preclude analyzing the challenges to disseminating education research in exhaustive detail. For additional discussion, see Berends, Kirby, Naftel, & McKelvey, 2001; Constas & Sternberg, 2006; Elmore, 1996; Fullen, 2000; Ramey & Ramey, 1998; Slavin & Madden, 1996.)
The Antiscientific Culture in Education
A decidedly antiscientific spirit has had a pervasive influence on the perceived value of research throughout the history of education (see Adams & Bruck, 1995; Lagemann, 2000; Lagemann & Shulman, 1999) and increasingly in the past two decades (Adams & Bruck, 1995; Lyon, Shaywitz, Shaywitz, & Chhabra, 2005). Academic resistance to even the thought that education could be a research-driven field was present in universities as early as the mid-1800s (Lagemann, 1999; National Research Council, 2002). Lagemann (1999) explains that a prevailing early view of teaching was that it was “women’s work”—implying a low-status vocation—requiring the ability to nurture rather than impart instruction via sophisticated and technical teaching skills deemed essential by researchers. Teaching was viewed as more akin to a genetically endowed affective ability than a learned intellectual skill to be developed through scientific methods. Thus, few were interested in applying scientific principles in education to develop and enhance teachers’ competencies.
Today, although few would argue that particular temperaments and dispositions make some teachers particularly effective, it is encouraging that irrelevant and invalid gender issues no longer enter the argument and that many scientists and educators understand that these qualities can be developed along with other essential knowledge and skills. Still, the role of scientific research in education is debated ad nauseam. Why might this be the case?
The most current cycle of education’s reluctance to use scientific evidence stemmed from the philosophy of postmodernism, which in its most dramatic form states that truth is in the eye of the beholder. It is relative and framed only by one’s own experience and culture (Ellis & Fouts, 1997; Gross, Levitt, & Lewis, 1997; Lyon et al., 2005; McKenna, Stahl, & Reinking, 1994). Research methods, such as ethnography, that are appropriate for collecting in-depth information about an individual teaching or learning experience became favored over those that would allow for establishing cause-effect principles about teaching and learning that could be generalized. The most hard-line version of postmodernism claimed that because such general cause-effect principles do not exist in the world, scientific methods that seek to identify them are not useful for education decision making and practice. Thus, over the past two decades, data were often not accumulated or disseminated that could dispel assertions about student learning and achievement that arose from philosophical, ideological, and political beliefs. As a result, untested claims have often trumped facts from science (Lyon et al., 2005; Stanovich, 2000). Moreover, many professional development programs that prepare teachers and education administrators view research through this postmodern lens; consequently, many prospective and veteran administrators and teachers are often unfamiliar with the value that scientific research can bring to their professional life or, worse, have been taught to discount scientific research in guiding practices and school policies.
It does not help when, as in the application of any behavioral science, many citizens, no matter their profession or background, believe they are already experts on human behavior and, in this case, on education. Most Americans have attended school and have widely different views on that experience. Whatever its nature and quality, many think the experience provided sufficient background for speaking authoritatively on how to improve education (Kaestle, 1993). Who needs scientific research to guide policies and practices when you have the necessary expertise in your own house? Indeed, it is fascinating that a substantial number of educators and noneducators who see the value of scientific research in fields such as health care and industry fail to see the contributions it can make to education. Some of this distrust for research and what it can contribute to the educational enterprise is related to the quality, or lack thereof, of education research.
Historically, the dissemination of research has not been successful or even a high priority in education partly because of the limited quality of the science. What do we mean by quality? Carnine (1997) proposes that research quality be judged on the basis of its trustworthiness, usability, and accessibility. Trustworthiness, defined by technical considerations of methodology and analysis, refers to the confidence practitioners can have in research findings. Usability addresses the likelihood that the research will be used by those responsible for making decisions that affect students, teachers, and other relevant educators. Accessibility reflects the ease and rapidity with which practitioners can obtain research findings, extract them, and understand them. To be sure, outstanding scientists from numerous disciplines have contributed and continue to contribute trustworthy, usable, and accessible findings that have led to effective educational innovations and practices (see, e.g., Constas & Sternberg, 2006; Fletcher, Foorman, Denton, & Vaughn, 2006; Lyon, Fletcher, Fuchs, & Chhabra, 2006; Marzano, Pickering, & Pollock, 2001; National Reading Panel, 2000; National Research Council, 2002; Schoenfeld, 2006). However, much of the research published in education journals, books, and other outlets is viewed by both scientists and educators as superficial, lacking in rigor, emphasizing breadth over depth, or as being abstruse (Levine, 2005, 2006). In the past 20 years in particular, the field has largely, for the philosophical reasons explained earlier, overrelied on qualitative methodologies, such as ethnographic data with doubtful reliability and validity, survey research, and case studies (Levine, 2005; Lyon, 2001; Lyon & Chhabra, 2004; Stanovich, 2000).
Indeed, as long as educational research has been published, it has been the target of criticism (Kaestle, 1993; Lagemann, 2000; Lagemann & Shulman, 1999; Levine, 2005, 2006; National Research Council, 2002; Vinovskis, 2001). And the criticism has not been confined to commentators in the United States (see Australian Research Council, 1992; Beveridge, 1998; Grace, 1990/1991). In 1974, for example, J. Nesbitt, the first president of the British Educational Research Association, complained that much educational research was “badly done, amateurish, gimmicky, … [and] often simply wrong” (Nesbitt, 1974, p. 4). Similarly, more recently, Labaree (1998) noted that “the broadly confirmed general perception of both the research and instructional programs of education schools in Britain is that they remain weak” (p. 8).
The reasons for the uneven quality of education research are too numerous to review comprehensively here. What we can conclude from reviews of the relevant literature (e.g., see Fletcher & Francis, 2004; Levin & O’Donnell, 1999; Levine, 2005; Mosteller & Boruch, 2002; National Research Council, 2002) is that educational research has been poorly funded (National Research Council, 2002; Vinovskis, 2001), frequently politicized (National Research Council, 2002; Vinovskis, 2001), and guided more by philosophy and ideology than by scientific principles (Lyon, 2005; McDaniel, Sims, & Miskel, 2001; Stanovich, 2000). It is important to note here that the scientific method also stems from philosophy. Indeed, few would refute that the methods that have materialized from philosophical perspectives have spawned trustworthy knowledge that has enhanced human health and well-being and otherwise made human progress possible (for historical and current discussions of this issue, see Ginev, 2005; Kuhn, 1962; Magnusson & Palincsar, 2006; Morrison, 2006; Popper, 1959). Nonetheless, what complicates this situation even further and makes change toward a more scientifically based culture even more difficult is that many colleges of education do not offer their undergraduate and graduate students, and especially doctorial candidates, sufficient training in the philosophical underpinnings, principles, and methods of scientific inquiry. In 2004, David Cohen, a highly respected educational scientist and professor at the University of Michigan, lamented that schools of education “chronically” allow graduate students to matriculate with very weak research skills—“It seems that any student who persists long enough can get a degree as [an educational] researcher” (Viadero, 2004). But, as yet, the academy has been refractory in its willingness to change research training priorities and practices. Cohen (2004) continued to lament at a meeting organized by the National research Council, “I haven’t seen any movement in our field to deal with it…. It troubles me deeply that so few of my colleagues seem to be taking it seriously” (p. 1).
Largely missing from today’s research studies are two major elements of science that contribute to trustworthiness: (a) the suitability of the proposed research design and methodology to address the specific research question(s) posed by the investigator(s) and (b) the scientific rigor of the methodology itself (Fletcher & Francis, 2004; Lyon, 2005; National Research Council, 2002). The frequent mismatch between the research question and the appropriate design and methodology in education research is exemplified in the persistent debates on the relative value of quantitative or qualitative research methods in identifying whether an instructional program is effective.
Not only are these debates time-consuming, but they are irrelevant and thus detract from the conversations essential to moving the field forward. That is, obviously we must use both types of designs and methods if we are to develop the most comprehensive understanding of whether educational programs and policies have any genuine impact on critical educational outcomes. The type of research design and method to be used depends on the question under study. Qualitative research simply cannot test and confirm that an instructional approach, for example, causes improvements in student learning and academic achievement—only well-designed randomized controlled experiments (or, to a lesser extent, quasi-experimental studies) can determine what works and what does not. Likewise, the types of designs and methods used to establish cause-effect principles do not allow in-depth understanding of individual experiences of the teacher or student or reveal characteristics of the context that might help or hinder the intervention’s effectiveness. But integrating methods in a thoughtful and competent manner demands that researchers, and especially doctorial education students, are sufficiently trained to use both quantitative and qualitative methods, including understanding the assumptions underlying them and the purposes for which they are best suited.
As we have argued before (Lyon, 1999b), we believe educational research is at a crossroads. If the field neglects modern scientific methods, it risks isolating itself from the mainstream of scientific thought and progress. To develop the most effective instructional approaches and interventions for dissemination, we must support the education field in developing the knowledge and skills needed to clearly define which interventions work and the conditions under which they are effective. Educating the field about recent advances in statistical methods—for example, hierarchical linear modeling—can help educational leaders and teachers have confidence that science can objectively identify effective approaches to teaching despite the complexity of the educational environment. These tools can help sort out the range of complex school, teacher, student, and family factors, such as a student’s socioeconomic status, a student’s previous teachers, and the demographics of schools and families, that also can influence student achievement and the success of program implementation and sustainability (Raudenbush & Bryk, 2002).
The Effects of Limited Quality and Relevance on the Marketplace for Educational Research
As we’ve explained, many schools of education do not typically prepare future teachers to value the use of research to inform their instructional practices or provide them with the knowledge and skills to keep abreast of the scientific literature in their field. In agreement, Debra Viadero, reporting for Education Week in 2003, wrote, “Even if teachers have the fortitude to plow through academic journals, chances are their professional training did not include coursework in how to distinguish good research from bad” (p. 1).
Less than one third of the instructional practices used by teachers are based on scientific research findings (Kennedy, 1997). Even when evidence-based practices are used, teachers cannot clearly explain why they have used them (Kennedy, 1997). When asked, many teachers report that educational research is trivial and irrelevant to their teaching, is too theoretical and complex, and communicated in an incomprehensible manner (Kennedy, 1997; Lyon et al., 2005). School principals and school administrators trained at the doctoral level are equally reluctant to use research to guide their leadership efforts and infuse research-based practices in their districts and schools. Levine (2005) reported that only 56% of principals, for example, found that the research courses they took in their degree programs were valuable to their jobs. A major reason cited was the courses were too abstract, emphasizing mechanistic aspects of research designs and methods without clear examples of how research can be applied to inform practice in schools and classrooms. In short, the courses emphasized form over function without integrating the two.
Moreover, teachers and educational administrators are frequently buffeted by unpredictable pendulum swings, such as fads in instructional practices. This situation is highly demoralizing to teachers and schools. Many teachers find themselves attempting to implement the latest highly touted instructional “magic bullet” only to learn, after it fails, that the research on which it was based was seriously flawed. Even more frustrating for educational consumers is that the majority of such programs, methods, and educational products today claim to be “research based” despite having little or no scientifically sound evidence of effectiveness. Until educational practitioners are properly trained to ensure that they can discriminate between rigorous and valid research and research that was poorly designed, frustration will increase, and the market for even the best research will continue to decrease (Blaunstein & Lyon, 2006).
Given the historical limitations in the quality and relevance of educational research and the constant sale and labor-intensive implementation of ineffective “research-based” programs, it is not surprising that school leaders and teachers have little respect for or confidence in research. However, we are optimistic that this situation can change and that education can be transformed into an evidence-based field that values reliable research, encourages the use of scientifically based methods, and appreciates the dissemination of trustworthy research. No doubt, it is unrealistic in the near future to guarantee that all educators can or want to develop a common knowledge base to become more effective consumers of educational research. In the meantime, it is our view that the results of science will not translate effectively into practice unless at least three conditions are met.
First, as we discussed previously, scientists must give the education community valid and useful information, and the education community must be provided the necessary background to evaluate and use that information. Although teachers and school administrators need not be researchers themselves, they must know how to access findings from science and be familiar with how research is done. They must possess a basic understanding of the differences among research designs, why a researcher chooses one design over the other, and why a study does or does not yield valid information (McCardle & Chhabra, 2004). In essence, teachers and administrators must become educated consumers of research. Without this basic knowledge, those expected to implement new programs will not be able to evaluate the veracity of the claims of effectiveness made by publishers and vendors of various educational curricula.
Second, in addition to producing scientifically sound research, those who design dissemination strategies must appreciate the conditions essential to implementing findings from research with fidelity in the complex world of schools and classrooms (Sternberg et al., 2006). These conditions include (a) ensuring that teachers and administrators implementing the program have realistic expectations of the time involved in successful upscaling, (b) ensuring that the implementation strategy provides sufficient scaffolding so that teachers can clearly understand the steps in the implementation process, (c) providing teachers with sufficient professional development to be comfortable with the content and the implementation strategy, and, very important, (d) ensuring that those implementing the program are committed to both the form and function of the innovation. This is not a trivial matter. Elmore (1996) and others (Sternberg et al., 2006) have reported that even during highly visible reform efforts, as little as 25% of teachers are interested in implementing the program or innovation.
Third, teachers must be explicitly supported by the school system and its culture, as well as by a capable technical assistance system to sustain the effective use of scientific innovations. For example, Cooper, Madden, and Slavin (1998) found in their examination of factors related to the successful implementation of the Success for All Program (Slavin & Madden, 1996) that the quality of instructional leadership within the school was critical, as was teacher acceptance and commitment. Success for All provided professional development to school building-level administrators, such as principals, to assist them in providing a supportive culture for change and strategies to overcome resistance to change. Moreover, building-level leaders demonstrated a strong commitment to the program and fostered implementation and sustainability by hiring facilitators to support teachers over the change process.
False Starts and Broken Promises: The Dissemination of Unproven Practices
There has been no shortage of well-meaning efforts, particularly at the federal level, to provide educators with information from research. The growth of knowledge dissemination and utilization in education began with large-scale efforts to improve the nation’s schools in the 1950s and 1960s. By the 1970s, the U.S. Department of Education expanded its dissemination efforts with two major programs: the Educational Resources Information Center (ERIC) and the National Diffusion Network (NDN) (U.S. Department of Education). The two programs shared a number of goals, one of which was the dissemination of “promising practices”2 and innovations to the educational community.
The NDN was chartered to create a national mechanism for disseminating curricula and other instructional products that would assist public and private educational institutions in sharing exemplary programs, products, and processes. Before a program could be disseminated through the NDN program, it had to be approved by a Program Effectiveness Panel, which examined data provided by the program developers.
The ERIC initiative was designed to give educators and the public access to a wide-ranging body of literature related to education. It was developed in response to the widespread belief that the inaccessibility of research was partly to blame for the infrequent use of research in education. Unfortunately, the quality of the research was not emphasized nearly as much as its availability. As noted, ERIC’s strategy for identifying material for its clearinghouses was and continues to be to identify “promising practices.” For example, the ERIC Clearing House for Science, Mathematics, and Environmental Education contacts state, county, and local coordinators and curriculum specialists for nominations of programs and materials they consider promising and exemplary.
Given the previous discussion about the limited quality and relevance of educational research, it is not surprising that much of what has been historically disseminated through both NDN and ERIC may not be the result of sound science. Indeed, the process of identifying “promising practices” on the basis of nominations from researchers and practitioners has not required an external evaluation and validation of the practice. Moreover, neither NDN nor ERIC adopted a set of common criteria for evaluating the quality of evidence submitted as proof that a practice was effective. Information on programs and curricula that are accessed through both NDN and ERIC is frequently contained in non-peer-reviewed journals and other information outlets that are not scrutinized for scientific quality. In short, both NDN and ERIC, despite the best of intentions, have disseminated a significant amount of information on “promising practices” that may not have been worth disseminating.
In addition to receiving information about many untested curricula and programs, potential NDN and ERIC users could be overwhelmed easily by the sheer volume of available information. Consider that between 1991 and 2001, there were 334,647 abstracts entered into the ERIC database. Though only some of the abstracts summarized research findings relevant to promising practices, the consumer would have had to read them all to distinguish research from nonresearch information. As we previously pointed out, the typical NDN and ERIC consumer would most likely not have the formal training and background to determine whether a study is valid. Thus, the utility of these dissemination engines was compromised not only by sheer volume but by the lack of a systematic process for ensuring quality control. As to evidence, the bar was set so low that the word itself almost did not apply. In short, this was a case of “let the buyer beware.”
Why Policy Had to Drive Research in Education Rather than the other Way around
As the field of education continued to debate the value and utility of scientific research in guiding practice and policy through the 1990s, a substantial amount of converging scientific evidence from basic and applied multidisciplinary research had emerged that helped to answer practical questions about teaching and learning in the nation’s schools and classrooms, not just in the laboratory. Much of the work was being led and funded by the National Institute of Child Health and Human Development (NICHD) within the National Institutes of Health (NIH) through its research program in child development and behavior. Within this program, a major focus was on the study of reading development, reading difficulties, and reading instruction and, more recently, the study of early childhood education, mathematics, and science.
The NICHD had supported scientists studying reading development and reading difficulties since 1964 and had made substantial progress in identifying the environmental, instructional, neurobiological, and genetic correlates of reading difficulties. During the 1990s, NICHD scientists had identified, through several experimental studies, the essential elements of reading instruction and several effective instructional strategies that prevent and remediate reading disabilities. The NICHD research was multidisciplinary and longitudinal in nature with teams of scientists from education, psychology, linguistics, neurology, radiology, genetics, anthropology, and sociology studying more than 44,000 children and adults, some for as many as 25 years. The importance of reading proficiency to the education and health of the nation’s children, combined with the agency’s experience in studying a complex academic skill from a rigorous and practically useful scientific perspective, led Congress to identify the NICHD as the lead federal research arm on many educational topics.
As data from the 44 NICHD reading research sites began to converge in the mid-1990s on how best to prevent and remediate reading difficulties, more than one third of the nation’s fourth-grade students were unable to read and understand simple children’s books, and an increasing number of adults were unable to read a bus schedule (Sweet, 2004). The NICHD research was not typically viewed as important by the reading community, however, because the quantitative and experimental research methods (though frequently combined with descriptive and qualitative methods) used were at odds with the postmodern view of evidence prevalent at that time. Yet, the national mood was beginning to reflect a significant concern about the prevalence and consequences of reading failure.
By 1996, the extent of reading failure across the country began to get the attention of many congressional members and policy makers as well as President Clinton. In his 1996 State of the Union Address, the president announced that almost 40% of the nation’s fourth graders could not read at grade level and subsequently proposed the America Reads Initiative. Simultaneously, the House Education and Work Force Committee, under the leadership of Chairman William Goodling, and the Senate Health, Education, Labor, and Pensions Committee, chaired by Senator James Jeffords, were holding hearings on the national reading deficit. It was within these House and Senate hearings that the first author gave what was to be the first of eight testimonies over the next 8 years on how scientific research could dramatically reduce reading failure in the United States. In essence, the NICHD data, derived from translational prevention and intervention studies carried out in many inner-city public schools, clearly indicated to the congressional members that there were effective approaches and programs that could prevent and remediate reading failure while many others could not. Unfortunately, far too many children in our nation’s classrooms were being taught with the ineffective ones.
During the late 1990s, both House and Senate education committees were given data on the gap between scientific evidence and how children were being taught (McDaniel et al., 2001; Song, Coggshall, & Miskell, 2004; Sweet, 2004). Their question was, Why did the gap exist? The basic answer included the complexities discussed in this chapter. Namely, the field of education did not typically base curricular and instructional practices on scientific research. If research information was used, it was communicated in a manner that frequently confused the educational consumer and did not take into account the complexity of the school culture and environment. Neither school administrators nor teachers had been prepared in their training to be knowledgeable consumers of research and to distinguish between the bad and the good. Because of this, schools were inundated with educational fads that invariably failed.
A second question to the first author from congressional members and staff was whether the educational community could begin to integrate the scientific research into their instructional practices. Our conclusion at that time was that forging an evidence-based culture in education would be extremely difficult, particularly if one expected the field to take the initiative on its own (see Lyon, 1997, 1998, 1999a, 2000, 2001, 2002, 2005; Walsh, 2006). Despite the overwhelming scientific evidence on how best to teach reading, the majority of colleges of education institutions were reluctant to adequately prepare prospective teachers for the classroom with information about valid, evidence-based instructional programs (see Levine, 2006, for a review of current teacher preparation limitations). There was no perception of urgency about the need to reduce reading failure, and the antiscientific barriers within the academy and in the professional development sector were simply too formidable to expect meaningful progress. Clearly, these same barriers existed in other education domains beyond reading (see Levine, 2006).
Given this cultural landscape, a number of activities undertaken by several legislators, legislative staff, and scientists, including the first author, were conducted to develop legislation toward establishing scientific standards for the federal funding of educational products and programs available to the schools (McDaniel et al., 2001; Song et al., 2004; Sweet, 2004). In essence, the purpose of these efforts was to make federal funding for educational programs contingent on documenting that the products, as well as professional development associated with the products, were based on scientifically based reading research (SBRR). This strategy is consistent with the notion that a highly effective and desirable way of disseminating findings from science is to attempt to incorporate them directly into the policies that govern the operation of practices and programs.
To ensure that a valid scientific foundation existed on which to base educational policy, Congress charged the director of NICHD (Dr. Duane Alexander), in consultation with the U.S. Secretary of Education (Richard W. Riley), with convening a National Reading Panel (NRP) that was composed of researchers, members of higher education institutions, teachers, and parents. The goal of the NRP was to assess the state of reading research and to identify scientifically based interventions found to be effective with well-defined subgroups of students (NRP, 2000; Song et al., 2004). The NRP was the first federally convened panel to examine the effectiveness of different instructional approaches on specific skills required for proficient reading using established objective review criteria and statistical meta-analyses. The findings of the NRP, combined with the results documented in the National Research Council consensus report, Preventing Reading Difficulties in Young Children (PRD; NRC, 1998), and frequent presentations of scientific data on reading to House and Senate congressional committees, subsequently informed the development of the Reading Excellence Act (PL 105-277) and the Reading First and Early Reading First programs within the No Child Left Behind Act (NCLB) of 2001. The Reading First Legislation significantly increased the federal investment in scientifically based reading instruction in the early grades and stressed accounting for results and systematic monitoring and evaluation to ensure that approved programs were implemented with fidelity.
The major contributions of these pieces of legislation were to infuse legislative language into NCLB and Reading First that explicitly called for evidentiary standards to guide the use of federal education funds. However, making this change in how federal funds should be used to support effective instruction was not without difficulty, and the resulting legislation did not fully achieve the initial intent of those developing Reading First. Specifically, the first author was convinced that the legislation should only provide funding for those programs that had been found to be effective through the most rigorous research possible. If this strict criterion was not in place, it was anticipated that vendors and publishers of programs would claim that their products were research based without having to provide the published data demonstrating the genuine benefits of their specific program. However, there were very few programs that met the more stringent effectiveness criteria at that time. As such, Congress made the decision to soften the criteria and allow funding for those programs that were based on principles of scientific evidence rather than based on evidence of program-specific effectiveness. This decision had both political and practical foundations.
Politically, vendors without the highest level of scientific evidence would have placed (and did place) significant pressure on Congress if their programs could not be considered. Practically, with so few programs eligible for funding, implementation of the Reading First program would have been difficult. As was expected, given the softening of the eligibility criteria, several programs without sufficient evidence of effectiveness in their own right were purchased and implemented using federal funds. In the first author’s opinion, this reduced the capability of the Reading First legislation to reach its potential in establishing policies that held publishers and vendors accountable for the effectiveness of their programs, but this example illustrates the importance of compromise and the need to be practical (and often incremental) when attempting to move complex systems toward evidence-based approaches.
Nevertheless, the Reading First legislation, though not as stringent as it could have been, did require states to review and identify educational programs that were based on principles of instruction that had been demonstrated to be effective through rigorous scientific research. This level of accountability was clearly a first in the history of federal educational funding. To ensure that the educational community developed a clear understanding of what constitutes SBRR and how to implement it in schools and classrooms, two major dissemination programs were launched as a second step of the strategy. These programs were developed and implemented for two major reasons. First, the bulk of peer-reviewed scientific research relevant to classroom instruction and student learning and achievement was in the area of reading. Far less scientific evidence had been obtained in early childhood education, mathematics, and science. Second, as discussed previously, the existing federal programs that disseminated education research—ERIC and NDN—were not designed and implemented to ensure that information accessed by the public was in fact evidence based.
To overcome these hurdles, substantial input from the first author and others on research capacity in education was provided through testimony and one-to-one meetings with congressional members. Robert Sweet, a professional staff member with the House Committee on Education and the Workforce, and Grover Whitehurst, at that time director of the Office of Educational Research and Improvement (OERI), spearheaded these efforts, which ultimately contributed to the passage of the Education Sciences Reform Act of 2002 (ESRA). The specific purpose and rationale for the passage of the ESRA was articulated by the act’s sponsor, Michael Castle (2002), chair of the House Education Reform Subcommittee:
This bill transforms the Office of Educational Research and Improvement into a streamlined more independent “Institute of Education Sciences.” Essentially this legislation attempts to address, what I have come to know as, serious shortcomings in the field of education research.
Grover Whitehurst, a developmental and clinical psychologist and a highly respected researcher, became the first director of the Institute of Education Sciences.
The contributions of the ESRA to the slow but steady culture change toward evidence-based practices in education are many, but three stand out for the purposes of this chapter. First, to build scientific capacity across all educational domains, the ESRA required that federally funded research meet high standards of quality based on the definition of “scientifically based research standards” set forth in NCLB. Second, research applications submitted to the Institute for Educational Sciences must, for the first time, be evaluated against the rigorous peer-review standards similar to those in place at the National Institutes of Health. Third, the ESRA, like the NCLB, provided legislative language to develop dissemination programs that, through rigorous scientific review, could evaluate the effectiveness of different programs, products, and strategies intended to enhance academic achievement and other important educational. These dissemination resources are described in the next section.
Light on the Horizon
Because of the recalcitrance of the educational research community to improve the quality of research from within the field, as well as the foreseeable lack of progress predicted for the future, federal legislation was written, in part, to help the educational community progress from making instructional decisions on the basis of untested assumptions and beliefs to decision making based on converging evidence. We also had no reason to believe that consumers of education would increase their demand for research of quality and relevance unless federal funding was made contingent on the use of scientifically based programs and instructional strategies. Why? As the Coalition for Evidence Based Policy noted,
Federal, state, and local governments spend more than 330 billion per year on public elementary and secondary education, and religious and independent schools spend an additional 30 billion. This vast enterprise encompasses thousands of educational interventions (curricula, strategies, practices, etc.), the vast majority of which have been implemented without a basis in rigorous evidence. (Bringing Evidence-Driven Progress, 2002, pp. 1-2)
But creating an interest in using scientifically based research in educational practice and policy is one thing—providing educators with access to the best research and helping them understand it is another. Within this context, several dissemination programs were established through the NCLB and ESRA to give the educational community the most current and accurate scientific information relevant to instruction, student learning, and achievement.
In the NCLB legislation (2001), Congress established an important role for the National Institute of Literacy (NIFL). The law directed NIFL, the U.S. Department of Education, NIH, and NICHD to
- Disseminate information on scientifically based reading research pertaining to children, youth, and adults
- Identify and disseminate information about schools, local educational agencies, and state educational agencies that have effectively developed and implemented classroom reading programs that meet the requirements of the Reading First legislation
- Support the continued identification and dissemination of information on reading programs that contain the essential components of reading instruction supported by scientifically based research that can lead to improved reading outcomes for children, youth, and adults
In addition, within the scope of the NCLB and Reading First legislation, the Department of Education established the National Center for Reading First Technical Assistance in 2004 (Sweet, 2004). The major functions of the center include giving educators and the public at large high-quality resources on SBRR; identifying scientifically based instructional assessments, programs, and materials; helping states and local districts build capacity to establish reading programs and accountability systems based on scientific research, such as teacher education academies and training for administrators in SBRR; providing technical assistance to states on curriculum and program selection; and supporting states in working with institutions of higher education to evaluate and improve preservice teacher education programs.
In 2002, under the aegis of the Education Sciences Reform Act, the Department of Education also established the Institute for Educational Sciences (IES). That same year, the IES began a partnership with the Council for Excellence in Government’s Coalition for Evidence-Based Policy, whose mission is to promote government policy making on the basis of rigorous evidence of program effectiveness. The coalition initiative continues to promote rigorous evaluation of education programs and strategies and plays a major role in ensuring that program funds support educational activities that research shows to be effective. The initiative draws on successful precedents for evidence-based approaches and their dissemination from medicine and welfare policy, where rigorous evaluation has played a major role in policy and funding decisions.
As part of its charter, the IES established the What Works Clearing House (WWC) (http://www.whatworks.ed.gov) to summarize evidence on existing programs across all areas of education and to make the summaries available to educational consumers. Within this context, the WWC reviews the research base underlying educational programs or innovations to evaluate, among many other factors, the methodological quality of the research, the degree to which the research was conducted with representative samples and subgroups for whom the program was intended, the appropriateness of the measures used to assess response to the program, and the fidelity of implementation of the program. Moreover, by virtue of its availability to the public, the WWC serves as a reference to enable researchers and consumers of research to better understand the scientific criteria against which educational programs and interventions are evaluated.
It remains to be seen whether the federal legislation and initiatives we’ve described will make a genuine and long-lasting impact on the learning and achievement of students in the United States. Initial reports show very positive results (Government Accounting Office, 2007; Jennings, 2006). While some imperfections in the legislative initiatives are apparent, they stem from inadequate implementation safeguards and the need for more relevant outcome measures such as authentic performance measures that more accurately represent the actual content that is being learned rather than a narrow set of skills assessed via standardized testing (see Reyna, 2005, for discussion), although others disagree (see Carlson & Levin, 2005, for a range of opinions). Not surprisingly, the federal requirement for evidence-based practices has led some in the educational community to characterize both NCLB and ESRA as examples of government interference in the marketplace of research ideas and practices (Lyon, 2005). These opinions may reflect philosophical, ideological, and turf issues or genuine concern about the role of the federal government in setting any policies that affect localities, even if done to help improve research quality and dissemination. Still, it must be reiterated that the NCLB and the ESRA were not conceptualized in a vacuum. The legislation was driven by a concern that the future of our nation’s children was being compromised by a failure to apply what scientific research had indicated was effective, particularly in the area of reading. Although much work remains to be done, we believe that NCLB and ESRA represent a new day in education—a day in which the scientific method has greater potential to supersede tradition, anecdote, ideology, and political persuasion in guiding educational practices and policies in schools and classroom across the country.
Summary and Recommendations
The consequences of disseminating research information of questionable quality and relevance to schools, teachers, parents, and policy makers are devastating. In this chapter, we have focused on the need to develop information that is worth disseminating. This emphasis is a reflection of where the education field is in thinking about how to advance the dissemination of science. But increasing the quality, relevance, and trustworthiness of research is only part of the challenge. The educational community must now take on the complex task of identifying the conditions under which research-based innovations and programs can be implemented and sustained in districts, schools, and classrooms. But the community does not have to start from scratch. In 1997, Gersten, Vaughn, Deschler, and Schiller reviewed the literature relevant to the implementation and sustainability of interventions and identified six factors essential for success:
- The intervention must be practical, concrete, and specific, accounting for the realities of schools.
- The intervention must have a scope and purpose that is neither vague nor narrow.
- The intervention must be accompanied by collegial support networks within the school and between the school and the implementers of change.
- Teachers must be given many different opportunities for not only implementing new procedures but also for receiving feedback from the implementers.
- The implementers must emphasize the professionalism of teachers by involving them in problem solving and emphasizing the connection between everyday classroom practice and research.
- Teachers must see the results of their efforts in improved student outcomes.
More recently, a number of educational scientists, working collaboratively with practitioners, have begun to use these principles along with implementation factors identified more recently in developing models and strategies for translating research into practice at a statewide level. For example, Fletcher et al. (2006), Schoenfeld (2006), and Calfee, Miller, Norman, Wilson, and Trainin (2006) have argued for programmatic research that begins with rigorous experimental investigations that proceed from small-scale research through large-scale curricular implementation, accompanied by explicit strategies to ensure sustainability. Within this context, Fletcher et al. describe how the Texas Reading Initiative (TRI) exemplifies this process. The TRI was built on more than two decades of basic scientific research designed to answer three focused questions: How do children learn to read? Why do some children have difficulty learning to read? Which instructional approaches, methods, and strategies have the highest probability of success in improving reading skills? This research was instrumental in the development of assessment instruments to identify children at risk for reading failure through ongoing screening of all students in Texas from kindergarten through the second grade. The research also led to the development of preventative and remedial strategies to increase the probability that all children would be reading proficiently by the third grade. Successful small-scale implementation efforts were carried out by the NICHD using this same research in Washington, D.C., Atlanta, Georgia, Tallahassee, Florida, Seattle, Washington, Albany, New York, and Toronto, Canada.
However, the implementation of these research-based practices in the TRI required a number of additional initiatives and collaborations to ensure uptake and sustainability. For example, the business community in Texas played a critical role in reinforcing the need for the state, districts, and schools to base all reading practices on the best science available and to build accountability systems to continuously measure whether programs were having beneficial effects. The business community was also instrumental in articulating the need for increased funding to support the reading initiative. Second, during the development of the TRI, partnerships were forged with universities that included researchers in the area of reading. These partnerships were essential because the research-based reading interventions had to be implemented by teachers who were well prepared to do so. The university partners developed statewide reading academies to ensure that teachers received the most current information and also provided continuous support to teachers to reinforce their practices in the classroom. Third, partnerships were developed with large urban districts throughout the state. By demonstrating that the implementation of research-based reading programs could foster significant improvements in student learning and achievement in challenging urban environments, a clear message was sent to all districts and schools that prevention and early intervention programs were indeed scalable and could lead to positive outcomes.
The TRI example demonstrates that solid research is simply not enough. Implementation and sustainability rely on a confluence of factors that includes support from stakeholders outside of the educational enterprise, strong collaborations with universities and committed school districts, and extensive teacher professional development. But the implementation of the TRI, as well as programs such as Success for All, has certainly not been easy, and much remains to be learned to close the gap between research and practice in the educational environment. Indeed, there are many more questions than answers at this juncture. And this means that new research must be undertaken to address a host of complex questions.
Consider that even in fields thought to be more advanced in their approaches to dissemination, studies have documented difficulties in applying new research findings in clinical practice, although the findings are packaged and ready to use in the form of guidelines (Cabana et al., 1999; Lomas, 1997). Within education, research and practice communities must gain a much better understanding of how, when, by whom, and under what circumstances research evidence spreads through colleges of education, federal and state departments of education, school districts, schools, and classrooms. For optimal uptake and sustainability, we must study options for creating the incentive systems that will be critical to helping educators at all levels of practice develop the attitudes, knowledge, and skills needed for incorporating new concepts into their practices. We must identify the amount of time and resources required to enhance learning and adaptability in teachers, administrators, and other decision makers in education so that dissemination efforts reach motivated consumers. We must understand more about how to educate the public that rigorous scientific research is our best hope of helping consumers of education distinguish which educational practices work and which do not and, in that process, help to eliminate the continuous disagreements that characterize the field.
Very importantly, we must learn how to give teachers the basic knowledge and skills needed to translate research into effective classroom practices. Specifically, teachers must have sufficient preservice training to know the subject they are teaching, to guide instruction on the basis of student performance, and to recognize and tailor their approaches to meet the needs of individual students. Without this scientifically based foundation in content knowledge and pedagogy, advances in dissemination, implementation, and sustainability will not be realized. Beyond this, however, the role of teachers in getting research to practice must be redefined. Instead of treating teachers as empty vessels into which research information is poured, teachers must become partners and given opportunities to work collaboratively with scientists in designing and conducting the research that is expected to influence their instructional practices in the classroom. Realizing this goal will require substantial changes in the current training of both teachers and educational researchers.
In summary, the time has come to more systematically (a) identify strategies that motivate the adoption and use of knowledge and practices developed with scientifically sound education research, (b) carry out longitudinal studies to assess the long-term impact of dissemination and the conditions that help sustain improvements in teaching practices and student progress, and (c) support and conduct experimental studies to test the relative effectiveness of different strategies for educating various consumer subgroups about the value of evidence-based practices, such as administrators, teachers, researchers, and policy makers concerned with education in particular environments (urban, rural, suburban, and so on) from college to preschool.
Moving knowledge to practice is a complex challenge. And, to be sure, it will take time to see the effects of distributing new knowledge on everyday practices in the education profession and even more time to see improved student learning and achievement. In the impatient world of education, the lack of immediate observable results can be frustrating and often leads to fragmented and ineffective implementation. In the end, no matter how robust the educational research enterprise becomes and no matter how stellar the research conducted, it will mean little if sustained efforts are not undertaken to help ensure that the research is infused into practice.