Robin Lin Miller. American Journal of Evaluation. Volume 36, Issue 1. December 2014.
Within psychology, community psychology occupies what might be labeled the “social justice” branch of the field. Formalized as a new subdiscipline in 1965, community psychology’s mission is to create societies characterized by equality and wellness. From its beginning, community psychology was more than a psychological science of the individual-community interface. As Julian Rappaport noted in a 2005 paper entitled “Community psychology is (thank God) more than science”:
Community psychology begins with a vision of what ought to be. The vision is not a general one; it is quite specific to a set of core values. Among these core values are notions of the just society. This requires Community Psychology to be more than science. It does not stop it from using science; but it is not a field delimited or defined by science. Therefore, community psychologists learn from, use, and wish to contribute to the knowledge base of many disciplines and scholarly traditions, as well as to the practices of many different action traditions. (Rappaport, 2005, pp. 236-237)
Promoting the redesign of social systems to prevent disorder and promote well-being, particularly for groups who have been ill served by society drives the field. To suit its social change agenda, community psychologists have evolved unique perspectives on and approaches to community research and action.
Identifying optimal social ecologies and creating social change to promote community well-being puts evaluative activity at the front and center of community psychology practice. Although community psychologists conduct evaluations routinely, very little has been written on how training in community psychology informs the way a community psychologist might practice evaluation. Evaluators have a small number of examples of evaluation approaches and concepts that draw on seminal work in community psychology, such as empowerment evaluation (Fetterman, 1994; Fetterman & Wandersman, 2005) and Kirkhart’s (1995) concept of multicultural validity or reflect elements of a kindred worldview (e.g., Merten’s Transformative paradigm; Mertens, 2009). Empowerment evaluation draws explicitly on seminal theoretical work by community psychologists (e.g., Maton, 2000, 2008; Rappaport, 1981, 1987; Zimmerman, 1990, 2000; Zimmerman & Rappaport, 1988). Other concepts from community psychology, perhaps less well known outside the field, are defining of community psychologists’ evaluation practice.
In the remainder of this essay, I highlight three features of community psychology that have had an especially salient impact on my own evaluation practice. I also address what has been for me a key tension in straddling community psychology and evaluation perspectives over the prior 27 years of my evaluation career.
Social Regularities
Within the history of the field, my lineage traces to Edward Seidman, a close collaborator of Julian Rappaport. I was among Seidman’s original New York-based research assistants when he started the Adolescent Pathways Project. In that project, Seidman and his colleagues Larry Aber, LaRue Allen, and Christina Mitchell followed roughly 1,400 urban youth longitudinally across their transitions from elementary school to middle school or from middle school to high school. The youth came from three different cities in the Northeast. All lived in neighborhoods that were racially or ethnically homogeneous (White, Latino, and Black) and that were also characterized by concentrated poverty. The project evaluated the impact of school transitions on youth behavioral, cognitive, social, and emotional outcomes. On that project, I was responsible for recruiting the participating schools and students who were in the study’s primarily Black New York City neighborhoods and for tracking those youth over time as they moved from school to school. Seidman supervised my master’s thesis, which was my first large solo evaluation of a school-based contraceptive education program offered by a Planned Parenthood affiliate based in New Jersey.
In 1988, Seidman introduced the concept of social regularities to his evolving theory of social intervention. Social regularities capture the idea that a constancy of relationships exists within and between systems over time, place, and ecological level (Seidman, 1988). Seidman suggested that there are patterned transactions between people and their contexts and that it is these patterns that must be altered to promote durable changes in wellness and equality. Seidman’s thinking was influenced by Watzlawick, Weakland, and Fisch’s (1974) book on creating sustainable change, in which they point out that accepting the rules and assumptions of the system in which one intervenes is unlikely to produce meaningful and lasting change. Formative action research experiences studying efforts to divert juveniles from becoming enmeshed in the juvenile justice system (e.g., Davidson et al., 1977) and mutual help organizations for persons with psychiatric disabilities (e.g., Salem, Seidman, & Rappaport, 1988) were also critical in the development of his ideas.
As a graduate student training under Seidman at New York University at the time that he developed his first paper on social regularities, I was taken by the idea that an intervention could fail to produce any long-term effect because it focused on creating first-order changes—those that accept the rules and assumptions of a system, rather than second-order changes—those that alter the rules and operate from a different set of assumptions than those that are in force. According to Seidman, first-order changes will fail to alter the social regularities that produce a social issue.
In classes, Seidman drew on the seminal works of Roger Barker, Kurt Lewin, and of a number of system theorists to make his point. He was especially fond of using Seymour Sarason’s work to illustrate the pieces of his evolving theory. He would use Seymour’s description of an alien observing an elementary school to identify the regularities in classrooms: One big person spends the bulk of his time with many small persons and little if any time with other big persons; the big person asks the majority of the questions over the course of time. Seymour’s larger point (and Seidman’s) was that you cannot learn how to improve settings such as schools if you focus solely on the attributes of the people in those settings. Instead, it is the frequency and pattern of interaction among actors and their environment over time that will point you to the regularities in social interactions that maintain those settings’ status quo. In his 1998 essay, Seidman held up Kurt Lewin’s definition of discrimination as one example of a social regularity that demonstrates the value of paying close attention to social transactions and their constancy. Seidman quotes Lewin:
He defined the degree of discrimination as the: “number of refusals and permissions, orderings and yieldings, which indicate open and closed possibilities for individuals in their daily living … We are dealing with a process which, like a river, continuously changes its elements even if its velocity and direction remain the same.” (Seidman, 1998, p. 10)
Seidman wanted his students to attune themselves to the velocity and direction of the pattern of such elements (e.g., plotting cumulative income growth by income percentile over 30 years) rather than to focus on elements and individuals in a nontransactional matter. He believed doing so would aid us to think about what might be required to alter enduring and unjust social patterns successfully.
As an evaluator, I use the concept of social regularities as an overarching sensitizing concept to understand an evaluand and what it has the potential to achieve, as well as a lens by which to critique it. I try to follow Seidman’s suggestion to pay particular attention to examining the time-based rates and ratios that reflect person-setting transactions, as these provide clues to identifying the regularities in force. In a recent evaluation of a program to facilitate access to medical care for HIV-positive persons being released from state prisons, Seidman’s ideas proved critical to our evaluation approach. As in many other states, HIV-related reentry services focus on the initial 30 days following an ex-offender’s release because ex-offenders on highly active antiretroviral treatment are released with only that much antiretroviral medication. These men and women have a very short time window to connect with an infectious disease specialist for a new prescription and establish or reestablish medical and medication assistance benefits before they are at risk of treatment interruption. Treatment interruption has significant negative health consequences for persons living with HIV and for others with whom they might have unprotected sex or share needles.
The natural inclination in an evaluation of reentry services is close focus on the first 30 days to determine what proportion of persons can gain access to a new prescription before those 30 days are over; the proportion of persons accessing care anywhere from 60 days to 6-months postrelease is also often used as an indicator of success in linking ex-offenders to care because of how complicated it is for these men and women to establish benefits while also dealing with the basic tasks of reestablishing a life outside of prison. However, constancy of care—the rate of routine accessing of care over extended periods of time—provides a better indication of the social regularities surrounding access to treatment and is a predictor of achieving the objective of viral suppression (Crawford, Sanderson, & Thornton, 2013). Constancy of care provides an indication of whether reentry services change the regularities governing access to care in an enduring way. Constancy of care also better represents the notion that accessing care is a transaction that unfolds over time between an individual who might benefit from care and the multiple individuals and systems that facilitate or impede her access and use.
In the evaluation we conducted, the indicators of short-term linkage (e.g., the 30-day linkage rate and initial 6-month linkage rate) provide a much more positive picture than is gained by taking a social regularities perspective and examining rates over time. The proportion of persons accessing care in the first 6 months after release, slightly more than half the population, was atypical of the pattern over the subsequent 42 months. A very stable and undesirable pattern characterizes the longer series: In any single 6-month period, only about a third of the population accessed HIV-related care. The dominant pattern was inconsistent use of HIV care over time, which helped us to explain why reasonable rates of immediate linkage to care were not followed by low rates of mortality and good mental and physical health over the 3 years following release.
This finding describes only one of the regularities we observed that independently and in combination undermine the ability of HIV-positive ex-offenders to access care, comply with medical regimens, and live healthy, satisfying lives in the community. The reentry service, although perceived as helpful, did not alter the complex regularities that impact on long-term health care access and use; the status quo of the systems with which ex-offenders interact in the community was allowed to continue to operate as always. Michigan’s Departments of Community Health and Corrections are currently working together to use insights from our work to attempt to alter some of the key social regularities that negatively impact HIV-positive ex-offenders’ health after release.
The Ecological Metaphor
James G. Kelly, a founding father of the field and one of its most important guiding spirits, pushed community psychologists to view people in communities using an ecological metaphor grounded in natural sciences (Kelly, 1966, 1968). Kelly argued that human ecologies are dynamic systems, constantly adapting to events and recycling and redistributing resources. He urged paying close attention to these dynamic processes. As articulated by Kelly, the ecological metaphor positions evaluands as events that occur in systems. Penny Hawe and her colleagues have argued persuasively for applying the ecological perspective to evaluands. Hawe notes:
An intervention may then be seen as a critical event in the history of a system, leading to the evolution of new structures of interaction and new shared meanings. Interventions impact on evolving networks of person-time-place interaction, changing relationships, displacing existing activities and redistributing and transforming resources. This alternative view has significant implications for how interventions should be evaluated and how they could be made more effective. (Hawe, Shiell, & Riley, 2009, p. 267)
As an evaluator, the notion of evaluands as events in systems focuses attention on the interaction between evaluands and their contexts. Evaluating the consequences of evaluands requires more than examining changes at the individual level of analysis, even if it is at the individual level where desired consequences are principally directed.
The ecological metaphor and the perspective that evaluands are events in systems prefigure the integration of concepts from systems thinking and the techniques associated with system dynamics modeling into my evaluation practice and have deepened my appreciation of attempts to integrate evaluation and systems thinking (e.g., Williams & Imam, 2007). Although I use the latter principally as a formative exercise to improve my own thinking about context-evaluand interactions, the insights from the ecological metaphor stimulate me to pose questions about how evaluands change settings and respond to events in settings in terms of factors such as resource dynamics. I have found ecological thinking to be especially productive to guide planning evaluations in which context-evaluand interactions and contextual consequences are of interest (e.g., Miller, Forney, Hubbard, & Camacho, 2012: Miller, Levine, McNall, Khamarko, & Valenti, 2011; Reed, Miller, & Francisco, 2014). I adopt an ecological perspective routinely and have come to appreciate how much community psychology has to offer evaluation when it comes to thinking about context and cross-level relationships. In addition to the ecological metaphor articulated by Kelly and elaborated on by Edison Trickett, my dissertation advisor Marybeth Shinn has made particular contributions to advancing cross-level theory and measurement (Shinn & Rapkin, 2005) and applying these ideas to her own work in the area of homelessness policies and practices (Shinn, 1992, 2007).
Kelly’s ideas have had noteworthy influence on how community psychologists conduct ourselves in the settings in which we do our work, on how we form relationships to the people in those settings, and on our tendency to include first-hand observation in our work. Kelly urged community psychologists to engage in careful reconnaissance to understand the settings in which they are engaged and to do so for its own sake, rather than as prelude to something else (Campbell, 2012; Campbell, Patterson, Fehler-Cabral, 2010; Kelly, 1988). Kelly also championed the importance of humility in carrying out one’s work. In a classic 1970 essay, he observed:
The spirit of the community psychologist is the spirit of a naturalist who dotes on his environment; of the journalist, who bird-dogs his story; and of the conservationist, who glows when he finds a new way to describe man’s interdependence with his environment. The recommended way to prevent professional extinction is participation in the local community; the preferred antidote for arrogance is an ecological view of man. (Kelly, 1970, pp. 524)
In this essay, Kelly goes on to wonder whether his interest in communities occurs in spite of his training, as he sees that part of becoming professionally trained can have the consequence of making the communities we desire to serve seem beneath us. He admonishes that we often become “too hung up on ourselves.” Although it is not always possible to engage his ecological metaphor fully in a single project, it is always possible to recognize that reconnaissance occurs throughout the life of an evaluation, and to adopt an ecological view that situates oneself as person and evaluator in the scene as a basis for reflective practice.
Social Justice
Community psychologists are principally concerned with meeting community wellness goals on behalf of traditionally underserved and socially disenfranchised populations. Most of us meet this goal, in part, by focusing on a particular social issue (e.g., violence against women, GBLTQ [gay, lesbian, bisexual, transgender, queer] rights), a community setting that is or can be a resource to promote social justice (e.g., newcomer schools for immigrant and refugee youth, community organizing and advocacy entities), or by developing tools that enable community self-determination (e.g., The Community Toolbox developed by Stephen Fawcett and his colleagues at the University of Kansas).
My personal experiences and beliefs, coupled with community psychology training, have led me to focus my career on evaluands that address health disparities, HIV in particular. In the areas of the world where my evaluation work is focused (the United States and Eastern Caribbean), HIV disproportionately impacts people of color and gay and bisexual males. Stigma associated with HIV and the people who are most affected by it has impeded effective national responses to the epidemic in these parts of the world, as it has in many other countries. I want to improve efforts to undo the systematically unjust distribution of HIV in these societies and evaluation provides me a useful tool to accomplish this. Similar to Donna Merten’s transformative evaluation approach (Mertens, 2009), in my evaluations I attempt to serve the needs, interests, and concerns of the disenfranchised communities who bear the burden of HIV. I do so by making particular effort to reflect the perspective of HIV-affected people in every evaluation, whether I use participatory methods or not, and by the kinds of programs I evaluate. Most of my work has focused on programs to reduce exposure to HIV among high-risk males, specifically young Black gay and bisexual men and male sex workers.
I have been lucky that some of my work has succeeded in promoting social justice. For example, in 2007, the American Psychological Association (APA) established a Task Force on Therapeutic Responses to Sexual Orientation. The five-member Task Force was charged with recommending to APA a policy position on how clinicians should intervene when confronted with people who are experiencing distress over their same-sex sexual attractions. The Task Force was asked to consider whether APA needed a new resolution on this issue and, if so, to develop that resolution. The political stakes surrounding the Task Force were high, as it was quite possible that the practice of conversion and reparative therapies could be deemed unethical by the Task Force, which, some feared, might jeopardize the professional standing and licenses of those psychologists who engage in these practices. The Task Force’s work had the potential to position APA in opposition to conservative religious ministries that advocate “curing” gay, lesbian, and bisexual people of their same-sex attractions.
In light of the brewing controversy and high political stakes associated with this Task Force, the then APA president, Sharon Stephens Brehm, appointed me as a 6th Task Force member to bring evaluation expertise to the work. The other members brought substantive clinical or research expertise.1 I was specifically charged with the responsibility to assess the available evidence on the benefits and harms that are experienced by those who undergo reparative and other therapies designed to change a person’s sexual orientation and underlying sexual attractions. In this role, I evaluated the evidence produced over a roughly 50-year period on reparative therapy and related therapeutic approaches. We found no credible evidence to support the use of reparative therapy and some evidence of harm.
The APA adopted the Task Force report and its associated resolution in 2009, after a period of extensive review. When the report was released in August 2009, it received widespread attention in domestic and international news media. Since that time, the report has been widely used, and, most recently, it was cited as providing a basis for legislation banning the practice of reparative therapy with minors in the states of California and New Jersey. The California Law held up under appeal in August 2013. The report has also influenced key policy position statements on the topic by leadership organizations around the globe including the Pan American Health Organization. The report’s careful grounding in available evidence and balanced reporting has been characterized as fair by advocates on both sides of the issue, although it certainly does not please everyone, as few reports would be likely to meet with universal approval.
The Tension
In the piece I cited at the beginning of this essay, Rappaport argued that community psychology is both science and social critique. He said:
I want to emphasize that although community psychologists can and do use science, I do not think Community Psychology is only a science: thus, my title. In my view, Community psychology is a unique blend of science and social criticism. It is a field of practice with explicit goals that might be thought of in Freire’s (1996, 1998) terms as critical consciousness. Combining the goals of fostering critical consciousness with the methods of science makes for an unusual combination (Rappaport, 2005, pp. 233).
Evaluation also blends systematically gathered evidence with critique to achieve a better world. Perhaps this is one reason why community psychology’s theory and practice literature mirrors many aspects of evaluation’s literature or at least reflects overlapping interests in promoting social change, appreciating diversity in perspective and values, engaging in collaborative practice, and insuring the relevance and utility of gathered evidence. However, there are significant tensions between the two. Although my training as a community psychologist informs my evaluations, there are critical differences in perspective among evaluation, writ large, and community psychology, and key differences in practice.
Theories of evaluation and community psychology differ on how evaluators and evaluations influence social justice. For community psychology, pursuit of social justice is the overarching aim of science and practice. Our vision of a just society points us to where to act and on whose behalf. We are advocates for particular social causes and groups and taking direct action on social issues is part of our practice. We are as likely to read community organizer Saul Alinsky’s Rules for Radicals (1971) in our first year of training as we are a text on field experiments. In our official vision statement for our APA division, the Society for Community Research and Action (SCRA), we say that we “promote social justice for all people by fostering collaboration where there is division and empowerment where there is oppression.” Community psychologists advocate for equality and we play many roles in pursuit of social justice outcomes as part of our professional practice. We are prescriptive in our valuing and openly advocate our value positions.
SCRA’s vision appears to be harmonious with arguments that evaluation is a moral activity (House, 1980). However, in practice, SCRA’s vision contrasts with the visions set forth by many evaluation theorists of how professional evaluators fulfill their moral function. For many evaluation theorists, although not all, evaluators facilitate just outcomes by contributing fair and impartial evidence to democratic debate and dialogue, by advancing well-reasoned and warranted claims about evaluands, and by transparently considering diverse value positions in a balanced fashion. Evaluators support the ability of others to act by providing useful evidence to inform their thinking about what course to take. It is in this sense that community psychology and many theories of evaluation differ in their assumptions about how one influences social change through the production and use of evidence and how to enact one’s moral professional responsibilities. The Task Force on which I served proved itself a good example of the potential for tension between acting credibly as a community psychologist and as an evaluator as prescribed in many theories of evaluation practice.
The Task Force’s consideration of what to do when confronted with a person who is distressed by their same-sex attractions had to take into account the values of very diverse constituencies who have opposing views on homosexuality and who remain deeply at odds with one another. In taking on the evaluator role for the Task Force, the credibility of my work rested on my willingness to see the issue from all of those viewpoints. It rested on my willingness to say that therapeutic efforts to change a person’s sexual orientation work and do more good than harm, should the evidence support me in arriving at such a conclusion. It rested on my ability to represent the range of value positions on the issue and particularly those in diverse orthodox faith traditions who believe strongly that eliminating a person’s same-sex sexual attractions is the right and best thing to do. Obviously, finding evidence that reparative therapy eliminated same-sex sexual attractions (which no solid evidence suggests it can accomplish) and never harmed people (which evidence suggests these practices can do) might put an evaluator in an unenviable position, depending on her own beliefs. But she could be viewed in a defensible position by having contributed an impartial synthesis of evidence to a larger dialogue about whether psychologists ought to be in the business of trying to change a person’s sexual attractions at all. Would she be viewed as having compromised her role as a community psychologist?
I am not convinced, given the moral position community psychology takes on human rights and sexual orientation, that I would have interpreted how best to carry out my moral professional duty on the Task Force in quite the same manner as I did approaching the work with the lens of an evaluator. Yes, I would consider the empirical evidence in nearly the same way, but given community psychology’s prescriptive value position, I may not have been cued to invest in understanding why eliminating same-sex attraction was so important to some people and why reparative therapists care so passionately about what they are trying to accomplish. My training as an evaluator outside of community psychology signaled me to understand that worldview as respectfully and as well as I possibly could and to write about our findings in a manner that reflected that understanding. I was also pushed to view the task in this way by the APA president who, at our first meeting, leaned over and whispered in my ear that it was my individual responsibility to make sure that the work remained impartial and grounded in whatever available evidence might show. I suspect that she grasped what would be required to move a resolution in the APA council successfully, given the diverse ideological positions on the topic and interests in the particular therapeutic practices within and outside the profession. I think the experience reinforced for me that how community psychology and evaluation each function to support social critique in democratic society may and perhaps should differ. Each offers different means to a similar end.
The Task Force took careful account of the perspectives of those from conservative religious backgrounds in reaching conclusions about the meaning of the evidence. I think, though I cannot prove it, that the care that we took to be balanced and fair to the evidence allowed our work to be influential in democratic debate in ways that would not have occurred had we taken the partisan approach that was originally envisioned by some and feared by others. Indeed, in addition to influencing policy, I think our balance and fairness allowed us to contribute to an environment that influenced Dr. Robert Spitzer to publicly request that his research be retracted from the Archives of Sexual Behavior because of its serious methodological flaws and the paper’s ongoing misuse to support reparative therapy, and Exodus International, the largest ex-gay ministry in the world, to disassociate itself with reparative therapy. Exodus International announced that 99.9% of the “ex-gays” they once believed they had helped are still same-sex attracted. (The evidence suggests that people who undergo these treatments continue to have same-sex attractions, even if some learn strategies to ignore their attractions.) Exodus International decided to shut its doors as an ex-gay ministry in the summer of 2013, after issuing an apology for the trauma and suffering it had caused.
Conclusion
Engaging in evaluations raises important dilemmas for community psychologists, as our role as evaluators requires commitment to the evidence whether we like that evidence or not, whereas our role as community psychologists requires commitment to evidence in the service of our field’s prescriptive values. We place priority on social critique and activism in carrying out our work, activities to which many evaluators are strongly averse and believe compromises evaluation’s credibility. Evaluators do advocate, but we advocate for rigorous evaluations, for the findings produced in evaluations that meet standards of professional quality, and for the use of evaluation processes and results. Each field, one through its advocacy for evaluation and the other through its advocacy of justice and empowerment, has carved out very different pathways to serving the development of a better society.
What protections must the community psychologist observe when she takes on the role of evaluator in order to avoid compromising the evaluator role and the credibility of evaluation as an impartial evidence-driven profession? I have only two suggestions, both borne out of my experiences straddling these two fields. Perhaps most obvious, when a specific evaluation opportunity presents itself, community psychologists may want to engage in self-reflective exercises in which they anticipate how easy or difficult it will be for their team to produce and disseminate evidence that runs counter to the field’s values. Consciously engaging in this kind of self-assessment by imagining their worst-case values scenario may help community psychologists to identify when they are at risk of producing a compromised evaluation or compromising on values beyond what they can tolerate professionally. Community psychologists might then be able to develop specific strategies to increase their ability to manage an impartial evaluation process successfully. Self-assessment of this kind might also highlight that the value positions and perspectives that are most vulnerable to underrepresentation are not necessarily those that come first to mind or signal to the evaluator that she must make special note that these positions and perspectives are absent from and did not inform the evaluation. And, if nothing else, such an exercise might help the community psychologist to be clear about when they ought to pass on specific evaluation work for values-based reasons.
It may also help community psychologists who do evaluations to recognize that there is a critical difference between backpedaling on their core values and using evaluative evidence to create opportunities for dialogue among diverse stakeholders about differences in what may be valued. To that end, community psychologists might establish a series of interdisciplinary charrettes in which seasoned evaluators from diverse policy and practice arenas, applied researchers from diverse disciplines oriented around solving social problems, and community psychologists come together to address the challenges and problems of advocacy and prescriptive valuing in evaluation. Cross-disciplinary interaction may lead to insights on what makes an evaluation socially just and could lead to improved theories of practice on how we can contribute to the creation of better societies by producing evaluations that are assets to societal improvement processes.