| Evaluating Programs For Women: A Gender-specific Framework (2000 Revised Edition)
Usefulness of the Model
Use of the Framework resulted in a strong, comprehensive evaluation process and outcomes that helped the programs to identify their strengths and support base, areas needing improvement, and recommendations for improving the programs. In both cases, the evaluation proved useful for program improvement. The BCUPC program appreciated the depth and usefulness of the data developed. The GGVPE program viewed the evaluation as thorough and useful, and found that the recommendations were particularly helpful in making changes to the program.
Awareness of and Support for Women's Participation. The evaluations showed the need for awareness of, and arrangements for, support for women's participation. In both evaluations, not paying attention to women's unique needs can prevent them from participating. For example, costs of the following items should be included in the evaluation budget either in whole or in part:
The Process Is Empowering. The stakeholder members of the two Evaluation Advisory Committees (EACs) commented on their sense of increased knowledge about the program and about evaluation as a process in which they could experience true participation and impact. The pilot evaluations demonstrated that the use of a stakeholder EAC gives real strength and grounding to the evaluation process. From the first questions set out by the Committee through to their input on final recommendations, the EACs were sources of varied perceptions which then were discussed and refined. The EACs were helpful in sorting out best ways to collect data and for verifying facts and perceptions that emerged. They supported and helped focus the evaluation process. The evaluations would not have been as rich and comprehensive if the EAC had not participated. The responsive constructionist process empowers the stakeholder groups that are involved. This is a distinct difference from traditional evaluation in which power is in the hands of the program funder or sponsor to whom the evaluator reports.
In the GGVPE evaluation, a plain language version of the process was useful in presenting and discussing concepts. The collaborating evaluator developed a plain language version of the process, which was used with the EAC. This step helped participants to understand the process and results throughout the process. The oldest of the grandmothers involved in the GGVPE evaluation, a wise Elder, commented that she "could understand every word" of the training and discussion, and the concepts used in explaining and working through the process with the EAC. The plain language version helped with the empowerment process, and was an important feature of the collaboration between the evaluators and the EAC.
Celebration Is Important. Women's needs to feel connected and appreciated for their contributions to the evaluation were evident in the second evaluation. The EAC held a celebration upon completion of the work with a social afternoon with all the members, the evaluators, the program staff, funders, members of the school staff and community agencies, and the parents of student participants. Sharing a brief ceremony and food helped to provide an opportunity for thanks and congratulations for a job well done, and to bring closure to the process. The circle of sharing and grandmothers' prayers made the occasion a special one. Such a closing celebration is a valuable way to express appreciation to all of those involved.
Indicators Can Be Used for Other Evaluations. The evaluators noted that the safety and comfort indicators used in the study were measurable, differed by gender, and verified research findings reported in the literature. Using rating scales ranging from "0 = did not have any good qualities at all" to "10 = Perfect in every way," was a tool that was culture-free in that focus group and interview participants could use it with ease, and it rendered useful data. It also gave the evaluators insight into participants' thinking and consideration of issues. It also provided respondents with a way to think about the project and the evaluation process that appealed to them, and a way to communicate about it.
Gender and Other Societal Elements Are Intertwined
While gender and societal elements such as socio-economic levels and race can be viewed separately to some extent, the pilot evaluations demonstrated how intertwined such elements are and how difficult it is to deal with specifics in isolation. On the other hand, some gender findings were clear and unmistakable. In both evaluations, gender was a clear focus.
In the BCUPE evaluation, there was clear awareness of gender and the impact of socio-economic levels, race and access to resources. Gender issues did impact on the evaluation, including difficulty in gathering client information related to women's feelings of stigma in accessing reproductive and services, and for some, lack of power, resources and privacy in their own lives.
In the GGVPE evaluation, some participants were not sufficiently aware of how gender affects girls and classroom activities. For example, the project was originally established for girls, but staff seemed unaware that group gender composition plays a role in how girls are able to discuss and deal with issues such as violence, and how the experience of violence may differ for girls and for boys. In addition, the high proportion of Aboriginal students, and students from First Nations schools who were bussed to a mixed-race school starting in junior high, meant that racial and cultural issues received concentrated attention, whereas gender was not a focus. Classes had a larger proportion of boys than girls, which emphasized the tendency to overlook girls' issues since boys were harder to control and tended to receive higher levels of attention than did girls.
The Importance of Awareness of Gender Issues. With programs like the BCUPC, a high level of gender awareness had been developed and was evident. The evaluation raised awareness of gender, race, age, cultural, socio-economic and power issues. But the Framework also showed that not everyone is aware of how gender affects the dynamics and evolution of programs. In program settings where there is limited gender awareness, it may be necessary to provide gender training for evaluation committee members and others. Some specific examples drawn from project experience included:
The GGVPE program's original mandate was to develop a program for girls. A school mandate to involve both girls and boys changed the entire program focus to one which included both boys and girls. This diffused the original intent and focus of trying to help girls deal more effectively with family, school and community violence. The changed focus was not discussed with the organization funding the program or with the participant grandmothers. Program sponsors were unaware of the magnitude of impact of the changed direction on girls. Although the program achieved good outcomes, it did not provide the opportunity to focus on girls' needs regarding violence. This demonstrates what can happen when women-oriented programs are changed to serve other interests and lose their focus on women.
Lack of awareness of the dynamics at school affecting girls occurred at two levels: Teachers commented they were not aware of how girls are affected differently than boys in a mixed group versus an all-girl group. They saw no reason to have a program just for girls, nor to separate the sexes for group work or focus groups. Girl students stated that there would not be any difference in their behaviour and reactions in a mixed group as opposed to an all-girls group. The research literature clearly indicates large differences in the ability to be open, to share feelings, and even to be heard or acknowledged in mixed groups. This was confirmed in that the disclosures were made by girls only in an all-girls group. In settings where there is limited gender awareness, it may be necessary to provide gender training for evaluation committee members and others.
Evaluators Can Easily Overlook or Find Difficulty in Addressing Gender Issues. Though both evaluators were skilled in feminist approaches, they realized how easy it is to overlook gender aspects that, when examined, were significant (for example, not planning more thoroughly for an all-girl focus group; overlooking the need for specific gender analysis on student questionnaire responses which, when done, revealed significant differences that confirmed research findings on mixed group behaviours of girls). In retrospect, the evaluators noted how difficult it was to keep gender awareness functioning at all times and at all levels of the evaluation including planning, data collection and analysis. For example:
Unexpected Difficulties in Data Collection. Data collection problems occurred in both pilot evaluations. For example:
Much was learned about how to improve the Framework during the pilot evaluations, particularly with respect to the need to explain elements and processes in more detail. Among the more major revisions are:
Social structures and processes affect health and the quality of life. A key social factor influencing health is gender. At all levels of society, awareness is expanding about the intimate links between gender and health. Gender-specific health programming is emerging as a significant focus across Canada and internationally, stemming from a growing awareness of the need for effective, gender-sensitive, woman-centred programs and a concomitant need for gender-based program evaluation approaches to examine these programs.
Program evaluation is recognized as an important part of operating programs well. If evaluation and other processes do not reflect gender differentiation, they perpetuate old models that overlook gender needs and differences, and fail to support the empowerment of women. Yet a search of the program evaluation literature reveals that little has been reported in the area of gender-specific, woman-centred evaluation models or processes. A shift to gender-specific evaluation affects how evaluation structures and processes are conceptualized, utilized, managed, analyzed and reported. In turn, the way evaluation is employed effects how woman-centred services are developed and delivered, and how effective they will be.
As policy-makers interested in women's health and women's programming review their progress in addressing key health determinants and attempt to identify what approaches are most effective, questions that have fundamental relevance to these issues emerge:
This study was undertaken for the Prairie Women's Health Centre of Excellence (PWHCE) to develop a flexible program evaluation framework to address these questions while acknowledging the unique evaluation needs of every program and jurisdiction. The objectives of the project were to:
A program is defined as an organized system of services, or a related series of activities, designed to address specified health needs of clients. Some theoretical background, gender lenses, and models of programs that support woman-centred health and development are examined. The study sets out characteristics for programs in which there is an interdisciplinary approach and individual accountability for the program administration.
In a gender-specific, woman-centred program, four key phases are involved: gender-sensitive, woman-centred needs assessment; planning; implementation; and evaluation of the extent to which the program meets women's needs. The planning, implementation and evaluation phases of the program cycle are organized around the outcomes, processes, and structures of gender-specific health services. This enables consideration of what results are achieved, as well as the incorporation of those service strategies and resource approaches appropriate for achieving the desired results.
Program evaluation is a process that studies the extent to which desired outcomes were achieved, optimal resources were employed, and/or adequate structures were in place for undertaking the program processes. A gender-specific, woman-centred evaluation framework builds in gender- and woman-sensitive considerations at each step, and uses gender-based analysis as a key element.
ORGANIZATION OF THE REPORT
PART 1: Characteristics of Effective Gender-specific and Woman-centred Programs
PART 2: The Gender-specific and Woman-centred Program Evaluation Framework
PART 3: The Steps of Conducting a Gender-specific and Woman-centred Program Evaluation
The study sets out the goals, purposes, approach and principles reflected in the framework. It suggests the use of woman-centred and equity-sensitive processes, and considerations focussing on involvement and empowerment in establishing the evaluation committee, gathering data, analyzing results and developing recommendations. It is based on a set of ten generic steps:
At each step, the framework outlines the ways in which gender-specific considerations must be brought into play to ensure gender-sensitive and woman-centred program evaluation process and results. It outlines questions and a considerations at step, and invites those involved in evaluation of woman-centred programs to consider gender issues.each
Although women's organizations and community groups have long advocated that a greater proportion of health research and service delivery funding be spent on woman-centred activities, little evidence exists to indicate significant increases have occurred. To support the contention that women's health concerns merit gender-specific approaches, the framework can help to support the view that gender-specific programs provide effective outcomes.
When the desirable characteristics of gender-specific, woman-centred programs and a program evaluation framework have been identified, their application to specific programs enables us to conduct useful program evaluations that can influence both programs and policies, and elicit the cooperation and participation of program staff, their clients and other stakeholders.
The framework should be viewed as a flexible instrument rather than a rigid format for achieving evaluation objectives. The framework is not a definitive work, but a provisional one upon which future efforts can be built. In that spirit, we can learn together, and continue to use the collective process essential for the progress we pursue.
1. Joan McLaren, Evaluating Programs for Women: A Gender-specific Framework, Winnipeg, Manitoba: Prairie Women's Health Centre of Excellence, 1999.
2. The program evaluation report entitled Working with Women: Program Evaluation of the Birth Control/Unplanned Pregnancy Volunteer Counselling Program, Women's Health Clinic by Joan McLaren, was completed in June 1999.
3. The program evaluation report entitled More
Than Just Worry: Violence Prevention Education--An Evaluation in
a Gender-specific Framework by Joan McLaren and Jayne
Melville Whyte, was completed in April 1999.