Appendix 2: Case Study Answers

Karen Tote has been overseeing a 12-hour hotline for students coping with mental health and addictions and/or in crisis at Lakeside College for 2 years. The hotline is staffed by five (5) telephone counsellors who work on rotation. The staff are consistently working at more than full speed, and can’t keep up with the demand. They are turning over quickly because of volume of work. Karen wants to expand to a 24-hour service and hire more staff. She decides to conduct an evaluation and gather data to strengthen her case for increased funding. There is not much room to adjust her workload but she is committed to overseeing the evaluation herself, as no one else is able to lead the work. Her budget is limited as the bulk of her funding is dedicated to running the hotline and is already stretched. However, Karen does have a dedicated group of stakeholders that are willing to support the evaluation.

Karen decides she wants to assess the following:

1. How well the service delivery model is working;
2. How useful the hotline has been in offering mental health supports and crisis intervention
3. If the hotline has been useful for diverse students such as LGBTQ students, students of diverse cultural and ethnic backgrounds and students with disabilities.

Consider the Following Questions:

What might some limitations be for Karen in conducting the evaluation?

Limitations include budget and time constraints. Karen’s questions are both process and outcome focused and require time and resources to plan and conduct effectively. Given the staff workload and high turnover, Karen might not be able to rely on any staff support for evaluation activities and may burn out if she takes too much on herself.

What are some ways in which Karen can prepare for her evaluation work?

Karen’s first step would be to assess her readiness for evaluation and list what she has in place and what she will need to work on in order to get the most out of her evaluation efforts. She can also rally her stakeholders and identify any activities they may be able to lead. Karen can also consider connecting with evaluators on campus or applying for an evaluation grant if possible.

How might she prioritize her needs?

Given the staff workload and turnover, Karen’s priority may be to focus her time on securing evaluation supports as getting ready for her evaluation. If she must prioritize her evaluation questions then she may consider focusing on the first two. Determining the effectiveness of the hotline will allow her to make the case for increased supports. This can also allow her to seek resources to then target diverse student populations to ensure the hotline is effectively serving LGBTQ students, those from diverse cultural and ethnic backgrounds and/or students with disabilities.

Erica is a Mental Health Educator at Mapleleaf College and works with the Student Health, and Wellness Centre on campus. Over the last three years she has been organizing training sessions for counsellors, clinical staff, administrators and student body leaders to recognize early signs of addiction among students accessing their services.

The training is conducted twice a year and Erica has decided to evaluate how effective these sessions have been. She spent a considerable amount of time (above her existing workload) developing and delivering a questionnaire to the 54 representatives who had previously attended the training sessions and received an overwhelming response with 48 completed surveys. Once she received the completed surveys she approached her supervisor for the go-ahead to hire someone to help her analyze the surveys. Her supervisor is surprised to hear of this request as she had not been aware of Erica’s evaluation efforts and does not have a budget to assist Erica with her evaluation. Close to 6 months later the completed surveys have yielded no action or analysis and no updates have been shared with the people who completed them.

Consider the Following Questions:

What were some factors at play in this scenario that impeded Erica’s evaluation efforts?

Erica’s first step should have been to communicate her interest in evaluation with her supervisor or team and collectively determine how to move forward. Without assessing her team’s readiness for evaluation, her efforts may have compromised any future evaluation work undertaken as well.

What effect do you think her actions may have on any future evaluation work?

Her lack of communication with her supervisor about her evaluation efforts could sour her relationship with her team especially on any topic related to evaluation. She may be feeling resentment for not being able to follow-up on the analysis or may feel regret for not undertaking the proper steps required. Her supervisor may feel left out of her decisions and disconnected to the work she has initiated with stakeholders. The evaluation questions she asked may not have reflected the goals of her program, as she had not consulted with her team. The stakeholders themselves may be confused about the lack of follow-up and may feel as though their participation has been a wasted effort. This may make them unlikely to participate in any future evaluations.

What might she have done differently?

Erica should have begun by consulting with her team and collectively working on assessing their readiness to plan, conduct and share any evaluation work they engage in.

Ruby has been working at the Student Accessibility Office at Waverly University as a Mental Health Educator for over five years. As part of her work she coordinates an annual mental health fair and speakers’ event to engage students, school staff and faculty on key issues related to student mental health and addictions. Every year she partners with mental health service providers from organizations on and off campus such as the Canadian Mental Health Association, Community Health Centres, Student Counselling Services and/or Health Clinics to attend the event as guest speakers and exhibitors. Her goals are:

1. To reduce stigma associated with mental health and addictions on campus among students and administrative staff by offering a platform for open dialogue and learning
2. To increase student knowledge of mental health such as ‘recognizing signs of depression’, supportive ways of acknowledging mental health’ or ‘dealing with anxiety’
3. To connect students to relevant services on and off campus to help them navigate and access the services they need

The event is in its fifth year and Ruby wants to conduct an evaluation to see whether they are meeting their goals. Through her event logs Ruby knows that 360 students and campus staff have attended the event in the last five years, and through the feedback forms completed at the event each year she knows that 70% of attendants learn something new about mental health. She has also made considerable partnerships by securing speakers and exhibitors for her event, 23 in total. This has increased the list of services students may be referred to by 40% when presenting with a mental health disorder at the Student Accessibility Office where she works. Over the last five years her office has also noted an increase of students accessing their in-house peer counsellors, an increase in referrals and an increase in student leaders who want to support mental health work on campus as peers or volunteers.

Here are some output and outcome related questions that Ruby could consider useful in her evaluation. Consider how Ruby might answer these questions using the information above

Outputs:

• How many events were held?
  5 events over 5 years
• How many participants were in attendance year over year ?
  360 students and campus staff
• How many partnerships have resulted from the event?
  23 partnerships

Outcomes:

• How has the mental health fair affected student access to services at the Accessibility Office?
  We know that access has increased by peer counselling appointments, referrals and volunteers however, we need additional data to determine if there are other contributing factors to these increases in service access
• How have the new service partnerships affected student access to services?
  We know that there is a 40% increase in the referral list so there are more partners and services that students may be referred to but determine the outcome of this on student access will require additional information
• Has the mental health fair led to changes in the way students perceive mental health?
  We may infer from the increases in student leaders wanting to support mental health on campus that the event has led to changes in perception but again, more information would be needed to qualify this as an outcome.

What might some additional output or outcome related questions be?

• How many participant feedback forms have been collected over the years – OUTPUT
• Have there been year over year increases in attendance – OUTPUT
• Are there partnership agreements in place? – OUTPUT
• Have the annual fair had any effect on mental health stigma on campus? – OUTCOME
• What other effects might the annual fairs have had on campus, among students, staff or other stakeholders? – OUTCOME

Jonathan is a psychotherapist specializing in youth addiction work and has recently joined the Bold Prairie College Student Counselling Office. In the first six (6) months of his appointment he has only seen an average of two (2) students per week and the office’s efforts to outreach this service has seen little uptake.

Jonathan’s quantitative data seems to indicate that having a psychotherapist on the team is not what is needed. Jonathan decides to gather some evaluation data and hosts three focus groups with a cross section of students. The decision to host focus groups (i.e., qualitative data gathering) is made so as to be able to learn more deeply about the student experience.

Through the focus groups the Student Counselling Office learns that while many students wish to access Jonathan’s service they are too afraid of being seen making appointments with him and are fearful of the stigma associated with accessing mental health services. Using both qualitative and quantitative data points allow Jonathan and his office to understand not only what is happening but also why it is happening.

Consider the following questions:

What other types of quantitative or qualitative data might Jonathan have collected?

Quantitative data:

• How many students have been referred to external psychotherapists?
• How many students are accessing other services at the Student Counselling Office – is there a trend of low uptake of available services?

Qualitative Data:

• What could the Student Counselling Office do to support students feeling safe in accessing psychotherapy services on campus?
• What could be some activities to challenge mental health stigma on campus? How may student leaders be involved in supporting some of these activities?
  
• What actions might the Student Accessibility Office take to continue to stay informed about how students feel about their services?
  They can incorporate feedback forms at their offices and methods for students to offer suggestions online through their website. They can also engage in ongoing evaluation efforts with follow-up focus groups.

Lee is a student leader at Coolwaters University who organizes diverse social awareness events on campus along with the Student Life Centre. He has been learning more about alcohol addiction and mental health among students and has organized a seminar with guest speakers from the local Addictions Treatment Centre. The seminar is planned as an all-day event with two workshops and one keynote speaker. Lee is also developing feedback surveys for all those planning to attend. His learning goals are: (1) to determine how useful the seminar was for students who attended; and (2) if students would like to see more events such as these on campus in future. Lee has developed the following questions:

1. The following information was most useful to me:
   ☐ Early signs of mental health and addiction challenges
   ☐ Information about available services in the area
   ☐ How to talk about addiction
2. How satisfied were you with the speakers of workshop 1 and workshop 2:
   Unsatisfied         Somewhat Satisfied         Satisfied             Very Satisfied
3. List three (3) things you learned today that you did not know before:

                                                           
                                                           
                                                           

• Has this seminar affected the way you think about alcohol addiction or other forms of addiction?

Consider the following:

Are all these questions well formulated? Which ones can be improved and how?

Question a.) could include more options and should also have an ‘other’ line. Question b.) should be separated into two questions so respondents can comment on workshop 1 and workshop 2 independently.

What other types of information should Lee include in this survey?

Lee should also include a preamble for the survey; information for how students may contact the Student Life Centre with additional questions; instructions for how to complete the survey; and, a note about confidentiality, how the survey data will be used and by whom.

What might be some other kinds of questions Lee can ask to meet his goals?

Based on his goals, Lee should also include a question on whether students would like to see more events such as these on campus. Since there is a keynote speaker planned, one of the questions should ask about how satisfied people were with this talk. Lee may also consider some demographic questions to determine which students have attended. For example asking respondents to state their gender may yield the realization that 70% of attendees are male, this can lead the Student Life Centre to consider ways in which to create a more gender inclusive space for events in future.

Onye is a student leader at Threerivers College and is very passionate about women’s mental health on campus. She volunteers with the Maya Women’s Health Clinic in her community and has spearheaded a partnership with the Clinic and the Campus Safety Office to develop a poster campaign specifically promoting women’s mental health.

The campaign is officially run by the Campus Safety Office and calls attention to the mental health stresses disproportionately faced by women such as sexual harassment and low body image or self-esteem. Onye was actively involved in developing the campaign and promoting it on campus through student groups, social clubs and student residences. One year later the Campus Safety Office is undertaking a combination of a process and outcome evaluation of the campaign and has reached out to Onye as a key stakeholder. She is helping the lead evaluator with their data collection efforts. The Safety Office is looking to collect information that can answer the following evaluation questions:

1. How did female students on campus respond to the poster? Did they feel like the message resonated with them?
2. Did the poster result in dialogue on female mental health, sexual harassment and/or other issues disproportionately faced by women on campus?
3. Has the poster campaign increased awareness of the issues female students struggle with on campus and how these issues affect their mental health?
4. How effective and/or useful has the partnership between the Maya Women’s Health Clinic and the Campus Safety Office been in running the poster campaign?
5. How has the poster campaign effected the general student population on campus regardless of gender?

Onye and the lead evaluator have determined that they will need to collect a mix of qualitative and quantitative data to help answer these questions.

Consider the following:

What data collection tools could Onye employ to gather the information needed?

The questions Onye is looking to answer require could best benefit from both quantitative and qualitative responses and a mix of surveys and focus groups would be useful in gathering the kind of information needed

Should different tools be employed for different audiences/purposes? How?

• Data to determine how useful the partnership has been can be collected through a survey administered online with key partner representative from both organization and any student leaders like Onye who were involved in developing the campaign.
• Data about how the campaign has affected student populations can be conducted through focus groups. Sessions with only female student will be required to explore the effect the campaign may have had or how and why the message resonated with women.
• A focus group with male students could also be conducted to determine if this changed the way in which students understand how safety and other pressures affect female students? It may be useful to involve them in exploring any role they can have in creating safer or more supportive environments on campus.