Supporting Elderly Female Victims of Abuse: Evaluation of Quality and Impact

WHOSEFVA Working with Healthcare Organizations to Support Elderly Female Victims of Abuse Train the trainers on evaluation Giorgi Davidovi Evaluation of Quality and Impact of Project Workstream 4 University of Tartu UT Co-funded by the Rights, Equality and Citizenship (REC) Programme of the European Union under Grant Agreement no JUST/2015/RDAP/AG/VICT/9320

Agenda 1.Introduction of Evaluation 2.Evaluation Framework 3.Evaluation Methodology: participatory and formative evaluation 4.Indicator type 5.Evaluation of the trainings 5.1 Train the Trainers 5.2 Mutual Learning Workshops 5.3 Training for the health and social care settings 5.4 Implementing Protocols for Identifying and Supporting Elderly Female Victims of Abuse 6.Evaluation of the Best Practice Protocols 6.1 The Best Practice Protocols 6.2 Focus groups/Interviews 7.Evaluation of the policy recommendations 8.Logistics of evaluation: the role of local evaluator

The evaluation action of WHOSEFVA aims: To evaluating the quality and impact of those actions undertaken within the project Framework for evaluation 1 To elaborating on the final evaluation report of WHOSEFVA actions

Task No. 1 Task Task description Unit responsible UT, WLF, AOF List of indicators, data collection plan and general instructions for data analysis Identification of relevant indicators need to evaluate the engagement process; Developing the data collection format (template); Defining goals for the preliminary analysis and providing general instructions for data analysis. Working together with the curriculum designers, an evaluation strategy is developed for soliciting feedback and analysing data collected from training and health care protocol implementation activities. Two other important elements of quality that will be monitored and evaluated are the quality of outputs and impact of activities. Data collection Local evaluators are responsible for collecting the needed data at the local level (documenting activities, administering and translating feedback forms, etc). This data is sent to the evaluator to synthesize and analyse. Data must be collected in the pre-given format (changes must be coordinated with UT). UT, PPs 2 UT train partners on how to collect local evaluation data Pre/Post tests for training participants to demonstrate increased knowledge. Preliminary analysis Regular updates on the evaluation is given every six months, so that corrective action can be taken if possible. UT, WLF, AOF, WAVE UT 3 4 Evaluation report The report will be prepared by the evaluator to address the key indicators: Collection of best practices on treating elderly female victims in healthcare Development of online course and training manual Evaluation of healthcare policy and recommendations in each partner country and EU All information gained from WS0-WS4 is collected into a final analytical report written in English. The closing evaluation will be carried out during the final meeting. The following questions will be addressed: - To what extent were the project objectives achieved? - To what extent were the project outcomes achieved? - How useful and beneficial are the results and outputs of the project to the target groups? - How useful and beneficial are the results and outputs of the project to the organisations involved and for national and EU-level work? The results are also evaluated against the EU regulations and recommendations. - How will the project outputs be sustained? How can we promote the use and utility of the outputs?

Framework for evaluation 1 The evaluation framework includes sessions with partners during the partner meetings. The progress will be monitored against expected results on a monthly basis, ensuring partners are well prepared for the meetings and the timeframes set for each task are followed. The evaluation questions are used as a tool for evaluating the projects process. In collaboration with all partners, the evaluation framework will address the following questions: - Are the appropriate methods being used to achieve the objectives? - Has this meeting and workshop helped to achieve the objectives? - Are we following the project plan/what should we change and why? - What did we learn during this meeting and workshop? - Can we see unexpected results and effects of the project; if yes, what effects do these have on the project activities? - How would we rate the quality of the partnership, communication, collaboration and working atmosphere among partners.

Evaluation Methodology The evaluation methodology used in the current evaluation plan is described through two concepts: participatory and formative evaluation. The two concepts are not contradictory, but rather touch upon different dimensions of the evaluation used in the given case. Participatory evaluation refers to how the evaluation is carried out, whereas formative evaluation refers to when the evaluation is carried out.

Participatory evaluation Participatory evaluation can be contrasted against conventional evaluation, in which trained evaluators are the experts who make the final judgement. Formative evaluation formative evaluation means that the evaluation is carried before the end of the evaluated process (either at the beginning of the program or during it). Participatory evaluation emphasizes the role of participants in the evaluation process which on the one hand is more democratic when compared with some of the other evaluation methods Formative evaluation may lead to substantial changes to the program as it enables clarification of the program goals and the true needs of the key stakeholders.

Indicator type A set of evaluation indicators (or questions) is presented in the following tables (tables 3 to 8). The tables include the evaluation indicators, the indicator type and the approximate time period when data should be collected. With regards to indicator type, five types of indicators are requested: Descriptive; Categorical; Binary; Ordinal; Numeric. Descriptive indicators require textual answers and are needed to provide in-depth information and context to other indicators. Categorical indicators are used in cases, where categorical options are pre-given and open-ended questions are unnecessary. Binary indicators are a special case of categorical indicators with only two possible categories. Ordinal indicators are used when the indicator can be measured on an ordinal scale, but the differences between different categories are difficult to quantify. Numeric indicators are used if the indicator is directly quantifiable.

Evaluation of the trainings

The trainings are the second shaping action, meant to improve the capacity of health and social care settings. Successful training program has the potential to amplify the effects of WHOSEFVA activities. Therefore, significant effort is assigned to the development of the training programs and its execution. WHOSEFVA project contains capacity building activities, including the Train the Trainers workshops, Mutual Learning Workshops, and Training for the health and social care settings.

WS/T ask Name of TtT The content of the training Number of participants Target Group of the Training 1,2 Train the Trainer for DV organizations Experts train partner DV organizations on working with healthcare (M4). The important part of the training is to give awareness raising tools in challenging ageism and gender discrimination and promoting positive attitudes towards elderly and older women. 16 Staff of partner organizations 1,3 Train the Trainer on Mutual learning workshops Experts train partners on how to conduct mutual learning workshops (M4). Partners will learn which themes to address, as they specifically relate to the training program and protocols being introduced. Partners will also be shown how to document the results in a way that is useful to the experts. 16 Staff of partner organizations 1,8 Experts train DV workers on political advocacy and policy analysis Train the Trainer on how to lead Elderly DV Support training for health care professionals The partners will assess the implementation of the regulations in their countries. 16 DV organizations, staff of partner organizations 2,4 Experts will conduct an in-depth 'train-the-trainer' session with staff from project partners who will conduct trainings with target audiences. Topics will include methods of engaging different audiences (nurses vs. doctors; emergency vs. primary care). 16 Partner DV organizations staff 4,2 Train partners on how to collect local evaluation data Pre/Post tests for training participants to demonstrate increased knowledge (M8-9). Statistics collected from selected DV/healthcare organizations before trainings and after trainings (and before and after implementation of protocols in test sites) to assess change in number of victims identified and types of services offered (psychological, counseling, referral, legal, etc.) 16 Staff of partner organizations

Target audience of the evaluation Qualitative indicators Date for collection the evaluation data Curriculum developers How partners identified the training needs of Trainees? November 2017 Curriculum developers Who were the trainers? How were the trainers identified? November 2017 TtT participants How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? November 2017 Curriculum Developers/ Trainers at TtT workshops Curriculum developers How the received feedback contributed the development of the pf the training manual? November 2017 How were the trainings developed? Who was developing the TtT programmes? November 2017

Local evaluators collect the data Quantiative indicators Date collection the evaluation data for Number of training participants (1.2 Train the Trainer for DV organizations ). November 2017 Number of training participants (1.3 Train the Trainer on Mutual learning workshops ). November 2017 Number of training participants (1.8 Experts train DV workers on political advocacy and policy analysis). November 2017 Number of training participants (2.4 Train the Trainer on how to lead Elderly DV Support training for health care professionals) November 2017 Number of training participants (4.2 Train partners on how to collect local evaluation data) November 2017 Number of workshops. November 2017

Mutual Learning Workshops Name of action The content of the training Number of event 1 even per Partner Number of participants 1. shelters/women s services/elderly services: 20 -EE LV and 30 persons per country in FI, AU, UK, GR. 2.medical professionals: 5 -EE LV and 10 persons per country in FI, AU, UK, GR. 3. social worker: 5 - EE LV and 10 social workers per in rest of participating countries. WS/Tas k 1.5 MLW 1st and 2nd 1) to discuss on challenges the growth of the elderly population (ageing population) brings especially to health care system services. 1) to give basic knowledge on elder abuse 1) to raise awareness of human rights/women s rights perspective in elder abuse 1) to share knowledge how professionals working with traumatized older victims of violence can maintain own wellbeing 6 in total 210 participants in total from health and social care settings

Target audience of the evaluation Date for collection the evaluation data Qualitative indicators Mutual Learning Workshops Who were the participants of the MWL? Demographic information? Participants October 2017/ February 2018 Who were the facilitators? How were the facilitators identified? Project partners October 2017/ February 2018 Which type of organizations received the training? Participants October 2017/ February 2018 How the attitude and knowledge of participants have changed after MLW? Participants October 2017/ February 2018 How the received feedback from participants was integrated in the development of the training manual? Manual developers October 2017/ February 2018 How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? Participants October 2017/ February 2018 Local evaluators collect the data Date for collection the evaluation data Quantitative indicators Number of participants from shelters/women s services/elderly services October 2017 Number of medical professionals October 2017 Number of social worker October 2017

Pre-evaluation form Post-evaluation from proposition 1 = not at all; 5 = completely I think elder abuse is a serious problem in my country What do you think is the level of problem of elder abuse in (add your country)? Serious problem Moderate problem Minor problem Not at all a problem What do you think is the priority level of recognizing and treating older victims of violence in your workplace? In my opinion recognizing and treating older victims of violence should be at higher priority level than before in my workplace High priority Medium priority Low priority Not a priority How familiar are with elder abuse in your work? Very familiar Moderately familiar Somewhat familiar Not at all familiar I am now more familiar with elder abuse How important you think is to recognize the older victims of violence in your work? Very important Moderately important Slightly important Low importance It is very important to recognize the older victims of violence in my work How often do you ask about violence experienced by your clients/patients if you suspect a case? Every time Almost every time Sometimes Never I feel I should ask more often about violence experienced by my clients/patients if I suspect a case

Training for the health and social care settings (II Mutual Learning Workshop) WS/Ta sk Name of action The content of the training Number of participants Number of event 2 session for 4x45 minute lectures in total. 2 sessions are divide into 2x45 minutes + 2x45 minutes in separate dates 6 event in total 2.7 Conduct local health and social care trainings 1. healthcare professionals: 25 - EE; 20 - LV, and 40 person per country in FI, AU, UK, GR. 2. Social workers: 5 person per country will participate from Estonia and Latvia, and 10 person per country in FI, AU, UK, GR. 210 in total

Target audience of the evaluation Date for collection the evaluation data Qualitative indicators Training for the health and social care settings How the training was developed? Manual developers June 2018 Who was developing the training? How the cooperation worked among the training developers? Manual developers June 2018 What were the sources of the training program? Review of literature used? Manual developers June 2018 What is the structure of the training? Training manual/progra m June 2018 What types of care providers were present? Participants June 2018 What was the demographics of participants in terms of gender? Participants June 2018 What was the knowledge and attitude of training participants before and after the training? participants June 2018 How well the participants integrated the information from the trainings into the everyday practice? Project partners/manu al developers June 2018 How the training participants were involved in the training development? Manual developers June 2018 How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? Participants June 2018

Local evaluator s collect the data Date for collection the evaluation data Quantitative indicators How many people were reached out? June 2018 How many participated? June 2018 How many trainings and session was conducted? June 2018

Number of event WS/Tas k 2.8 Name of action Implementing Protocols for Identifying and Supporting Elderly Female Victims of Abuse The content of the training Number of participants 5 persons will participate in EE and LV and 10 person in, AU, UK, GR, UK. Training will be provided for the health care staff and dv organizations on the protocols, best practices and tools developed. 1 per partner 6 event in total 50 in total

Target audience of the evaluation Date for collection the evaluation data Qualitative indicators Training for the health and social care settings What types of care provider were presented? Participants March 2018 What was the demographics of participants in terms of gender? March 2018 Participant s What was the knowledge and attitude of training participants before and after the training? Participants March 2018 How well the participants integrated the information from the trainings into the everyday practice? Manual developers March 2018 How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? Participants March 2018 Local evaluators collect the data Date for collection the evaluation data Quantitative indicators How many people were reached out? March 2018 How many participated? March 2018 How many trainings and session was conducted? March 2018

Evaluation of the Best Practice Protocols

Evaluation of the Best Practice Protocols WHOSEFVA experts introduce the protocols based on information gathered from the Mutual Learning Workshops, earlier Daphne projects and Emergency Care Procedure (PAKE) currently used in Finland. For the detailed information see the executive plan. The evaluation data regarding the protocols will be collected before and after the implementation. The data such as how many victims were identified, consulted before the implementation will be the main pre- evaluation indicator. Local evaluator will be obliged to conduct the interviews with the health and social care setting to document the pre-implementation the situation and post-implementation. A survey will be used for collecting the statistical data regarding the victim identification and referrals.

Target audience of the evaluation Qualitative indicators Date What is the structure of the protocols? What is the variation among the PPs? Protocol developers/ Project partners October 2018 How the protocols are implemented? Project health professionals partners/ October 2018 care Type of services offered Project health professionals partners/ October 2018 care How relevant were the BP for the health care settings? health professionals care October 2018 Type of injuries reported before the BP implementation Project health professionals partners/ February 2018 care Type of injuries reported before the BP implementation Project health professionals partners/ October 2018 care

Target audience of the evaluatio n Quantified indicators (perception) Date Number of victims identified before/after the BP implementation Interview conducted by evaluators with health care professional s/Raw data collected from health care settings February 2018 Number of referrals to police before/after the BP implementation February 2018 local Number of victims served before/after the BP implementation February 2018 Number of documented DV case before/after the implementation February 2018

Focus groups/Interviews Number of event WS/Tas k 1.6 Name of TtT The content of the training Number of participants 3 set support/focus groups or interviews. In Estonia and Latvia 5 individuals will be recruited and in Finland, Austria, Greece and UK 10 individual in each set of support/focus groups or interviews. Target Group of the Training Professionals at health and social care settings, domestic violence organizations Local focus groups/intervi ews To gain knowledge on the victims experiences and identification of the best treatment 3 per partner 18 event in total 150 in total

Target audience of the evaluation Qualitative indicators Date Focus groups/Interviews Who were the participants of the focus groups/interview? Demographics? Participants October 2017, June/October 2018 Who were the facilitators? Project partners October 2017, June/October 2018 How the collected feedback of 1st focus group/interviews was integrated in the training manual? Manual developers October 2017 How the collected feedback of 2st focus group/interviews was integrated in the training manual? Manual developers June 2018 How the collected feedback of 3st focus group/interviews was integrated in the training manual? Manual developers October 2018 How satisfied the participants were with the atmosphere, content and logistics of focus groups/interview? Participants October 2017, June/October 2018 Local evaluators collect the data Quantitative indicators Date Number of participants October 2017, June/ October 2018 Number of events October 2017, June/ October 2018

Evaluation of the policy recommendations

Target audience of the evaluatio n Qualitative indicators Date How was the policy recommendation developed? Project partners April 2018 The structure of the policy recommendations Project partners April 2018 Variation among the policy recommendations Project partners April 2018 How the advocacy meetings contributed the advocating the policy recommendation? Results of advocacy meetings Project partners April 2018 Local evaluators collect the data Quantitative indicators Date Number of policy recommendation April 2018 Number of people introduced with the PR April 2018 Number of participants at advocacy meetings April 2018

Logistics of evaluation: the role of local evaluator

The selected methodology of evaluation requires to closely coordinate the evaluation action on the local level. This means that every partner needs to have a local evaluator. The decision on who should be evaluator should be made inside the organization and coordinated with the evaluator (UT). Idea of local evaluator The local evaluation task can be done by the team, but there must be identifying one designated person who will be a responsible for carrying out the local evaluation actions.

First, partners are responsible for adjusting the evaluation templates to their local environment. That mainly includes translating the questionnaires and giving the feedback on developed evaluation materials based on the local context. First steps..

Second, if the localized evaluation plan is coordinated with the UT, partners will start with active data collection phase. That includes documenting activities, administering and translating feedback forms, conducting interviews with relevant target participants on local level, etc. To ensure the synergy of evaluation the actions undertaken by the local evaluator should be coordinated with the evaluator from UT. Following steps The raw data will be used for preliminary analysis conducted by the individual project partners and it will be sent to the evaluator (UT) for further analysis. Data must be collected in the pre-given format. The collected data should be translated and summarized also into the given evaluation templates by the local evaluator.

Cell:+37256097981 Skype: davidovi333 Thank you! Contact: Giorgi Davidovi Cell:+37256097981 Skype: davidovi333