Better Evaluation by Rhys Jones at Prifysgol Aberystwyth University
This project focuses on evaluating the impact of youth work policy and interventions, addressing challenges and opportunities faced in the field. It emphasizes evidence-based approaches and the need for justifying public spending. The strategy underscores the importance of strengthening the evidence base on the impact of youth work across Wales. The framework of Demonstrating Success aims to capture the progress and achievements of young people receiving services through policy-based initiatives.
Uploaded on Feb 20, 2025 | 1 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Gwerthuso gwell?/Better evaluation? Rhys Jones (Prifysgol Aberystwyth University)
Cefndir/Background Prosiect ar hunaniaethau pobl ifan (ESRC) Yn rhan o Ganolfan Ymchwil WISERD ar Gymdeithas Sifil WISERD Education Project on the identities of young people (ESRC) Part of the WISERD Civil Society Research Centre WISERD Education
Strwythur/Structure Cyd-destun allanol: gwaith polisi a gwaith ieuenctid Cyd-destun mewnol: heriau a chyfleon Ymyrraeth, canlyniad ac effaith Dull y Newid Mwyaf Arwyddocaol Cymorth a chamau nesaf External context: policy work and youth work Internal context: challenges and opportunities Intervention, outcome and impact The Most Significant Change method Support and next steps
Cyd-destun allanol/External context Cyd-destun allanol 1 External context 1 Polisi ar sail tystiolaeth (Test, Learn, Adapt) Mae ymyrraethau n methu yn aml Mae angen cyfiawnhau gwariant cyhoeddus Evidence-based policy (Test, Learn, Adapt) Interventions often fail The need to justify public spending
Cyd-destun allanol/External context Cyd-destun allanol 2/External context 2 Youth work organisations and all engaged in youth work are expected to assess outcomes and impacts and to have systems for the planning, monitoring and evaluation of all aspects of their work with young people. Organisations and individual youth workers should be able to use self- assessment and self-assessment tools to identify positive benefits for young people and to reflect on and develop their practice (Youth Work in Wales: Principles and Purposes).
Cyd-destun allanol/External context Cyd-destun allanol 2/External context 2 The strategy recognises... the need to strengthen the evidence base on the impact of youth work across Wales. much more needs to be done to systematically and robustly evidence the contribution of youth work (National Youth Work Strategy for Wales 2014-18).
Cyd-destun allanol/External context Cyd-destun allanol 2/External context 2 Demonstrating Success is a framework for capturing the progress and success achieved by young people who receive services and support through policy based initiatives and programmes in different sectors and settings. As part of this wider framework, this handbook for practitioners describes methods for assessing the distance travelled in relation to skills that have been termed soft or generic skills (Demonstrating Success: Handbook for Practitioners). Youth Work Reference Group (2015): to develop and implement a National Outcomes Framework .
Cyd-destun mewnol/Internal context I ba raddau rydych chi n asesu canlyniadau a thraweffaith eich gwaith? Sut a pam ydych yn gwneud hyn? Oes arfer dda? Beth yw r heriau neu rwystrau? Oes potensial? To what extent do you evaluate the outcomes and impact of your work? How and why do you do this? Is there good practice? What are the challenges or obstacles? Is there potential?
Cyd-destun mewnol/Internal context Allbwn, canlyniad ac effaith Allbwn: gweithgareddau, prosesau, cynnyrch Canlyniad: canlyniad y gweithgareddau a phrosesau Effaith: effaith ehangach a mwy hir dymor Output, outcome and impact Output: actions, processes, products Outcomes: the results of actions and processes Impact: broader and longer-term effects
Cyd-destun mewnol/Internal context While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants lives on a broader scale (Andrew Harding, LSE). Youth organisations encourage and support the involvement of young people in the generation of social capital but would find it very difficult to assess the impact of that support and encouragement (Rob Norris, Towards and Outcomes and Impact Approach for the Youth Sector in Wales).
Cyd-destun mewnol/Internal context Time Skills Distance travelled The passage of time Society as a complex system Capasiti: Amser Sgiliau Pellter a deithiwyd Treigl amser Cymdeithas fel system cymhleth Capacity
Cyd-destun mewnol/Internal context Cyfleon Opportunities Rydych chi n adnabod eich cleientiaid yn dda iawn Mae gennych chi storfa fawr o wybodaeth amdanynt Mae sefydliadau eraill yn barod i gydweithio gyda chi Mae hwn yn amser da i lunio ffordd Gymreig o werthuso You know your clients very well You possess a large store of information about them Other institutions are ready to cooperate with you This is a good time to be shaping a Welsh approach to evaluation
Cyd-destun mewnol/Internal context Ideally, there should be some standardisation of categories across the youth service (voluntary and statutory) whilst allowing for appropriate organisational amendments and additions and for specific development programmes. Data to be collected by organisations working directly with young people should, where possible, include: statistical data; skills and competence data; youth worker data; achievement of the organisation s mission, strategic and operational plans; achievement of external stakeholder objectives. Norris, Towards and Outcomes and Impact Approach for the Youth Sector in Wales
Dull y NMA/The MSC method Damcaniaeth newid Deiagram sy n dangos dylanwad rhaglen neu ymyrraeth ar gleientiaid Amlinellu yr holl weithgareddau a phrosesau (allbwn) Sut mae r prosesau ayb yn arwain at ganlyniadau Natur effaith ehangach y canlyniadau Theory of change A diagram that represents the effect of a programme or intervention on clients Outline all of the activities and processes (output) How these processes etc lead to outcomes The broader impact of the outcomes
Dull y NMA/The MSC method We believe that a clear, concise and convincing explanation of what you do, what impact you aim to have and how you believe you will have it is a vital foundation of any programme, and a prerequisite for effective evaluation (NESTA, Guidance for Developing a Theory of Change for Your Programme).
Dull y NMA/The MSC method The most significant change (MSC) technique is a form of participatory monitoring and evaluation Essentially the process involves the collection of significant change (SC) stories and the systematic selection of the most significant of these stories by panels of stakeholders or staff When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact (Davies and Dart 2005: 8). MSC focuses on monitoring intermediate outcomes and impact (ibid: 9).
Dull y NMA/The MSC method Cwestiwn i r person ifanc /A question for the young person : Looking back over the last year, what do you think was the most significant change in terms of your engagement with the community? Terfynau (amser a gofod)/Boundaries (time and space) Gwerthuso/Evaluate (what do you think) Blaenoriaethu/Prioritising (the most significant, your engagement) Newid/Change (y gwahaniaeth a wnaethpwyd/the difference made)
Dull y NMA/The MSC method Casglu straeon: Gweithwyr ieuenctid yn ysgrifennu straeon maen nhw n ymwybodol o hwy Holi person ifanc yn uniongyrchol Trafodaeth gr p Person ifanc yn ysgrifennu r stori Collecting stories Youth workers writing stories that they are familiar with Asking a young person directly Group discussion The young person writing the story
Dull y NMA/The MSC method Pan adolygir yr atebion gan unigolion ar haen uwch/When answers are examined by individuals at a higher level: From among all these significant changes, what do you think was the most significant change of all? Darllen y straeon/Read the stories Trafod a dewis/Discuss and choose Esbonio pam/Explain why Adborth (atebolrwydd a dilysu)/Feedback (accountability and verification)
Dull y NMA/The MSC method Yr agwedd feintiol: The quantitative aspect 1. Agweddau meintiol i bob stori (faint o bobl ayb) 2. Holi am esiamplau ychwanegol o r newid mwyaf arwyddocaol 3. Meintioli y straeon amrywiol a gasglwyd 1. Quantitative aspects to all stories (how many people etc) 2. Seeking additional examples of the most significant change 3. Quantifying the various stories collected
Dull y NMA/The MSC method Manteision Data sy n cael ei gasglu gan sawl person Ansoddol i gychwyn (gan adeiladu ar gryfderau r sector) Modd ychwanegu elfennau meintiol Ffordd dda o fesur rhaglenni gyda canlyniadau amrywiol Hawdd i w esbonio Advantages Data collected by many people Qualitative to begin (building on the strengths of the sector) Scope to add quantitative elements A good way of measuring programmes with numerous outcomes Easy to explain
Dull y NMA/The MSC method Manteision Advantages Ffordd dda o adnabod canlyniadau annisgwyl Ffordd dda o adnabod gwerthoedd sefydliadol/ penderfynu ar flaenoriaethau Annog dadansoddi (angen cyfiawnhau penderfyniadau) Adeiladu capasiti ymysg staff a gwirfoddolwyr A good way of identifying unexpected outcomes A good way of identifying organisational values/deciding on priorities Encourages analysis (the need to justify decisions) Builds capacity among staff and volunteers
Dull y NMA/The MSC method Manteision Advantages Ffordd dda o adnabod canlyniadau annisgwyl Ffordd dda o adnabod gwerthoedd sefydliadol/ penderfynu ar flaenoriaethau Annog dadansoddi (angen cyfiawnhau penderfyniadau) Adeiladu capasiti ymysg staff a gwirfoddolwyr A good way of identifying unexpected outcomes A good way of identifying organisational values/deciding on priorities Encourages analysis (the need to justify decisions) Builds capacity among staff and volunteers
Dull y NMA/The MSC method Manteision Advantages It is already being used in the youth sector The work of Sue Cooper in the south-west of England Similar work in the Manchester area Mae n cael ei ddefnyddio yn y sector gwaith ieuenctid yn barod Gwaith Sue Cooper yn ne- orllewin Lloegr Gwaith tebyg yn ardal Manceinion
Dull y NMA/The MSC method O r holl newidiadau arwyddocaol hyn, pa un ydych chi n credu yw r newid mwyaf arwyddocaol? From among all these significant changes, what do you think was the most significant change of all? Read the stories Discuss and choose Explain why Darllenwch y straeon Trafodwch a dewiswch Esboniwch pam
Dull y NMA/The MSC method Nid dyma r ateb o reidrwydd ond mae n un opsiwn? Dull annibynnol neu wedi ei integreiddio gydag eraill? Oes angen ei addasu neu ei deilwra i r sefyllfa yng Nghymru? This isn t necessarily the answer but it is one option A standalone method or integrated with others? Does it need adapting or amending to reflect the situation in Wales?
Camau nesaf/Next steps Cymorth Help Cyrff yng Nghymru yn barod i helpu Prifysgol Aberystwyth WISERD Institutions in Wales are ready to help Aberystwyth University WISERD Pa gefnogaeth sydd ei angen? What support is needed?
Camau nesaf/Next steps Camau nesaf Next steps Ysgrifennu dogfen ar sail y cyflwyniad, yr ymchwil sy n sail iddo a r mewnbwn gennych chi Writing a document on the basis of the presentation, the research that lies behind it and your input Parhau i gydweithio? Continuing to collaborate?
