Consistency of Reviews in ENQA IQA Group Seminar" (62 characters)
In this seminar, Sandra Marcos Ortega from ACSUCYL in Spain presented findings on the consistency of reviews in the ENQA-IQA Group. The mission, work plan, and analysis of outcomes were discussed along with proposals for future actions. Details on understanding and interpreting ESG 2015 standards were highlighted, focusing on agency quality assurance activities and stakeholder involvement. The session also included a template for reviewing reports and ensuring adherence to ENQA's standards. (387 characters)
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
CONSISTENCY OF THE REVIEWS CONSISTENCY OF THE REVIEWS ENQA ENQA IQA GROUP IQA GROUP Sandra Marcos Ortega. ACSUCYL (Spain) 2nd ENQA Reviewers Seminar Manneheim 10-11 November 2016 1
INDEX WHAT S ENQA IQA GROUP MISSION OUR WORK SOME FINDINGS WHAT S NEXT DISCUSSION 23.2.2025 2
WHATS ENQA- IQA GROUP Mandate for the ENQA IQA group. April 2015 Permanent working group in charge of the internal quality assurance of the agency external reviews. At the service of the Board Six members Aurelija Valeikien , SKVC (Lithuania) N ria Comet, AQU Catalunya (Spain) Ole Espen Rakkestad, NOKUT (Norway) Julien Lecoq, HCERES (France) Oana Sarbu, ARACIS (Romania) Sandra Marcos, ACSUCYL (Spain) Chair 23.2.2025 3
ENQA-IQA GROUP MISSION To analyse and provide knowledge on the outcomes of the ENQA external review process (and particularly the self- evaluations and evaluation reports) 23.2.2025 replace txt View menu > Header and footer 4
OUR WORK - Definition of the Work Plan - Familiarising with EQAR interpretation Since April 2015. 3 meetings - Understanding and interpretation of ESG 2015 - List of possible evidences ... 23.2.2025 replace txt View menu > Header and footer 5
OUR WORK ENQA understanding and interpretation of ESG 2015 Proposal of ENQA understanding and list of evidences. Analysis of SAR and ERR Two Agencies analysed. First evaluations against new ESG: ACPUA ECCE 6
OUR WORK Template for IQA-group s analysis of the review reports ECCE and ACPUA How the criteria were interpreted by the panel (match with ENQA s understanding? Any additional?) How the panel considered the evidences in the external review report? (Everything fully covered? Any additional evidence? Something not considered?) Agency interpretation in the SARs (match with ENQA s understanding? Any additional?) Evidence provided in the SARs (indicate the number) IQA group s observation (proposed changes in the initial document) ENQA understanding (IQA group s initial document) ESG Part 3 3.1 Activities, policy and processes for quality assurance Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a regular basis. They should have clear and explicit goals and objectives that are part of their publicly available mission statement. These should translate into the daily work of the agency. Agencies should ensure the involvement of stakeholders in their governance and work. - Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis. Organisations occasionally organise reviews of institutions or programmes do not comply with the standard. QA should be the main/one of the main activities of the agency. Match with ENQA s interpretation? Yes/No If no, how it was interpreted: Additional comments: Match with ENQA interpretation? Yes/No: If no, how it was interpreted: Additional comments: that only Evidence provided in the SARs (indicate the number): Additional comments: Everything fully covered? Yes/No: If no, specify what wasn t not considered? Any additional evidence? Yes, No. If yes, please specify: Additional comments: 3.1. Activities, policy and processes for quality assurance (Evidences) 1. Mission statement, description of values or principles of work. 2. How the mission is presented by the agency to the evaluated bodies. 3. Strategic Plan 4. The Annual Work Plan stating objectives, activities, units/persons responsible, timeline, budget. 5. Monitoring procedures of annual work plan implementation. Corrective actions taken. 6. Web page 7. its mandate. ... 7
OUR WORK ENQA understanding and interpretation of ESG 2015 Interviews with secretaries involved in external evaluations using new ESG - GUIDING QUESTIONS 1. Transition to the revised ESG: what are the ESG standards the agencies, in your opinion, find the most difficult to understand/to comply with? 2. Which were the most challenging ESG standards to evaluate, from the panel points of view? Why? 8
SOME FINDINGS FROM OUR ANALYSIS: Difficult to get consistency: differences in the interpretation between the members of the group when making the analysis Differences in understanding of some standards by Agencies (SAR) Similar interpretation by the experts but not in the final decisions Similar interpretation by the Agencies and experts as the first proposal of the ENQA IQA group 2.1 and 3.4 need more reflection on it 9
SOME FINDINGS FROM INTERVIEWS: Important the context of the Agency Well appreciate to prepare the review with the secretariat Interesting that someone understand the local language Share responsibilities between different members of the panel 2.1 and 3.4 the most difficult Mission is not always translated into daily work 10
FIRST FINDINGS: SOME EXAMPLES 3.4. THEMATIC ANALYSIS Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities. ENQA UNDERSTANDING The reports/thematic analyses published by the agencies should be based on findings from their EQA-activities, NOT on other kinds of information about the agency, the performance of the agency, or on data from the higher education sector in general. The reports/analyses should be clearly distinguishable from the agencies yearly activity reports. Expected evidence in the agencies SAR Procedure/process description for the production of thematic analyses Examples of analyses done: reports, analyses, magazine articles, journals, books etc. Description/documentation of how the agency s thematic analyses are typically disseminated 11
FIRST FINDINGS: SOME EXAMPLES ECCE Compliance: The ECCE President s annual report to the general council is published on the ECCE website each year. Additionally, the President prepares reports on ECCE s activities for the general newsletter of the ECU ( Backspace ). This report is also available on ECCE s website. Evidences: No evidences highlighted in the SAR Interpretation of the Standard by ECCE: Understood as annual activities report and accountability reports for Council. Panel Conclusion: Substantially compliance 12
FIRST FINDINGS: SOME EXAMPLES ACPUA Compliance: producing reports that are useful for HEIs and for the society as a whole, Evaluation reports. summary reports on global results at the end of every programme accreditation process. carrying out studies upon request of the Aragon Government about different key topics, e.g. employability of university graduates, connections of higher education with other education levels, learning outcomes, etc, annual report with information about all its activities and results. Report on the status of EQA in Spanish universities, , summarizing global results of all Spanish QA agencies in different evaluation processes. design of a new University Information System, defining a set of indicators to reflect the reality with accuracy and reliability and allowing comparative research about the different university systems. ACPUA s Board of Directors approved the Basis for an Excellence Programme aimed at fostering and recognizing excellence and best practices in the Aragon university system (for both institutions and programmes). Organizes monthly seminars, as they offer an open forum to reflect, discuss, share ideas and perspectives around quality and improvement in a non-evaluating environment. - Organizes meetings and sessions on different QA topics of interest. It also attends forums and working sessions organized by other institutions 13
FIRST FINDINGS: SOME EXAMPLES Interpretation of the Standard by the Agency: - To some extend the interpretation match with the Standard - Still including annual activities report , evaluation reports, Evidences: A list of evidences of each report and seminar organized and describe in the SAR. Panel Conclusion: Substantially compliance 14
WHATS NEXT To polish the list of evidences and the interpretation Meeting in January 2017 TO IMPROVE CONSISTENCY ON REVIEWS Work on a draft of guiding questions and a list of evidences FUTURE Work on the analysis of other SARs and ERR to improve consistency 15
WHATS NEXT Guiding questions for Agencies Guiding questions for experts EXPECTED OUTCOMES List of evidences 16
DISCUSSION 1. DO YOU THINK IT WILL BE USEFUL THE DOCUMENTS WE ARE WORKING ON. 2. IN YOUR OPINION, WHAT WILL BE MORE USEFUL : A LIST OF GENERAL EVIDENCES FOR ALL CRITERIA OR A LIST FOR EACH ONE SEPARATELY? 3. DO YOU FIND ANY CRITERIA MORE DIFFICULT TO UNDERSTAND THAN OTHERS? 18
