Quality Assurance Practices and University Evaluation Systems by ANVUR President Stefano Fantoni

Quality assurance practices and University evaluation systems Stefano Fantoni President, ANVUR Euro-Mediterranean University Network TETHIS meeting University of Palermo 10 -11 of April 2014

THE ITALIAN NATIONAL AGENCYFORTHE EVALUATIONOF UNIVERSITIESAND RESEARCH INSTITUTES ANVUR is the Italian National Agency for the Evaluation of Universities and Research Institutes. It started operating in May 2011. ANVUR is in charge of Evaluating the quality of processes, results, and products of all the activities concerning research, teaching and technological transfer Defining methods and criteria to be used for the periodical accreditation and evaluation of Universities (undergraduate, postgraduate and Ph.D. courses) Providing general advices to internal evaluation bodies Evaluating public funding programs aimed at research and innovation activities Every two years, ANVUR will edit a general Report on the italian Higher Education status 2

THE ITALIAN NATIONAL AGENCYFORTHE EVALUATIONOF UNIVERSITIESAND RESEARCH INSTITUTES ANVUR, in its evaluation process, refers to the Quality Assurance system as described in European Standard Guidelines ESG/ENQA 2005/2009. ANVUR has performed the research evaluation exercise for the period 2004-2010 (VQR 2004 - 2010, Valutazione della Qualit della Ricerca), whose results have been presented in July 2013 ANVUR is currently performing the accreditation and assessment program for courses and institution (AVA, Accreditamento, Valutazione e Autovalutazione) ANVUR is currently developing the accreditation and the evaluation models for the PhD courses. 3

THECONTESTOFTHEEVALUATION Evaluation of Research Evaluation of Universities INCOMING RESEARCHERS EVALUATION COURSES EVALUATION MANAGEMENT EVALUATION PUBLIC FUNDING PLACEMENT & TRANSITIONTOWORK 4

THE EVALUATIONOF RESEARCH VQR 2004-2010 Each university professor or researcher has been asked to submit three research products (six instead of three to the researchers of the Research Institutes) published in the 2004-2010 period. Each research product (for a total of about 180 thousands) has been evaluated by bibliometric or/and peer review to evaluate NOT THE INDIVIDUALS but RESEARCH AREAS, DEPARTMENTS, UNIVERSITIES International character of the italian production: 62,1% of products is in English; mainly in scientific and technical areas (maximum value 88,6% in informatics and communication engineering). Humanities, law and in general social areas have the majority of works written in italian ( 5,7% of not italian in Law; 13.2% for Humanities and historical sciences). 5

THE EVALUATIONOF RESEARCH 2004 2010 (VQR 2004-2010) Products are classified in 14 scientific areas, as established by the National University Council (CUN). For eight areas (mathematics, physics, chemistry, earth Sciences, biology, medicine, agricultural and Veterinarian Sciences, part of Architecture and Civic Engineering, Industrial and communication Engineering) the research quality has been assessed mainly on the basis of bibliometric indicators ( impact factor and number of citations of the article in a mixed way). For six areas (Humanities and Social Sciences) the products have been assessed through peer review. Bibliometric indicators were used also in part of Architecture and Civic Engineering, Psychology Sciences, Economics and Statistics. Some areas were importing or exporting products, giving an image of the processes of crossed fertilization or of boundaries processes in Italian research 6

SOME NUMBERS 95 Universities, 67 public and 28 private, 11 of which are at distance institutions 12 Research Institutes under the control of the Ministry of Education and University 26 Research Institutes which have voluntary submitted an application to be evaluated by ANVUR 450 national and international Scientific experts from all the 14 Areas of Research 14.770 reviewers, 10.150 Italian and 4.620 working in non Italian institutions 18 months the full time-length of the evaluation 184,878 research products have been submitted for evaluation; 5.1% of missing products from - not active, or partially active- researchers. The typology of products varies in different scientific areas: book and book chapters (average value 19.9%) are more common in humanities and social sciences, articles (average value 73.5%) in scientific and technical areas. 7

EXPECTEDVS. SUBMITTEDPRODUCTS 30.000 25.000 # Prodotti attesi (Area del soggetto valutato) # prodotti conferiti (Area del soggetto valutato) # prodotti conferiti (Area del prodotto) 20.000 # Prodotti 15.000 10.000 5.000 0 01 02 03 04 05 06 07 08 09 10 11 12 13 14 GEV 8

SUBMITTEDPRODUCTSBY AREAOFTHEPRODUCT 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Articles Books, chapters Proceedings Others 9

THE EVALUATION RESULTS A research product has been classified as follows: Excellent, when it falls in the top 20% in terms of quality according to the international scientific community in the field Good, when it falls in the 60-80% Adequate, when it falls in the top 50-60% Limited, when it falls in the bottom 50% Penalized (not pertinent or copied): 2076 products (1,1%) TOTAL** 181878 100,0 Excellent 66973 36,8 Good 47925 26,4 Acceptable 22542 12,4 Limited 42362 23,3 10

NOTE on the Evaluation Results Comparisons only within the same area, NOT between different areas The products evaluation results have been used to elaborate the final assessment of the structures. Its weight was 50% The others indicators for the final assessment were ability to attract funds in a number of competitive calls (10%) mobility index, calculating the results of the new coming researches in the examined period (10%) Internationalization index (incoming and outcoming researchers, products with non Italian co-authors), 10% High qualification index (ph.d. students, post graduation grants ), 10% Funds for research coming from the institution, 5% Improvement index, as a difference from the former evaluation process (5%) 11

AVA ANDTHE QUALITYASSESSMENT COURSES & INSTITUTIONS ASSESSMENT ACCREDITATION: CRITERIA FOR QUALITY ASSESSMENT OUTCOMESASSESSMENT INTERNATIONALCOMPARISON ON-SITE VISITS QUALITYASSESSMENTGROUP 12

SUA: THE ANNUAL SUMMARIZING CHART Each dean or responsible of the course of study is required to fill the names of three professors and their teaching subjects yearly in charge of the course the aims and methods of the course the professional profile for the graduates The labor market needs the curriculum (characterizing compulsory or optional subjects, laboratories, stages, international programs ). SUA chart provides information for the students enrolling the following year 13

THEANNUALREVIEWREPORT Each University has a committee for the audit of the Quality Assessment (Nucleo di Valutazione). In November, the Nucleo Analize the SUAs of the former year the results of the questionnaires filled by students and graduates evaluates Quality Assurance system writes a rewiev report Suggestes the changes to improve the quality 14

THE EXTERNAL EVALUATION ANVUR will also perform the evaluation of the system as a whole Universities have an external evaluation every five years Courses of study have an external evaluation every three years The external assessment is performed by a group of experts appointed by ANVUR (CEV, Experts of Evaluation Commission). The CEVs work starting from the review of the Nucleo and from other documents produced by the course/university Then they do an on site visit of three to five days, depending from the dimension of the university. Their report is submitted to the University governance for comments . The final report gives an A, B or C classification. 15

AVA PROCESS Teaching service Self-assessment Student questionnaire Project to improve quality Review report Final assessment report On site visit Provisional report discussion Stakeholders and useers CEV commission of experts 16

17