ANVUR Experience: Research Evaluation Approaches and Controversies

Evaluating research on a massive scale in the ANVUR experience. Approaches and controversies Andrea Bonaccorsi Board of ANVUR In collaboration with Marco Malgarini Director, Research Assessment

Outline 1.The Italian National Agency for the Evaluation of Universities and Research Institutes 2.The Research Evaluation for the period 2004-2010 (VQR 2004-2010) 3.The National Scientific Habilitation Program 4.The ANVUR bi-annual Report on the Italian Research system 5.Classification of journals in Humanities and Social Sciences 2

The Italian National Agency for the Evaluation of Universities and Research Institutes According to the legislation established in 2010, the Italian National Agency for the Evaluation of Universities and Research Institutes (ANVUR) is in charge of: External evaluation of the quality of activities of Universities and Public Research Organisations (PROs) (a) accreditation and quality assurance of academic courses according to European Guidelines (ENQA) (b) research assessment Giving general advices for the activities of internal evaluation bodies Evaluate public funding programs aimed at enhancing research and innovation activities 3

The Italian National Agency for the Evaluation of Universities and Research Institutes More specifically, since 2011 ANVUR has been involved in several projects of research assessment or which have used the results of research assessment The quality of research evaluation exercise for the period 2004-2010 (VQR 2004- 2010, or Valutazione della Qualit della Ricerca), results of which have been presented on July 16, 2013 The National Habilitation program, which is currently completed (1st wave) The bi-annual Report on the Italian Research system, published in March 2014 The accreditation of doctoral courses, due to be published in June 2014. 4

The Evaluation of Research 20042010 (VQR 2004-2010) The Decree regulating the second Italian research quality evaluation exercise has been issued in 2011. According to the Decree, all the Italian Assistant, Associate and Full Professors participate to the evaluation exercise, together with Researcher and Technicians of PROs directly funded by the Ministry of University and Education (MIUR) Evaluation refers to Areas of research, Universities and Departments it does not concern individual researchers Evaluation has already affected the distribution of public funding to the Italian research system (2013). 5

The Evaluation of Research 20042010 (VQR 2004-2010) Each University has chosen the three best research products (six products for PROs) published in the 2004-2010 period by each researcher Research products considered for evaluation are: Articles Books, books chapters and proceedings (with ISBN code) Critical editions, translations, scientific comments Patents Design, Performance, Dataset, Software and other research products Research evaluation is classified according to official organisation of Scientific Areas of research established by the National University Council (CUN) 6

Italian Scientific Areas of Research More specifically, Italian academia is officially divided up in 14 Scientific Areas (Aree CUN) The 14 Areas may be distinguished into: Areas for which is common to evaluate research quality on the basis of bibliometric indicators regarding the impact of the Journal on which the article is published and the number of citations of the article (so called Bibliometric sectors) Mathematics (Area 1) Physics (Area 2) Chemistry (Area 3) Earth Sciences (Area 4) Biology (Area 5) Medicine (Area 6) Agricultural and Veterinarian Sciences (Area 7) Part of Architecture and Civic Engineering (Area 8) Industrial and communication Engineering (Area 9) Humanities and Social Sciences Among Social Sciences, bibliometric indicators are also used in Economics and Statistics (Area 13) and in a small part of Psychology (Area 11) 7

Institutions and experts involved Institutions involved in the VQR 2004-2010 include: 67 + 28 public and private Universities 12 PROs under the control of the Ministry of Education and University 27 Research Institutes, Consortia and Foundations which have voluntary submitted an application to be evaluated by ANVUR Evaluation has been performed by: 450 national and international Scientific experts from all the 14 Areas of Research Almost 14,500 national and international reviewers 8

Research products submitted for evaluation A total of 184,742 research products have been submitted for evaluation Among them we find: 73,5% of review articles 19,9% of books, books chapters 5,8% proceedings, translations, scientific comments 0,9% miscellaneous other products Research products may be classified according to the area to which they are submitted for evaluation and to the area of the subject submitting them for evaluation 9

Evaluation Methods and Criteria Evaluation is based on a combination of bibliometric and peer review methods In any case, overall at least 50%+1 of the products of research submitted for evaluation is subject to peer review (by legal obligation in the decree creating ANVUR) In the bibliometric sectors defined above, there is an intense use of bibliometric methods Bibliometric evaluation concerns research products indexed either in Scopus or Web of Sciences databases 10

The bibliometric method of evaluation According to the VQR definition, a research product may be defined as follows: Excellent, when it falls in the top 20% in terms of quality according to the international scientific community in the field Good, when it falls in the 60-80% segment in terms of quality according to the international scientific community in the field Adequate, when it falls in the top 50-60% segment in terms of quality according to the international scientific community in the field Limited, when it falls in the bottom 50% in terms of quality according to the international scientific community in the field According to the aforementioned criterium: A product of research randomly extracted in an homogeneous set of all the products published in the world in a given sector and a given year has a probability of being classified as Excellent, Good, Fair or Limited respectively equal to 0,2; 0,2; 0,1; 0,5 11

The calculation of bibliometric indicators Threshold (# citations) for each evaluation class for some subject category according to the ISI database, year 2009 According to this methodology, the classification of research products is performed in two steps: 1.Definition of the appropriate bibliometric indicators to be used 2.Definition of an appropriate algorithm assigning the value of the chosen indicator(s) to one of the defined class of merit (E; G; A; L) 12

The calculation of bibliometric indicators The indicators chosen for the VQR 2004-2010 are the following: 1.An indicator of the impact of the journal on which the research product has been published, such as Scimago Journal Rank or the Journal Impact Factor. Such an indicator provides information on the quality of the publication linked to: a)Referee procedure b)Reject rate c)Diffusion of the journal d)Reputation of the Journal 1.The number of citations received by the research product 13

The bibliometric algorithm - 1 For each article, we first identify the Journal, the year of publication and the scientific field We then calculate the empirical cumulative distribution of the SJR for all the Journals in a given scientific field in a given year The distribution is divided up in four areas, with probability of inclusion respectively equal to 0,2; 0,2; 0,1 and 0,5 14

The bibliometric algorithm - 2 We then calculate the empirical cumulative distribution of the number of citations for all the articles in a given scientific field in a given year Also this distribution is divided up in four areas, with probability of inclusion respectively equal to 0,2; 0,2; 0,1 and 0,5 15

The bibliometric algorithm - 3 As a result, we have two different classifications respectively based on the quality of the Journal (as measured by the SJR) and the number of citations of the paper Bibliometry (SJR) 2 3 4 1 1 A Starting from these information, we build up the 4x4 matrix showed here 2 When the two indicators give equal indications, the class is settled B When the two indicators give diverging signals, we may choose to give an higher weight either to the impact or to the citational indicator 3 C # Citations 4 D 16

The bibliometric algorithm - 4 If classification according to the two criteria diverges, the final class may depend: On choosing whether to give more emphasis to one of the two criteria On the value of the difference among the two different classifications As an example, the matrix on the right present a possible classification for older articles, in which more emphasis is given to the number of citations criteria The U cell will be attributed with informed peer review The choice about the allocation of undetermined cells is left to the responsibility of each group of experts for each scientific disipline 17

Impact of VQR Funding of universities Share of block granting (Fondo di Finanziamento Ordinario, FFO) allocated according to the relative merits of universities as stated by VQR Decree August 2013- share will increase steadily in following years However, current maximum change +/- 5% with respect to previous year Decree 2014 under approval Internal allocation of resources of universities Use of VQR indicators by department and/or scientific areas used to allocate resources (e.g. PhD grants, post-doc grants, research funds) Use for allocation of new positions more sensitive Accreditation of doctoral courses Reform of doctoral courses aimed at strenghtening the international orientation and rationalizing the offer Accreditation criteria include the scientific quality of the faculty ANVUR is responsible for accreditation- used VQR data to evaluate the quality of members of the faculty 18

Critical issues in Humanities and Social Sciences Peer review Need to test extensively the Evaluation form before submitting to referees Qualitative comments should be mandatory and communicated to the evaluated subjects List of referees made public Meaning of internationalization Not all disciplines share the same meaning of internationalization Avoid an interpretation by referees as a negative score on products in Italian language and/or in Italian journals/ books Yet there is no evidence of a systematic bias Classification of journals in the area of Economics and Business Journals classified according to the percentiles of the world distribution of IF in four broad areas (Economics, Accounting and management, History, Mathematics and Statistics) No Italian journals in class A; only 20 in class C Yet the low score is mainly due to books and book chapters 19

The National Habilitation Law n. 240/2010 has innovated the recruitment procedures for Italian Academia Candidates aspiring to become Professor in Italy are now selected through a two stage procedure: In a first stage, they apply to be granted a National Habilitation to become either Associate or Full Professor in their scientific field Only those being granted the National Habilitation may participate to local selection procedures promoted by Universities In 2012, Ministerial Decree n. 76 has established criteria and parameters for the National Habilitation In July 2012, the Italian Ministry of Education has started the selection procedure for the National Habilitation Program 20

The National Habilitation Program According to the Ministerial Decrees, ANVUR: Defines parameters and criteria to be used for the selection of both Full Professors participating to the Commissions and of candidates in the National Habilitation Program Selects the Italian and Foreign candidates to be member of the sector-specific Commissions in charge of granting the National Habilitation 21

The National Habilitation Program: parameters and indicators In the bibliometric areas the indicators are: Number of articles published either in the Scopus or ISI-Web of Science databases in the last 10 years Number of citations received over the entire academic career of the candidate from all indexed sources H-index In the non-bibliometric sectors the indicators are: Number of books Number of book chapters and journal articles Number of articles in A-rated journals All the above indicators are normalised for the academic age of the candidate. The academic age is defined as the number of years since the first publication included in the submission Normalization of h-index is fone using the contemporary h-index formula 22

The National Habilitation Program: parameters and indicators For each indicator, the Median values of the distributions for the two Populations of Italian Associate and Full Professors are calculated According to the Decree, to be part of the Commissions granting the National Habilitation an Italian or Foreign Full Professor should fall in the top 50% of the distribution in two out of three indicators, calculated for the Population of Italian Full Professors (only one for Humanities and Social Sciences) The same criterium applies for the selection of candidates to be appointed as Associate or Full Professors, respectively However, according to the Decree, Judging Commissions are allowed to grant Habilitation also to candidates not fully complying to these requirements, provided that the decision is adequately motivated ex ante (= before opening the list of candidatures) 23

The National Habilitation Program: the information set The information set on which the indicators are calculated is obtained from the combination of: The Archive of all the Italian Associate and Full Professors. Data are drawn from access to personal websites of professors, held by the Minister of University and Research and managed by a university consortium of IT infrastructure and services (Cineca) Bibliometric information obtained from the Scopus and ISI-Web of Science databases Distributions are calculate separately for Associate and Full Professors, with reference to the each specific Italian Scientific Sector (SSD, i.e. Sector Specific Scientific Sector) 24

The National Habilitation Program: calculation of indicators On the basis of the archive of Italian Professors, on July 15, 2012, the number of publications in bibliometric sectors was equal to 1,316,000 research products The procedure of calculation of the aforementioned indicators can be divided in three major steps: 1.Matching of authors listed in the Italian archive with the Scopus database, on the basis of the surname, initial of the name and affiliation. A specific algorithm was used in order to: a)Minimize errors due to mistakes in the Italian archive b)Minimize errors due to mistakes in the Scopus database (in particular, with reference to the author surname) c)Minimize errors due to ambiguity in surnames (especially for composite surnames) d)Maximize identification of authors not recognised in the Scopus database 2.Extracting bibliometric information from the matched dataset 25

The National Habilitation Program: calculation of indicators 3. Calculating the three bibliometric indicators: a)On the period 2002-2012, given that it has been impossible to correctly distinguish the papers published after July 2002, as it was originally requested from the Decree b)For each Scientific Sector, separately for Associate and Full Professors c)For the citation indicators, it has been chosen the max number of citations available for each article on the basis of the two available datasets At the end of the procedure, the percentage of articles contained in the Italian archive correctly identified in the bibliometric datasets was equal to 96% Correct identification is lower for conference proceedings (32%, with higher levels where proceedings are particularly relevant, such as mathematics 52% - and engineering 45%) 26

The National Habilitation Program: selection of the Commissions ANVUR has also been responsible for ascertain the scientific qualification of Italian and Foreign Full Professors aiming to become part of the Commissions granting the National Habilitation ANVUR activities in this field may be distinguished according to the affiliation (National or International) of the candidates For National candidates ANVUR took care of: 1.Calculation of individual indicators on the basis of the information contained in the dataset on July 21st, 2012 2.Communication to the candidate of a first assessment ( green or red light) on the basis of this information, leaving the possibility to withdraw the candidature before August 28, 2012 3.New calculation of the indicators on the basis of the information available on August 28, 2012 4.In case of a negative assessment, candidates have been notified of the decision and informed about the calculations. Candidates had the possibility of appealing to the decision and ANVUR has re-calculated the indicators for those providing new evidence about their publication activities 27

The National Habilitation Program: selection of the Commissions For Foreign candidates ANVUR activities were the following: Creation of a list of candidates for the position, following the publication of an invitation to apply on Nature Manual calculation of the bibliometric indicators: only those with a value of two out of three indicators above the median of the sector have been admitted to a random draw selecting the Foreign Member of the Commission 28

Classification of journals In Humanities and Social sciences the indicators required the identification of: - scientific journals (as opposed to popular, professional, cultural or other journals) - A-rated journals Classification of journals has been done in several waves: - startup (based on lists of journals on the personal website of professors and researchers)- appr. 20,000 titles, of which 15,000 considered scientific (titles were evaluated separately in each of scientific disciplines in which the authors were active, summing up > 42,000 occurrences) (July- September 2012) - new journals included by candidates to the Habilitation in their publication list- appr. 9,000, of which 3,000 considered scientific (January-March 2013) - revision of rating- a procedure aimed at allowing journals to apply for upgrade- 550 applications, for 2,200 positions overall. Evaluation largely based on peer review (procedure close in June 2013) - new journals included by candidates to the Habilitation in their publication list- appr. 3,000 new items (January-March 2014) 29

A case study. Number of journals registered in personal archives of researchers and professors Area 10 (Humanities and Arts) July 15, 2012 Number of journals (with repetition) 10.989 Number of journals (without repetition) 5.000 Number of articles registered in personal archives 42.157

Distribution of journals by number of articles Number of journals Number of articles More than 100 articles 16 3.109 10-99 articles 909 19.059 Less than 10 articles 10.064 19.989 Total 10.989 42.157

Distribuzione delle riviste per numero di articoli per SSD 600 500 400 300 Numero di articoli 200 100 0 226 5851 7876 451 676 901 1126 1351 1576 1801 2026 2251 2476 2701 2926 3151 3376 3601 3826 4051 4276 4501 4726 4951 5176 5401 5626 6076 6301 6526 6751 6976 7201 7426 7651 8101 8326 8551 8776 9001 9226 9451 9676 9901 1 10126 10351 10576 10801

Average number of articles per journal Disciplinary field Number of journals Number of articles 10/A1 10/B1 10/C1 10/D1 10/D2 10/D3 10/D4 10/E1 10/F1 10/F2 10/F3 10/G1 10/H1 10/I1 10/L1 10/M1 10/M2 10/N1 10/N3 370 178 192 168 175 145 176 217 324 147 222 309 154 162 206 124 61 184 70 1943 824 1005 914 1063 831 936 875 1279 592 880 1279 676 569 760 371 205 788 214 5,25 4,63 5,23 5,44 6,07 5,73 5,32 4,03 3,95 4,03 3,96 4,14 4,39 3,51 3,69 2,99 3,36 4,28 3,06 Totale 3584 16004 4,47

Classification of journals Procedure Creation of an Expert Group - Areas covered: Architecture (4 experts), Literature and Art (8), Philosophy and History (4), Law (4), Economics and Business (4), Social sciences (4) - 2-year appointment - Part time appointment (only reimbursement of expenses) Triangulation approach - Collect data on institutional repositories of Italian academics on journal lists (June 2012) - Ask opinion of scientific societies (July 2012) - Compare opinion of scientific societies with referees (July-August 2012) - Suggest consensu opinion by experts (September 2012) 34

Classification of journals Revision of rating Opening of a procedure for the revision of rating Annual procedure Platform open to editors of journals (April-June 2013) Questionnaire to be filled Expert group in charge of revision- more intense use of anonymous referees Text of referee report made available to editors Second revision (2014) Complete renewal of the Expert group Better coverage of disciplines Revision platform available since September 24, 2014 35

Classification of journals Robustness of the classification as scientific journal Criteria for scientific nature (not defined in legislation) - Existence of a scientific committee - Editor (academic vs non academic) - Existence of an international scientific committee - Audience (academic, professional, popular or general public) - Formato of articles (lenght, references, footnotes) Peer review generally considered a sufficient condition for scientific nature Questionnaire addressed to editors in the revision procedure (April- June 2013) Issues - Errors in initial classification (due to time pressure) in September 2012- several clearly non- scientific journals (e.g, newspapers or popular science journals) wrongly included in the list Media campaign (up to front page covering by Corriere della Sera) Ex-officio correction of errors (publication in March 2013) Few complaints about cancelation from the list of scientific journals (to date, one journal in business, one in public law) No legal complaints about the scientific nature of journals No media attention or complaints after corrections of errors - - - - - 36

Classification of journals Robustness of the A-classification Criteria for A-rating (loosely defined in legislation) - Scientific nature - Peer review - Impact on the scientific community - Reputation Questionnaire addressed to editors in the revision procedure (April- June 2013) Operationalization - Nature of the peer review procedure (double-blind vs single-blind; referees only within the scientific committee or in the larger scientific community; communication to authors; publication of the list of referees; rate of acceptance of manuscripts) - Diffusion in libraries (national and international) - Reference by scientific associations - Questionnaire to referees (typically 3 per journal) 37

Classification of journals Controversies about the A-classification Legal controversies - Some complaints to the Administrative Tribunal (TAR) tried to cast doubt on the overall procedure, on grounds of excess power, irrationality, violation of principles of equity and trasparence in administrative action - The Tribunal rejected the opposition (March 2014) - Other complaints addressed, on the contrary, specific evaluations, claiming there was lack of justification. These complaints mostly come from journals in Law and History (mostly Contemporary History) - In some cases the Tribunal forced ANVUR to repeatthe evaluation. ANVUR nominated a different group of referees (typically, 3 professors different from those who proposed the initial classification). In most cases the classification was confirmed, in a few cases it was changed Political controversies - Some scientific societies (mainly in part of the Humanities) have argued against the overall journal classification system - After the revision procedure (2013) intense correspondence with editors of non-A journals 38

Special projects for Humanities and Social sciences Humanities and Social Sciences do not have bibliometric indicators. Special projects at ANVUR: - support HSS Italian journals wishing to apply for admission into WoS or Scopus - create a Database of Italian journals aimed at extracting citational information Technical feasibility for a Database: - IT platform - delivery of full text - current issues + backlog (10 years) - automatic extraction of citations from the text of articles, via sophisticated computational linguistics system of rules - indicators: (a) production (b) use (c) citations - Altmetrics under experimentation - test to be carried out on A-rated journals Workshop January 20, 2014 Consultation with scientific societies and editors of A-rated journals 39

Research in Humanities and Social sciences June 2014 Creation of High Level Research Groups with a mission of debating hot issues listening to opinions (including scientific societies) examining the international state of the art producing recommendations Set of research questions to be addressed Start June 2014 Areas: - Humanities - Law - Management (to be decided) Research Call - Research projects to be carried out by individual researchers or teams - First call (September 2014) on evaluation in HSS - 12 projecte received- selection by end of October 2014 - Future call on evaluation of teaching + advancements in bibliometrics 40

The ANVUR Report on the Italian Research System ANVUR delivered in March 2014 its first Report on the Italian Research System The Report contains detailed analysis of the situation of the Italian Research system, in terms of demand and supply for higher education, academic staff, labor market outcomes and other indicators The Report also contains a chapter devoted to the analysis of the position of the Italian research system in the international scenario The benchmarking analysis is based on standard bibliometric information stemming from the major international databases in the field In this respect, choices have been made regarding: The level of aggregation of the analysis The countries involved in the comparison The indicators used in the analysis The period covered in the report 41

The ANVUR Report on the Italian Research System: level of aggregation The report analyses the position of Italian research at 3 different levels of aggregation: At the level of the Scopus Subject Areas At the level of the Italian Aree CUN For all the scientific sectors considered together In order to be able to gather relevant information at the level of the Italian Aree CUN , ANVUR has realised a mapping of the Scopus subject areas into the Italian classification The mapping has been based on information provided within the VQR 2004-2010: We exclude from the calculation the subject areas in which less than 10 products have been submitted for evaluation We calculate the number of publications presented in each Scopus subject area from each Italian Area CUN We assign each Scopus subject area to the Area CUN with the highest share in that Scopus s.a. In case the highest share is below the 75% threshold, we considered the possibility of multiple assignment if the second highest share is equal or superior to 20% (N.B: a subject area is assigned at most to 2 Aree CUN) 42

The ANVUR Report on the Italian Research System: countries involved in the comparison The analysis referred to the following countries and aggregation of countries: European countries and aggregates Eu 15 aggregate oItaly oFrance oGermany oNetherlands oSpain oUnited Kingdom oSweden European Union (EU 27) aggregate Other European: Switzerland 43

The ANVUR Report on the Italian Research System: countries involved in the comparison The analysis referred to the following countries and aggregation of countries: OECD OECD aggregate oAustralia oCanada oJapan oSouth Korea oUS Non-OECD BRIC aggregate oBrasil oChina oIndia oRussia World total 44

The ANVUR Report on the Italian Research System: indicators The analysis will be based on the following indicators: Indicators measuring research productivity Publications (whole) Publications (fractional) Indicators measuring the impact of published research Citations (whole) Citations (fractional) Citations per paper Field weighted citation impact Indicators measuring the level of national and international collaboration: Collaborations single authorship Collaborations domestic collaborations Collaborations international collaborations Collaborations cross sector collaborations Indicators measuring the quality of publication activities in terms of excellence: The full percentile distribution of the number of publications with respect to the most cited publications. All the information will be available for the period 1996-2012 45

The ANVUR Report on the Italian Research System: indicators The Report addresses the following questions: What is the position of the Italian research system vis- -vis its main competitors in terms of: Scientific production (number of products) Scientific impact (number of citations) How Italy compares with its main competitors in terms of publications in top rated journals? How Italy compares in terms of collaborating activity? What has been the evolution of relative scientific specialisation in Italy in the last 20 years 46

The ANVUR Report on the Italian Research System: indicators In order to answer those questions, bibliometric indicators will be combined with information stemming from official sources in order to calculate appropriate productivity indicators More specifically, research production and impact will be assessed in terms of: Production and impact per capita, dividing by (active) population Production and impact per researcher, dividing by the number of (public and private) researchers Production and impact per euro invested in R&D activity, dividing by R&D (Private, Governement, Higher Education) expenditures Indicators of relative specialisation will also be calculated in order to investigate on the pattern of specialisation of the Italian research system The report has been published in March, 2014. 47