Publication Forum Steering Group & Panel Chairs Meeting Highlights

Publication Forum Steering Group and Panel Chairs meeting Online 16.12.2021 13:00-15:00

PROGRAMME 1. 2. 3. 4. 5. 6. 7. Opening of meeting Presentation round of participants Publication Forum background and goals Instructions for JUFO panels 2022-2025 Evaluation processes and timeline 2022-2025 Discussion Ending the meeting

2. Presentation of participants

STEERING GROUP MEMBERS 2020-2023 Chair Ulla-Maija Forsberg (University of Helsinki / Federation of Finnish Learned Societies) Vice president Taina Pihlajaniemi (University of Oulu / Universities Finland UNIFI) Professor Paula Eerola (University of Helsinki / Natural Sciences) Professor Heikki Hiilamo (University of Helsinki / Social Sciences) Director of Services Kristiina Hormia-Poutanen (The National Library of Finland) Professor Helmi J rviluoma (University of Eastern Finland, Humanities) Counsellor of Education Matti Kajaste (Ministry of Education and Culture) Professor Satu Lahti (University of Turku / Medical and Health Sciences) Library Director Riitta L hdem ki (Council for Finnish University Libraries) Vice President Riitta Maijala (Academy of Finland) Emeritus Professor Risto Nieminen (Aalto University / Engineering and Technology) Development Manager Hanna-Mari Puuska (CSC IT Centre for Science)ration of Finnish Learned Societies Secretariat: Secretary General Janne P l nen (Federation of Finnish Learned Societies) Senior Planning Officer Elina Pylv n inen (Federation of Finnish Learned Societies) Executive Director Lea Ryyn nen-Karjalainen (Federation of Finnish Learned Societies) Development manager Eeva Nyr vaara (Finlands universitet UNIFI ry) 4

23 EXPERT PANEL CHAIRS 2022-2025 1. Kaisa Miettinen (JYU): Mathematics and statistics 2. Samuli Pekkola (TAU): Computer and information sciences 3. Petriina Paturi (UTU): Physical sciences, space science and astronomy 4. Kari Rissanen (JYU): Chemical sciences 5. Matti Kummu (Aalto): Geosciences and environmental sciences 6. Tanja Pyh j rvi (HU): Biosciences I 7. Lea Sistonen ( A): Biosciences II 8. Minna Lanz (TAU): Civil and mechanical engineering 9. Heikki Handroos (LUT): Electrical and electronic engineering, information engineering 10.Satu Ojala (UO): Chemical engineering, materials engineering and environmental engineering 11.Anna-Liisa Levonen (UEF): Medical engineering, biotechnology and basic medicine 12.Risto Kerkel (UO): Clinical medicine I 13.Jouko Miettunen (UO): Clinical medicine II and dentistry 14.Ursula Schwab (UEF): Health sciences and other medical sciences 15.Jarmo Valaja (HU): Agricultural sciences 16.Kaisa Kotakorpi (TAU): Economics and business 17.Kathrin Komp-Leukkunen (HU): Social sciences, media and communications, interdisciplinary social sciences 18.Auli Toom (HU): Psychology and educational sciences 19.Jukka Snell (UTU): Political science, public administration and law 20.Matti H yry (Aalto): Philosophy and theology 21.Anne M ntynen (HU): Languages 22.Sofia Pantouvaki (Aalto): Literature, arts and architecture 23.Pasi Ihalainen (JYU): History, archaeology and cultural studies 5

3. Publication Forum background and goals

BACKGROUND Publication count has been used as one funding criteria of Universities since 2007, however there was insufficient attention to quality. Finnish Council of University Rectors (UNIFI) constituted a working group to develop quality assessment, considering different options: Citation analysis is well-established in natural sciences and medicine but sources cover poorly engineering, social sciences and humanities. Norway (2005) and Denmark (2009) developed a quality index of publications channels that cover the international and national output of universities in all languages and formats: journal articles, proceedigns, book chapters and monographs. According to the UNIFI working-group, the main benefits of a JUFO classification is that it allows the Finnish scientific community to determine what constitutes good research and the criteria in place to assess it. 7

AIMS & PURPOSE In 2010, the Finnish research community represented by the Federation of Finnish Learned Socities (TSV) was entrusted with the task of developing the Publication Forum classification with funding from the Ministry of Education and Culture. The main task of JUFO classification is to identify peer-reviewed journals/series and book publishers (level 1), and to indicate those that have the strongest quality and impact and the widest respect in the research community (levels 2 and 3). The development of the quality assessment of publications based on JUFO serves several interests: 1. Provide quality indicator for the output of universities in the funding-model used by the Ministry of Education and Culture. 2. Produce transparent and comparable data on publication activities of research organisations and fields. 3. Encourage researchers to publish their results in reliable and leading international and national outlets of their field. 8

MACRO-LEVEL FUNDING-MODEL Since 2015, public funding of universities from the Ministry of Education and Culture is partly based on publications: 13% in 2015-2020 and 14% since 2021. Funding-model focuses on the more than 25,000 peer-reviewed publications produced annually. Publications are weighted based on the publication type and JUFO-level of the publication channel. Extra-weight of 20 % was introduced in 2021 for openly available publications. Publication type Level 0 Level 1 Level 2 Level 3 A1 Journal article (refereed), original research 0,1 1 3 4 A2 Review article, Literature review, Systematic review 0,1 1 3 4 A3 Book section, Chapters in research books A4 Conference proceedings C1 Book 0,1 1 3 4 0,1 0,4 1 4 3 12 4 16 C2 Edited book, conference proceedings or special issue of a journal 0,1 1 3 4 9

COMPLETE COVERAGE OF OUTPUTS Web of Science and Scopus databases provide incomplete coverage of Finnish universities peer-reviewed output, especially in engineering, social sciences and humanities. Complete coverage is important in a funding-model, which is based on both quantity and quality of outputs. Web of Science indexed journals Scopus indexed journals 97% 96% 96% 100% 94% 93% 92% 91% 90% 90% 87% Share of peer-reviewed output 90% 84% 84% 80% 70% 70% 65% 63% 56% 60% 48% 48% 50% 45% 40% 33% 29% 28% 27% 30% 23% 20% 10% 0% 10

EQUALITY BETWEEN FIELDS On average, peer-reviewed publication outputs from the various main fields yield a fairly similar amount of JUFO-points per output, and the differences have become smaller. 2.5 2.2 2.1 2.1 2.0 Average points per publication 1.9 2.0 1.8 1.8 1.8 1.8 1.8 1.8 1.8 1.8 1.7 1.7 1.7 1 Natural Sciences 2 Engineering 3 Medicine 4 Agriculture 5 Social Sciences 6 Humanities All fields 1.7 1.5 1.5 1.5 1.5 1.3 1.0 0.5 0.0 JUFO 2012-14 levels JUFO 2015-18 levels JUFO 2019-22 levels 11

SCIENTIFIC IMPACT Finnish Web of Science indexed output from 2011-2015 in level 3 journals has on the whole stronger citation impact than output in level 2 journals, or in level 1 journals. MNSC and Top10% indexes of the Finnish output in level 3 journals are around 3 times higher (2.8 and 3.1) than world average (=1). MNSC = Mean Normalized Citation Score Top10% = Share of articles among the most highly cited 10% of articles Cited = Articles with at least one citation 12

DEVELOPMENT OF FINNISH UNIVERSITIES PUBLISHING IN JUFO LEVELS INCLUDING ALL FIELDS Number and share of peer-reviewed publications (articles, monographs and edited volumes) has increased in levels 1-3 and decreased in level 0. 2011 2012 2013 2014 2015 2016 2017 2018 2019 20000 57 % 18000 Peer-reviewed publications 16000 14000 51 % 12000 10000 27 % 8000 21 % 21 % 6000 11 % 4000 5 % 7 % 2000 0 13 Level 0 Level 1 Level 2 Level 3

DEVELOPMENT OF FINNISH UNIVERSITIES PUBLISHING IN DOAJ JOURNALS AND PREDATORY JOURNALS (CABELLS) Publishing in DOAJ Gold OA journals is increasing, mostly in APC-based journals. Publishing in predatory OA journals is very marginal, possible concern is the grey-zone. 25% 0.2 % 5.5 % Cabells predatory reports 20% MDPI journals 1.7 % 0.7 % 1.7 % Frontiers journals 15% 1.8 % PLoS journals 10% BMC journals 8.3 % Springer-Nature journals 5% Other APC journals 5.0 % Diamond OA journals 0% 2012 2013 2014 2015 2016 2017 2018 2019 2020 14

CHALLENGES Publication Forum rating should reflect the research community s appreciation of quality and impact of publication channels. Uniform distinction between levels 0 and 1. Important to maintain level 1 criteria despite the advantage of adding journals to level 1. Reliable identification of leading outlets to levels 2 and 3. Important to focus on scientific quality and impact of channels, regardless of the current publication profile of the Finnish researchers. Avoid consideration of strategic interests of specific field, organization, group or individual. Fair representation of different fields in levels 2 and 3. Important to pay special attention to interdisciplinary areas and channels, and subfields and smaller specialties. Prepare for the critique strong grounds for ratings. 15

4. Instructions for JUFO panels 2022-2025

LEVEL 1 CRITERIA 1. Transparency: the publication channel's website has a transparent description of the editorial board and the peer review process. (A book publisher may meet Level 1 criteria even if the editorial board and peer review process are not described on the website). 2. Scientific focus: the publication channel is specialised in the publication of scientific or scholarly research outcomes and it publishes peer-reviewed scientific publications on a regular basis. 3. Editorial board: the publication channel s editorial board constitutes of experts, who mainly include researchers working in universities or research institutes. 4. Peer-review: the entire manuscripts of scientific or scholarly articles or books are subject to peer review, which is carried out blindly or openly by external scientific experts invited by the editors. (A book publisher may meet the Level 1 criteria if it has a credible quality assessment by the book s editors.) 5. Scope: the publication channel is used by a national or international scientific community, with over half (1/2) of the editorial board or authors coming from a different research organisation than the publisher organisation. 6. Credibility: the publication channel is scientifically relevant in its field for the international or Finnish scientific community, and its procedure for ensuring scientific quality is credible. 17

LEVEL 0 New feature coming to JUFO portal: grounds for level 0. Panel members can choose grounds for level 0 from the list and panel chairs will choose which of them will be publicly visible. List of grounds: Publication channel is professional or general. The scientific publishing activity of the publication channel is irregular. Information about editorial board is insufficient. Editorial board does not meet the criteria. Information about peer review practices is insufficient. Peer review practices do not meet the criteria. Publication channel is local. Scientific level of the publication channel is questionable. Publication channel is too new for evaluation. Publication channel does not have a working website. Publication channel can not be evaluated because of the publishing language. 18

NEW PUBLICATION CHANNELS AND LANGUAGES New publication channels: Even new publication channels can be granted level 1 if they fulfill the criteria. If publication activities are less than one year old, or if there are not enough issues or publications in the channels that allow content to be evaluated, the panels can rate it level 0 in the JUFO portal and choose the following grounds: Publication channel is too new for evaluation . The secretariat will bring publication channels with grounds publication channel is too new for evaluation regularly to the panels for re-evaluation. Publication channels in foreign languages: Publication channels using any language can be granted level 1 if they fulfill the criteria. However, most panels find it challenging to evaluate publication channels in other languages than Finnish, Swedish or English. If the panel is not able to evaluate the publication channel because of its language, it can rate it level 0 in the JUFO portal and choose the following grounds: Publication channels can not be evaluated because of the publishing language . 19

OTHER CONSIDERATIONS REGARDING LEVEL 1 Limited authorship: Some research funders have limited authorship on their publication platforms only to those who have received the funding. These can be granted level 1 if they fulfill the criteria. Publisher: Journals/series are evaluated individually from the publisher. Publication channels from any publisher can be granted level 1 if they fulfill the criteria. OA publishing model: publication channels are evaluated according to the same criteria regardless of the publishing or funding model. Publication channels operating with APC (article processing charges) can be granted level 1 if they fulfill the criteria. 20

PREDATORY JOURNALS AND THE GREY ZONE The Secretariat will warn the panellists of the journals on the Cabell s list and those removed from the DOAJ, as well as other suspicious publication channels that meet the characteristics of predatory journals, or the publication channel publishes a particularly large number of special issues and/or its publication volume is growing rapidly. The Secretariat will also bring to the panels for re-evaluation cases where the JUFO level is 1-3 but the publication channel is on the Cabell s list or level 0 or X in Norway. If the panel determines that it is a predatory journal or publisher, the panel may rate it level 0 in the JUFO portal and choose the following grounds: Scientific level of the publication channel is questionable . 21

LEVEL 2 & 3 CRITERIA INTERNATIONAL JOURNALS/SERIES In the evaluation of publication channels for level 2 and 3, the priority is scientific quality, impact and appreciation, regardless of whether the publication channel is used a lot or little by Finnish researchers. Level 2 Level 3 Level 2 can be awarded to leading scientific publication channels of the various disciplines that meet the following criteria: Has a wide reach and high respect among international experts in the field. Researchers from different countries seek to publish their best results. Editors, authors and readers represent various nationalities. Panels must choose, within their own Level 2 quota, the publication channels attracting and selecting the highest quality publications. Finnish channels aimed at international audiences are evaluated in relation to other international publication channels. Level 3 is a sub-category of the Level 2, and they need to fulfill the following additional criteria: The research published in them represents the highest level in the discipline and has very high impact (e.g. as measured through citation indicators). The series cover the discipline comprehensively, not limited to the discussion of narrow special themes. Both the authors and readers are international and the editorial boards consist of the leading researchers in the field. Publication in these journals and series is highly appreciated among the international research community of the field. The panels will prepare a proposal for publication series to be rated as Level 3, and the chairs of the panels make consensus decisions to include publication series in the Level 3 category. 22

LEVEL 2 & 3 CRITERIA NATIONAL LANGUAGE JOURNALS/SERIES In the humanities and social sciences (Panels 14 and 16-23), Level 2 can also include leading Finnish or Swedish-language publication channels meeting all the following criteria: The quality assessment of the scientific writings must be in line with the best practices. The publication series must be among those that cover the research in the respective discipline and the book publishers in their main discipline most comprehensively and be used by the entire national scientific community in that particular discipline. The context of the research problems is strongly focused on Finnish society or Finnish or Swedish-speaking culture. Publishing in these channels is regarded to be equal merit to publishing in foreign Level 2 channels. The SSH panel chairs, acting in corpore, are to make a consensus decision on the classification of Finnish and Swedish-language publication channels in Level 2 (the level 2 journals consume the level 2 volume quota of the panels concerned). The SSH panel chairs will be instructed based on analysis of national publication volumes to adjust the classification of Finnish- and Swedish-language journals/series so that in the SSH fields the proportion of Finnish- and Swedish-language articles and monographs in level 2 or 3 is not significantly higher or lower than the proportion of foreign-language publications. 23

LEVEL 2 & 3 CRITERIA ADDITIONAL CONSIDERATIONS Open access: If candidates for Level 2 or Level 3 in the same field have equal impact or prestige, an open access journal or one permitting immediate self-archiving of the peer- reviewed version of the manuscript is chosen for higher level, over the channel that does not support immediate open access. For book publishers and book series, an embargo period may be allowed: 6 months in science and technology, and 12 months in human sciences. Limited authorship: In the re-evaluation of levels 2 and 3, the panels may take into account whether authorship is limited too much and use this as grounds for downgraded level. Review journals: The best review journals can be classified as Levels 2 and Level 3 as long as they do not fill the entire quota. In the natural and medical sciences review articles typically gain the most citations because of their nature, so the impact factors for journals that only publish reviews are higher than those publishing original research papers. 24

LEVEL 2 & 3 BOOK PUBLISHERS The book publisher list is common to all panels, a total of about 100 publishers can be classified as Level 2, and a total of about 10 publishers can be classified as level 3. The panels make suggestions for Level 2 and Level 3 book publishers, and the social sciences and the humanities panel chairs prepare a preliminary proposal to all panel chairs, who confirm the classification by consensus. The SSH chairs will be instructed based on analysis of national publication volumes to adjust the book publisher classification so that in the SSH fields the proportion of book articles and monographs in level 2 or 3 is not significantly higher or lower than the proportion of journal and conference articles. 25

TRANSPARENCY OF EVALUATIONS For Level 2 and Level 3, the panels need to provide, in general terms, the principles and criteria that form the basis for their classification. In addition, the aim is to record channel-specific justifications for decisions that significantly deviate from the impact factors or the Norwegian and Danish ratings of expert panels: The panels may grant level 2 or 3 without separate justifications if the publication channel meets one of the following criteria: SNIP >= 1.5 Norway level 2 Denmark level 2 or 3 If the publication channel does not meet any of the above criteria, the panels must justify classification to level 2 or 3. 26

PUBLICATION VOLUME FOR JOURNALS/SERIES Quotas for Level 2 and Level 3 are determined for journals/series in each panel on the basis of their total annual publication volume (3-year average number of all peer-reviewed articles rounded up to the nearest twenty-five). Publication volume information is updated only for the journals/series covered by the Scimago Journal Rank (SJR) service and for all other journals/series the volume is estimated to be a minimum (25). Any compensation for removal of conferences from the panels lists of journals/series in 2016 is removed. A journal/series consumes the level 2 or 3 quota only up to 2000 publication volume, even if its publishing volume exceeds 2000. In individual cases, if two or more panels agree on the level 2 or 3 rating of a journal/series with a large publication volume, they may propose to the steering-group to divide the publication volume. If necessary, a journal/series can be transferred from the list of an individual panel to panel 24 by decision of the Steering Group. This requires that at least 30 peer-reviewed Finnish publications have been published in this journal/series in the previous three years, there are publications from at least three main disciplines, and no panel accounts for more than 50% of the publications. 27

LEVEL QUOTAS VOL 100 25 25 100 125 25 25 50 100 425 1000 % 10.0 2.5 2.5 10.0 12.5 2.5 2.5 5.0 10.0 42.5 100 LEVEL 3 2 2 2 1 1 1 1 1 1 1 New quotas are applied in 2022 in re-evaluation of levels 2 and 3: Levels 2 and 3 may cover at most 25 % of the total publication volume of level 1-3 journals. Level 3 journals may cover at most 10 % and Level 2 journals may cover at most 15 % of the total publication volume. The quotas are calculated on the basis of the publishing volume, i.e. the annual number of publications published in the issues of the journal. PANEL S LIST Journal 1 Journal 2 Journal 3 Journal 4 Journal 5 Journal 6 Journal 7 Journal 8 Journal 9 Journals 10-25 Total 28

LEVEL QUOTAS FOR 2022 REVIEW OF LEVELS 2 & 3 A PRELIMINARY ESTIMATE (to be updated) Publication volume New level 2 & 3 quota 25% Old level 2 & 3 quota 20% 30% 28.2 % 25.9 % 23.9 % 25% 22.7 % 21.9 % 21.9 % 20% 15% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Publication volume New level 3 quota 10% Old level 3 quota 5% 10% 7.0 %6.1 % 6.7 % 5.6 % 5.5 % 5.5 % 5% 0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 In the beginning of 2022 review of levels 2 & 3, panels exceeding their level 2 & 3 quotas will be compensated so that the combined level 2 & 3 share of the publication volume does not exceed 22.5% (i.e. all panels have at least 2.5% for upgrades). 29

5. Evaluation processes and timeline 2022-2025

EVALUATION PROCESSES Levels 1 and 0: New channels are added continuously and they are brought to panels for evaluation approx. 4 times a year. Suggested new additions and upgrades from level 0 to level 1. New channels identified from the publication data universities report annually to VIRTA Publication Information Service. Levels 2 and 3: Classification is updated every 4 years. Corrections: Individual changes to the level 2 and 3 ratings can be made only in case panel finds an important mistake in the classification. Downgrades: Panels can make decisions about level downgrades between 1.1.-30.6. each year. The downgraded levels will be effective from the beginning of next year. Upgrades will be effective without delays. 31

MEETINGS 2022-2025 YEAR 2022 PANEL MEETINGS 1. meeting in January-February: Introduction to panel work (Zoom) 2. meeting in April-May: Preliminary classification (On site) 3. meeting in October-November: Final classification (On site?) 2 SSH panel chairs meetings: book publishers and national journals. 1 all panel chairs meeting: level 3 and panel 24 (including book publishers). Additional meetings for multidisciplinary areas, open for all panellists. 1. meeting in April-May 2. meeting in October-November 1. meeting in April-May 2. meeting in October-November 1. meeting in April-May 2. meeting in October-November 2023 2024 2025 32

REVIEW OF RATINGS 2022 Panels will prepare a new classification and re-evaluate levels 2 and 3 in 2022. Preliminary classification in spring 2022. Final classification in autumn 2022. New classification will be effective from the beginning of 2023. Research community is encouraged to make suggestions for level amendments in the JUFO portal. Suggestions require reasonings (consider journal X in relation to level 2 or 3 journals in the same research area, and other reasons than just the impact factor). Suggestions received before the end of March 2022 are used in preparation of a preliminary classification. 33

ETHICAL ASPECTS The evaluation work must be based on good scientific practice (honesty, general care and accuracy), including the declaration of engagements. Panel members are responsible for complying with the National Board of Research Ethics Guidelines for Responsible Conduct of Research, the principles of which are universally accepted by universities and research institutes. Panelists must declare their engagements each time a panel discusses evaluation of channels linked to them. This applies to publishing channels where the panelists have been editors or members of the Editorial Board. In these cases, the panelist will not be excluded from the decision-making, but the panel can draw attention to the engagements in the assessment situation. 34

FOCUS OF REVIEW OR LEVEL 2 & 3 Level 2 & 3 quotas have been increased to allow panels to improve fair treatment leading journals/series from various sub-fields and smaller specialties, as well as interdisciplinary areas and channels. Given the increased quotas, focus should be on improving the classification with upgrades to levels 2 and 3, rather than changing/adjusting the ratings with a lot of downgrades. In the evaluation of publication channels for level 2 and 3, the priority is scientific quality, impact and appreciation, regardless of whether the publication channel is used a lot or little by Finnish researchers. 35

6. Discussion 7. Ending of meeting

THANK YOU!