Data Science Ethics Discussion and Reviewing Research Papers

Ch. Eick Ch. Eick News November 5, 2024 The results of the Midterm exam were reasonable ; however, there were about 7-9 very low scores in the midterm exam; I view all performance in this exam who have scores below 24 points as poor There will be a discussion centering on ethics for data science lead by group K on November 7; there will be a similar discussing, focusing on a different subtopic, lead by group N on November 21. Task 4 is due on Monday, November 11 end of the day in MS Teams (no grace period) Task 5 is due on Saturday, November 16 in Kritik; however, some peer reviewing will take place in the window November 17-22. The second course exam is scheduled for Tuesday, November 26, 2:30p; this will be the last activity of this course. We taught the makeup class on Nov. 1 as the class on Tuesday, November 12 has been cancelled. Topic for the remainder of this lecture: Reading and Reviewing Data Mining Papers and Task5. 1

Ch. Eick Christoph F. Eick

Ch. Eick Ch. Eick Already covered in Sept. 2024! On Reviewing Reviewing has many roles in our society: To help people to make selections To determine which research is most valuable/worth publishing To determine which research gets funded and how to distribute research funds and other resources Reviewing is also important of your everyday life: To choose the best methods and products To choose what to allocate your resources on To choose entertainment options In order to be successful in your research and professional life you need to be a good reviewer! Challenge: Reviewing requires you to form an opinion about something, and students are usually not often asked about their opinion about something too often 3

Ch. Eick About Reading Scientific Papers

Ch. Eick Ch. Eick Questions to ask when reading papers 1. What is the field of research of the paper? If the paper is part of a well established field, you should describe the field and its current state. 2. What is the problem area with which the paper is concerned? 3. What problem does the paper solve? Is the problem the authors solve defined clearly in the paper? 4. What is the author's thesis? That is, what is he/she trying to convince you of? 5. Summarize the author's argument. That is, how does the author go about trying to convince you of her thesis? 6. Does the author describe other work in the field? If so, how does the research described in the paper differ from the other work? 7. Does the paper succeed? Are you convinced of the thesis by the time that you have finished reading the paper? 8. Does the author indicate how the work should be followed up on? Does the paper generate new ideas? 9. Some papers implicitly or explicitly provide a new way of doing things or of thinking about problems. If your paper does so, describe the approach. 10. Can the paper be trusted? Are there any indications that the authors are flagrantly lying? Does the paper contain unusual inconsistencies? 11. What other things did you learn when reading the paper? 12. What is the educational value of the paper?

https://docs.google.com/forms/d/e/1FAIpQLSdbygAtXMLJpwn-O01ACTQvLSqkaQ97hnrHXhttps://docs.google.com/forms/d/e/1FAIpQLSdbygAtXMLJpwn-O01ACTQvLSqkaQ97hnrHX Ch. Eick Ch. Eick Some Hints on Reading Papers In general, reading, understanding and reviewing conference papers is challenging. If you understand 60% of the paper you have to review, you should be happy! Discussing paper with peers helps understanding papers; if each students understands 30% of the paper, together you might actually understand 55% of the paper. Misconceptions and misunderstanding pose another challenge for understanding papers If you read a paper and do not understand a term, look it up, e.g. in Wikipedia or google it. As you might be lost understanding the paper; I suggest that you initially scan through the paper, and then do some websearch and do some reading of background material, and then read the paper again, discuss it with your teammates and continue this loop Lack of Math knowledge might be another challenge. Do get frustrated initially things will likely improve as the work on this task evolves. 6

Ch. Eick Ch. Eick News November 7, 2024 There will be a discussion centering on ethics for data science lead by group K on November 7; there will be a similar discussing, focusing on a different subtopic, lead by group N on November 21. Task 4 is due on Monday, November 11 end of the day in MS Teams (no grace period). We taught the makeup class on Nov. 1 as the class on Tuesday, November 12 has been cancelled. Today s Lecture a. Reading and Reviewing Papers and Task5 b. Ethics for Data Science Discussion lead by Group K Advanced Clustering (will discuss 2-3 more clustering algorithms) c. d. Finish Discussion on Sequence Mining (likely on Nov. 14) 7

Ch. Eick Ch. Eick Teaching Material Paper Reading / Surving Graduate School http://www.wikihow.com/Read-a-Scientific-Paper http://mindfulconstruct.com/2008/12/27/15- tips-for-reading-a-scientific-research-paper/http://violentmetaphors.com/2013/08/25/how-to- read-and-understand-a-scientific-paper-2/ http://www.huffingtonpost.com/jennifer-raff/how- to-read-and-understand-a-scientific-paper_b_5501628.html http://web.stanford.edu/~siegelr/readingsci.htm http://www.cs.cmu.edu/~jrs/sins.html (3 sins COSC/MATH) More general, although related issues: (15) Seven Qualities of Highly Effective Graduate Students | LinkedIn 12 Tips for Surviving and Thriving in Grad School Preparing for Graduate School: Advice for New Student Success | Harvard Extension School

Ch. Eick Ch. Eick Challenges Task5 The paper you review is, in our opinion, not easy to read and has a high technical depth. However, it won a paper award for the IEEE ICDM 2022 Conference. Welcome to ICDM: IEEE International Conference on Data Mining! That is, if you want to assess what is going on in a particular research field---here data mining---you will need to at least partially understand papers, like the one you will be reviewing. 9

Ch. Eick Ch. Eick Task 5 Reviews Task 5 Paper Reviews will differ from traditional conference paper reviews in several aspects (purple means not part of a traditional review): The section that summarizes the paper will be about 2-3 times as longer The educational value of the paper for graduate students need to be assessed in your review You will assess the broader impact of the paper You will also conduct web-search trying to find similar papers and summarize your findings You will give an assessment why you believe this paper won a KDD Best Paper award. Usually you should also check if the paper or parts of it are already published; however, this is not required for Task 5! A paragraph is assumed to have 4-10 sentences. Follow the review template outlined in the Task 5 specification! 10

Ch. Eick Ch. Eick KDD 2012 Reviewing Criteria: Research Track KDD 2012 Reviewing Criteria: Research Track Below we have provided some guidelines to reviewers on how to write reviews, both the content of reviews and also how the numerical scoring system works. Many of the suggestions below have been liberally borrowed from other conferences - so thanks to the many folks who have contributed to writing these types of "guidance" pages in the past. Writing Reviews: Content (Edited by Ch. Eick) For each paper you will provide written comments under each of the headings below. Your review should address both the strengths and weaknesses of the paper - identify the areas where you believe the paper is particularly strong and particularly weak - this will be very valuable to the PC Chairs and the SPC. Novelty: This is arguably the single most important criterion for selecting papers for the conference. Reviewers should reward papers that propose genuinely new ideas or novel adaptations/applications of existing methods. It is not the duty of the reviewer to infer what aspects of a paper are novel - the authors should explicitly point out how their work is novel relative to prior work. Assessment of novelty is obviously a subjective process, but as a reviewer you should try to assess whether the ideas are truly new, or area novel combinations or adaptations or extensions of existing ideas, or minor extensions of existing ideas, and so on. Technical Quality: Are the results sound? Are there obvious flaws in the conceptual approach? Did the authors ignore (or appear unaware of) highly relevant prior work? Are the experiments well thought out and convincing? Are there obvious experiments that were not carried out? Will it be possible for later researchers to replicate these results? Are the data sets and/or code publicly available? Did the authors discuss sensitivity of their algorithm/method/procedure to parameter settings? Did the authors clearly assess both the strengths and weaknesses of their approach? 11

Ch. Eick Ch. Eick KDD 2012 Reviewing Criteria: Research Track KDD 2012 Reviewing Criteria: Research Track Below we have provided some guidelines to reviewers on how to write reviews, both the content of reviews and also how the numerical scoring system works. Many of the suggestions below have been liberally borrowed from other conferences - so thanks to the many folks who have contributed to writing these types of "guidance" pages in the past. Potential Impact and Significance: Is this really a significant advance in the state of the art? Is this a paper that people are likely to read and cite in later years? Does the paper address an important problem (e.g., one that people outside machine learning and data mining are aware of) or just a problem that only a few researchers are interested in and that won t have any lasting impact? Is this a paper that researchers and/or practitioners might find useful 5 or 10 years from now? Is this work that can be built on by other researchers? Clarity of Writing: Please make full use of the range of scores for this category so that we can identify poorly-written papers early in the process. Is the paper clearly written? Is there a good use of examples and figures? Is it well organized? Are there problems with style and grammar? Are there issues with typos, formatting, references, etc? It is the responsibility of the authors of a paper to write clearly, rather than it being the duty of the reviewers to try to extract information from a poorly written paper. Do not assume that the authors will fix problems before a final camera-ready version is published - unlike journal publications, there will not be time to carefully check that accepted papers are properly written. Think of future readers trying to extract information from the paper - it may be better to advise the authors to revise a paper and submit to a later conference, than to accept and publish a poorly- written version. Additional Points (optional): this is an optional section on the review form can be used to add additional comments for the authors that don t naturally fit into any of the areas above. 12

Ch. Eick Ch. Eick KDD 2012 Numerical Paper Evaluation General Advice on Review Writing: please be as precise as you can in your comments to the authors and avoid vague statements. Your criticism should be constructive where possible - if you are giving a low score to a paper then try to be clear in explaining to the authors the types of actions they could take to improve their paper in the future. For example, if you think that this work is incremental relative to prior work, please cite the specific relevant prior work you are referring to. Or if you think the experiments are not very realistic or useful, let the author(s) know what they could do to improve them (e.g., more realistic data sets, larger data sets, sensitivity analyses, etc). Writing Reviews: Numerical Scoring For KDD-2012 we are using a 7-point scoring system. We strongly encourage you to use the full range of scores, if appropriate for your papers. Try not to put all of your papers in a narrow range of scores in the middle of the scale, e.g., 3s, 4s, and 5s. Don t be afraid to assign 1s/2s, or 6s/7s, if papers deserve them. If you are new to the KDD conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent KDD conferences to help calibrate your scores. The scoring system is as follows: 7: An excellent paper, a very strong accept. I will fight for acceptance, this is potentially best-paper material. 6: A very good paper, should be accepted. I vote and argue for acceptance, clearly belongs in the conference. 5: A good paper overall, accept if possible. I vote for acceptance, although would not be upset if it were rejected because of the low acceptance rate. 4: Decent paper, but may be below KDD threshold I tend to vote for rejecting it, but could be persuaded otherwise. 3: An OK paper, but not good enough. A rejection. I vote for rejecting it, although would not be upset if it were accepted. 2: A clear rejection. I vote and argue for rejection. Clearly below the standards for the conference. 1: A strong rejection. I'm surprised it was submitted to this conference. I will actively fight for rejection. 13