Enhancing Automatic Response-to-Text Assessment Through Topic Coherence

incorporating coherence of topics as a criterion n.w
1 / 33
Embed
Share

Explore the incorporation of topic coherence as a criterion in automatic response-to-text assessment for evaluating the organization of writing. The study focuses on analytical text-based writing and aims to score students' essays based on quality and organization. Various aspects such as making claims, marshaling evidence, and evaluating organization style are examined. The research delves into the assessment of students' ability to find evidence to support their claims and emphasizes coherence in writing. Discover how topic coherence plays a crucial role in assessing the quality of essays.

  • Text assessment
  • Topic coherence
  • Writing organization
  • Analytical writing
  • Student scoring
rayaanacharya
anan206 Follow

Uploaded on | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. INCORPORATING COHERENCE OF TOPICS AS A CRITERION IN AUTOMATIC RESPONSE-TO-TEXT ASSESSMENT OF THE ORGANIZATION OF WRITING ZAHRA RAHIMI ZAR10@PITT.EDU DIANE LITMAN DLITMAN@PITT.EDU ELAINE WANG RICHARD CORRENTI ELW51@PITT.EDU RCORRENT@PITT.EDU BEA 2015 University of Pittsburgh

  2. Goals Automatic scoring of students writing Analytical text-based writing Quality of essays in terms of organization 2 6/4/2015

  3. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 3 6/4/2015

  4. Response-to-Text Assessment (RTA) (Correnti et al., 2013) Analytical text-based writing Making claims Marshalling evidence from a source text to support a viewpoint Evaluated on five-traits rubric. Thinking about the text Skill at finding evidence to support their claims (Rahimi et al. 2014) Organization Style (Mechanics, Usage, Grammar, Spelling) 4 6/4/2015

  5. Text and Writing Prompt Excerpt from Text ( A Brighter Future by Hannah Sachs from Time for Kids) : The people of Sauri have made amazing progress in just four years. The Yala Sub-District Hospital has medicine, free of charge, for all of the most common diseases. Water is connected to the hospital, which also has a generator for electricity. Writing Prompt: The author provided one specific example of how the quality of life can be improved by the Millennium Villages Project in Sauri, Kenya. Based on the article, did the author provide a convincing argument that winning the fight against poverty is achievable in our lifetime? Explain why or why not with 3-4 examples from the text to support your answer. 5 6/4/2015

  6. A Sample High Quality Essay they showed many example in the beginning and showed how it changed at theThis story convinced me that winning the fight against poverty is achievable because end. One example they sued show a great amount oF change when they stated at first most people thall were ill just stayed in the hospital Not even getting treated either because of the cost or the hospital didnt have it, but at the end it stated they now give free medicine to most common deseases. Anotehr amazing change is in the beginning majority of the childrenw erent going to school because the parents couldn t affford the school fee, and the kdis didnt like school because tehre was No midday meal, and Not a lot of book, pencils, and paper. Then in 2008 the perceNtage of kids going to school increased a lot because they Now have food to be served aNd they Now have more supplies. So Now theres a better chance of the childreN getting a better life The last example is Now they dont have to worry about their families starving because Now they have more water and fertalizer. They have made some excellent changes in sauri. Those chaNges have saved many lives and I think it will continue to change of course in positive ways Hospitals (before) Hospitals (after) Hospitals Schools Farming 6 6/4/2015

  7. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 7 6/4/2015

  8. Focus of the Study Develop a task-dependent model that is consistent with the rubric criteria Ability to provide feedback that is better aligned with the task Organization as conceived by the RTA How well the pieces of evidence are organized to make a strong argument Coherence around the ordering of topics related to pieces of evidence. Assessment of coherence (Foltz et al., 1998; Higgins et al., 2004; Bursteinet al., 2010; Somasundaran et al.,2014) Evaluate the writing of younger students in grades 5 through 8 8 6/4/2015

  9. Lexical Cohesion is Insufficient Assess coherence using: Entity grids (Burstein et al., 2010) and lexical chains (Somasundaran et al., 2014) Repetition of identical or similar words according to external similarity sources Interested in evaluating the coherence around pieces of evidence, not just the lexical cohesion Hypothesis: existing models are not as well on short and noisy essays as on longer and better written essays The hospitals were in bad situation. There was no electricity or water. There would be no transition between these two sentences 9 6/4/2015

  10. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 10 6/4/2015

  11. How to Model Coherence around Topics and Pieces of Evidence? By: Topic Grid and Topic Chains 11 6/4/2015

  12. Example Topic and Pieces of Evidence The people of Sauri have made amazing progress in just four years. The Yala Sub-District Hospital has medicine, free of charge, for all of the most common diseases. Water is connected to the hospital, which also has a generator for electricity Excerpt from the text 1. Yala sub-district hospital medicine 2. medicine free charge 3. water connected hospital 4. hospital generator electricity 5. Medicine common diseases Pieces of evidence around the topic hospitals for the state after 12 6/4/2015

  13. Topic Grid One example they sued show a great amount oF change when they stated at first most people thall were ill just stayed in the hospital Not even getting treated either because of the cost or the hospital didnt have it, but at the end it stated they now give free medicine to most common deseases. 1 2 3 4 5 6 7 8 9 10 1. Yala sub-district hospital medicine 2. medicine free charge 3. water connected hospital 4. hospital generator electricity 5. Medicine common diseases Hospitals.before - x - - - - - - - - Hospitals.after - - x - - - - - - - Education.before - - - x - - - - - - Education.after - - - - x x - - - - Farming.before - - - - - - x - - - Farming.after - - - - - - - x - - General x - - - - - - - x x 13 6/4/2015

  14. Topic Chain 1 2 3 4 5 6 7 8 9 10 Hospitals.before - x - - - - - - - - One chain for each topic Hospitals.after - - x - - - - - - - Education.before - - - x - - - - - - Each node carries two pieces of information: The index of the text unit it appears in Whether it is a before or after state Education.after - - - - x x - - - - Farming.before - - - - - - x - - - Farming.after - - - - - - - x - - General x - - - - - - - x x Topic Chain Hospitals (b,2),(a,3) Education (b,4),(a,5),(a,6) Farming (b,7),(a,8) 14 6/4/2015

  15. Features Goal: design a small set of rubric-based features that performs acceptably and also models what is actually important in the rubric. Surface Discourse structure Local coherence and paragraph transitions Topic development Topic ordering and patterns 15 6/4/2015

  16. Features (Based on Literature) Surface Number of paragraphs Average sentence length Discourse structure HasBeginning HasEnding (based on general statements from the text and the prompt) LSA-similarity of 1 to 3 sentences from the beginning and ending of the essay with respect to the length of the essay. Local coherence and paragraph transitions The average LSA (Foltz et al., 1998) similarity of adjacent sentences Average LSA similarity of all paragraphs (Foltz et al., 1998) For one paragraph essays, we divide the essays into 3 equal parts and calculate the similarity of 3 parts. 16 6/4/2015

  17. Topic-Based Features (Based on Literature) Average number of nodes in chains Max distance between chain s nodes Sum of the distances between each pair of adjacent nodes Average number of nodes in chains divided by average chain length Number of topics covered in the essay divided by the length of the essay Count and percentage of discourse markers from each of the four groups adjacent to a topic 17 6/4/2015

  18. Topic-Based Features (New) Number and percentage of chains which discusses both aspects ( before and after ) of that topic. Before-only, After-only Number of chains starting and ending inside another chain Levenshtein edit-distance 18 6/4/2015

  19. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 19 6/4/2015

  20. Data (Correnti et al. 13) First dataset: Grades 5-6 Second dataset: Grades 6-8 Number of essays 1580 812 Number of doubly scored essays Around 600 802 Avg number of words 161.25 207.99 Avg number of unique words 93.27 113.14 Quadratic weighted kappa 0.68 0.69 20 6/4/2015

  21. Distribution of Organization Scores Short, many spelling and grammatical errors, and not well-organized Score on a scale of 1-4 Dataset/score 1 2 3 4 5 6 grades 398 (25%) 714 (46%) 353 (22%) 115 (7%) 6 8 grades 128 (16%) 316 (39%) 246 (30%) 122 (15%) 21 6/4/2015

  22. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 22 6/4/2015

  23. Does our rubric-based model perform better than the baselines? On grades (5-6): significantly higher performance than either baseline or the combination On grades (6-8): no significant difference between the rubric-based model and the baselines Model grades (5 6) grades (6-8) 1 EntityGridTT (Burstein et al. 2010) 0.42 0.49 Baselines 2 LEX1 (Somasundaran et al. 2014) 0.45 0.53 3 EntityGridTT+LEX1 0.46 0.54 4 Rubric-based 0.51 0.51 5 EntityGridTT+Rubric-based 0.49 0.53 6 LEX1+Rubric-based 0.51 0.55 7 EntityGridTT+LEX1+Rubric-based 0.50 0.56 Quadratic Weighted Kappa 23 6/4/2015

  24. Is the new model generalizable across different grades? For both experiments: the rubric-based model performs at least as well as the baselines. Test on the shorter and noisier set of (5-6): the rubric-based model performs significantly better than the baselines. Model Train on(5 6) Test on (6-8) Train on(6-8) Test on (5-6) 1 EntityGridTT (Burstein et al. 2010) 0.51 0.43 2 LEX1 (Somasundaran et al. 2014) 0.43 0.41 Baselines 3 EntityGridTT+LEX1 0.52 0.42 4 Rubric-based 0.56 0.47 5 EntityGridTT+LEX1+Rubric-based 0.58 0.45 Quadratic Weighted Kappa 24 6/4/2015

  25. How important are the topic-based features? we believe that the topic-based features are more substantive and potentially provide more useful information for students and teachers. Improve performance by enhancing the simple topic alignment of the sentences. Model (5-6) cross val (6-8) cross val Train on(5 6) Test on (6-8) Train on(6-8) Test on (5-6) 0 EntityGridTT+LEX1 0.46 0.54 0.52 0.42 Baseline 3 Topic-Based 0.42 0.45 0.46 0.40 4 Surface 0.32 0.40 0.42 0.35 5 LocalCoherence+ParagraphTransition 0.20 0.21 0.23 0.18 6 DiscourseStrucutre 0.25 0.19 0.26 0.22 Quadratic Weighted Kappa 25 6/4/2015

  26. Outline Goals Response-to-Text Assessment (RTA) Focus of the Study Approach and Model Data Experiments and Results Conclusion and Future Work 26 6/4/2015

  27. Conclusion We present the results for predicting the score of the Organization dimension of a response-to-text assessment. New task-dependent rubric-based model performs as well as either baseline on both datasets. On the shorter and noisier essays, the rubric-based model based on coarse- grained topic information outperforms state-of-the-art models based on syntactic and lexical information. In general, the rubric-based features can add value to the baselines. 27 6/4/2015

  28. Future Work Use a more sophisticated method to annotate text units Test the generalizability of our model by using other texts and prompts from other response-to-text writing tasks Extract topics and words automatically, as our current approach requires these to be manually defined by experts Although this task needs to be only done once for each new text and prompt 28 6/4/2015

  29. Thank you! 29 6/4/2015

  30. Levenshtein Edit-Distance Edit-distance of the topic vector representations for befores and afters normalized by the number of topics in the essay Good organization of topics Cover both the before and the after examples on each discussed topic Come in a similar order The greater the value, the worse the pattern of discussed topics befores=[3,4,4,5] , afters=[3,6,5] befores=[3,4,5] , afters=[3,6,5] The normalized Levensthein = 1/4 30 6/4/2015

  31. Can the lexical chaining baseline be improved with the use of topic information from the source document? Lexical chaining uses both external sources to measure semantic similarity and also our list of topics extracted from the source text Model grades (5 6) grades (6-8) 1 LEX1 0.45 0.53 2 LEX1+Topic 0.48 0.54 31 6/4/2015

  32. Surface > TopicOrdering > LocalCoherence+ParagraphTransitions > DiscourseStructure > TopicDevelopment 32 6/4/2015

  33. Related work on measuring coherence in student essays Vector-based similarity methods measure lexical relatedness between text segments (Foltz et al., 1998) Between discourse segments (Higgins et al., 2004) Centering theory (Grosz et al., 1995) addresses local coherence (Miltsakaki and Kukich, 2000) Entity-based essay representation (Burstein et al., 2010) Lexical chaining addresses (Somasundaran et al.,2014) Discourse structure is used to measure the organization of argumentative writing (Cohen, 1987; Burstein et al., 1998; Burstein et al., 2003) 33 6/4/2015

Related


No related presentations.

More Related Content