Weakly Supervised Models of Aspect-Sentiment in Online Course Discussions
Massive open online courses (MOOCs) present unique challenges with classroom interactions, requiring innovative approaches like weakly supervised models for aspect-sentiment analysis. Explore the complexities of MOOC discussion forums, predicting fine-grained problems, and related work in the field.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Weakly Supervised Models of Aspect-Sentiment for Online Course Discussion Forums ARTI RAMESH ARTI RAMESH SHACHI H. KUMAR SHACHI H. KUMAR JAMES FOULDS JAMES FOULDS LISE GETOOR LISE GETOOR I S Q L N
Massive: attracts thousands of participants Open: open access, content, and assessment Online: hosted online by education companies in partnership with top universities 2
Classroom MOOCs Classroom Face-to-face interaction between instructor and students MOOC Discussion Forums Primary means of interaction between instructor and students Large number of students, posts: Hard to monitor manually Posts discuss problems in course - course material, errors, feedback 3
Example MOOC Posts MOOC Post The video is very choppy. Can somebody fix this? Fine-grained Topic Lecture-Video Will subtitles be made available for the lectures for this week? I liked the transcripts from last week. Lecture-Subtitles Will everyone get a certificate or only people in the signature track? Certificate When is quiz 4 due? Quiz-Deadlines 4
Predicting fine-grained problems: Challenges Labeled data hard to obtain 5-10% posts contain problems Privacy concerns around data sharing Problems differ across courses Unsupervised/weakly supervised approaches desirable System not fine-tuned to one course, but can adapt across courses 5
Related Work Aspect-sentiment in Online Reviews Semi-supervised generative model, with seed words to identify aspect clusters [Mukherjee et al., 2012] Unsupervised Aspect-Sentiment Model for Online Reviews [Brody et al., 2012] Hierarchical Aspect-Sentiment Model for Online Reviews [Kim et al. 2013] MOOCs Predicting Instructor Intervention in MOOC Forums[Chaturvedi et al., 2014] 6
SeededLDA for MOOC Forums SeededLDA Guide topic discovery by specifying representative seed words seededLDA uses seeds to bias topic-word and word- document distributions seededLDA gathers words related to seed words SeededLDA for MOOCs Many classes but a common set of seed words Seed words for MOOCs from syllabus and forums Jagarlamudi et al. 2010 7
Hinge-loss Markov Random Fields & Probabilistic Soft Logic Hinge-loss Markov Random Fields (HL-MRFs) Logic-based MRFs that can reason about both discrete and continuous graph data scalably and accurately Efficient Inference: convex optimization in continuous space Probabilistic Soft Logic (PSL) Templating language for HL-MRFs Weighted logical rules to model dependencies Continuous variables in [0,1] Bach et al. 2012 8
Predicting fine-grained problems and sentiment: Joint Prediction Problem Analogous to predicting aspect-sentiment in online reviews Aspect hierarchy connecting course elements HL-MRF framework Combining different features Encoding coarse-to-fine aspect hierarchy Encoding dependencies between aspect and sentiment Jointly modeling aspect and sentiment 9
Our Contributions Identify fine-grained aspects in online courses Extract course-specific features from posts using SeededLDA Construct coarse-to-fine aspect hierarchy to model aspect dependencies Construct weakly-supervised joint model for aspect-sentiment using HL-MRFs Validate system using crowdsourced posts sampled from 12 courses 10
MOOC Aspect-Sentiment Models: SeededLDA Coarse Aspect seeds LECTURE: lecture, video, download, transcript, slide, note QUIZ: quiz, assignment, question, midterm, exam, submission CERTIFICATE: certificate, score, statement, signature SOCIAL: name, course, introduction, study, group Sentiment seeds POSITIVE: interest, exciting, thank, great, happy, glad, enjoy NEGATIVE: problem, difficult, error, issue, unable, misunderstand NEUTRAL: coursera, class, hello, everyone, greet, name 11
SeededLDA Model Fine Aspect seeds LECTURE-VIDEO: video, problem, download, play, player, LECTURE-AUDIO: volume, low, headphone, sound, audio, hear LECTURE-LECTURER: professor, fast, speak, pace, follow, speed LECTURE-SUBTITLES: transcript, subtitle, slide, note, lecture, LECTURE-CONTENT: typo, error, mistake, wrong, right, incorrect QUIZ-CONTENT: question, challenge, difficult, understand, typo QUIZ-SUBMISSION: submission, submit, quiz, error, unable, resubmit QUIZ-GRADING: answer, question, answer, grade, assignment, quiz QUIZ-DEADLINE: due, deadline, miss, extend, late 12
PSL-Joint: Combining Features SeededLDA score for fine aspect and coarse aspect to predict fine aspect of post P 13
PSL-Joint: Combining Features SeededLDA score for sentiment and fine aspect to predict fine aspect 14
PSL-Joint: Encoding Dependencies Dependency between coarse aspect and fine aspect 15
PSL-Joint: Encoding Dependencies Dependency between sentiment and fine aspect 16
Experimental Evaluation F-1 scores for SeededLDA and PSL-Joint for coarse aspects Model Lecture Quiz Certificate Social SeededLDA 0.632 0.657 0.459 0.654 PSL-Joint 0.630 0.706 0.621 0.659 SeededLDA and PSL-Joint for sentiment Model Positive Negative Neutral SeededLDA 0.182 0.517 0.356 PSL-Joint 0.189 0.615 0.434 17
Experimental Evaluation SeededLDA and PSL-Joint for coarse aspects Model Lecture Quiz Certificate Social SeededLDA 0.632 0.657 0.459 0.654 PSL-Joint 0.630 0.706 0.621 0.659 PSL-Joint SeededLDA and PSL-Joint for sentiment outperforms SeededLDA for most coarse aspects and sentiment Model Positive Negative Neutral SeededLDA 0.182 0.517 0.356 PSL-Joint 0.189 0.615 0.434 18
Experimental Evaluation Fine-grained aspects under coarse aspect lecture Model Content Video Audio Lecturer Subtitles SeededLDA 0.08 0.240 0.684 0.06 0.397 PSL-Joint 0.410 0.485 0.582 0.323 0.461 Fine-grained aspects under coarse aspect quiz Model Content Submission Deadlines Grading SeededLDA 0.011 0.437 0.214 0.514 PSL-Joint 0.36 0.416 0.611 0.550 19
Experimental Evaluation Fine-grained aspects under coarse aspect lecture Model Content Video Audio Lecturer Subtitles SeededLDA 0.08 0.240 0.684 0.06 0.397 PSL-Joint distinguishes between lecture- content and quiz- content PSL-Joint 0.410 0.485 0.582 0.323 0.461 Fine-grained aspects under coarse aspect quiz Model Content Submission Deadlines Grading SeededLDA 0.011 0.437 0.214 0.514 PSL-Joint 0.36 0.416 0.611 0.550 20
Experimental Evaluation Fine-grained aspects under coarse aspect lecture Model Content Video Audio Lecturer Subtitles Significant improvement in scores for lecture-lecturer and quiz-deadlines SeededLDA 0.08 0.240 0.684 0.06 0.397 PSL-Joint 0.410 0.485 0.582 0.323 0.461 Fine-grained aspects under coarse aspect quiz Model Content Submission Deadlines Grading SeededLDA 0.011 0.437 0.214 0.514 PSL-Joint 0.36 0.416 0.611 0.550 21
Interpreting PSL-Joint Predictions There is a typo or other mistake in the assignment instructions (e.g. essential information omitted). SeededLDA Prediction: Lecture-content PSL-Joint Prediction: Quiz-content Thanks for the suggestion about downloading the video and referring to the subtitles. The audio is barely audible, even when the volume is set to 100% SeededLDA Prediction: Lecture-subtitles PSL-Joint Prediction: Lecture-audio 22
Conclusion: Fine-grained aspect- sentiment in MOOC forums Automatically detecting problems in forum posts useful for instructors Weakly supervised probabilistic framework to automatically detect aspect and sentiment in online courses SeededLDA and PSL-Joint models as means to encode domain information and predict aspect and sentiment PSL-Joint significantly outperforms SeededLDA for many fine aspects, coarse aspects, and sentiment Structural dependencies among aspect and sentiment helps in prediction 23
