Exploring Multimodal Learning Analytics Techniques
"Discover the world of Multimodal Learning Analytics (MMLA) techniques, involving the collection and analysis of data from various sources like video, logs, audio, and biosensors. Understand the process and benefits of MMLA in examining learning in diverse environments. Gain insights into the importance of multimodal design and the specialized expertise required in this field."
Uploaded on | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Week 8 Video 5 Multimodal Learning Analytics
Thank you Thank you to Yiqiu (Rachel) Zou for feedback and comments on this video
Multimodal Learning Analytics A set of techniques that can be used to collect multiple sources of data in high-frequency (video, logs, audio, gestures, biosensors), synchronize and code the data, and examine learning in realistic, ecologically valid, social, mixed-media learning environments. (Blikstein et al., 2013)
In response to Earlier work that focused on interaction log data because it was easiest to obtain, relatively easy to work with, and relatively easy to scale and apply in the real world looking for solutions where it is easy to search, not where the real solutions are most probable to be found. (Ochoa, 2022)
A range of types of data out there Keystroke Mouse movement Eye movement/pupillometry EEG Electrodermal activity/galvanic skin response Motion sensors Skeleton tracking/posture Prosody, cadence Video (zoom, webcam, video cameras in the world) Audio (zoom, personal microphones, room microphone) Location-tracking badges
Where do you even start? Keystroke Mouse movement Eye movement/pupillometry EEG Electrodermal activity/galvanic skin response Motion sensors Skeleton tracking/posture Prosody, cadence Video (zoom, webcam, video cameras in the world) Audio (zoom, personal microphones, room microphone) Location-tracking badges
Process of MMLA (Ochoa, 2022) Define Construct(s) of Interest Design System ( Execution ) Multimedia Recording Multimedia Feature Extraction Multimedia Fusion Behavior Detection and Construct Estimation Use Model ( Feedback to Participants ) 1. 2. 3. 4. 5. 6. 7.
Whats Special about Multimodal Designing the combination of sensors Multimodal Feature Extraction Feature Engineering practices have developed for each type of sensor (often beyond EDM/LA communities) Specific expertise needed in this
Whats Special about Multimodal Multimodal Fusion (Lahat et al., 2015; Chango et al., 2022) One approach: feature engineering on each type of data, then all types of data thrown into classifier together Alternate approach: Conduct feature engineering multimodally create features that involve multiple data streams Third approach: Build classifier from each input stream, then ensemble classifiers afterwards
Key Challenges (Sharma & Giannakos, 2020; Ochoa, 2022) Every project is a one-off Different goals lead to different sensors and measures and extraction processes Hard to build from one project to next Sensor outputs may be context-dependent Many multimodal learning analytics studies still conducted in laboratory settings or relatively-controlled/constrained classrooms
Key Challenges (Ochoa, 2022) Multiple types of technical expertise needed Each type of sensor you use Plus the multimodal fusion
Key Challenges (Ochoa, 2022) Practical challenges Cost Robustness Scalability Dealing with Different Grain-Sizes Ethical Concerns about Multimodal Measurement Some people get very concerned about video or EEG or GSR of kids
Any EDM/LA method can be multimodal In one particularly ambitious paper, Yan and colleagues (2022) conduct Correlation Mining Prediction Modeling Discovery with Models (will be discussed next week) Epistemic Network Analysis And a qualitative methodology On a single multimodal data set
Directions that make things easier Improving foundational technology Better tools for high-speed auto-transcription (AI Whisper) Better algorithms for eye-tracking from webcams (Hutt et al., 2022) Cheaper gadgets for everything
Towards the Future MMLA is demonstrating a lot of potential Some challenges for it to fully achieve that potential and scale Watch this space!
