Panopto Captions Accuracy
Learn about the accuracy of Panopto captions through ROUGE-L analysis and student feedback. Discover how editing captions improves accuracy and helps students focus. Find out how Panopto enhances the lecture experience for students with disabilities.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Panopto Captions Accuracy Canvas & Teaching Technology Monthly Meeting Gemma Sinclair Product Owner September 2022
Captions accuracy We can calculate the accuracy of the captions using a correct transcript (or reference text) and comparing to the automatically generated captions. This gives a confidence score, e.g. in this sample text with M ori terms the score was 50.82%. Rouge-L analysis between the reference text and the Panopto automatic captions gives a confidence score of 0.5081967 (50.82%) 2
Captions Accuracy In Summer School we ran a pilot with Note Takers who edited Panopto captions for accuracy in 5 courses. Time taken to edit captions to 100% accuracy per 1 hour lecture: 1 hour (Finance paper) 1.5 to 2 hours (Engineering papers, English paper) 3 hours (Statistics paper) Proper Nouns / names e.g., Author names Audio volume e.g., lecturer moves away from the microphone NZ accente.g., bed becomes beard , says becomes sees ; uptalk statement becomes a question Filler words'ums , 'ahs etc. are ignored therefore equations/formulae not captured accuratelye.g., Sum and R Punctuatione.g., not picking up end/start of new sentence Acronyms and Homophones e.g., to, too, two In addition to M ori language and technical terms/jargon there were common issues with Proper nouns, NZ accent, equations and acronyms. 3
Captions accuracy We applied the ROUGE-L analysis to lectures from the Summer School note taker pilot to calculate accuracy scores. We can see the accuracy scores which reflect the time taken to edit the lectures for accuracy (1 hour to 3 hours). 4
Student feedback COMSCI/FINANCE/MATHS student CHEMMAT student (5 papers) Although not 100% accurate, helps stop the mind from wandering. Student watches at 2x speed and reads the captions to help focus. Very helpful as each lecturer has a different accent, so adjusts speed accordingly. I wanted to try out Panopto's captions before requesting a note taker. I'm happy to report that the captioning has been a huge help with my experience in lectures and I won't need note- taker assistance this year. LAW/GEOG student (3 papers) LAW and CRIM/SOCIOL student (5 papers) Lectures with no subtitles are a pain. Most courses do have subtitles. In one they intentionally turned off, as one thing was offensive. They said, if they have special circumstances they can email and request a transcript. The captions are better than the transcript. Student enrolled in these courses said they didn t need note taker support as they had Panopto recordings (with captions). 6 https://www.auckland.ac.nz/en/students/student-support/students-with-disabilities/support-for-current-students/note-takers.html
Custom Dictionary While testing M ori phrases in Panopto we were amazed to find there are some kupu reo in Panopto global English. M ori kupu in Panopto: Akaroa, Hapu, Ihumatao, Iwi, Kaitiaki, Kahu, Kaipara, Kaka, Karua, Katoa, Kaumatua, Kaupapa, Kauru, Kotahi, Kura, Mahuta, Maori, Matara, Nuku, Oranga, Rotorua, Puna, Wahine, Waitangi, Wiki Tamaki, Tangaroa, Tangata, Tamariki, Tarawa They are missing the macrons (e.g. Maori instead of M ori). Some are places (Ihum tao) and people (Nanaia Mahuta) that were in the news in the last few years. 7
Custom Dictionary What we also find is that there are several other languages (e.g. Japanese, Spanish, French) included in Global English. E.g. Surnames (Boneham, Hiroko, Khadija, Riina) and place names (Kyoto, Detroit, Copacobana, Phi Phi, Ottawa, Timor etc) 8
Crowd Captions Crowd Captions is a Browser Extension tool that allows students to edit captions from Panopto. The tool is currently being piloted in four courses this semester: BIOSCI 106 (1,075 students) BIOSCI 220 (333) ENGGEN 131 (1,039) MECHENG 222 (244) 9
