Data Collection and Qualitative Analysis in HCI and Interactive Technologies

CS 598 AK Experimental HCI & Interactive Technologies T ext Chapter 4 DataCollectionandQualitativeAnalysis

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis InterpretingT extualData: Analyzing textual data involves identifying clear themes that emerge from the text anddetermining thosethat aremostcommon.Themes are not always identically phrased, so some subjective judgment is required in deciding whether two phrasesbelong to the sametheme.

Data CollectionandQualitative Analysis InterpretingT extualData: Analyzing textual data involves identifying clear themes that emerge from the text anddetermining thosethat aremostcommon.Themes are not always identically phrased, so some subjective judgment is required in deciding whether two phrasesbelong to the sametheme. Forexample, the phrases too cluttered, very squidged up, all on top of eachother, and not very spreadout might beconsideredto all belong to the themeof cluttered.

Data CollectionandQualitative Analysis InterpretingT extualData: Analyzing textual data involves identifying clear themes that emerge from the text anddetermining thosethat aremostcommon.Themes are not always identically phrased, so some subjective judgment is required in deciding whether two phrasesbelong to the sametheme. Forexample, the phrases too cluttered, very squidged up, all on top of eachother, and not very spreadout might beconsideredto all belong to the themeof cluttered. It is therefore best if this analysis is independently performed (and subsequentlyagreed on)byat least twopeopleto ensurethat it is valid andunbiased. Note: Quantitativemethodsareavailable for this purpose:Measuresof Inter-Rater Reliability.

Data CollectionandQualitative Analysis InterpretingVideoData: Alist of important actionsisagreed in advancebythe research team. Forexample,in anexperiment that asksparticipants to rearrange objectsin order of size using a drag-and-drop interface, one action category might be participant movesanobjectseveralplacesto the left andplacesit between two objectsthat arenot obviously of similar size.

Data CollectionandQualitative Analysis InterpretingVideoData: Alist of important actionsisagreed in advancebythe research team. Forexample,in anexperiment that asksparticipants to rearrange objectsin order of size using a drag-and-drop interface, one action category might be participant movesanobjectseveralplacesto the left andplacesit between two objectsthat arenot obviously of similar size. Thisaction mayindicatethat the participantisdoing acoarseseparationof big objects from small ones, perhaps before doing a later fine-grained separation: this is easierto seefrom the video screencaststhan from alog file that capturesallinteractions with the system.

Data CollectionandQualitative Analysis InterpretingVideoData: Specialethical approval and participantconsentwill berequired if the video recordingincludesparticipants facessothat theyareidentifiable. The same can be said for audio recordings (because people can be identified bytheir voice),althoughmakingacommitment to havethe audiorecordings completelytranscribedbeforedestroyingthem will appeasemost ethics committees. It ismuchmore difficult to fully transcribeavideo recording, andmost experimenterswill want to keepthe videodata.

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis Natureof DataCollected: The5Psof DataCollection: 1. Performanceindicateshow well subjectssucceededinatask. 2. Preferenceindicates subjectsrelative preferencesamongdesigns. 3. Perceptionindicatesother subject optionsabout astudy. 4. Processindicates the temporal processasubjectwasengagedin. 5. Product isacollectionof anysubject-createdartifacts,suchasdesigns.

Data CollectionandQualitative Analysis PerformanceData: The2Most Frequent Measures CAUTION:Making comparative (better, worse) conclusions between designs basedon error andresponsetime data often requires detailed consideration of speed-accuracytradeoffs

Data CollectionandQualitative Analysis

Data CollectionandQualitative Analysis PreferenceData: Participants are often asked to indicate their preferences for conditions indeed,it isadvisableto alwaysaskfor preferencedatain apost-experiment questionnaire or interview so as to collect as much data as possible from eachparticipant.

Data CollectionandQualitative Analysis PreferenceData: Participants are often asked to indicate their preferences for conditions indeed,it isadvisableto alwaysaskfor preferencedatain apost-experiment questionnaire or interview so as to collect as much data as possible from eachparticipant. If sufficient preference data have been collected, then an interesting additional analysis can be performed to determine whether there is any relationship between the performance dataandthe preferencedata:people donot alwaysprefer the conditionsthat they perform best.

Data CollectionandQualitative Analysis PreferenceData: Participants are often asked to indicate their preferences for conditions indeed,it isadvisableto alwaysaskfor preferencedatain apost-experiment questionnaire or interview so as to collect as much data as possible from eachparticipant. If sufficient preference data have been collected, then an interesting additional analysis can be performed to determine whether there is any relationship between the performance dataandthe preferencedata:people donot alwaysprefer the conditionsthat they perform best. Correlations between preference andaccuracy andbetween preference and response time can be calculated over all the trials. This will reveal whether (or not) participants performed better in the conditions that theypreferred.

Data CollectionandQualitative Analysis PerceptionData: Thisissimilar to collectingpreferencedatain that it focusesoneliciting participants opinions;however, in this case,noexplicit comparisonsare involved, and the questions asked are broader than asking for relative rankingor ratingof conditionsor stimuli.

Data CollectionandQualitative Analysis PerceptionData: Thisissimilar to collectingpreferencedatain that it focusesoneliciting participants opinions;however, in this case,noexplicit comparisonsare involved, and the questions asked are broader than asking for relative rankingor ratingof conditionsor stimuli. Qualitative perception dataallow for awide rangeof issuesto beexplored and for the participants to express their opinions about the experimental tasksandprocessfreely.

Data CollectionandQualitative Analysis PerceptionData: Thisissimilar to collectingpreferencedatain that it focusesoneliciting participants opinions;however, in this case,noexplicit comparisonsare involved, and the questions asked are broader than asking for relative rankingor ratingof conditionsor stimuli. Qualitative perception dataallow for awide rangeof issuesto beexplored and for the participants to express their opinions about the experimental tasksandprocessfreely. For example: What particular aspects of the stimuli did they focus on? Did they notice anything interesting about the stimuli? If so, what features did they notice? Did they devise a particular approach to performing the tasks? Didthey start to gettired or lose concentration at anytime? If so,whenand why?

Data CollectionandQualitative Analysis ProcessData: Processdatacanincludemouseclicks (or anyother input actions), the useof scrolling, zooming and panning features (display manipulations), eye tracking data, etc., and can (should) be stored automatically in a log file that records everything that the participant did with the system, as well as the time of everyaction.

Data CollectionandQualitative Analysis ProcessData: Processdatacanincludemouseclicks (or anyother input actions), the useof scrolling, zooming and panning features (display manipulations), eye tracking data, etc., and can (should) be stored automatically in a log file that records everything that the participant did with the system, as well as the time of everyaction. Collecting information about process produces a vast amount of often unwieldy datathat canbevery time consumingto analyze.It isagoodidea to determine in advance, therefore, what use this process data will have, andhow it might contribute to answeringthe research question.

Data CollectionandQualitative Analysis Product Data: Forsomeexperiments,the datacollectedareartefactscreatedbythe participants (e.g., a design, a diagram, a set of words, a pseudocode algorithm).

Data CollectionandQualitative Analysis Product Data: Forsomeexperiments,the datacollectedareartefactscreatedbythe participants (e.g., a design, a diagram, a set of words, a pseudocode algorithm). In these cases, it may still be the case that the response can be judged as correct or incorrect (givingquantitative performance data), but it isalso usual for these artifacts to be analyzed for other features of interest. These data are typically qualitative, but may be reduced into some quantitative measures.

Data CollectionandQualitative Analysis Conclusion:TheChallengeandtheT emptation It canbetemptingto collect all fivetypesof Pdata performance, preference, perception, process,andproduct to getaswide aview as possibleof the effectsof the different conditions.

Data CollectionandQualitative Analysis Conclusion:TheChallengeandtheT emptation It canbetemptingto collect all fivetypesof Pdata performance, preference, perception, process,andproduct to getaswide aview as possibleof the effectsof the different conditions. However, this approach has its pitfalls: experiments that collect everything require carefulplanningandadministration, maytakealongtime to conduct (longer than is acceptable for a participant to spend on an experiment), and require extensiveanalysisandcross-referencing.

Data CollectionandQualitative Analysis Conclusion:AUsefulApproachto aPracticalSolution-TRIANGULATION In social sciencestudies, triangulation isoften usedto verify results collectedbydifferent methods. If dataarecollectedusing two methods,andthe resultssupport eachother, then the overall conclusionsaremuchmore likelyto beaccepted.

Data CollectionandQualitative Analysis Conclusion:AUsefulApproachto aPracticalSolution-TRIANGULATION In social sciencestudies, triangulation isoften usedto verify results collectedbydifferent methods. If dataarecollectedusing two methods,andthe resultssupport eachother, then the overall conclusionsaremuchmore likelyto beaccepted. If the resultsarecontradictory, then youcannot besurewhichis more likely to becorrect.

Data CollectionandQualitative Analysis Conclusion:AUsefulApproachto aPracticalSolution-TRIANGULATION In social sciencestudies, triangulation isoften usedto verify results collectedbydifferent methods. If dataarecollectedusing two methods,andthe resultssupport eachother, then the overall conclusionsaremuchmore likelyto beaccepted. If the resultsarecontradictory, then youcannot besurewhichis more likely to becorrect. Athird form of data (the third point of the triangle ) will confirm which data to havemore confidence in (or, if all three are contradictory, will indicate that there are someserious problems with the researchquestion, the experiment, or the datacollection). NOTE:Not everyexperiment merits publication!!