Experimental HCI & Interactive Technologies Research
This research delves into defining specific research questions for experiments in the field of Human-Computer Interaction (HCI) and interactive technologies. It emphasizes the importance of formulating questions that can provide yes or no answers, focusing on designing experiments and understanding confounding factors that may affect data reliability. The discussion also touches on research questions as hypotheses and null hypotheses, shedding light on the logic behind statistical analysis of experimental data.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
C S 598 AK Experimental HCI & Interactive Technologies DefiningtheResearch
Definingtheresearch TheResearch Question EVEN THOUGH: Experiments are often (if not always) conducted as parts of larger, more substantial researchprojects,eachparticular experiment must begin with first defining a specific research question (or specific questions). Ausefulheuristic for doingsoisto ensurethat the experiment isstated ashavingthe goalof providing yesor noanswersto research questions phrased (for the purpose of this course, in English) as well formed (grammaticallycorrect) statements endingwith ? s. E.g.,Areour intendedusersmore able,onaverage,to learn how to successfullyconnect aBluetooth deviceto Bluetooth speaker while usinginterfaceAor whileusinginterfaceBfor thespeaker sdesign?
Definingtheresearch TheResearch Question
Definingtheresearch TheResearch Question It is useful now to appreciate that in the vast majority of HCI experiments, no research questions, so as those we have discussed, canbeansweredwith absolute certainty. Why? There will always be unavoidable risks in generalizing conclusions from samples (of subjects,of tasks,etc.)to populations, andthere will alwaysbesomelevelof unpreventable noiseor random error.
Definingtheresearch DesigningExperiments
Definingtheresearch DesigningExperiments Experiments maybeinterrupted, softwaremaycrash,thesunmight be shining on one day and not the next (affecting the perception of colors on ascreen),participants might bein abadmood(andnot willing to givetheir best performance), and/or somemight haveuseful backgroundknowledge not possessedbyothers." Extraneousfactorssuchasthesethat might affectthe reliability of the data arecalledconfoundingfactors: somewill beeasyto identify(e.g.,a computer that worksslower thanexpectedfor someparticipants,the different content in Web page stimuli), and some will not.
Definingtheresearch Research Question as Hypothesis and Null Hypothesis The research question is often best understood in terms of an expected change in the value of a dependent variable (s) as a function of manipulating an independent variable(s). Thus, the null hypothesis is that no expected change occurred. The logic of statistical analysis of experimental data is based on a logic of determining whether enough (sufficient or significant ) evidence exists to reject the null hypothesis.
Hypothesis Testing: An approximation Decision True False Correct acceptance (1- ) Type I Error ( ) True Null hypothesis (no effect) Type II Error ( ) Correct rejection (1- ) False Typically, alpha < beta
Definingtheresearch DesigningExperiments Those aspects of the experimental environment that the experimenter can keep the samethroughout the experiment arecalledexperimentalcontrols. Thus,different factorsof the experiment are(1)deliberatelyvaried (the independent variable); (2)deliberatelykept the same(the experimental controls or control conditions );or (3)accidentallyvaried(the confounding factors).
Definingtheresearch Validity and Control Validity: Establishing that the observed behavior (change in the value of a dependent variable) is caused by the manipulation of the independent variable(s).
Definingtheresearch Validity: Establishing that the observed behavior (change in the value of a dependent variable) is caused by the manipulation of the independent variable(s). - - - - Face validity Construct validity Internal validity External (or Ecological) validity
Definingtheresearch Construct Validity: The degree to which the experimenter: a) manipulated the independent variable they wanted, and b) measured the dependent variable they wanted. Examples: Say we are interested in the effects of fatigue on driving. How should we manipulate fatigue? What behavior should we measure as our dependent variables? These point to the utility of the fatigue construct.
Definingtheresearch Internal Validity: Refers to the situation where the causal or independent variable, and NO other variables, caused the change in the values of the measured dependent variables. Problem: Increasing control and eliminating confounds is the path toward internal validity. However, whether one has achieved perfect internal validity can often never be known with certainty.
Definingtheresearch External or Ecological Validity: The degree to which the observed findings can be generalized to other situations, tasks, settings and subjects. For human subjects (participants), a logic based on random statistical sampling is quite mature. Rarely is the same logic available for experimental settings. Instead, we typically make informal appeals to similarity between experimental and target conditions.
Definingtheresearch Face Validity: Refers to experimental situations or tasks having the same look and feel of a target situation and set of tasks. Arguably less important than other forms of validity Yet, can influence a human subject s acceptance of an experiment or a sponsor s acceptance of results.
Definingtheresearch Generalizability HCIexperimentsareall assumedto haveatarget population in mind. Experiments typically use a set of experimental participants (the sample ), collect responsesto theideafromtheseparticipants,andmake generalizations from theseresponses(i.e.,speculateabout how the wider population would respond).
Definingtheresearch Generalizability HCIexperimentsareall assumedto haveatarget population in mind. Experiments typically use a set of experimental participants (the sample ), collect responsesto theideafromtheseparticipants,andmake generalizations from theseresponses(i.e.,speculateabout how the wider population would respond). At oneextreme, asampleof just onepersonwill produce resultsthat cannot reasonably be generalized to the population; at the other extreme, a large sample that includesallmembersof the population iscomplete,anddoes not needto begeneralized.Thelarger andthe more diversethe sample of peopleused, the more likelyit is that the generalizationwill bevalid.
Definingtheresearch Generalizability It is useful to consider other relevant generalizations, in particular, what the participants are askedto do (the experimental tasks) and the meansbywhichthe abstract ideaisinstantiated in aconcreteform (theexperimental objects). If we ask our participants to perform only one task using the new HCI idea,thentheresults canonlyberepresentativeof performancewith that one task. And, as with the previous sample/population difference, it is typicallyimpossibleto askthe participants to perform everypossible task(apart from whentesting very limited systemsor ideas).
Definingtheresearch) Generalizability Just aswehaveapopulation of people(fromwhichweselect asample), wealsotherefore haveapopulation of possibletasks(from whichweselecta sample) andapopulation of experimentalobjects(from whichwealsoselect asample).
Definingtheresearch Generalizability Just aswehaveapopulation of people(fromwhichweselect asample), wealsotherefore haveapopulation of possibletasks(from whichweselecta sample) andapopulation of experimentalobjects(from whichwealsoselect asample). Inthe samewaythat weaggregateresultsover people(to generalize to the humanpopulation),wecanalsoaggregateresultsover sampletasks andsampleexperimentalobjectsto generalizethe resultsto applyto more than onetype of taskandmore than onetype of experimentalobject. Wecantherefore speculateasto what resultswould havebeenobtained if awider rangeof tasksor experimentalobjectshadbeenused.
Definingtheresearch Experimental Objects The HCI idea that is being tested, as well as the alternative conditions, needto beembodiedin aperceivableexperimentalobject: this isthe way in which the idea is presented to the participants in a concrete manner, rather thanasanabstract idea.
Definingtheresearch Experimental Objects The HCI idea that is being tested, as well as the alternative conditions, needto beembodiedin aperceivableexperimentalobject: this isthe way in which the idea is presented to the participants in a concrete manner, rather thanasanabstract idea. Experimental objects are the concrete means by which a participant can experiencethe HCIideaand its alternatives. If only oneexperimentalobject is used in the experiment, then the results of the experiment will only pertain to that particular experimentalobject.
Definingtheresearch Experimental Objects The HCI idea that is being tested, as well as the alternative conditions, needto beembodiedin aperceivableexperimentalobject: this isthe way in which the idea is presented to the participants in a concrete manner, rather thanasanabstract idea. Experimental objects are the concrete means by which a participant can experiencethe HCIideaand its alternatives. If only oneexperimentalobject is used in the experiment, then the results of the experiment will only pertain to that particular experimentalobject. OFTEN,in my experience, achieving reliable experimental results or good HCIdesignrequires aqualitative researchphase(ethnography, etc.) to be doneprior to experimentationto identify, selector designtheseobjects.
Definingtheresearch Experimental Objects
Definingtheresearch Experimental Objects
Definingtheresearch(Part 1of 2) Experimental Objects Case Study: A major customer (left) of a major Point-of-Sale (POS) research and technology company (right) asked for our help in designing a next generation of POS terminals with a focus on database and interface design targeted specifically on the special needs and roles played by on-floor sales staff andin particular, to put the customer into anewpair of shoestoday. Becausewewantedto useexperienced andknowledgeableemployeesinour userstudiesand experiments(to ensure immediately positive user acceptance and performance upon system implementation), we first conducted focus- groupsizeinterviews and discussionswith them, and observedthem in action.
Experimental Objects Astriking initial finding wasthat casesin which their inventory (the huge room of shoeboxes behind the door contained the exact shoebrought to them bythe customertook uponly atrivial fraction of their time.
Experimental Objects Astriking initial finding wasthat casesin which their inventory (the huge room of shoeboxes behind the door contained the exact shoebrought to them bythe customertook uponly atrivial fraction of their time. Close to 90% of their time and energy went into efforts to still sell the customerapair of shoeswhenthe exactshoesinitially requested were not ininventory, soworkersneeded to thenfinda closest match (or matches)to shoesthey did havein immediate inventory.
Experimental Objects Astriking initial finding wasthat casesin which their inventory (the huge room of shoeboxes behind the door contained the exact shoebrought to them bythe customertook uponly atrivial fraction of their time. Close to 90% of their time and energy went into efforts to still sell the customerapair of shoeswhenthe exactshoesinitially requested were not ininventory, soworkersneeded to thenfinda closest match (or matches)to shoesthey did havein immediate inventory. Appreciating that the organization of our databaseand interface design would need to be compatible with their strategies for finding suchnear matches,weexaminedhow theseworkers hadlearned to bestorganize their shoeinventory to helpthem readily find theseclosematches.
Experimental Objects Thinkof the shoeinventory in terms of anN-dimensionalspace,with the (most obvious) dimensionsof: 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??) Questions:Givenany2adjoiningpair of shoeboxeschosenat random, onwhichof the above dimensionswerethey closest?Most distant?
Experimental Objects Answer#1:Most distant: Ladies vs Mens Separatelocationsin stores. 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??) Questions:Givenany2adjoiningpair of shoeboxeschosenat random, onwhichof the above dimensionswerethey closest?Most distant?
Experimental Objects Q2.Answer:Surprise!The closestdimension differed basedonAnswer#1 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??) So,NewQuestions:What wasthe closestdimension for Mensshoes,and what wasthe closestdimension for Ladiesshoes?
Experimental Objects Answerfor Mensshoesclosestdimension:(5)Style 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??)
Experimental Objects Answerfor Womenshoesclosestdimension:(3)????????????????????? 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??)
Experimental Objects Answerfor Womenshoesclosestdimension:(3) Color! 1. Ladiesshoes versus Mens shoes in their parlance. 2. Brand 3. Color 4. Price 5. Style 6. Size 7. Veganor Not (just kidding,but learnedthat Zapposhasthis searchoption) 8. Anythingelseyou canthinkof (??)
Experimental Objects Aside from the requirement that each condition of the independent variable must be able to be represented within each experimental object and must provide the samefunctionality (seeSection2.2),experimentalobjectsneedto bechosencarefully. Althoughit istrue that havingexperimentalobjectsof adiffering nature permits generalizability of theresults to awider rangeof objects,they shouldnot betoo different from eachother acarefulbalance needsto bemaintained:
Experimental Objects If the experimental objects are too different from eachother, then it will be inappropriate to aggregate the data over them, and the data will need to be analyzedseparatelyfor eachexperimentalobject.
Experimental Objects If the experimental objects are too different from eachother, then it will be inappropriate to aggregate the data over them, and the data will need to be analyzedseparatelyfor eachexperimentalobject. For example, the performance data for using a mobile device for navigating one floor of a building may need to be analyzed separately from the data obtained when the device is used for navigating a large park, simply because the two environments aresodifferent that they bearnorelation to eachother, andbecauseany conclusionsmade about the floor navigation areunlikely to alsobeapplicableto thepark.
Experimental Objects If the experimental objects are too different from eachother, then it will be inappropriate to aggregate the data over them, and the data will need to be analyzedseparatelyfor eachexperimentalobject. For example, the performance data for using a mobile device for navigating one floor of a building may need to be analyzed separately from the data obtained when the device is used for navigating a large park, simply because the two environments aresodifferent that they bearnorelation to eachother, andbecauseany conclusionsmade about the floor navigation areunlikely to alsobeapplicableto thepark. HEURISTIC:Thinkof theseobjectsthe wayyouthink about subjects:youwant heterogeneity to ensure generalization from the sample to the population, yet too muchheterogeneity limits your ability to aggregatedataacrossthem.
Experimental T asks The definition of the experimental tasks is often one of the most difficult aspectsof experimentaldesign,especially if acognitiveactivity (e.g.,analysis of ascenario,comprehensionof adiagram)is involved.
Experimental T asks The definition of the experimental tasks is often one of the most difficult aspectsof experimentaldesign,especially if acognitiveactivity (e.g.,analysis of ascenario,comprehensionof adiagram)is involved. T asksfor more physicalactivities areeasier:participants canbeaskedto select objects on a screen using a cursor, or wave a remote control device in the correct manner soasto turn onscreenpages.
Experimental T asks The definition of the experimental tasks is often one of the most difficult aspectsof experimentaldesign,especially if acognitiveactivity (e.g.,analysis of ascenario,comprehensionof adiagram)is involved. T asksfor more physicalactivities areeasier:participants canbeaskedto select objects on a screen using a cursor, or wave a remote control device in the correct manner soasto turn onscreenpages. If weareinterestedin cognitiveactivities, then the tasksaremore difficult to define. It is important to remember that we cannot ever measure what is actually going on in a participant s mind, we can only measure observable behavior; therefore, we need to choose tasks that will produce behaviors that appropriatelyrelate to the cognitiveactivity in whichweareinterested.
Experimental T asks T asksMust:
Natureof thedomain Achoiceneedsto bemadeasto whether realworld dataor scenarioswill be used, or whether information will be fabricated for the purposes of the experiment.
Natureof thedomain Achoiceneedsto bemadeasto whether realworld dataor scenarioswill be used, or whether information will be fabricated for the purposes of the experiment. Using a real world domain is good if it is possible, but often doing so makes it more difficult to identify appropriate experimental objects, choose tasksthat havearangeof possibleanswers, andphrasethe tasksin terms of thesemanticsof thedomain.
Natureof thedomain Achoiceneedsto bemadeasto whether realworld dataor scenarioswill be used, or whether information will be fabricated for the purposes of the experiment. Using a real world domain is good if it is possible, but often doing so makes it more difficult to identify appropriate experimental objects, choose tasksthat havearangeof possibleanswers, andphrasethe tasksin terms of thesemanticsof thedomain. Sometimes it may be appropriate to use information from a real domain, but adapt it to make it appropriate for the experiment. [For experimental convenienceof onetype or another, or to makeits source anonymous, etc.]
Definingtheresearch Summary Thischapteraddressedthe decisions that needto bemadewhendesigning anexperiment... Takingtime over thesedecisions is crucialfor the successof your test of the HCIidea youwill not regret spendingtime on thinking thingsthrough very carefully in advance! It is important to note that, with sucharangeof decisionsto bemade,it iseasyto (consciously or otherwise) makedecisionsthat arebiased towards a desiredresult.
