Initial Thoughts on Social Inquiry Methods for Game Design

Methods of Social Inquiry for Game Design: Some Initial Thoughts Elizabeth Ellie Bartels Senior Associate, Caerus Associates ebartels@caerusassociates.com

Overview Purpose and Motivation Current Design Process and Critique Social Science and Game Design Best Practices for Game Design Broader Considerations 2

Key Research Questions How can we build games to facilitate meeting our objectives? How can we assess the validity of findings from games? How can sponsors and clients use these tools to understand findings of the games we build for them? 3

Goal of Improving Game Design Methods Reality of games is that they will always be imperfect instruments Witches of Wargaming Logistical constraints Huge strength of gaming is flexibility, cookie cutter approach not the answer Point is not to say anything short of a perfectly designed game is unacceptable, its to be able to talk about how these limitations on design limit our findings Set expectations before the game Defend findings after the game 4

Standard Game Design Process Game Structure, Scenario, Rules, and Roles Development Game Execution Post Game Analysis Objectives Research 5

On Game Design (Chap 5 Art of Wargaming) Focus on crafting game objectives based on sponsor requirements How do you translate broad objective (ex. Understand XX phenomenon ) into concrete focus areas? World building more art than science, no real formalisms available How do you decide which world is most useful to build? Define Information Requirements, all necessary information and no unnecessary information How do I know what is necessary? What is the potential cost of getting it wrong? 6

Question of Structure Structured Problem Often talk about desire to increase game structure, but we mean two different things: Wastes of Time Mods & Sims Structured Methods ask standardized questions that guide and standardize data collection, making systematic comparisons possible Structured Method Unstructured Method Matrix Gaming Reporting Structured Problems have clearly defined boundary, highlighting specific variables and strong hypothesis about their causal relationship Unstructured Problem 7

How to Improve Structure Structured Problem Where We Talk about Going Unstructured Method Structured Method Most Current Gaming Where We Need to Go Unstructured Problem 8

Proposed Game Design Process Based on Social Science Research Methods Objectives Define mental model/hypothesis/theory Strategy for instantiating the model Method of representation (Game Structure, Scenario, Rules, and Roles) Game execution Post-game analysis 9

Gaming in Social Science Gaming is used by some social scientists, but generally as a pedagogical tool (ex. Asal and Brynen) or as part of experimental treatment (ex. McDerrmott) Use is limited by perceived lack of rigor Game theory related, but often too simplified to do more than point to abstract types of systems (ex. Axelrod) Very limited published work situating gaming among other techniques at a theoretical level 10

Social Science in Gaming Social science is used by some gamers, as a supplement to more standard gaming techniques (ex. Wong and Cobb, Brightman) Use of social science often linked to study of Irregular Warfare issues Limited published work on applying social science thinking to overarching game design (ex. McCown, and Bartels, McCown, and Wilkie) 11

Introduction to Social Science Approaches Limiting to neopositivist approaches Three categories: Formal Models: Creation of theorized mechanism in artificial environment that can be compared to real world events Statistics: Measuring causal effect through correlation between large numbers of examples using quantitative measurements Case Studies: Measuring causal mechanism through tracing sequence of events across small number of rich qualitative examples All require: Developing a logically consistent model, deriving observable implications, testing those observables in the real world, and using results of test to infer about world more broadly 12

Games and Case Studies Case studies a promising, but relatively untapped methodology for structure research of problem at a range of different levels of structure Appropriate to exploring complex causal relationships through structured, but flexible, approach Methodology supports not only theory testing, but also the emergence of new theories and variables over the course of research Primary Reference: Alexander George and Andrew Bennett, Case Studies and Theory Development in the Social Sciences, MIT Press, 2005 13

Logic of Case Study Research Case(s) should be examples of a class or sub-class of phenomenon Historical events are usually examples of multiple classes, so we need to be specific about what lens we are using Method for structured, focused analysis: Structured: Questions, based on objective, are asked of each case to guide and standardize data collection, which makes systematic comparison possible Focused: Motivated by a specific research question Both can be harder to achieve in collaborative environment, important to ensure all parties share understanding 14

Logic of Case Study Research for Games Games can most often be thought of a single case study, where the comparison is: Shown by variation over time (process tracing) Counter factual analysis to posit a control Can get stronger findings when there are clear comparisons between the game and another case, but not as common Historical reality Paired Games 15

Best Practices for the Research Design Process Case Study Term Problem and Research Question Research Strategy and Specification of Variables Case Selection Describe Variance in Key Variables Data Requirements Gaming Equivalent Purpose and Objectives Game Concept Scenario Setting Defining Scenario, Roles, and Rules Data Collection Plan 16

Best Practices for Objectives Case study objectives derive from research problem, just as game objectives derive from sponsor problem Central problem can be well structured or poorly structured, which will shape objectives 6 broad types of objectives of case studies, which can also apply to games: Descriptive Pattern detection Identify variables, hypotheses, causal mechanism, and paths Plausibility probes of new hypotheses Testing validity and scope of theory Explain case with existing theory 17

Best Practices for Game Concept: Research Strategy Hypothesis: What is the current understanding of what is occurring Because games are often on un- or poorly-structured problems, often these are messier in games than in social science Gamers frequently skip this step, but this is what drives all future decisions about what is, and is not, important to include in the game Critical when linking games to other analysis, as need to ensure consistent definitions and assumptions are in place to make compatible In single cases (which include most games) important to consider all available alternative hypotheses, including those suggested by participants, to ensure left out options don t threaten validity 18

Best Practices for Game Concept: Variable Definition Definition of variables: Outcome to be explained or predicted (dependent variable) Inputs and intervening variables that might cause outcome Which variables are we interested in variation, and which should be constant Confounding variables are factors that might produce the outcome being studied but are not the input of interest. Ideally, these are constant across the case, but often need to show through process tracing that while present, they don t appear at the right time point in events to create the outcome 19

Best Practices for Game Concept: Decisions as a Key Variable Games by definition have player decisions as a key variable, but depending on objectives they are different types of variables Independent: Participant decisions produce what outcomes? These games are focused on the product of decision Dependent: What decisions are made by participant given specific inputs? These games are focused on the process of decision making 20

Best Practices for Game Concept: Other Key Variables Other variables in games: Environmental factors included (or excluded) in the scenario Actors represented (or not) by the players Rule formalized (or not) in the rules When possible, define variables clearly ahead of game play: Outcome to be explained/predicted (Dependent) Variables that contribute to the outcome (Independent) Constants 21

Best Practices for Scenario Setting First criteria should be relevance of case to research question Case/setting must be an example of a class of events being studied Cuban missile crisis can be used as a case of deterrence, or coercive diplomacy, crisis management, etc. Studying most and least likely cases, or deviant cases more likely to yield interpretable results Most likely: favorable case, possible to strongly undermine hypothesis Least likely: unfavorable case, possible to generate convincing support for hypothesis Good tests are often unrepresentative, as a result it is important not to overgeneralize Often case is starting point, rather than theory, but still requires clear objectives and research strategy (including clearly stated hypothesis) 22

Best Practices for Defining Scenario, Roles, and Rules Understanding range of variation that can occur is important for specifying what causal relationships are illustrated Think carefully about what variation you need in the case Cases with clear variances are more important for theory testing Limits in variation considered by designer is based on habit as well as practical constraints Environment limited by time and data available Player roles limited by available participants, number of cells by space Rules limited by staff available to support communication and adjudication Critical to be honest about how these choices restrict game findings 23

Best Practices for Defining Scenario, Roles, and Rules: Variation in between Game and Reality has a Decisions are Independent Variable/Process Games Decisions are Dependent Variable/Product Games Roles Medium Impact on Validity Scope (who is involved in decision making) Level of analysis (Essence of Decisions) Medium Impact on Validity Scope (who is involved in decision making) Level of analysis (Essence of Decisions) Environment Small Impact on Validity Information availability Large Impact on Validity Problem type Timelines Information availability Context Rules Large Impact on Validity Planning paradigms Information sharing Small Impact on Validity Information sharing 24

Best Practices for Defining Scenario, Roles, and Rules Critical to be honest about how these choices restrict game findings Internal Validity: how strong a claim can you make about causality External Validity: how much can you generalize the findings of the game to real world cases Common problems: Elimination of alternative explanations for the outcome Failure to consider alternative hypothesis Limited applicability to other cases 25

Best Practices for Data Collection Plan Consistent, thoughtful data collection plan provides the structure needed to support systematic analysis Generalized questions to be asked: Not a mechanical process, questions must be integrated with other decisions made during research design Does not preclude collecting more data Better defined research design supports more precise data collection 26

Consequences for Games as Part of Larger Studies Focus on good design can make it easier to link games to other research projects Allows better variable matching by making sure consistent variables are actually comparable across studies Often there s a desire to move from unstructured to more structured problems, or from the specifics of a particular empirical case to findings that have a more general scope, this method highlights those opportunities 27

Abductive, Inductive, or Deductive? We are only able to undertake the process of simplifying the complexity of the world into the artificial form of game by making choices about what to include or exclude We make these choices based on our existing hypothesis or mental models of the problem set We as designers have a theory, and use that theory to shape the game world deductively, we just generally don t acknowledge it Deductive design method forms boundary conditions for the abductive or inductive analytic outputs of games 28