Insights on Experimental Methods in Accounting with an Economics Emphasis

Experimental Methods in Accounting: An Economics Emphasis Independent study in experimental accounting and economics* *Adapted from Joan Luft, by Rick Young

What is the purpose of an experiment? To test theory under highly controlled conditions, and thus make it easier to decide among alternative explanations for observed phenomena Establishing Causality is, after all, the goal Real world data, sometimes referred to as archival data, does not always support decisive tests of plausible competing theories 2

Caveat External Validity is often called into question in lab settings Is Internal Validity a necessary condition to establish causality? IMO, yes. Is Internal Validity a necessary condition for External Validity? IMO, yes. 3

Critiques of experiments should be based on . . . Quality of theory on which experiment is based Quality of operationalization of theory Quality of experimental control and analysis Not on: whether the theory is supported Unfortunately, not how publishing usually works 4

Critiques of experiments often are based on . . . Observations of differences between experimental setting and real world. Such criticism may be (a) devastating, or (b) vacuous 5

On the Idea of Re-creating the real world Someone asked an American poet: Should literature try to re-create the real world? Answer: No. One of the damned thing is enough. 6

Which features of the real accounting and decision environment must be captured in experiments? Context matters Framing! 7

How do we know what does matter? Which factors should be present, manipulated, measured, controlled for . . . .? Answer should be theory-based 8

Experimental Research in Accounting Research Draws on Two Traditions Experimental economics Experimental psychology Different traditions of theory and control but both stress value of abstraction for control purposes. 9

Common errors to avoid . . . Don t work with underspecified, vague, or nonexistent theories and try to generalize anyway by applying findings directly. Swieringa and Weick, JAR 1983 Similarly Brown, Evans and Moser, JMAR 2009 10

Examples of Things to Avoid The purpose of an experiment is to test theory. Statements like this are not theories: More accurate information is better. Teamwork is better. Fair reward systems are better. 11

A study is unrealistic if . . . . Nineteen-year olds interact via computer networks Small sums of money are involved Individuals, not institutions, make the decisions Assumes a one-period world 12

How to Respond Does the experiment misrepresent the underlying theory if it uses nineteen-year olds? Unlikely for most economic theories Pays only small sums of money? Same answer Does the theory capture important aspects of the real-world phenomenon? It better! Does it misrepresent important real-world phenomena? It better not! 13

Theories . . . Economics A theory of the domain Other social sciences don t tell us much about production functions, cost functions, market structures. Psychology A theory of the people in the domain Economics does not have very plausible theories of cognition or preferences. 14

Experimental Control Experimental Economics: control through specification If it s not in the theory, find ways of keeping it out of the lab! Experimental Psychology: control through comparisons If it s not in the theory, find ways of holding it constant (or randomizing its effects away) Diff in diff studies: Schwartz et al. 2008 Or using ad hoc control variables common approach in archival research, too 15

Experimental Economics Start with a model that specifies (e.g.) Actions Outcomes & probability distributions of outcomes conditional on actions Payoffs Utility functions Information & communication structure Resulting equilibrium 16

Control through specification If it s in the model, operationalize it in the lab! Induce utility functions in model (sometimes) If it s not in the model keep it out of the lab! Keep task and information abstract (mundane realism) 17

Example: Concrete vs. Abstract Economic theories framing : U = U(w, a), w = wealth and a = effort, ?U/?w > 0, ?U/?a < 0 Gift Exchange expeirments use monetary payoffs to represent both w and a. Is this a problem? That is, U(w-c(a)) is used Operationalize the math, not the words. 18

A few problems . . . Most models have simple utility functions, U = U(w,a). People bring other preferences to the lab with them. Experimental economics solution: be sure monetary payoffs dominate other considerations. Theory of revealed preferences (Vernon Smith) 19

How easy (possible) is it to achieve dominance of payoffs in the lab? Depends on context. What game do people think they re playing? This probably matters in both lab and real world. 20

A few problems, cont. . . . We want to test interesting models Interesting models often have Surprising (unintuitive) solutions, or Solutions that are too hard to work out intuitively 21

Implications of these problems People in the lab will not (at first) do what interesting models say they will do. They might do so, eventually, with appropriate incentives and learning opportunities, but . . . 22

Lab tests often acknowledge difficulty of task Allow appropriate time and practice to figure it out . Often dealt with by using multiple trials, even though model is one-period. Data analysis attempts to account for the learning curve , but it is ad hoc since difficulty of task is outside the model. 23

Experiments, models, and the real world Empirical research outside the lab (archival studies) define important problems, documents prevailing practices, and provides limited evidence for or against theory. Analytical modeling develops theories about how key variables affect each other. Experimental research tests (competing) theories under highly controlled conditions. 24