Independent Evaluation Insights and Solutions

Independent Evaluation: Insights from Public Accounting Abigail Brown (Harvard) Jacob Alex Klerman (Abt Associates) Presentation for Improving the Quality of Public Services, Moscow, June 2011

This Paper/Presentation Reviews the history, theory, and experience in independent audits for insights about independent evaluations Our thesis: In both fields The disjunction between who oversees the audit/evaluation and the ultimate consumers of the results Leads to problematic incentives for the auditor/evaluator and bad outcomes But that recognizing the disjunction and the experience in auditing suggest several feasible changes that would improve the situation Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 2 2

Outline The Auditing Problem(s) Some Auditing Solutions The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 3 3

Owners and Managers Owners Provide the capital Managers Managers run the firm day to day Brown & Klerman May 9, 2011 4

But Who Watches the Mangers? Owners are too busy to watch well But need to watch Malfeasance Insufficient effort Hire/Fire Incentive compensation (bonuses, stock options) Owners Managers Brown & Klerman May 9, 2011 5

Enter Independent Auditor Paid by owners (corporate funds) Origins in 1840s England Required by 1934 (U.S.) Securities Exchange Act Owners Auditor To watch (and report on) managers Managers Brown & Klerman May 9, 2011 6

But Why Should this Work? Managers Choose auditors this time and next time and how much to pay them Oversee auditor s contract, including scope Can give auditors other (non- auditing/consulting) work So why should auditors report truthfully to owners? Why not just report what managers want reported? Paid by owners (corporate funds) Owners Auditor To watch (and report on) managers Managers Brown & Klerman May 9, 2011 7

Outline The Auditing Problem(s) Some Auditing Solutions The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 8 8

Four Partial Solutions Professionalism Reputation Liability Owner Oversight Brown & Klerman May 9, 2011 9

AICPA Code of Conduct Professionalism Reputation Liability Owner Oversight A distinguishing mark of a profession is acceptance of its responsibility to the public. In discharging their professional responsibilities, members may encounter conflicting pressures from among each of those groups. In resolving those conflicts, members should act with integrity, guided by the precept that when members fulfill their responsibility to the public, clients' and employers' interests are best served. .. Those who rely on certified public accountants expect them to discharge their responsibilities with integrity, objectivity, due professional care, and a genuine interest in serving the public. Brown & Klerman May 9, 2011 10

A Simple Equilibrium Argument Professionalism Reputation Liability Owner Oversight Suppose there are two types of auditors independent and not independent The auditor selected by management is publicly observable What would owners conclude if management chose a not independent auditor? So management has to choose an independent auditor Brown & Klerman May 9, 2011 11

Sue Them Professionalism Reputation Liability Owner Oversight Auditors can be sued for errors Have been large payouts Should be a strong financial incentive for independence But, liability requires Clear standards for malfeasance which the profession has tried hard to avoid Strong liability which the profession has successfully weakened Brown & Klerman May 9, 2011 12

SOX/Sarbanes-Oxley Act of 2002 Professionalism Reputation Liability Owner Oversight If the problem is too little owner (too much management) involvement in the audit, fix that directly! SOX requires that an Independent Audit Committee of BoD/Board of Directors (i.e., excluding management) Hire the auditor Oversee the audit Approve other business with the auditor (to prevent side payments) Brown & Klerman May 9, 2011 13

Outline The Auditing Problem(s) Some Auditing Solutions The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 14 14

Evaluation is Like Auditing Evaluation Auditing Government agency Ideally wants both the appearanceof independent evaluation and the ability to claim that the program works Corporation audit the books are correct/ we are making profits What are the implications of experience in independent auditing for independent evaluation ? Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 15 15

Applying these Strategies to Evaluation Professionalism Reputation Liability Owner Oversight Brown & Klerman May 9, 2011 16

Same Problems as in Auditing Professionalism Reputation Liability Owner Oversight Also problematic in evaluation Evaluation is also a business Stakes are high: own employment and bonus Perhaps professionalism more effective in evaluation Public release of results and methods Results published in academic journals Results reviewed and used by academics Brown & Klerman May 9, 2011 17

Does Reputation Matter? Professionalism Reputation Liability Owner Oversight To some extent But clients also seem to value (perhaps value more) a reputation for Client service Not revealing client s secrets Not publishing without permission Brown & Klerman May 9, 2011 18

Liability Does Not Seem Operative Professionalism Reputation Liability Owner Oversight There does not seem to be any statutory basis on which to sue evaluators for failure to be independent Who would sue? For what? What are the damages? Ideas for making this work? Brown & Klerman May 9, 2011 19

0wnerOversight Seems Promising Professionalism Reputation Liability Owner Oversight Currently, agency Selects the evaluator Oversees the evaluation Reviews the report and any presentations Up to and including (indefinitely) preventing publication and presentation For evaluation seems easier to change By analogy with SOX Audit Committee reforms Brown & Klerman May 9, 2011 20

By Analogy with SOX, Audit Committee Expand membership of TWG/Technical Working Group to include Congress (e.g., CBO, GAO) Higher executive (e.g., OMB, agency s Inspector General, agency s PA&E) General policy community (academics, think tanks from both sides) Change role of TWG Select the evaluator Oversee the evaluation Review the report and any presentations Decide what and when to release reports Professionalism Reputation Liability Owner Oversight Brown & Klerman May 9, 2011 21

Encourage Independence by Professionalism Reputation Liability Owner Oversight Agency comments on, but need not approve, materials going to TWG Remove restrictions on evaluator contact with the TWG Encourage reporting of attempts by agency to subvert independence of the evaluation Brown & Klerman May 9, 2011 22

Outline The Auditing Problem(s) Some Auditing Solutions The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 23 23

Owner Oversight in Practice Explicitly specify which tasks are independent evaluations Sometimes in statute specific evaluations, certain evaluation funds Sometimes by programs wanting strong evidence of effectiveness Define independent evaluation in the FAR/Federal Acquisition Regulations; i.e., A TWG that includes key consumers, and Rules for how the TWG would operate Limit other business contact between independent evaluators and agencies (to prevent side payments ) Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 24 24

Plus, Facilitate Market for Reputation Publicize deviations attempts to subvert independence By the agency, and By the evaluator Distinguish independent evaluations from others In making funding decisions (e.g., official policy reviews by NAS/IOM) At conference presentations and at journal publication Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 25 25

Outline Introduction The Auditing Problem(s) Some Auditing Solutions The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 26 26

Independent Evaluation: Insights from Public Accounting Abigail Brown (Harvard) Jacob Alex Klerman (Abt Associates) Presentation for Improving the Quality of Public Services, Moscow, June 2011

Evaluation is Like Auditing Evaluation Auditing Government agency Ideally wants both the appearanceof independent evaluation and the ability to claim that the program works Corporation audit the books are correct/ we are making profits Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 28 28

Owners and Managers Owners Provide the capital For most of human history, these were the same. Managers Managers run the firm day to day Brown & Klerman May 9, 2011 29

Owners and Managers Separation of owners (i.e., stockholders) and control (i.e., management) is a cornerstone of the modern corporation Allowing: Poor entrepreneurs to start companies/implement their ideas Owners to invest all of their money and to diversify Owners Managers Brown & Klerman May 9, 2011 30

And the Issues Are Subtle Pure fraud is clear, but How much to spend looking? Where to look? Management often has a desired spin ; owners may want something else GE s earnings smoothing Lehman s Repo 105 Good accounting requires judgment When judgment is required, which way does the accountant lean? Paid by owners (corporate funds) Owners Auditor To watch (and report on) managers Managers Brown & Klerman May 9, 2011 31

But Professionalism Reputation Liability Owner Oversight Their own private (financial) interests are large Continued employment, repeat audit business, consulting business Auditing s professional ethos emphasizes confidentiality and client (i.e., management) service Not independence (and serving owners) Brown & Klerman May 9, 2011 32

Deviation Should be Costly Professionalism Reputation Liability Owner Oversight Now suppose management suggests creative accounting Perhaps with some side payment to the auditor (future auditing, consulting work, etc.) Auditor weighs Side payment vs. Loss of reputation as an independent auditor if side payment is discovered Brown & Klerman May 9, 2011 33

Does this Reputation Work? Professionalism Reputation Liability Owner Oversight Yes: Arthur Anderson collapsed after criminal conviction in relation to Enron scandal No: Other accounting scandals did not cause similar collapse (Arthur Anderson was the exception) Some evidence of collusion among firms (rather than competition on independence ) Too few to fail (currently only four large accounting firms) Brown & Klerman May 9, 2011 34

Will SOX Work? Professionalism Reputation Liability Owner Oversight Duh! Wasn t this obvious? This was best practice before SOX; but inconsistently applied Why not? And what does that suggest? Management often controls the BoD Management often suggests BoD members Who are not independent relatives, interlocking BoD, other business relationships And board membership is lucrative, so members want to stay in management s good graces Brown & Klerman May 9, 2011 35

Evaluation is Like Auditing Evaluation Auditing Government agency Ideally wants both the appearanceof independent evaluation and the ability to claim that the program works Corporation audit the books are correct/ we are making profits Brown & Klerman Brown & Klerman May 9, 2011 May 9, 2011 36 36