Unlocking Research Questions and Contributions in Public Management

Session 2 Research Question and Lit Reviews PUAD 5319 wwehde@ttu.edu Office: 14 Holden Hall, hours 3:30 5:00 pm Wednesdays

Clearly explain what the research question is Why it is important, and what your contribution is going to be - contribution to existing knowledge - implication for public management, policy making, or urban planning etc. Assignment 1: Problem Statement You want to clearly state your unit of analysis. DUE: September 10th

In this Session: Review good research questions Discuss Christensen Ethics Literature reviews Start? Conceptual Frameworks

What we skipped Your boss asks you to provide the best possible evidence on the following question (this is a real priority questions for SBA): How does providing application assistance impact equity in access to small business COVID-19 relief funds? Take 5 minutes to write down how you would design a study to address the question and provide as much detail as possible.

Evaluating your Research Topic Question Does providing application assistance expand access to small business COVID-19 relief funding? Possible Answers No (why?) Yes (why?) Dependent (on?) Possible Theoretical Reasons (literature can be your guide here): No, because small businesses from underserved communities didn t even know about the program. Dependent on whether city did outreach Yes, but only for Veterans and Minority-owned businesses, not Women

A general research question can be further broken down into several sub-problems. It makes your research question more specific. Phase 2: Specifying Research Questions Three main concerns that help the researcher narrow down the problem to researchable questions: 1. What entities (individual, people, groups, formal organizations, nations) are to be studied Unit of analysis 2. What aspects or characteristics of these entities/phenomena are of interest Variables 3. What kinds of relationships among the characteristics are anticipated Relationships between variables

What makes for a good research question? 1) 2) 3) 4) 5) 6) 7) It passes the so what test It is stated in researchable terms. It is answerable with empirical evidence It helps us understand the way the world works It is feasible (i.e. data is available & you can measure the outcome) It has an appropriate unit of analysis It is narrow enough for a class paper Does the question tackle a problem that is too large for one research paper? (e.g. What are the determinants of social mobility?)

Discuss Christensen Article

Ethical Issues in Social Science Research

What to NOT DO San Antonio contraceptive study: In 1971, an oral contraceptive study was conducted on 70 poor Mexican-American women to evaluate the efficacy of different kinds of female contraceptive pills. They agreed to participate in a study to determine the side-effects of contraceptives. The randomized half received oral contraceptives and the others a placebo. They were not informed that they were subjects of this kind of research or that they might receive inactive medication. As expected, there were high numbers of unplanned pregnancies in the placebo group; ten of the 76 participants became pregnant during the study.

What to NOT DO RETRACTED: "I am not alone we are all alone: Using masturbation as an ethnographic method in research on shota subculture in Japan This year The author has now explained that the work described in this note was carried out as a piece of independent research in Germany, without institutional ethical oversight. The author stated that they followed professional ethical guidelines for their discipline, including the primary injunction to do no harm The Editors are mindful of arguments that moral offense generated by the publication of the note provides problematic grounds for the censoring of scholarship. However, the Editorial Team recognises the argument that the note legitimizes sexual activity involving sexually graphic illustrated images of children and young people, both as an activity in itself and as a research method.

Ethical Issues in Research Obligations to the scientific research community: Honesty and accuracy Obligations to participants in your research (IRB process) -Voluntary participation and informed consent -Anonymity and confidentiality -No harm to the participants Value-free research? 12

Ethical Obligations to the Scientific Research Community: Honesty and Accuracy Scientific progress is based on mutual trust among investigators who build their ideas and research on the work of one another. Therefore, dishonesty and inaccuracy in reporting and conducting research undermine science itself. - Use Turnitin (or whatever it is called here) for assignments in this class - Many journals require the researchers to provide their data and the codes/programs they use to analyze the data. Research should be replicable. 13

Ethical obligations to participants: Informed consent and voluntary participation It is conceivable that research participants may be harmed physically, personally (e.g., by being humiliated or embarrassed), psychologically (e.g., by losing their self-esteem), and socially (e.g., by losing their trust in others). Subjects informed consent is obtained by making it clear to them that their participation is voluntary, and by providing them with enough information about the research so that they can make an informed decision about whether to participate. 14

Ethical obligations to participants: Anonymity and confidentiality The right to privacy typically is secured in surveys by promising anonymity, which is provided by having respondents fill out unmarked questionnaires, or confidentiality, which is provided by removing identifiers from the data, not disclosing individual identities in research reports, and not divulging individual information without the respondent s permission. This right is protected in field research by altering reports of field studies so as to prevent recognition, and by asking the research participants themselves if the material is objectionable. 15

Ethical obligations to participants: Protection from harm -Informing subjects about reasonable or foreseeable risks; -Assessing the impact of potentially harmful procedures after an investigation, in order to counteract possible negative effects. -The basic safeguard against the potentially harmful effects of deception is debriefing explaining the true purpose of the study and nature of the deception at the conclusion of the study. 16

IRB Approval for Human Subjects Research Institutional Review Boards (IRBs) are committees at research institutions that are responsible for reviewing research proposals involving the use of human and animal subjects. Such boards determine if the investigator has thoroughly considered the potential for harm and has provided adequate safeguards for the protection of subjects rights. Don t forget to get IRB Approval when you develop your proposal if the subjects are individual human beings. TTU IRB website: Human Research Protection Program | Research Home | TTU 17

Concluding Remarks Doing scientific research in public administration is not only about science, but also art of storytelling. - Appealing: research questions that address important policy or management problems, have implications for policy making and management practice - Convincing: rigorous research design and implementation - Ethical: The research process should be ethical. 18

Developing a Literature Review Creswell Ch. 2

Goals of Literature Reviews To evaluate the state of knowledge or practice on a particular subject this involves critical analysis and synthesis Lit reviews aren't about you: how much you've read, how harshly you can critique. They're about your reader: what does s/he need to know to position your research question and proposed answer in the field?

Goals of Literature Review Process Highlight the areas of agreement and disagreement in past research this can help refine your research question! Organize based on themes, not individual studies Identify the gaps that exist in past research: Theoretical Methodological Sample/Data Empirical Findings

Steps in Developing Literature Review Step 1: Search & find key studies Step 2: Summarize Step 3: Synthesize, cite & write Step 1: Define the scope of the literature review: What key words? What disciplines? What timeframe? What research methods? Literature Review Tutorial

Steps in Developing Literature Review Step 1 Search for key studies 1. Use search engine or library database to search those articles. - Google Scholar - Web of Science 2. Consult with the subject librarian 3. Use this new resource, that provides a map of key studies and how they are connected https://www.connectedpapers.com/

Go Down the Rabbit Hole Step 1 Search for key studies Identify the relevant literature to review: 1. Start with relatively recent literature reviews. 1. Like today s article coherent, theory based. 2. Or, more systematic.

Back tracking 1. Look at key studies identified in these works Look at the references of those studies. 1. Foundations

Forward tracking 1. Look at the cited by on Google scholar to get the most recent articles. 1. The Future

Step 1 Search for key studies (20-25 minutes) 1. LET S DO IT!

Identify 2 key studies that you can use in your paper. For these studies, go to the cited by and identify a third study that is perhaps the most recent Go to https://www.connectedpapers.com/ and identify key authors/papers Save in a folder on your computer so you can read later. Organize into subtopics if you have the time!

Steps in Developing Literature Review: Step 2 summarize Read and take organized notes on each paper based on key categories you want to focus on:

NOTES Research questions examined Theoretical framework Research design Data sources, sample, and measures Key results Strengths & Limitations Future research/gaps

Extra Resources Wesley Wehde, Ph.D. (wwwehde.com) Use Academic Article reading worksheet? Organize the results of your literature review into themes using the spreadsheet?

Step 2summarize (10-20 minutes) Open the Literature Review spreadsheet & save on your computer Enter key information for the 2 studies you identified earlier If you have time, enter links for other studies to look at later Alternative way to take notes & save studies: Citation software Zotero demonstration

Steps in Developing Literature Review Step 3 synthesize *Organize your literature review around themes and not discrete studies*

Pull the results of the literature review together What are the areas of agreement or disagreement? What are the reasons for the disagreements? (different theoretical lens, different methods, different samples, etc.) What do the disagreements indicate about gaps in research and possible contributions you can make. What hasn t been said about your topic? gaps

Example wording: e.g. Empirical studies on X generally support a positive relationship between A and B (Author A, 1999; Author B, 2003; Authors C & D, 2010). However, empirical support for significant positive effects is not universal (Author E, 2012).

Example of organizing literature review results in a table Table 1. Policy Designs and Findings from Published Promise Research on Postsecondary Outcomes Program Structure Eligibility Requirements Findings Increase in college application, enrollment, credits attempted, persistence and attainment (Bartik et al. 2019; Andrews et al. 2010) Kalamazoo Promise First-dollar aid covers tuition & fees at any public institution in Michigan Attend Kalamazoo Public School; 2.0 GPA and 12 credits per semester in college First-dollar scholarship covers tuition at any college in the country (capped at the highest tuition and fees at an Arkansas public university) Increase in college enrollment and bachelor s degree completion. Null impact on associate s degree completion (Swanson & Ritter 2018) Attend high school in El Dorado School District from kindergarten-12th for 100% or 9th-12th for 60% of scholarship El Dorado Scholarship 40 hours community service in high school, 3.0 GPA or higher in high school, 90% attendance rate in high school, Attend school and reside in New Haven, no expulsions in high school. In college, student must maintain a 2.0 GPA in first year and 2.5 GPA afterwards. Unclear impacts on college enrollment and persistence (Daugherty & Gonzalez, 2016). None consistently statistically significant and positive. First-dollar aid up to $10,000 per year covers tuition at all public college in Connecticut. Includes student support system. New Haven Promise Last-dollar aid covers cost of attendance up to $5,000/$10,000 max award per year to attend any institution in Pennsylvania Attend Pittsburgh Public Schools, 90% attendance rate and high school GPA requirement (varies over time) Pittsburgh Promise Increase in college enrollment & persistence (Page et al. 2018) Increase in two-year college enrollment and credit accumulation. Reduction in four-year enrollment (Carruthers & Fox 2016) Last-dollar aid covers tuition and fees in any Tennessee community college Knox Achieves Attend Knox County high school Increase in college enrollment, shift from four-year to two-year colleges in first cohort (Gurantz 2018) Last-dollar aid covers tuition and fees in any Oregon community college Oregon Promise Oregon residents with 2.5 GPA Increase in enrollment in public and private four-year and two-year colleges. Increase in enrollment in selective colleges (Bifulco et al. 2019) Buffalo Say Yes to Education Last-dollar aid covers tuition at in-state two or four-year colleges Graduate from public high school in Buffalo & fill out FAFSA Attend and graduate from eligible high school in Milwaukee Public Schools, make 2.5 high school GPA, attend class 90% of the time & FAFSA, need at least $1 of unmet need & enroll within four years of high school Last-dollar aid providing up to cost of attendance or $12,000 per year to one cohort of students to attend any two or four-year public college in Wisconsin. Increase in two-year college enrollment and attainment (Harris et al. 2018) The Degree Project Source: Bell 2021. Does Free Community College Improve Student Outcomes?

As you gather more studies, key themes/variables will become more obvious These themes will be how you organize your literature review. Some find it helpful to create a Mind Map of the literature: http://www.raulpacheco.org/2019/08/from-review-of-the- literature-to-mind-map-of-the-field/ https://app.mindmup.com/map/new/1643471874825

Steps in Developing Literature Review Step 3 synthesize Justifying your research and show your scholarly contribution One of the major goals of your literature review - Exposing the inadequacy in the current research & summarizing the gap (theoretical inadequacy, or empirical inconsistence) - Positioning your research in the current debate or gap & be persuasive

Citation References: using APA Style - Harvard APA Tutorial http://gseacademic.harvard.edu/~instruct/articulate/apa_mod/AP A_Intro/player.html Two general ways to integrate source material into your review - Scholar A (2016) finds, acknowledges, suggests that... - Paraphrase what the citing material says (Scholar A, 2016).

Citation management tools allow you to store and organize citations and files, as well as to create bibliographies formatted in major citation styles (MLA, APA, Chicago, etc.). RefWorks | Zotero | EndNote I just use Google Scholar because I m an old dog

Conclusions on Literature Review Key elements of final product: 1. Concisely summarize from a set of relevant sources the collective conclusions most pertinent to your own research interests 2. Comment on the agreement and disagreement on a particular issue 3. Identify the research gaps (theoretical or empirical) 4. Highlight your contribution to the literature 5. Cite at least 15 sources! (APA format)