Challenges in Survey Implementation Without Response Bias

Non-response & other implementation challenges February 27, 2020 Nadi, Fiji

Example: Fictional HIES, sample 1000 hhs 100 households are well-to-do ,rich households. 900 households are not rich , some are poor This is information we do not have when we do the survey. -------------------------- 50 of the rich households refuse to participate in the survey Of the 900 not rich households, very few, only 20 hh s, refuse to participate.

Assume that the average monthly income among the rich hh,s in the sample is 10,000 dollars Assume that the average monthly income among the not rich hh,s is 2,000 dollars We get the estimate from the sample: (50 x 10,000 + 880 x 2,000)/930 = 2,430 If no nonresponse we would have got: (100 x 10,000 + 900 x 2,000)/1000 = 2,800

Non-Response bias If the households that refuse to participate are systematically different in any way from those that choose to respond, non-response leads to non- response bias (a systematic error in our results

The only way to prevent non-response bias is to prevent non-response. Supervisors and interviewers should be trained in techniques to maximize response rates. In addition, HQ should monitor refusal rates by interviewer and team to identify problems early.

Handling of nonresponse, two ways Replacement: If 10 households are needed for an EA, 12 households may be selected, and from that list of 12, 10 can be selected as target households and 2 households as replacements. Careful records should be kept of replacements. Include anticipated nonresponse in sample size calculation. If 10 households are needed, select 12. No replacements

Nonresponse Unit nonresponse Noncontacts Refusals Item nonresponse Individual questions skipped

Nonresponse Bias Total Population

Nonresponse Bias Nonrespondents Total Population Respondents

Nonresponse Bias Nonrespondents Respondent Population Respondents

tR= proportion of population that will respond YR = mean of respondent population YNR= mean of nonrespondent population

Y = mean of entire target population = tRYR + (1 tR) YNR

Bias Due to Nonresponse BNR = (1- tR)(YR YNR)

Relative Bias Due to Nonresponse RBNR = (1- tR) ( ) YR YNR Y

Nonresponse bias is a function of nonresponse rate difference between respondents and nonrespondents

Response Bias as a Function of tR and the Relative Difference Between YR and YNR 25 tR = .50 Relative Response Bias (%) 20 tR = .70 15 10 tR = .90 5 0 -50 -40 -30 -20 10 20 30 40 50 -10 0 -5 -10 -15 -20 -25 Relative Diff Btwn Respondents & Nonrespondents (%)

tR= Response rate for a tele. survey = 75% Example

tR= Response rate for a tele. survey = 75% YR = Av. income for respondents = 107 Kr. Example

tR= Response rate for a tele. survey = 75% YR = Av. income for respondents = 107 Kr. YNR = Av. income for nonrespondents = 89 Kr. Example

tR= Response rate for a tele. survey = 75% YR = Av. income for respondents = 107 Kr. YNR = Av. income for nonrespondents = 89 Kr. Y = .75 (107) + .25 (89) = 102.5 Kr. Example

( 107 89 102.50 ) RB= (.25) = .044 or 4.4% Example

Other implementation problems Household listing Mapping Identify households

Effect of old household list (5years) They got a 7.4% sample loss due to imperfect household list (dwelling vacant/address not a dwelling (4.7%), dwelling destroyed (2%), other (0.4%) dwelling not found (0.3%)).

In an EA of 100 households there will be on average 7 households that are on the census list but will not be found if selected to the sample. Since the census the total number of households will have grown by approximately 6% at the time of the survey (population growth). So, in a typical enumeration area (EA) of 100 households there will be on average 6 new households that are not on the census household list

Conducting a High Quality Listing Operation The objectives of the listing are : Provide an updated list of all the households in the EA from which the households to be interviewed can be selected using systematic random sampling. Obtain an updated count of the households in the EA as an input to the weight calculations.

Conducting a High Quality Listing Operation (2) 1. Meet with local leaders to determine the boundaries of the EA. Use the map provided by HQ as the EA boundaries may not match the traditional boundaries. Examples: A single village being divided into two EAs A small separate settlement being included with a nearby village.

Conducting a High Quality Listing Operation (3) 2. Segmenting large EAs. Dividing EA into equal sized segments. The boundaries of the segments should ideally follow well defined landmarks so that they can be easily identified by the interviewers during the survey. One segment is randomly selected, then listed and enumerated.

Conducting a High Quality Listing Operation (4) 3. The listers go systematically through the area and mark dwelling structures on a sketch map. 4. In each dwelling structure (serially numbered), the households are listed and serially numbered. Info on the household is entered in the listing form: Name of household head Street address (if available) Telephone number

Conducting a High Quality Listing Operation (5) 5. The household listers should make a note on the household list if the EA has significantly more or significantly less households now than what was registered in the census.

Consequences of Listing Errors If the listing includes areas that should not be covered, this would lead to incorrectly high weights, and over-estimate the total population for the area. If the listing leaves out areas that should be covered, this would lead to incorrectly low weights, and under-estimate the total population for the area. Both situations can lead to bias in the final results.

Household A householdconsists of a person or a group of related or unrelated persons, who reside together in the same dwelling unit, are answerable to the same head and share a common cooking arrangement.

Head of household The head of householdis defined as a usual resident member of the household who is acknowledged by the other members of the household as the household head. Households are found in dwelling units, dwelling units in structures, and structures in clusters. In some cases, one may find a group of people living together in the same house, but each person has separate cooking arrangements. In this case, each person constitutes a one- person household.

Litmus test The litmus test for identifying a household would be asking oneself the following three questions: 1. Do they reside in the same residential structure? 2. Are they answerable to the same head? 3. Do they share the same cooking arrangement?

If the answer to each of the above questions is YES then you have adequately identified a household. If the answer to any of the questions is NO then you have more than one household. Note that domestic servants and other workers living and eating in the same household are to be included as household members. Some difficult cases may be found. Under these circumstances consult your supervisor.

Case 1 You may come across two families living in one residential structure. Each family owns a farm but they share their crops and they cook and eat together. Here the answers to the three questions will be: Yes, No, Yes. This means that each family has a household head and therefore they are two households.

Case 2 Three bachelors occupy a house, share rent, water and electricity expenses equally but they eat out separately. The answers to the three questions are: Yes, No, No. This implies three households.

Case 3 Suppose two families and a bachelor live in one house. Each of the families cooks separately and the bachelor eats out. Apply the test and the answers are: Yes, No, No. These are three households.