Evolution of Network Scale-Up Method (NSUM) and Population Estimation Techniques

The Network Scale-Up Method (NSUM) Christopher McCarty October 30, 2012

The NSUM team H. Russell Bernard (University of Florida) Peter D. Killworth (Southampton Oceanography Centre) Christopher McCarty (University of of Florida) Eugene Johnsen (UC-Santa Barbara) Gene A. Shelley (Georgia State University/CDC)

Outline Origin and Evolution of Network Scale-up Method (NSUM) How to do it Compromising assumptions Current application in international health

Origin and Evolution of Network Scale-up Method (NSUM)

Populations of Interest HIV positive Public health and public policy advocates are interested in certain populations IV Drug Users Migrants We know a lot about each population, but we don t know how many there are Homeless Men Who Have Sex with men (MSM) For many reasons these populations are virtually impossible to count Female Sex Workers

The Problem How do you estimate the size of a population that you cannot count?

Government estimates were 7,000 dead Mexico City Earthquake - 1985 In a city of 18 million people, could the number of dead be 7,000 if everyone knew someone who died? Everyone seemed to know someone who died H. Russell Bernard

Estimate network size Estimating the Size of an Average Personal Network and of an Event Subpopulation H. Russell Bernard, Peter D. Killworth, Eugene C. Johnsen, Scott Robinson In: The Small World, ed. M. Kochen, 159-75 (1989) Funded by NSF grant BNS-8318132 and UF Graduate School Mexico City divided into 20 grids, 20 respondents per grid, or 400 respondents 91 of 400 (23%) knew someone who died But this did not reveal how many people died For this they needed to know the personal network size of each respondent

Relationship between network size (c) and the probability of knowing someone who died c (t/e) * p where: c = personal network size t = total population e = subpopulation size (those who died) p = probability of being in the subpopulation If e = 7,000, t=18 million and p=.2275 then c = 585

Back estimate an unknown Estimating the Size of an Average Personal Network and of an Event Subpopulation: Some Empirical Results H. Russell Bernard, Peter D. Killworth, Eugene C. Johnsen, Scott Robinson Social Science Research 20: 109-121 (1991) Presented at American Statistical Association (1987) Funded by NSF grant BNS-8318132 and UF Graduate School Survey in Mexico City of 2,260 respondents Subpopulation Event population size (e) Probability of knowing someone (p) Network size (c) Solved for the personal network size that best fit all subpopulations (c=220) Doctors 30,426 .3889 173 Mailmen 14,728 .1473 116 Bus Drivers 11,696 .2571 272 Applied this to reported unknown m (rape victims). Quake Victims 10,000 .2668 332 TV Repairmen 4,013 .2619 810 Best estimate for the number of rape victims in Mexico City (the unknown) was6,303 e 8,811 Priests 1,595 .2854 2254

A Primitive model Everyone s network in a society reflects the distribution of subpopulations in that society t= the size of a population (e.g. the U.S.) e = the size of some subpopulation within it (e.g. diabetics). m = the number of members of the subpopulation known by any one person (e.g. diabetics) c = personal network size

How to do it

This suggests that... Personal network size c=(m*t)/e Size of subpopulation e=(m/c)*t where: m=average of all reported unknowns t=total population size c=average network size where: m=sum of all reported knowns t=total population size e=sum of all known population sizes

To recap: NSUM is a 2-step process Step 1: Estimate personal network size Use estimates of the number known (m) in a set of known populations to estimate network size for each respondent Step 2: Use this to scale-up unknowns Use the reports of the unknowns to scale-up to the unknown population Note that under-reporting of known m will result in lower estimates of c, and bigger e. Under-reporting of unknown m will result in lower e.

NSUM is a survey approach With any survey you must: Define the population Choose a sample size Determine who is an eligible respondent With this approach you also must Choose a set of populations of known size

Select respondent population Respondent population is not the same as the population to be estimated (target population) Respondents are sampled from the population within which the subpopulation exists Interview U.S. respondents to estimate homeless population in the U.S. Interview Barcelona respondents to estimate heroin users in Barcelona You must know the size of the respondent population (t) (e.g. the U.S. or Barcelona)

Choose sample size As in all surveys, sample size should be based on expected margins of error. If the thing you are estimating varies a lot, then the sample should be larger to capture that Margins of error could be calculated on either network size (c) or on the values for the unknowns (m) Matthew Salganik is working no this problem Our data suggest that a survey of 400 respondents would generate a margin of error of 26 network alters A survey of 1,000 in the U.S. would generate a margin of error of 16 network alters

Alter boundary Definition of who is an alter can have enormous effects on the estimate of network size, and therefore then size of the unknown subpopulation Early studies of network size used ever known, while our studies used currently known Our definition: You know them and they know you by sight or by name. You have had some form of contact with them in the past two years and you could contact them if you had to

Selecting known populations Known populations should vary in size and type DO NOT limit study to populations related to, say, health conditions, although plentiful, may introduce error DO NOT use only large populations (such as men or people over age 65). This introduces recall error DO NOT use only small populations. This introduces error from very few hits A rough guide is to use populations between .1% and 4% of the total population Ideally collection of sub-population data will be recurring so that they can be used in subsequent years It is important that the data all reflect the same year (be aware that some population data lags)

Potential Sources in the U.S. U.S. Statistical Abstract U.S. Census FBI Crime Statistics State and city-level Vital Statistics Surveys by organizations Voter Registration databases

We experimented with names The U.S. Census provides estimates of both first name and last names The advantage of names is that they vary in size and are typically ascribed Countries and cultures vary in the way they use names They are also prone to what s called barrier error (more about this in a minute)

Peter D. Killworth, Christopher McCarty, Eugene Johnsen, Gene A. Shelley and H. Russell Bernard. (1998) A Social Network Approach to Estimating Seroprevalence in the United States. Evaluation Review 22:289-308 Funded by NSF grant SES-8803577 Estimating the Size of Hard-to-Count Populations Telephone survey of 1,524 Florida respondents Respondents estimated: How many they knew in six known subpopulations (four different sets of six) If they knew someone with certain first names How many they knew who tested positive for HIV and those with AIDS Average network size was estimated to be 105 (s.d. 92) This approach was limited by using only six known populations for each respondent and not asking how many people were known with first names

Peter D. Killworth, Christopher McCarty, H. Russell Bernard, Gene A. Shelley and Eugene Johnsen. (1998) Estimation of Seroprevalence, Rape and Homelessness in the U.S. Using a Social Network Approach. Evaluation Review 22:289-308 For the next survey we made two critical changes We asked each respondent about a lot of known subpopulations (29 of them) We asked how many they knew, not just if they knew someone in the known subpopulations Telephone survey of 1,554 U.S. respondents Back-estimation resulted in average network size of 286 s.d. 291 (the first instance of an estimate near 290)

Summation method Previous attempts estimated network size by back- estimating from known subpopulations We thought of an alternative approach: Ask respondents how many people they know in mutually exclusive and exhaustive relation categories Sum the estimates in those categories to get network size

Relation categories H. Russell Bernard, Peter D. Killworth, Christopher McCarty. (1982) INDEX: An Experiment in Social Structure. Social Forces 61:99-133 Immediate family Other birth family Family of spouse or significant other Co-workers People at work but don't work with directly Best friends/confidantes People know through hobbies/recreation People from religious organization People from other organization School relations Neighbors Just friends People known through others Childhood relations People who provide a service Other

Advantages of the summation method Unlike back-estimation from known populations, the summation method should not be subject to certain kinds of error (also, more in a minute). It does not require finding known populations, which could be a problem in some countries where there are no reliable data on the population

Christopher McCarty, Peter D. Killworth, H. Russell Bernard, Eugene Johnsen and Gene A. Shelley. (2001) Comparing Two Methods for Estimating Network Size. Human Organization60:28-39 Two telephone surveys in U.S. Included back-estimate and summation method for each respondent 29 known populations for back-estimating 16 relation categories for summation Virtually the same result as previous study Correlation between methods is either fully or only .56 N Back Estimation Summation 796 290.8 290.7 574 291.2 281.2 1,554 (previous study) 286 n.a.

Reliability check Change the data We changed reported values at or above 5 to a value of 5 precisely. The mean dropped to 206, a change of 29%. Survey of clergy We surveyed a national sample of 159 members of the clergy people widely thought to have large networks. We set values of at least 5 to a uniformly distributed random value between 5 and 15. We repeated the random change (5 15), but only for large subpopulations (with >1 million). The mean increased to 402, a change of 38% -- in the opposite direction. Mean c = 598 for the back- estimation method Mean c = 948 for the summation method

So, 290 was not a coincidence 1. Two different methods of counting similar results 2. Changing the data produced large changes in the results, and in the expected directions 3. People who are widely thought to have large networks did have large networks

The distribution of c

The data track

Over- and under-estimation Tendency for people to overestimate small populations (<2 million) and to underestimate large ones (>3 million). The two largest populations are people who have a twin brother or sister and diabetics. Without these two outliers, the correlation rises from r = .79 to r = .94

Compromising assumptions

NSUM Assumptions 1. Everyone in t has an equal chance of knowing someone in e Violation of this is called Barrier Error 2. Everyone knows everything about everyone they know Violation of this assumption is called Transmission Error 3. Respondents can accurately report the number of people they know in any given subpopulation Violation of this is called Recall Error

Barrier Error exists: Correlation between the mean number of Native Americans known and the percent of the state population that is Native American is 0.58, p = 0.0001.

Known populations and their relationship to demographic variables (Barrier Error there are many dots!) Population State Sex Race Age Education Marital status Work status Religion Political Party Native Americans Gave birth in past 12 months Adopted a child in past year Widow(er) under 65 years On kidney dialysis Postal worker Commercial pilot Member of Jaycees Diabetic Opened a business in year Have a twin brother or sister Licensed gun dealer Came down with AIDS Males in prison Homicide victim in past year Suicide in past year Died in wreck in past year Women raped in past year Homeless HIV positive

Names and their relationship to demographic variables (Barrier Error again, many dots) Population State Sex Race Age Education Marital status Work status Religion Political Party Michael Christina Christopher Jacqueline James Jennifer Anthony Kimberly Robert Stephanie David Nicole We assumed that a representative sample will eliminate barrier error. In fact, we don t know so far, research to improve estimates in NSUM is on transmission error.

What to do about Barrier Error? We have always assumed we could eliminate the effect of barrier error by: 1. ...using a large and representative sample of respondents and 2. ...using a lot of subpopulations We don t know that A potential area of research would be to adjust data for barrier error using information about respondents

Transmission Error Recall that errors of transmission occur when you know someone in a subpopulation but do not know they are in it I might know a diabetic person, but do not know they are diabetic More critical to the method, I might know someone who is an IV drug user, but not know they are an IV drug user

Transmission Error Study We recruited 30 people who were members of one of the known populations (e.g. diabetics) We elicited 25 alters from each using first names The respondent provided the alter s phone number (30x25=750 alters) We contacted 220 of 750 named alters and asked them things about themselves and about ego.

Results of Transmission Study Population % who knew % who did not know Respond ents # of alters Am. Ind. Diabetic Birth in last 12 mos. Gun dealer 100 55 93 92 58 88 0 45 7 8 42 12 2 6 3 1 1 5 12 44 27 12 12 26 Member of JC s Dialysis Business in last 12 mos. Postal worker Has twin Widowed <65 75 100 88 97 25 0 12 3 4 1 2 4 16 10 24 38

Findings from the alter study Some things are more visible (kidney dialysis versus diabetic) Some things are very easy to know (99% know marital status of ego) Some things are hard to know (52% knew how many siblings ego had)

Some people withdraw Gene A. Shelley, Peter D. Killworth, H. Russell Bernard, Christopher McCarty, Eugene C. Johnsen, Ronald E. Rice. (2006) Who knows your HIV status II: Information propagation within social networks of seropositive people. Human Organization 65: 430-445 Gene Shelley conducted ethnographic work with a sample of HIV+ respondents People said that they withdrew from their network in order to limit the number of people who knew their HIV status Eugene Johnsen confirmed that HIV+ people have, on average, networks that are one-third the average of others in the U.S.

What to do about Transmission Error? From the transmission study we tried to develop weights for classes of characteristics about subpopulations Things that carry a strong stigma (HIV+) Things that carry a moderate stigma (diabetes) Things that just don t come up in conversation (being a twin) These weights did not improve our estimates Recent work in this area focuses on transmission error on the unknown m values (e.g. HIV, drug user)

Our estimates using NSUM Killworth et al. 1998 of 1,554 adults in the U.S. in 1994. HIV+: 800,000 43,000 Homeless: 526,000 35,000 Women raped in the last 12 months: 194,000 21,000 These were all close to other estimates

Charles Kadushin, Pater D. Killworth, H. Russell Bernard and Andrew a. Beveridge (2006) Scale-up Methods as Applied to Estimates of Heroin Use. Journal of Drug Issues. 36: 417-439 National survey n=17000 to estimate the prevalence of crimes in 14 cities, large and small, in the U.S. Reported the number of people they knew who had been victims of six kinds of crime and the number of people they knew who used heroin regularly.

Tian Zheng, Matthew Salganik and Andrew Gelman (2006) How many people do you know in prison?: Using overdispersion in count data to estimate social structure in networks. Journal of the American Statistical Association 101(474) 474: 409-423 Re-analyzed NSUM data from Killworth et al. (1998) Developed a method to estimate social structure using NSUM survey data Respondents varied in the number of acquaintances Respondents varied in propensity to form ties to people in some groups, and not in others This was a critical turning point as this was the first independent group to work on statistical improvements to the method

Current application in international health

MARPs Certain populations are at high risk for contracting and spreading HIV Most At Risk Populations (MARPs) typically fall into one of three categories Female Sex Workers Men Who Have Sex With Men IV Drug Users Much is known about the prevalence of HIV among these groups, but not much about the size of the groups, especially at the country level

Methods to Estimate the Size of MARPs (http://data.unaids.org/pub/Manual/2003/20030701_gs_estpopulationsize_en.pdf) Methods that require a sample frame Census Counting all members Enumeration: Counting members in a sample frame then scaling up Population Survey: Draw a representative sample (similar to enumeration) Methods that do not require a sample frame Capture-Recapture Multiplier