Role of Taxonomy in SHIN: Enhancing Information Organization

Taxonomy Validation: Case Study Richard Morey, Public Health Advisor, SAMHSA Office of Communications Joseph A. Busch, Founder and Principal, Taxonomy Strategies

Agenda SAMHSA Health Information Network Role of Taxonomy Taxonomy Validation 2

What Does SAMHSA Do? Substance Abuse and Mental Health Services Administration Part of the US Dept of Health and Human Services (HHS) Supports State & local efforts to improve quality & availability of SA & MH treatment & prevention services More Specifically: Federal Grants to States and Community Orgs Drug and alcohol treatment & mental health services referral Statistics on Drug Abuse & Mental Health

What Does SAMHSA Do? What Does SAMHSA Do? SAMHSA s Health Information Network Call center, publications & information Customer Touch Points * Customer requests (600K+ Inquiries) Customer orders (10.7M shipped) Customer on the Web (8M+ Web visits) Customer community (40K email subs) Customers at conferences (100+ exhibits) How do we make sense of all this rich data? *stats are from the SAMHSA Health Information Network Annual Report 2008

What Does SAMHSA Do? What Does SAMHSA Do? SAMHSA s Health Information Network Mission: To connect people working in substance abuse and mental health fields and the general public to the latest information on prevention and treatment Vision: Share data to develop a more proactive, interactive knowledge management system

SAMHSA Knowledge Management SAMHSA Knowledge Management Project Project

Role of Taxonomy in SHIN Tagging content Navigating (top level, drill down & contextual filters) Searching (categories & synonyms) Aggregating & reporting 7

SAMHSA Taxonomy Eight facets. High-level; especially Substances. Designed to create compelling groups or buckets of similar content; but not point to individual pieces of content. 8

SAMHSA SHIN Taxonomy overview (image) Substance Abuse and Mental Health Services Administration Taxonomy Subjects Contextual Filters 38 Population Groups Audience 23 Filter Facets 59 terms 33 Substances 36 Content Type Conditions & Disorders 36 Subject Facets 301 terms Intervention & Treatment Topics 48 Professional & Research Topics 37 Geography & Locations 109

SAMHSA SHIN Taxonomy validation exercise Demonstrate that SAMHSA staff will be able to use the Taxonomy to easily tag and find content. Validation tests: Walk through & review candidate taxonomy. Place 30 most popular queries (words & phrases) from search logs in the correct Taxonomy facet. Tag content from a test collection and compare those tags to an established baseline. 5 items selected from collection of 20. 14 SAMHSA & IQ Solutions staff participated in one-hour one-on- one test sessions over three days.

Term sorting data collection form Intervention & Treatment Topics Content Type Population Groups Conditions & Disorders Professional & Research Topics Geographic Locations Term Alcohol Anger Anxiety Autism Binge Eating Bipolar Disorder Change Management Child Co-occurring Disorders Counseling Depression Fulton County Health Literacy Immigrant Adolescents Inhalants Low Income Methadone Native American Culture Ohio Opium Posters PTSD Residential Treatment Rohypnol Smoking Stress Substance Abuse Suicide University Students Vicodin Webcast Audience Substances

Blind sorting of popular search terms (n=12) 84% of terms were correctly sorted 60- 100% of the time. 25-50% (6%) < 25% (3%) 50-60% (7%) Results: Excellent Difficulties For Methadone, confusion when, in this case, a substance is a treatment. For general terms such as Smoking, Substance Abuse and Suicide, confusion about whether these are Conditions or Research topics.

Search terms sorting task user rating (n=12) Medium 33% Easy 42% Easy/Medium 25% No one rated the task as difficult!

Tagging template filled in Title: Image Number: American Indian/Alaska Native Substance Abuse Treatment Services: 2004 18 Content Type Series Report Audience Prevention Program Planners Subjects Population Groups American Indian & Alaska Native Substances Conditions & Disorders Intervention & Treatment Topics Professional & Research Topics Substance Abuse Geographic & Locations Add any additional keywords that you think would be helpful in finding this item (that are not in the title or taxonomy): _JB_ Initials Was it easy / medium / difficult to tag this item? (circle one)

Tagged examples test collection Title of Test Content Item Times Tagged Alcohol Awareness Month 12 Older Adults with Mental Illnesses 11 DASIS Report: Homeless Admissions 9 Underage Drinking Prevention PSA 7 Tips for Teens: Methamphetamine 4 Total 43

Content tagging consensus (n=244) Test subjects tagged content consistent with the baseline 41% of the time. Incorrect 4% Over-Tagged 13% Results: Good Consensus 41% Alternatives 42% Observations Many other tags were reasonable alternatives. Correct + Alternative tags accounted for 83% of tags. Over tagging is a minor problem.

Tagging exercise test subject rating (n=42) Difficult 7% Easy 40% Medium 48% Easy-Med 5% Only 7% rated the task as difficult!

Examples of taxonomy changes informed by validation exercise Facet Substances Conditions & Disorders Type of Edit Restructuring Added Change Removed hierarchy Alcohol Abuse Eating Disorder Fetal Alcohol Spectrum Disorder Mental Illness Underage Drinking Autism Employment Services Housing Services Motivational Incentives Motivational Interviewing Primary Care Treatment (in general) Screening & Assessing Added many scope notes Deleted Added Intervention & Treatment Topics Split Scoping

Contact Info Richard Morey, 240-276-2131, rich.morey@samhsa.hhs.gov Joseph Busch, 415-377-7912, jbusch@taxonomystrategies.com