Innovative Changes to Commodity Flow Survey Methodologies

Using Multiple Methodologies to Test Innovative Changes to the Commodity Flow Survey Rebecca Keegan Economic Statistical Methods Division 04/11/23 Panel: Respondent-Centered Innovations in Data Collection Instruments for Business Surveys at the Census Bureau Any views expressed are those of the author(s) and not those of the U.S. Census Bureau.The Census Bureau has reviewed this presentation for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (Approval ID: CBDRB-FY23-ESMD002-015) 1

Overview This talk will explore how researchers applied several methodologies to aid the development of, and refine, several new features added to the Commodity Flow Survey (CFS), while keeping the respondents at the center of decision-making. Due in part to the respondent centered research conducted over several years, the CFS has been able to collect substantially more data, while reducing respondent burden. 2

Background: Commodity Flow Survey Commodity Flow Survey (CFS) U.S. Department of Transportation (DOT), U.S. Census Bureau Administered every 5 years, the CFS provides data on the movement of goods in the United States including commodities shipped, their value, weight, and mode of transportation, as well as the origin and destination of shipments. o Used to evaluate the demand for transportation facilities and services demand for transportation facilities and services; ; to analyze trends/ trends/ forecast demands in the movement of goods, and associated infrastructure equipment. analyze infrastructure and CFS respondents report on a multitude of commodity shipment details, including the value, weight and description of commodities shipped during a given week. 3

Re-imagining the CFS Prior cognitive testing and direct feedback from respondents had revealed the most difficult aspects of compiling the data for the Commodity Flow Survey: o Often manual process of creating a sample creating a sample of their shipments o Process of matching their company s commodity descriptions matching their company s commodity descriptions to the Census Bureau s commodity codes (had to look up codes in SCTG Code list) o There was no method of correcting errors no method of correcting errors in the data upload 4

Re-imagining the CFS There was an effort to re-imagine how data could be collected for the CFS before the next mailout in 2022. A series of respondent centered research studies were conducted to explore whether innovations could address these concerns: o Could respondents be offered a path to response wherein they could choose to provide all their shipments all their shipments for the week, eliminating the need to create a sample? o Could a machine learning tool machine learning tool be implemented that would read the text descriptions of the commodities and map it to the Census commodity code on the back end? o Could a program be implemented that would consolidate any errors, and allow respondents to easily fix errors easily fix errors within the instrument? 5

Engaging With Respondents: Before & After Production Throughout the testing iterations, researchers wanted to keep respondent feedback at the forefront of the decision- making process. Across the multitude of methodologies employed, research questions centered around these key guiding questions: Will the innovation reduce burden, or otherwise be beneficial to respondents? Are the changes clear and usable? Three rounds of research utilizing different methodologies were employed in support of a pilot test: o A round of exploratory work leading up to the pilot test o A mini round of usability testing informed the design of the new features o A round of debriefing interviews following the pilot In support of the final production instrument researchers conducted: o An expert review of the specs o Cognitive testing with respondents o Post production debriefing interviews 6

Timeline Usability Usability Testing Testing (2020) Pilot launch (2020) Respondent Respondent Debriefings Debriefings (2021) Exploratory Exploratory Interviews Interviews (2019) Cognitive Cognitive Testing Testing (2021) Production Mailout (2022) Respondent Respondent Debriefings Debriefings (2022) Expert Review Expert Review (2022) 7

What changed Since 2017? o A New Method of Data Collection o Machine Learning Tool o Fixing Errors How did respondents react to these changes? 8

Innovative Changes: A New Method of Data Collection Previous CFS Reporting Previous CFS Reporting In the 2017 mailout, respondents were asked to take a sample of their shipments. For example, if they shipped 50 commodities for the week, they would be asked to report every other shipment. o This was meant to reduce reporting burden, but in fact as record keeping became more automated, selecting a sample created an extra step for some, and manual work for others. Current CFS Reporting Current CFS Reporting Sampling Became Optional Respondents now have the option to upload all their shipment records, as opposed to selecting a sample, which was a burdensome process. o The respondent is presented with 3 primary choices that are clearly explained. A recommendation for response is provided. They can choose to: Provide all their shipments Take a sample of their shipments Or manually enter shipments within the instrument 9

New Method of Data Collection: Respondent Feedback Those who could use the new path were enthusiastic about the change. Testing revealed what type of establishment would benefit most from this change. o It s not for everybody! Generally larger companies with well organized, easy to access shipment records were the most likely to find the new option simpler than taking a sample Conversely, smaller companies with paper, or harder to access records might still opt to take a sample 10

New Method of Data Collection: Respondent Feedback Testing focused on clearly communicating the new option to respondents, and determining what tools could be made available to aid in their decision making o For example, the survey offers a clear recommendation for a path based on their number of shipments; providing a pdf preview of the survey questions before the respondent selects a path 11

Innovative Changes: Machine Learning Tool Previous CFS Reporting Previous CFS Reporting In 2017, one of the shipping related items respondents were asked to provide was a written product description, in addition to a Census- specific 5-digit code, called an SCTG code (standard classification of transported goods). o Because the code was for Census only, this was almost always a manual process to look up the code and enter it in for each commodity. *fabricated data 12

Innovative Changes: Machine Learning Tool Current CFS Reporting Current CFS Reporting The latest iteration of the CFS allowed respondents to forgo entering in Census Bureau specific product codes by applying the code on the backend. Respondents would interact with a machine learning tool if the system could not read the text description to apply the code. o The tool would provide it s best guess as to the appropriate text description and will get smarter overtime. *fabricated data 13

Machine Learning: Respondent Feedback It became clear during the earliest stages of testing that eliminating the need to provide SCTG codes would be a desirable change for respondents. o They already maintained and provided a text description of their products to the survey Research revealed what sorts of commodities the program might struggle with, such as items that were not written in plain English. o For example, chemical compounds, items with an SKU embedded in the description, or company-specific names, or even commodities that are too broad, such as coffee Research focused primarily on the usability of the tool, and what necessary help or assistance could be provided if needed. o For example, providing a mechanism to de-duplicate identical descriptions; and linking to the old SCTG code book as a guide 14

Innovative Changes: Fixing Errors Previous CFS Reporting Previous CFS Reporting In the 2017 version of CFS, error checking did not exist! oThe system would accept the data as written, unless it did not fit the upload template Current CFS Reporting Current CFS Reporting Post-upload, the system will consolidate all the errors on a dashboard, and allow respondents to make changes. The new fix-error feature allows respondents to correct any errors within the instrument, and with one click, fix all identical errors. 15

Fixing Errors: Respondent Feedback Through respondent testing, it became clear that respondents were concerned they would need to manually click through hundreds of errors, even if it was the same issue repeating itself, thus the team created a button to address this Respondents also reported that they wanted a sense of how many errors were left to correct, so the team included a clear count to eliminate any confusion. 16 *fabricated data

Conclusions In all, 84 interviews across several years and multiple methodologies were conducted in support of the 2022 CFS, and testing the benefits of its new features. The multitude of qualitative methodologies allowed for respondent input across the various stages of survey/instrument design, both prior to, and post implementation of brand new features and a fundamental shift in reporting. Across the studies, respondent interests were the guiding factor. This research helped the Commodity Flow Survey collect more data with less respondent burden. Due to the successful implementation of these changes, in 2022, the CFS is on track to collect 100 million shipments, compared to 6 million in 2017. 17

Thank you! Rebecca.Keegan@census.gov 18