VISIT STATE HEALTH COMPARE
Blog & News
U.S. Census Bureau Analytic Report Shows Significant Non-Response Bias in the 2020 American Community Survey
November 11, 2021:The U.S. Census Bureau recently released an analytical report that details and evaluates the impact of the COVID-19 pandemic on data collection and data quality in the 2020 American Community Survey (ACS) and provides evidence supporting its decision to release the 2020 ACS 1-Year estimates and public use microdata on an “experimental,” rather than official basis. This blog post highlights evidence from the report that the pandemic-related disruptions to data collection resulted in measurable non-response bias in the 2020 ACS, causing the survey to over-represent economically advantaged populations.
The analytical report examines trends in a number of measures that are thought to be reasonably stable from year to year or which can be validated against external data sources. The measures evaluated include:
- Building structure type,
- Medicaid coverage,
- educational attainment,
- non-citizen population, and
- median household income.
This post will highlight trends from a selection of these measures: Medicaid coverage, non-citizen population, and educational attainment.
Medicaid Coverage
Figure 1 below compares Medicaid coverage (millions of persons covered) information from the ACS to enrollment data from the Centers for Medicare and Medicaid Services (CMS). Though ACS data have consistently under-estimated the number of individuals covered by Medicaid relative to administrative data, the figure shows how trends in enrollment have been parallel in the past. However, in 2020, these trends diverged, with CMS enrollment data showing an increase in Medicaid coverage (consistent with previous experience that Medicaid coverage increases during economic downturns) and ACS data showing a decrease in Medicaid coverage. This divergence is evidence that households that responded to the ACS were more socioeconomically advantaged than in previous years, and therefore were less likely to report having Medicaid coverage.
Non-Citizen Population
Results in figure 2 show that the number of non-citizens has been largely stable year to year from 2016 through 2019, but the number of non-citizens decreased substantially in 2020, falling by 1.6 million persons. The report’s authors propose that this decrease, though potentially due in part to changes in international migration, is likely due to non-response bias since “foreign born—and non-citizens in particular—disproportionately respond to the ACS via in-person response follow-up methods,” which were substantially cut back in 2020 due to the COVID pandemic.
Educational Attainment
The share of the population age 25 years and over with a bachelor’s degree or higher was relatively stable year-over-year from 2016 through 2019, changing by less than one percentage point. However, from 2019 to 2020, the percent of the population with a bachelor’s degree increased by 2.3 percentage points, which, if accurate, would translate to an increase of about 6 million people with a bachelor’s degree or higher between 2019 and 2020. In previous years, the ACS has shown a year-over-year increase of approximately 2 million in the number of bachelor’s degrees (figure 3).
Below, figure 4 compares the year-over-year change in the number of bachelor’s degrees in the ACS to administrative data from the National Student Clearinghouse (NSC), an organization that provides enrollment and degree verification. Contrary to the ACS estimates, the NSC data do not show a large increase in the number of bachelor’s degrees in 2020. This is another piece of evidence demonstrating that the 2020 ACS data had measurable non-response bias that also over-represented a more highly educated population.
Conclusion
The measures presented here and in the Census Bureau’s analytic report on the 2020 ACS provide fairly clear evidence that the data collected in the 2020 ACS over-represent a more socioeconomically and educationally advantaged population, an issue that is the product of pandemic-related disruptions to data collection as well as resulting non-response bias. Because of these demonstrable data quality issues, the Census Bureau has chosen to release a limited set of 2020 ACS 1-Year estimates and the public-use microdata sample (PUMS) as “experimental” data products, which will be available later this month.
We commend Census for its transparency in providing this level of detailed evidence of the complications with the 2020 ACS and for making the 2020 ACS data available in an experimental format at least, rather than withholding them entirely from public release. Though the lack of reliable 2020 ACS data creates a critical information gap for a range of data users, given this evidence and the "gold standard" nature of ACS data, it is clear that the Census Bureau acted prudently in not releasing the 2020 ACS as usual.
Related Resources
Census Bureau Announces Major Changes to 2020 American Community Survey (ACS) Data Release (SHADAC Blog)
Changes in Federal Surveys Due to and During COVID-19 (SHADAC Brief)
Current Population Survey (CPS) will Serve as Primary Source of 2020 State-level Data on Health Insurance (SHADAC Blog)
