When the coronavirus pandemic besieged Colorado last spring, Boulder Mayor Sam Weaver and a few of his friends quickly developed a new hobby — downloading data on the virus’ spread in the state and plugging it into their own spreadsheets for analysis.
Weaver is an engineer by training; data is oxygen for him. He said having numbers for Boulder and other jurisdictions at his fingertips provided fast insight into the pandemic. Where was it spreading? Where was it in decline? What could his city do differently?
But there was one measurement where Weaver and his friends soon noticed something curious: Test positivity. The figure, expressed as a percentage, shows how many coronavirus tests in a given area are coming back positive. Generally, anything below 5% shows that the virus is under control, according to the World Health Organization. Currently, CDPHE says the positivity rate over the past week in Colorado is 8.1%.
COVID-19 IN COLORADO
The latest from the coronavirus outbreak in Colorado:
- MAP: Cases and deaths in Colorado.
- TESTING: Here’s where to find a community testing site. The state is now encouraging anyone with symptoms to get tested.
- VACCINE HOTLINE: Get up-to-date information.
The number comes with particularly high stakes now because Colorado health officials use it as one of three measurements to determine whether restrictions on businesses and other gathering spots should be increased or relaxed in a county. Elevated positivity numbers contributed to the Colorado Department of Public Health and Environment’s decision last week to move several counties to tighter restrictions.
In theory, test positivity is a simple calculation. You take the number of tests that come back positive, divide by the total number of tests administered and multiply by 100. But when Weaver and his friends did that for their own calculations, they could never recreate the positivity figure the state reported for Boulder.
Usually, it was off by just a couple of percentage points. But, during the spike in cases this fall that coincided with the return of students to the University of Colorado campus, Weaver said his calculations put test positivity at 10 percentage points higher in Boulder County than the state’s numbers did, meaning the virus’ spread was worse than what was being reported.
So what was going on?
“It took me awhile to sort all this out,” he said.
An incomplete dataset
The answer is that the test-positivity figure released by state health authorities is not quite what it seems. Data-reporting issues mean that state officials calculate the figure using an incomplete set of test numbers.
While CDPHE says it is confident that the large majority of tests make it into the calculation, it says it is also impossible to know exactly how many tests are being left out. In other words, it’s impossible to know whether the figure reported on a given day is too high or too low and by how much.
And that’s not the only data struggle state health officials have faced recently in trying to understand the trajectory of what could end up being the pandemic’s worst surge yet in Colorado.
Late last month, CDPHE announced that it had discovered nearly 16,000 previously unrecorded coronavirus test results. The tests, which included more than 1,200 positive cases, had been administered by Kaiser Permanente during October. But a state computer glitch meant they had been rejected and not included in CDPHE’s data. The people who tested positive had not been contacted by public health contact-tracers, either.
Adding the tests meant that Colorado’s total confirmed coronavirus cases for October at the time jumped by about 5%.
Though CDPHE knew the original dates of the tests’ results, it assigned the positive cases to the days on which it finally uploaded them into its system. That resulted in about 850 old cases — some, perhaps, dating back weeks — being included in the new-case totals reported for Sunday, Oct. 25. That, in turn, made that day appear as if the state had hugely smashed its previous record high for new cases reported in a day, as some media outlets reported.
“The daily case total is a total of cases based on their reported date, which is usually but not always the date we first receive them in our systems,” CDPHE wrote in an emailed response to questions from The Colorado Sun.
Coronavirus is undoubtedly surging in Colorado. Hospitalizations related to the virus have reached worrying new levels, and CDPHE for Saturday reported nearly 2,500 new cases for the day — more than even the old-case-inflated number reported the previous Sunday.
To Weaver, this surge makes CDPHE’s data challenges all the more troublesome.
“There continues to be what I think could be data gaps that need to be filled in,” Weaver said.
But CDPHE defends its data practices. The agency says the newly discovered cases — which had a 7.7% positivity percentage — did little to change overall positivity trends for the month and didn’t impact decisions about restrictions statewide or in individual counties.
“We have implemented a consistent approach to reporting tests and cases in Colorado since March that helps us maintain high quality data across multiple systems,” the agency’s communications staff wrote in an email to The Sun.
A “mess” of numbers
Faced with the urgency of the pandemic, there has never been quite such a need for reliable health data in the United States as there is now. But, across the country, actually gathering that data has been a constant struggle. That includes in Colorado, where Gov. Jared Polis has pledged to lead guided by data while the state’s data-collection systems have sometimes struggled to keep up.
Nationally, the positivity percentage has been an especially thorny subject. The COVID Tracking Project, an effort to collect coronavirus numbers from all 50 states, last month published a blog post titled “Test Positivity in the U.S. Is a Mess” in which it describes why it does not include the figure in its reporting.
“[T]he underlying numbers used to calculate test positivity are counted differently in different places,” the post states. “In the absence of federal standards, U.S. states and territories report their ‘total tests’ data in several different ways.”
Some states report by the number of samples tested, even if multiple samples came from the same person. Other states, including Colorado, report the number of people tested.
And there are also discrepancies among health agencies in Colorado about how positive cases are reported. While CDPHE calculates its positivity rates based on the days the tests were reported to the state, Kate Watkins, an epidemiologist for Jefferson County Public Health, said her agency calculates rates for its own dashboard using the days the tests were administered, along with a caveat that says “Testing data may be incomplete.”
“I prefer (using) test collection date because we have seen a few other instances where we get a large backlog of just positive cases, which would have affected our percent positivity,” she wrote in an email. “However, that means that there is a lag in our percent positivity rates of about a week as additional tests are reported.”
The revenge of the fax machine
In addition to the timing of the reporting, there is also the mechanism — and this explains why CDPHE doesn’t know exactly how many coronavirus tests are being run in Colorado and why Boulder’s official positivity rates were so low during the CU spike.
The state would prefer to receive test results electronically through a system known as ELR, electronic laboratory reporting. The American health care system being the American health care system, though, not all places that conduct tests are technologically equipped to submit results that way. Shockingly, some still use fax machines.
Positive coronavirus test results ultimately also land in a second system known as CEDRS, pronounced “cedars” and standing for Colorado Electronic Disease Reporting System.
For tests submitted through ELR, this whole process is pretty seamless. The ones that are faxed, though, have to be entered into the state’s systems manually, which is time consuming. As a result, CDPHE has chosen to manually enter only positive test results.
“Extending data entry to non-electronic negative case reports would result in an additional data entry burden on state and local staff in the tens of thousands of cases each week,” CDPHE communications staff wrote in an email (emphasis theirs). “We feel that investing in automation and electronic reporting is a much wiser use of public resources.”
This creates the blind spot in the positivity percentage. When calculating the figure, CDPHE uses only tests reported through ELR, an approach the agency says is “consistent with the approach used by the federal Centers for Medicare and Medicaid Services and many states to estimate county level percent positivity.”
It’s not clear how many tests get excluded this way. The best the agency can do is estimate the proportion of tests coming in through ELR by looking at the proportion of positive tests that come in that way. Using that method, CDPHE estimates that about 90% of all coronavirus tests are submitted through ELR.
But one large lab missing from the ELR system can significantly skew the positivity percentage.
The Wardenburg wrinkle
In late September, as the CU case explosion became the largest single outbreak the state had seen, Weaver sent an email to several Boulder County health officials.
“As you heard yesterday, I have very serious concerns about the County reporting of COVID positivity rates,” he wrote in his message, which was obtained by The Sun through a records request.
Then, Weaver proceeded to lay out his argument. Using numbers of new cases and total tests taken from the county’s own website, when Weaver calculated the daily positivity rate, he got 7.47%. So why did the county say the daily number was only 4.1%? And it was even worse looking at the five-day average, where Weaver’s calculation put the positivity rate at 14.4% while the county still reported only 4.1%
A day later, Megan Noel, Boulder County Public Health’s COVID-19 surveillance co-lead, got back to him with an answer. Boulder County was using the state’s method for calculating positivity by only looking at the ELR numbers.
“This has been a big discussion this week because one of the labs that has not been able to report to this dataset to date is CU-Wardenburg, and as you know there are a large number of active student cases contributing to our overall case count,” she wrote. “This is why it’s so difficult to match up the case and testing numbers.”
CU’s Wardenburg Health Center is the on-campus student health clinic, making it the most likely place for students to get tested. Not having its numbers included in the test positivity calculation rendered the metric useless, Weaver said.
“I was really disturbed by this huge discrepancy,” he said.
By the end of September, CDPHE had figured out a stripped-down way to get labs to submit results electronically, even if they weren’t yet hooked up to the full ELR system. Wardenburg was the first lab connected through this method.
But the process of bringing labs into some kind of electronic reporting system takes time. On Saturday, CDPHE announced that it had finally added roughly 9,000 testing encounters, including 105 positives, from another large lab, Biodesix, to its ELR system. The influx of old tests was so big that CDPHE said the numbers could change some counties’ restriction status on the state’s COVID-19 dial.
“CDPHE has notified affected counties and local public health providers and continues to provide support in their mitigation efforts,” the agency wrote in a news release.
As labs go through the process of getting connected to electronic reporting systems, though, CDPHE said it still has no way of knowing exactly how many tests are being conducted or how accurate its test positivity calculations are on any given day.
“It is impossible to provide an accurate percentage of all tests being captured in ELR,” the agency’s communications staff wrote in an email to The Sun. “Providers are implementing newer point of care tests all the time and we would be unaware of the volume that provider is testing until they complete onboarding to ELR.”
A belated discovery
CDPHE learned of a new issue with electronic reporting of tests late last month when it discovered the nearly 16,000 missing Kaiser Permanente tests.
CDPHE said Kaiser on Oct. 1 changed the format on the file it sends to the state health agency, causing the state’s computer system to reject the report. Kaiser spokesman Nicholas Roper said the formatting change was needed to comply with new federal regulations and said CDPHE had assured Kaiser initially that the test data was being received correctly. He said the error was not Kaiser’s fault. CDPHE later confirmed that the error occurred on their end of the handoff.
The issue meant that hundreds of tests a day between Oct. 1 and Oct. 22 — including as many as 100 positive cases per day — weren’t getting entered into CDPHE’s system.
Most of the tests were conducted in the Denver metro area, and the announcement that the previously rejected results had been discovered and added into the system appears to have caught county public health departments off guard.
CDPHE sent a news release about the tests to reporters on the evening of Oct. 24, a Saturday. But several local public health agencies told The Sun they were not informed of the newly discovered tests until the following Sunday or Monday, Oct. 25 and 26.
The county health agencies said the newly added tests didn’t much change their counties’ coronavirus trend lines or positivity percentages, in large part because the tests were distributed over a long period of time. But not learning about the positive cases earlier put counties in a bind.
Ashley Richter, the communicable disease epidemiology manager for Tri-County Health, wrote in an email to The Sun that her agency didn’t know about the newly discovered Kaiser tests until the morning of Oct. 26. Had it known sooner, it could have brought in additional staffing on that Monday to handle the sudden surge of work that came with so many positive cases flooding in at once.
“We would also have been able to work earlier with our local, school, and county partners to address impacts of this data glitch on the metrics they utilize to make decisions,” she wrote.
Kate Watkins, with Jefferson County Public Health, said not knowing about the positive cases sooner made for a missed opportunity to better contain the virus.
“We would have been able to contact these cases sooner, and in a more meaningful timeframe to their illness to promote them staying at home and informing their close contacts to stay at home as well,” she wrote in an email.
For its part, CDPHE told The Sun that state and local public health workers were focusing on contacting the most recent cases “for which contact tracing is still relevant.”
“This is so our capacity is used in the most efficient way to reduce the spread of COVID-19 to the greatest extent,” CDPHE’s communications staff wrote in an email to The Sun.
It remains unclear, though, how many of the newly discovered cases went through contact-tracing.
The cautious calculation
To Weaver, the Boulder mayor, the situation with the state’s coronavirus data is a little baffling.
Why isn’t there a better handle just on the total number of tests run? Why isn’t there better analysis of where in a county cases are spreading most?
He has asked for access to data reported through ELR, but Boulder County health officials have told him they can’t release it — it contains information that potentially could identify the individuals who were tested, so releasing it would violate medical privacy laws.
But, most on his mind, Weaver said, is the issue of reliability. How can leaders make good decisions when there’s at least a little uncertainty about the data they are using?
Weaver concluded there’s a logical answer: If test positivity could actually be higher than what the official number shows, he said, then the state should have a lower threshold for when it considers tightening restrictions. Caution is the best approach to what you know you don’t know.
“I just think,” he said, “everyone up and down the food chain needs to acknowledge that.”
This story was updated at 10:50 a.m. on Nov. 2, 2020, to include comment from a Kaiser Permanente spokesman.
This story was updated at 10:10 a.m. on Nov. 3, 2020, to include a new clarification from CDPHE that the error regarding the handoff of test data from Kaiser Permanente occurred on the state’s side and was not Kaiser’s fault.