In the early hours of Feb. 21, 2018, malware began spreading rapidly throughout the Colorado Department of Transportation computer network. As employees turned on their computers, messages popped up on screens that files had been taken hostage. If users wanted them back, they would have to pay — in bitcoin.
“The impact was fast and furious,” said Debbi Blyth, the state’s chief information security officer, as she shared the story of how CDOT’s network was taken down, brought back and then taken down again by malware known as SamSam. “Server administrators, database administrators, backup administrators and hundreds of calls to the help desk — and that was all in the hour or first couple hours of opening for business.”
The attack on CDOT was one of a number of high-profile ransomware attacks that year. Other SamSam victims included the University of Calgary, the Port of San Diego and the city of Atlanta. Some paid the ransom. CDOT didn’t, but it wasn’t pretty. Ultimately, it cost the state $1.7 million in overtime, meals and equipment.
But what’s not calculated in those costs are the security improvements made during and since the attack. Nor does it account for the increased security awareness among state employees, who better understand the importance of backing up data and using two-factor authentication to verify a user’s identity twice. And there was the formation of new partnerships between the disparate support crew from various state and federal agencies, private vendors and other cybersecurity professionals.
“It was just a unique atmosphere to be able to work together like that,” said Col. Isaac Martinez, who recruited a cybersecurity crew from the Colorado National Guard. “Even the vendors thought this is pretty unique, but pretty cool at the same time.”
Blyth, who works in the Governor’s Office of Information Technology, shared details of what went down that 2018 winter while speaking at the Cyber Summit at Regis University last week and what it took for her team to tackle the malicious miscreant with two first names.
A slip in security
The attack really started three days earlier.
CDOT wanted to try out a new cloud-based business process so an employee from Blyth’s office, which manages technology for 17 state agencies in 1,300 locations, set up a temporary server for testing. It was connected to the internet and to the CDOT domain using an administrator account.
“Because it was only going to be a temporary server and only up for a couple weeks, the system administrator who created this did not apply any of our standard security controls to it,” Blyth said. “The server was discovered almost immediately by attackers.”
That was Feb. 18, a Sunday.
Her office later learned from forensics experts that hackers found the insecure system on the same day the temporary server went live. They tapped in remotely and began guessing passwords. Some 40,000 attempts later, the account was unlocked thanks to its privileged features like unlimited failed logins. The system was compromised on Feb. 20 as invaders wandered into CDOT’s main network and began sniffing around to see what was available.
But it wasn’t until Day 3 that anyone noticed. Databases began failing. Applications stopped working. And, of course, there were the threatening messages that files would be deleted unless users paid up. The state never disclosed the amount of bitcoin demanded, but attackers included instructions on how to get the cryptocurrency to them. CDOT ordered employees to turn off their computers. Work phones, because they also used the internet, were taken offline.
Employees could still use personal devices or log in to online services like Google’s Gmail, recalls Amy Ford, who at the time served as CDOT’s spokeswoman. But to stop the spread, certain business operations, such as processing $100 million each month to pay employees and vendors, were inaccessible.
“The immediacy of trying to pull everyone offline (triggered) natural questions from employees. What about this? Can I do that? How do I handle my flash drive?” said Ford, who noted that the agency created an incident command center similar to the one it used during the 2013 floods. “We’d do a series of daily updates with employees to let them know what was happening and what they could do or could not do. … Much of my work I did on my personal computer and cellphone.”
Employee concerns were real, said Dan Likarish, a now-retired Regis University cybersecurity professor who helped with the recovery.
“The CDOT manager, he was worrying about ‘I gotta get people paid on Friday. Are my HR systems compromised?’” Likarish recalled. “If you’ve got somebody living paycheck to paycheck like we all do, that’s pain.”
Balking at bitcoin
Blyth’s team immediately took over from CDOT and made the decision not to pay the ransom.
The state had its reasons. There were backups, thanks to an effort a year earlier called Backup Colorado. And there was network segmentation, which meant the attacked CDOT system could be disconnected from other state agencies’ networks.
“We knew that was working in our favor,” Blyth said. “So for us, we didn’t even consider paying the ransom because we knew we could restore it.”
At this time in ransomware history, there was a growing awareness in this newer form of malware that encrypted files and threatened to delete them unless the victim paid a fee.
Paying the ransom was and still is frowned upon by federal authorities. That’s because you don’t really know if the attacker will give you the right decryption key. Or you might not be able to access the data anyway. And then there’s the thought of emboldening criminals.
“The last thing you want to do is pay,” said Likarish, who ran Regis University’s Center on Information Assurance Studies. “Nothing good happens if you pay because it’s criminal and they know what they’re doing. You pay once and then you pay more and more.”
By 2016, ransomware attacks had tripled from 2015, occurring at a rate of 4,000 a day, according to a report from the U.S. Department of Justice. A year later, then-U.S. Deputy Attorney General Rod J. Rosenstein estimated that total ransom payments were approaching $1 billion a year. Attacks were getting more sophisticated, taking down healthcare providers and local governments.
SamSam was first spotted in late 2015 and there were sporadic reports of attacks. A month before the CDOT incident, SamSam stung the city of Farmington, N.M., demanding three bitcoins, according to the Farmington Daily Times. The city didn’t pay because it had backups, but it took staff several months to recover everything, said Georgette Allen, the city’s spokeswoman, in an interview.
But others did pay. Hancock Health, an Indiana hospital, paid a ransom of four bitcoins, or about $55,000 at the time, according to the Daily Reporter in Greenfield. The hospital recovered the files and had its system up and running about three days later.
These days, ransomware attacks have declined against individual users but increased against larger enterprises, according to the FBI, which encourages victims to report attacks. Attacks are also more sophisticated and seek higher ransoms. Last October, for example, attackers demanded $1.2 million (U.S.) worth of bitcoin from a Canadian insurance company. The sum was negotiated down to $950,000 and paid. (Interestingly, a specialist hired by the firm tracked down the cryptocurrency payment to Bitfinex, a place where users can exchange cryptocurrency for dollars. The company sued the exchange and a London judge issued an injunction. The case is ongoing, according to New Money Review.)
But there may be another reason for the rise: cybersecurity insurance. The market grew 14.54% to $3.6 billion in 2018 from a year earlier, according to The Center for Insurance Policy and Research.
Investigative news site ProPublica calls the growing industry “The Extortion Economy.” Insurers often advise clients to pay the ransom because it’ll likely cost less and save on recovery time. As one insurance industry representative noted: the insurer’s mission isn’t to stop the bad guy but help clients get back to business.
Regis University was attacked by ransomware just before school started in August. The school had insurance and paid the ransom, though it won’t say how much. Salvador Aceves, Regis’ chief financial officer, said the decision to pay was based on the advice and confidentiality agreements with its third-party partners, such as its insurer. But it was also partly the school’s choice.
“It’s really a very difficult decision to find yourself in,” Aceves said. “Do you have the capacity to be able to recover?”
But a downside to paying the ransom, Likarish points out, is that you don’t always get to know what went wrong.
“As soon as you call them and activate that policy, your people are off to the side,” Likarish said. “And that’s miserable because as soon as the insurer’s team … turns the network back to you, you’re at the exact same spot as you were when things went down. Some of it is back up and running, but you have no good awareness of what happened. What did they do?”
CDOT didn’t have insurance. But Blyth felt confident recovery was doable. Only CDOT’s business operations were impacted. Those functions are separate from its traffic operations, so services like traffic alerts and cameras were protected. Had traffic operations been affected, she said, this would have been a public-safety event.
The CDOT system was restored seven days after the attacks began, and Blyth’s team felt pretty good. Everyone went home Feb. 28 “feeling so victorious,” she said.
That didn’t last.
“We came in the next morning and we found new attacker activity all over the place,” she said. “They had launched more instances of the SamSam ransomware and we could see their tracks all over the network. So it was on that day, March 1, that I said, we do not have this under control. I need help.”
Plan B
Blyth called the state’s Office of Emergency Management, which reached out to the governor who, in turn, declared a state emergency. It was the first time a state had declared an emergency based on a cyber attack. Since then, Ohio, Texas and Louisiana have done so.
The main reason she wanted an emergency declared was the Colorado National Guard, which is at the governor’s beck and call. The organization has a team of trained cybersecurity professionals. About a half-dozen folks from the National Guard, led by Col. Martinez, showed up to help the next day, a Friday, eight days after the initial attack.
Immediately, the Office of Emergency Management brought order to the chaos. Blyth’s team had been working 18-hour days and eating pizza or other junk food — and ordering too much of it. Trash piled up. Everyone was exhausted — and also dehydrated because there was only one working drinking fountain in the building.
“They had everybody who was working on the internet response team badge in on their way in and out. This meant they always knew exactly how many meals we needed, how many people to feed,” Blyth said. “When they started bringing in pallets of bottles of water, we started thinking more clearly. It made a difference to us.”
“I wished I had called them a week sooner,” Blyth said.
And the National Guard did its job — it offered relief.
“We’ll jump in so you get a good night’s rest so you can feel more refreshed and come back and tell us where you need some additional help,” said Martinez, whose team arrived dressed in plain clothes instead of military fatigues so it wouldn’t seem like they were taking over. “I think that part helped them out because then we got looked at more as value-added versus, ‘Oh geez, here’s another person.’”
When he stepped into the building, Martinez saw what he called “organized chaos.” Everyone knew what they were doing but would often get interrupted before the job was done. The Guard brought structure, such as having two daily reporting meetings. A list of priorities written on butcher paper was taped to the walls. And at every meeting Martinez would ask “Is there a new malware infection or has new malware been identified?”
“Anytime we met, we would ask those questions to make sure that are we all still solving the same problem or did a new one just arrive,” he said. “We needed to know if nothing was new or if it’s a new one. Then we’d have to ask for more help.”
The National Guard wasn’t the only group that arrived to help. An alphabet soup of agencies joined in: DHS, FBI and FEMA. At the peak, about 130 people were working on containment each day.
One of the problems with SamSam was that the strain was so new, CDOT’s anti-malware software didn’t detect it. Blyth said the security company later ended up using the CDOT malware sample to build a signature to protect its other users from SamSam.
When SamSam unexpectedly returned to CDOT computers, it was because the attacker left behind other tools on computers. The team was looking only for SamSam. Restoring the network to an earlier point from the backups didn’t eliminate the tools — and SamSam started up again.
This is one of the frustrations in the battle against malware. No one knows what’s bad until it hits. And while anti-malware makers can then build a program to protect against that bad guy, one small variation or another unknown and the malware can slip by unnoticed.
Blyth had a fix for that. Her team had already been rolling out a new tool that would detect unusual activity typical of malicious software. The detection technology was already purchased before the CDOT attack but it was being rolled out slowly. CDOT was scheduled to get the new technology, called an end-point detection and response tool, a week after SamSam struck.
The incident staff created a test network and put infected computer systems on it. The new tool detected the malware and quarantined the bad stuff to prevent it from spreading.
“We tested this a number of times. It gave us much more comfort that it was okay to go around and start turning on those systems again,” Blyth said. “…By March 6, we were able to declare that we had containment underway and on March 9, the Colorado National Guard and other partners were able to depart.”
And two weeks later, or 31 days after SamSam made its presence known, the agency announced that CDOT systems were 80% restored.
“It was beautiful,” Likarish recalled about the experience. “In that room, you’ve got state of Colorado people and they’re tired, they’re beat up. This is all new to them… What (Director of Emergency Management Michael) Willis provided was direct authority.”
Willis, who also served in the National Guard, would make sure things got done. At daily all-hands meetings, he’d check with each group for updates.
Likarish points to a photo that showed one such meeting. In the image, there are dozens of people in a war-room-like setting. Everyone’s eyes are on Willis. Likarish recalls the date was March 6, the day the team declared containment was accomplished.
“That was a critical point where we went from, ‘Have we defended and do we know where we’re at?’ to ‘Let’s begin the restoration.’ All of a sudden, the people that were defending, they started moving into restoring,” he said. “Within days, by the 23rd, everything is back up and running.”
Aftermath
Blyth now manages a $17.7 million cybersecurity budget, which has doubled during her five-year tenure.
As SamSam was still being contained, she had convinced the state legislature to increase her budget by $11.9 million to speed up many security projects, including installing the special detection tools across all 17 agencies. Thanks to the budget increase, it took 48 hours to accomplish the updates instead of four weeks.
“I always say never waste a good security incident,” Blyth said. “While you have everybody’s attention, like CDOT, we’d say something like, ‘We want to implement two-factor authentication,’ and they’re like, ‘Do it!’ And then we would say to the other agencies, we’re going to implement two-factor authentication. And they would be like, ‘Do it! Don’t let what happened to CDOT happen to us.’”
As her team was working through the process, they would notice security issues that were already on their to-do list and fix them. New firewalls were installed to track everything going in and out of state computer networks. Vendors were required to use two-factor authentication. The number of privileged administrator accounts was reduced to 50 from 500.
In November 2018, a federal grand jury indicted two Iranian men who are accused of attacking CDOT with SamSam. They’re also accused of unleashing the malware on the cities of Atlanta and Newark, N.J., LabCorp, several health care facilities and others — and collecting $6 million in ransom payments and causing another $30 million in additional losses for the victims. A representative from the U.S. Department of Justice said the charges are still pending and the defendants remain at large.
Ransomware is still around, but it’s already changed from the days of SamSam’s attack on CDOT, said Tyler Moffitt, senior threat research analyst at Webroot in Broomfield. Ransomware is now more targeted, focusing on small and medium-sized businesses and government agencies.
Those groups have something in common: “a lot of computers (but) not a lot of IT staff,” said Moffitt, who advises regular back ups as a low-cost way to recover data.
But worse, he added, there’s a newer and more malicious trend evolving: Ransomware is now making copies of valuable files and sending them to the attacker.
“What they’re telling people now is if you don’t pay this ransom because you feel you’ve got good backups or you don’t need it, we will now threaten to breach you and release that customer information that you had, forcing you to acknowledge that you are compromised,” he said.
Besides regularly backing up files, security experts advise using two-factor authentication. Managing firewalls can limit or block network traffic coming or going through connections like TOR, a popular software with hackers because it renders them anonymous.
Colorado’s state government sped up its security overhaul because of SamSam. In retrospect, Blyth said that she wouldn’t have waited so long to call the Office of Emergency Management for help. And she still wouldn’t have paid one bit of bitcoin.
“There will be security incidents,” Blyth said. “But the controls that we’re putting in place should hopefully reduce the likelihood (of them) and we’ll also reduce the impact because we will be faster at detecting and be able to contain it quicker.”
Already there have been results. A few months ago, malware known as TrickBot showed up and spread to 18 computers. But Colorado’s new detection software stopped it from executing, she said. Because of the additional investment Blyth received to modernize state security, she’s getting noticed.
“By taking the hit like she did, now she’s got improved funding,” said Likarish, the retired cybersecurity professor at Regis. “She got everybody’s attention. It could have gone south in so many different ways but it didn’t. Good things can happen.”