SolarWinds hack: What we know, and don't know, so far
Yes, this feels like the big one
I keep hearing this from people I respect: It’s hard to overstate how serious the SolarWinds hack is. So, yes, it seems to be the Big One. I suspect we’ll be hearing about the damage for years. This piece is a roundup of what I think we know about it on Friday at midday.
But note: While security experts continue to pick through the digital wreckage left behind, the forensics will take a long time. You’ll see hundreds of stories speculating on what really happened. In a situation like this, very few people know the whole story, so read everything — including this story — with a skeptic’s eye. Understand that almost everything we’ve heard is from a third party.
Quick Review: SolarWinds provides management software named Orion that is used by many major government agencies and more than 400 of the Fortune 500 companies. In March, criminals slipped Trojan horse software into an Orion update, ultimately giving the criminals access to many systems that interfaced with Orion at all these organizations. It could take years to undo the damage; or, organizations could never really know what kind of data was stolen during these past nine months.
My biggest unknown at the moment: What did Covid-19 have to do with this? The timing could be coincidental. But the infiltration seems to have occurred right as American companies and government agencies were scrambling to manage the abrupt transition to a work-from-home environment. It’s easy to see how that chaos could have contributed to this hack. Perhaps the timing was even intentional. That’s my speculation.
Whatever doubt remained that SolarWinds was a massive incident was lifted on Thursday, when the Department of Homeland Security’s Cybersecurity & Infrastructure Security Agency pulled the fire alarm with this “grave threat” notice:
“CISA has determined that this threat poses a grave risk to the Federal Government and state, local, tribal, and territorial governments as well as critical infrastructure entities and other private sector organizations …
“This is a patient, well-resourced, and focused adversary that has sustained long duration activity on victim networks.
The SolarWinds Orion supply chain compromise is not the only initial infection vector this APT actor leveraged.
…just in case you thought companies could remove the SolarWinds hack and wipe their hands clean.
The best piece I’ve seen so far (not a surprise) about the incident is from Robert McMillan and Dustin Volz at the Wall Street Journal. There are good nuggets in here about how the hack was discovered, and some sober realism about how long it might take to assess the damage.
“The SolarWinds attack so eluded U.S. security measures that it was discovered not by intelligence officials but, almost accidentally, thanks to an automated security alert sent in recent weeks to an employee at FireEye, which itself had been quietly compromised….
“The warning, which was also sent to the company’s security team, told the employee of FireEye that someone had used the employee’s credentials to log into the company’s virtual private network from an unrecognized device—the kind of security message that corporate workers routinely delete. Had it not triggered scrutiny from FireEye executives, the attack would likely still not be detected, officials say…
“But because it went undetected for so long and due to the expertise of the hackers, thousands of potential victims may never be able to know for sure whether they were compromised, security experts say….
“SolarWinds said it released a quick fix that patched the security issue for customers this week. But experts have warned that merely cutting off the access point for hackers won’t guarantee their removal, especially because they would have used their time inside those networks to further conceal their activity…..
“While intelligence officials and security experts generally agree Russia is responsible, and some believe it is the handiwork of Moscow’s foreign intelligence service, FireEye and Microsoft, as well as some government officials, believe the attack was perpetrated by a hacking group never seen before, one whose tools and techniques had been previously unknown.”
This Politico story suggests hackers may have access servers at the federal agency which manages nuclear weapons and that FERC — Federal Energy Regulatory Commission — might have gotten the worst of it. Remember, it’s early in the investigation, however.
“The hackers have been able to do more damage at FERC than the other agencies, and officials there have evidence of highly malicious activity, the officials said, but did not elaborate...
“The attack on DOE is the clearest sign yet that the hackers were able to access the networks belonging to a core part of the U.S. national security enterprise.”
Reuters alleged that Microsoft “was hacked” and its software was used to hack other firms, also, though Microsoft has not said so. It’s no surprise to hear conflicting reports at this stage.
“Microsoft also had its own products leveraged to attack victims, said people familiar with the matter. The U.S. National Security Agency issued a rare “cybersecurity advisory” Thursday detailing how certain Microsoft Azure cloud services may have been compromised by hackers and directing users to lock down their systems….
“Still, another person familiar with the matter said the Department of Homeland Security (DHS) does not believe Microsoft was a key avenue of fresh infection.
For its part, Microsoft’s Brad Smith penned a blog calling the incident “a moment of reckoning” for the world. He specifically called out private firms that sell hacking software, likening them to digital mercenaries. And he named names.
This phenomenon has reached the point where it has acquired its own acronym – PSOAs, for private sector offensive actors. Unfortunately, this is not an acronym that will make the world a better place.
One illustrative company in this new sector is the NSO Group, based in Israel and now involved in U.S. litigation. NSO created and sold to governments an app called Pegasus, which could be installed on a device simply by calling the device via WhatsApp; the device’s owner did not even have to answer. According to WhatsApp, NSO used Pegasus to access more than 1,400 mobile devices, including those belonging to journalists and human rights activists.
NSO represents the increasing confluence between sophisticated private-sector technology and nation-state attackers. Citizen Lab, a research laboratory at the University of Toronto, has identified more than 100 abuse cases regarding NSO alone. But it is hardly alone. Other companies are increasingly rumored to be joining in what has become a new $12 billion global technology market.
Early on, the Washington Post blamed Russia-based hacking group known as Cozy Bear for the attack. Sen. Richard Blumenthal (D-CT) appears to have publicly blamed Russia, too. Others have not been so quick to attribute the hack to the Russian gang.
The Russian hackers, known by the nicknames APT29 or Cozy Bear, are part of that nation’s foreign intelligence service, the SVR, and they breached email systems in some cases, said the people familiar with the intrusions, who spoke on the condition of anonymity because of the sensitivity of the matter. The same Russian group hacked the State Department and the White House email servers during the Obama administration.
For an interesting perspective on a potential root cause of the problem, here’s a blog post by an IT worker suggesting local governments are relying too much on automated tools, and not enough on human capital, to fight off hackers.
Rather than rely on the purchase of services and expertise, these agencies should invest in their staff so that they maintain the ability to detect and respond to hacks in real-time. Local, trained staff will notice unusual occurrences or patterns on established platforms more thoroughly than a software-only solution. Should the software solutions and consultants be abandoned? No. They usually provide solid reliable information that can be used to strengthen the defense against hacking. I prefer to think of them as a race car, and in-house, trained staff as the drivers.
Finally, I asked Ben Rothke, a long-time cybersecurity professional and author of several books, for his perspective on the SolarWinds attack. Rothke is now Senior Information Security Specialist at Tapad. Here’s what he told me. I’m particularly fond of the bit about companies using cheap storage to facilitate a dangerous pack-rat mentality about data.
“Wendell Phillips noted 150 years ago that "eternal vigilance is the price of liberty." With some poetic license, in 2020, it would be "eternal network vigilance is the requirement for Internet connectivity."
“It is easy to point fingers at SolarWinds, Microsoft, and the various federal agencies. But if a nation-state has teams of well-trained and experienced hackers, who are dedicated and politically motivated to penetrate your infrastructure, it is a challenging attack to defend against.
“Look at it this way; no one will tell you that Fort Knox is impenetrable. But the US Army has made it so incredibly difficult that there have been no direct attacks against the facility. Adding to that is the reality that a bar of gold weighs almost 28 pounds. So, running out with 70 gold bars, as they do in the movies, means the culprit can carry a ton of gold. That does not happen in the real world.
“But our new reality means attackers can move lots of data, which is the new gold, with ease, from far away.
“A complex and sophisticated problem like nation-state attacks is not quickly solved, contrary to what a lot of the security vendors may be telling you.
“So, what is the solution? John Kindervag, then of Forrester Research, created the notion of zero-trust network architecture. But creating a sophisticated architecture like that takes time and effort. Until then, network monitoring's eternal vigilance is the way to know if someone is attacking you and in your network.
“Finally, with storage so incredibly inexpensive, firms are storing far too much data than they need to. They need to start thinking of offloading and retiring data that is no longer needed.
“Ultimately, the current situation is akin to the reality of My 600-lb Life. There are no quick fixes; success is often elusive. But with enough effort and time, success can be achieved.