A Devastating Irony

 

I have been really busy with work lately. So busy, in fact, that I haven't been able to catch up on the latest cyberattack that's making it's rounds through the noosphere. I am referring to the SolarWinds incident in December 2020, just in case you are reading this from a temporally displaced context. When analyzing something like a network security incident for a customer or stakeholder, there are a few questions that need to be answered. These questions are questions that the popular media seems to have a phobia about answering for the non-technical readers. The reason for this is twofold and, quite simple - firstly that putting concrete terms on breaches limits (or takes the lid off of!) the amount of sensationalism that can be used, and secondly, it makes Fortune 500 IT security operations and enterprise software vendors look terminally incompetent - which actually can't be avoided. But, more on that in another piece. 

So, let's pretend I'm giving a report to a non-technical person in a professional capacity here. The news is dregs, so the Common Vulnerabilities and Exposures (CVE) database is where one finds facts about these things. Threats, once cataloged are assigned a CVE Number. Step one is to find the CVE number of the threat and read about it. 

Multiple Vulnerabilities in SolarWinds N-Central Could Allow for Remote Code Execution

Immediately we see it is a Remote Code Execution threat. These are among the most dangerous for a network operator, as this would grant the malicious actor the ability to run arbitrary programs on the compromised systems - e.g. "take over the computers on the network". This is how crypto-lockers work."Great" says our intrepid network and server administrator. "What's the damage and how many weekends will I be working to fix this?", he or she might say at this point. 

There are 5 CVE entries associated with this threat.  

CVE-2020-25617: The AdvancedScripts HTTP endpoint allows Relative Path Traversal by an authenticated user of the N-Central Administration Console (NAC), leading to execution of OS commands as root.

CVE-2020-25618: The sudo configuration has incorrect access control because the nable web user account is effectively able to run arbitrary OS commands as root (i.e., the use of root privileges is not limited to specific programs listed in the sudoers file).

CVE-2020-25620: Hard-coded Credentials exist by default for local user accounts named support@n-able.com and nableadmin@n-able.com. These allow logins to the N-Central Administrative Console (NAC) and/or the regular web interface

CVE-2020-25621: The local database does not require authentication: security is only based on ability to access a network interface. The database has keys and passwords.

CVE-2020-25622: The AdvancedScripts HTTP endpoint allows CSRF (Cross-Site Request Forgery) 

Every single one of these vulnerabilities is exteremely dangerous to the organization that is hosting them - be it externally or internally. The risk of bad actors is much lower on internal networks, but compromised laptops can be a source of attacks, and if the internal applications are still vulnerable, unpatched internal applications are still a threat vector. 

CVE-2020-25620 and CVE-2020-25621 seem particularly egregious to me. Since I am penning this from the perspective of analysis of the threat, I will not go down this particular thread other than to say that they are both the product of poor software design choices and a willingness to forgo security for convenience - and, that almost every software company in the world does something like this, somewhere in it's code. These 2 could be severe enough to consider losing a contract over. And, it makes the developers over at SolarWinds look really bad." 

So what is the impact and risk? To answer that question, we need to know what the software in question actually does. What is the "problem space" within which we are dealing with this threat? Let's see what SolarWinds N-Central does.  

(https://www.solarwindsmsp.com/products/n-central)

It is a tool for managing devices remotely (servers, desktops, firewalls, etc.) and, it appears to be riddled with backdoors. 

And that's where we are now. The thing that makes my head spin on this one is the sheer scope of this. And, to demonstrate the non-hyperbolicity of that statement, the National Cyber Awareness System in the US has this threat cataloged as Alert (AA20-352A): Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations. Bold text highlighted by myself. 

The Cybersecurity and Infrastructure Security Agency (CISA) is aware of compromises of U.S. government agencies, critical infrastructure entities, and private sector organizations by an advanced persistent threat (APT) actor beginning in at least March 2020. This APT actor has demonstrated patience, operational security, and complex tradecraft in these intrusions. CISA expects that removing this threat actor from compromised environments will be highly complex and challenging for organizations.

On December 13, 2020, CISA released Emergency Directive 21-01: Mitigate SolarWinds Orion Code Compromise, ordering federal civilian executive branch departments and agencies to disconnect affected devices.  

(From https://us-cert.cisa.gov/ncas/alerts/aa20-352a) 

This is an extreme step that would be used to contain an immediate, possibly active threat that utilized something specific to a specific type of device - and guidance like this is rare in my experience. I have done it once in an 8-year stretch operating a public network at the direction of an operating VP.

In my analysis, the best use of limited resources at impacted organizations would be to focus on threat confinement and mitigation. The proximate "causes" of this, as much as they can be defined, can be generally enumerated as: 

• Hard coding of default user accounts and passwords in application code
• HTTP endpoints which do not follow best security practices in use since the middle of the last decade.
• An application server configuration which allows for at least 2 ways to elevate application security context and 1 way to execute arbitrary code at that elevated context. 

Collective human error. In all seriousness, the list above can be read as a "How not to do application security" list. And my suspicion is each of those individual issues looked small to the last person who "passed" on the code review or whatever process happens in the development workflow at SolarWinds. In the context of this piece, it is worth calling out where we might begin to analyze the root causes - but first, the threat must be contained. 

I fear that this is bad news for the SolarWinds product here. Trust is a difficult thing to regain once it is violated, and trust has indeed been violated here. Our intrepid IT party has just been handed quite a bit of unplanned work to do - which is suddenly both very operationally impactful and "high-profile". The trust-violating aspect of it is that it is the very tool that was used to make their lives easier which is now responsible for the pain being felt. 

Remote administration is a very sharp edge. Any sharp tool can be used as a weapon. The network operator who refuses to think strategically is at a disadvantage among adversaries who do. The same can be said of software development teams. Somewhere, at SolarWinds, is a programmer who saw the code with the hard-coded credentials and signed off on the commit. It should actually be a straightforward matter to go back and see who made the change and who approved the change. There is exactly one scenario where I will retract the part of the blame form SolarWinds' development team  - that is if the attacker surreptitiously made the change in the codebase by bypassing procedural controls. I don't expect developers to treat their code repository as suspect every time they do a build. If that's the case, the network and infrastructure teams have failed at their mission. 

That last possibility concerns me because it would be quite sophisticated. The NCAS has plainly stated that this appears to be at a very concerning level of sophistication, perhaps state-sponsored. I will inject that it does not need to be state-sponsored - but someone will have been paying a skilled technical resource team or otherwise remunerating them for the time it would take to engineer and execute this attack. 

Indeed, if what I'm reading in the threat assessment is correct, that may have happened. 

The adversary added a malicious version of the binary solarwinds.orion.core.businesslayer.dll into the SolarWinds software lifecycle, which was then signed by the legitimate SolarWinds code signing certificate (emp. mine). This binary, once installed, calls out to a victim-specific avsvmcloud[.]com domain using a protocol designed to mimic legitimate SolarWinds protocol traffic.

I am legitimately impressed at the expertise of these threat actors. I also suspect some "inside knowledge" of the SolarWinds platform and it's protocols might have been involved. Some of the major takeaways from the US-CERT write up follow. 

SolarWinds Orion typically leverages a significant number of highly privileged accounts and access to perform normal business functions. Successful compromise of one of these systems can therefore enable further action and privileges in any environment where these accounts are trusted.

The adversary is making extensive use of obfuscation to hide their C2 (command and contrrol) communications. 

 

According to FireEye, the malware also checks for a list of hard-coded IPv4 and IPv6 addresses-including RFC-reserved IPv4 and IPv6 IP-in an attempt to detect if the malware is executed in an analysis environment 

 While not a full anti-forensic technique, the adversary is heavily leveraging compromised or spoofed tokens for accounts for lateral movement. This will frustrate commonly used detection techniques in many environments. Since valid, but unauthorized, security tokens and accounts are utilized, detecting this activity will require the maturity to identify actions that are outside of a user’s normal duties. For example, it is unlikely that an account associated with the HR department would need to access the cyber threat intelligence database.

Taken together, these observed techniques indicate an adversary who is skilled, stealthy with operational security, and is willing to expend significant resources to maintain covert presence.

We are dealing with a sophisticated threat here. I would like to point out that this appears to be using a level of patience and covert operation that I have not seen before in a widely-reported threat. Over the coming days and weeks, I'm sure more will come to light. Something tells me that this will be part 1 in a mini-series on the topic. 

Keep your systems patched.