14 DAY TRIAL // The Apache Infrastructure Team has released a detailed analysis of the recent attack that led to multiple apache.org servers being compromised. After outlining the mistakes that made the incident possible and their plan to strengthen security, the admins have been congratulated by the community for their openness.
The full report published on the Apache Foundation's blog starts by stressing that, "At no time were any Apache Software Foundation code repositories, downloads, or users put at risk by this intrusion," and explains that, "Providing a detailed account of what happened will make the internet a better place, by allowing others to learn from our mistakes."
It was confirmed that the point of entry for the attackers was the server hosting the Apache Conference website (apachecon.com), which was being maintained by a third-party company. The attackers gained root privileges on the machine, possibly by using a local privilege escalation exploit. There is few information available about how they got access, because they deleted the logs.
What's certain, though, is that they used the SSH key associated to an account the Apache Infrastructure Team had on that server for backup purposes, to jump to people.apache.org, the Foundation's "staging machine for our mirror network," as it is called in the report. The newly obtained access was used to write CGI scripts into the document root of the apache.org website, which then got propagated on all mirrors, due to automatic sync processes. These scripts were later executed by the attackers over HTTP in order to obtain remote shells.
The first thing that the Apache Infrastructure Team criticize themselves for is the SSH keys implementation, which, according to their own account, left a lot to be desired. "We did not restrict SSH keys appropriately, and we were unaware of their misuse," they write. The second one is leaving ExecCGI enabled, even though most of their websites don't require it. Finally, the current setup of the rsync and logging processes also contributed to the success of the attack.
The admins are in the process of making changes to address several of these problems. These involve, but are not limited to, requiring all users with elevated privileges to use OPIE for sudo on certain machines, recreating and using new SSH keys, one per host, for backups, while also enforcing use of the from="" and command="" strings in the authorized key file on the destination backup server, disabling CGI support on most website systems and re-implementing measures such as IP banning after several failed logins, on all machines.
"What really impresses me, however, is how well Apache handled the potentially highly embarrassing incident – taking swift action and keeping their users informed via blog updates. […] So bravo to Apache for responding to the problem rapidly and with openness, proving it is possible to turn a potentially bad story into a positive experience," Graham Cluley, senior technology consultant at antivirus vendor Sophos, comments.