The Apache Project's Infrastructure Team was forced to take its primary servers offline yesterday, after discovering that unknown hackers uploaded and executed malicious code on them. The attackers apparently used a stolen SSH authentication key associated with a backup account to break in.

The attack started during the evening of August 27 and targeted the minotaur.apache.org aka people.apache.org server. According to the Apache team, this is the "seed host for most apache.org websites" and also hosts accounts for all developers.

The perpetrators logged in to the server running FreeBSD 7-STABLE using the SSH key corresponding to an account employed to perform automatic backups for the ApacheCon website. Fortunately, they did not succeed in escalating the account's privileges on the server.

Using the compromised account's access to the directory housing the www.apache.org website, the attackers proceeded to uploading several CGI scripts and other files. These rogue files were then copied by automatic sync processes to most of the project's webservers.

The Apache Infrastructure Team notes that, during the morning of August 28, the CGI scripts were executed remotely via HTTP, resulting in unauthorized processes being created on eos.apache.org, which alerted the admins. "Within the next 10 minutes we decided to shutdown all machines involved as a precaution," the team notes.

A preliminary investigation revealed that one server called eris.apache.org was completely unaffected, so it was used to convey a downtime alert for most apache.org services. The administration later changed all websites to point to aurora.apache.org, the project's European backup mirror, which had the rogue files copied on it, but not executed, and was easier to clean.

Several servers are still offline, but most public services are available again, the Apache team announces. The investigation goes on and, while there is yet no reason to believe that Apache-related downloads have been affected, users are advised to employ the available digital signatures to check the authenticity and integrity of the files.