14 DAY TRIAL // GitHub has been hit by another distributed denial-of-service (DDOS) attack. Since the attack has been bigger than previous ones, the code repository has decided to share some details about it and the company’s mitigation strategies.
As most GitHub users will tell you, DDOS attacks against the service are highly common. In fact, they’re so common that GitHub has become pretty good at mitigating them. It usually takes 20 minutes to restore access to the service.
However, this time, the website was not prepared to handle an attack such as the one that started on March 11 at 14:25 PDT.
GitHub is prepared to handle both volumetric and complex attacks. While the latest attack did not generate more bandwidth than previous ones, it did generate a lot more packets than the code repository was used to.
“As we began investigating we noticed an apparent backlog of connections at our load balancing tier. When we see this, it typically corresponds with a performance problem with some part of our backend applications,” GitHub’s Mark Imbriaco wrote in a blog post.
“After some investigation, we discovered that we were seeing several thousand HTTP requests per second distributed across thousands of IP addresses for a crafted URL,” Imbriaco added.
“These requests were being sent to the non-SSL HTTP port and were then being redirected to HTTPS, which was consuming capacity in our load balancers and in our application tier. Unfortunately, we did not have a pre-configured way to block these requests and it took us a while to deploy a change to block them.”
At 15:35 PDT, the malicious requests were blocked, but the attack was also designed to exhaust GitHub’s SSL processing capacity so it took more time to address this vector as well. The site was completely restored at 16:34 PDT, but the attack carried on for some time even after that.
“We had the ability to mitigate attacks of this nature in our load balancing tier and in our DDoS mitigation platform, but they were not configured in advance. It took us valuable minutes to configure, test, and tune these countermeasures which resulted in a longer than necessary downtime,” Imbriaco said.
“We're happy that we were able to successfully mitigate the attack but we have a lot of room to improve in terms of how long the process takes.”
Now, GitHub is working on improving its DDOS mitigation capabilities even more to ensure that even uncommon cybercriminal operations can be blocked quickly.