14 DAY TRIAL // By monitoring the activity of web crawlers, a company offering distributed denial-of-service (DDoS) mitigation solutions noticed that in 4% of the cases the crawlers were impostors with fake user-agents that allowed them to pass as Googlebots.
Googlebots are crawlers that scour websites for information that can be indexed for the search engines. They are generally given limited access, since some parts of some websites contain sensitive information they should not have access to.
Incapsula has presented a new report showing that the spoofed Googlebots are used by threat actors to carry out malicious activities.
According to the data they collected, it appears that 34.3% of the fake crawlers were explicitly malicious, most of them (23.5%) being used for application layer DDoS attacks, while the rest dealt with scraping, spamming and hacking.
65.7% had purposes far from being nefarious, but still worrying, considering that they collected information used for marketing purposes.
Mitigating an application layer DDoS attack is not too easy, because the requests are directed at the application interface and mimic legitimate behavior, which makes filtering out the bad traffic more difficult.
Website admins that do not have solutions that can distinguish the fake web crawlers from the legitimate ones face a pretty difficult dilemma because they either risk loss of traffic by blocking the bots or suffer downtime by allowing them.
Incapsula says that “fake Googlebot visits originate from botnets - clusters of compromised connected devices (e.g., Trojan infected personal computers) exploited for malicious purposes.”
The researchers identified the top four sources of such botnets in the US (25.16%), China (15.61%), Turkey (14.7%) and Brazil (13.49%).
During the study, 400 million search engine visits to 10,000 sites were observed, over a period of 30 days. The company identified 4% of all the bots as being fake ones.