AnomalyDetection can be improved by anyone, it's open-source

Jan 7, 2015 22:47 GMT  ·  By

A tool signed by Twitter promises to help administrators with the detection of spikes regarding the information received by the service from the users, which comes in handy for identifying bot or spam activity.

The utility is called AnomalyDetection and the microblogging service released it under GPLv3 license on GitHub, making it freely available for anyone who wants to use it or contribute to its further development.

Activity spikes could be on account of malicious activity

Detecting variations in the usage of a service can play an important part in deciding the next step for a business, especially where social engagement is concerned, as it can reveal the effect an event has on the users.

“This package helps us monitor spikes in user engagement on the platform surrounding holidays, major sporting events or during breaking news,” Twitter software engineer Arun Kejariwal said in a blog post on Wednesday.   However, AnomalyDetection can also be used to determine if a spam campaign is currently taking advantage of the platform, as such activity does create significant ripples as far as regular usage is concerned; negative anomalies are also tracked.   Alternatively, the new asset could be used to identify the amount of links pointing to malware or fraudulent pages that are distributed through the platform to unsuspecting users. This type of usage can create an activity burst that may raise suspicions and prompt an investigation.   Moreover, an affluence of bots trying to artificially engorge the number of followers for certain Twitter accounts can also be observed thanks to this tool.

Better activity monitoring could lead to improved services

The necessity of building this activity measurement instrument stemmed from the need to monitor seasonal trends, such as those during a holiday or a particular season of the year. This is particularly useful with social networks.   Data management expert Varun Chandola says that anomalies are not the same in every field of activity and that for each domain a specific technique needs to be applied in order to achieve relevant results.   As for how the utility works, Kejariwal explains that the “primary algorithm, Seasonal Hybrid ESD (S-H-ESD), builds upon the Generalized ESD test for detecting anomalies. S-H-ESD can be used to detect both global and local anomalies.”   With support for rich visualization, AnomalyDetection is bound to be an asset for keeping an eye on the evolution of a community and the reasons behind it becoming more or less engaged.