14 DAY TRIAL // Headless browsers became a known technology back in 2009 when Google revealed it was opened to using one to parse and index AJAX-based websites.
Since then, the technology has seen more and more usage in various projects, sometimes for good reasons, sometimes for malicious purposes.
So what is it?
A headless browser is just like any regular browser, only stripped away of its GUI (Graphical User Interface). All other parts like the networking component, the JavaScript interpreter, the rendering and layout engines, still being present and helping it work just like any other browser when surfing the Web.
Developers have found this technology highly useful, allowing them to test websites, Web and mobile applications, without actually having to look at a browser window and press refresh for an updated view of the page.
Almost all headless browsers can be controlled via an API or a console, and developers have used them to automate their testing processes, take screenshots of what's rendered inside them, and retrieve the content of Web pages and then supplying it to other more complex software.
Of course, this technology has proven to be highly efficient to hackers as well, its automation features allowing them to create and launch complex, yet fully controllable attacks on various websites and Web services.
Headless browsers have been known to be used in DDOS attacks, brute-force attacks, and also for falsely increasing ad revenue by faking page loads and user interactions.
So what headless browsers can I use?
Most successful headless browser projects nowadays are based on a real-life browser, with the most widespread of them being PhantomJS, based on Chrome's and Safari's WebKit engine.
Similarly there's trifleJS originally built on Internet Explorer's former Trident engine, now ported to Google's V8, SlimerJS on Firefox's Gecko engine, Awesomium based on Chromium, and HtmlUnit built on Mozilla's Rhino engine.
As for headless browsers built on custom browser engines, there's Twill, but the project is inactive for 8 years now. This seems to be a problem with all similar projects, all reaching a wall when, due to a lack of resources and time, and because of the huge amount of work and maintenance a browser engine needs, the project is eventually abandoned.
Because of this reason we always recommend choosing a headless browser built on a known browser engine, since there's a high chance to have a lot of people contributing to it, and lots of in-depth documentation and tutorials at hand at any time.
P.S.: You should check Asad Dhamani's list of headless browsers and their derived technologies.