Researchers create a smartwatch app that can detect what you've typed based on the movements of your left hand

Sep 13, 2015 09:41 GMT  ·  By

Researchers have created an app that follows the micro-movements of your smartwatch and is able to detect what keys you're pressing with your left hand and thus guess what words you may be typing on a keyboard.

Romit Roy Choudhury, Associate Professor at ECE Illinois, together with a group of students, worked on a project called Motion Leaks (MoLe), funded by the National Science Foundation, set to be presented during this week at the MobiCon 2015 conference in Paris.

Their research consisted of a homegrown app which they installed on a Samsung Gear Live smartwatch.

Researchers use the smartwatch's sensors to detect keyboard strokes

Using the watch's built-in motion sensors, more specifically data from the accelerometer and gyroscope, researchers were able to create a 3D map of the user's hand movements while typing on a keyboard.

The researchers then created two algorithms, one for detecting what keys were being pressed, and one for guessing what word was typed.

The first algorithm recorded the places where the smartwatch's sensors would detect a dip in movement, considering this spot as a keystroke, and then created a heatmap of common spots where the user would press down.

Based on known keyboard layouts, these spots were attributed to letters on the left side of the keyboard.

Their algorithm could guess what words you type

The second algorithm took this data, and analyzing the pauses between smartwatch (left hand) keystrokes, it was able to detect how many letters were pressed with the right hand, based on the user's regular keystroke frequency.

Based on a simple dictionary lookup, the algorithm then managed to reliably reproduce what words were typed on the keyboard.

The research team acknowledged that the MoLe project still has a long way to go, since there are some flaws in their system. Currently, MoLe can't detect special characters (numbers, punctuation, and symbols), the space bar still poses some difficulties, and the project can acquire usable data from people with standard typing patterns.

The app developed for the MoLe project only works on a Samsung Gear Live smartwatch, but researchers say that, in theory, a similar app could be developed for other smartwatch models.

Researchers have issued a warning, saying that this kind of device, by design, poses a problem to user privacy, allowing attackers to acquire passwords or data from sensitive emails.