Skip Main Navigation

Community College of Rhode Island

All Mobile Devices are Watching You!

Presenter: Dr. Xinwen Fu, Associate Professor in the Department of Computer Science, University of Massachusetts Lowell

In this talk we introduce a novel computer vision based attack that automatically discloses inputs on a touch enabled device. Our spying camera, including Google Glass, Samsung Galaxy Gear, Webcam and smartphone, can take a video of the victim tapping on the touch screen and automatically recognize more than 90% of the tapped passcodes from three meters away, even if our naked eyes cannot see those passcodes or anything on the touch screen. The basic idea is to track the movement of the fingertip and use the fingertip's relative position on the touch screen to recognize the touch input. We apply various computer vision and machine learning techniques to automatically track the touching fingertip, locate the touched points and then map the estimated touched points to a software keyboard in a reference image. Extensive experiments were performed to demonstrate the impact of this attack. As a countermeasure, we design a context aware Privacy Enhancing Keyboard (PEK) which pops up a randomized keyboard on Android systems for sensitive information such as password inputs and shows a conventional QWERTY keyboard for normal inputs.

This page developed and maintained by Web Services. Send comments and suggestions to

Last Updated: 7/1/15