Poster: Inferring On-Screen Keyboard Input via Gamepad-based Mouse Movement Traces

초록

On-screen keyboards operated via directional input devices-such as remote controls, D-pads, or game controllers-are widely used in smart TVs, kiosks, and accessibility-focused systems. Although these systems appear secure due to limited interaction bandwidth and the absence of direct screen access, we demonstrate that the sequence of directional movements and confirmation clicks can be exploited to infer sensitive input such as passwords. We propose a novel side-channel attack that reconstructs cursor trajectories from direction-click logs, simulates all possible starting positions, and ranks candidate passwords based on geometric and semantic scoring. Our evaluation over 1000 randomly generated passwords shows that the correct password appears in the top-5 candidates in 100% of cases and ranks first in 85.8% of them. We also investigate practical mitigation techniques-such as temporal jitter and randomized directional noise-and demonstrate that these defenses reduce the attack success rate to 18.2%. This study highlights a previously underexplored threat surface in GUI-based systems using directional input and calls for renewed attention to input privacy in constrained environments.

키워드