언어모델 기반 Physical AI 에이전트 환경에서의 로컬 컨텍스트 프롬프트 인젝션 위협 분석 및 방어 아키텍처 제안

초록

This paper analyzes local context-based prompt injection and behavioral hallucination threats targeting language model-based agentsin Physical AI environments, and presents threat models and attack scenarios for evaluating them. Results from simulation-based iterativeexperiments showed that while clear physical malfunctions were limited, discrepancies between descriptions and actual behaviors wereconsistently observed. Furthermore, under certain conditions, incorrect judgments and records were found to influence subsequentinference, leading to an accumulation of errors. This demonstrates that in Physical AI security, not only the final behavior but also thereliability of state interpretation, the explanation layer, and the recording system are critical targets for protection. Additionally, theexperimental results showed that the agent’s judgment tendencies can vary depending on the prompt structure and policy expressionmethod; in particular, trust boundary series prompts can amplify the influence of malicious phrases contrary to intent. Accordingly, thispaper proposes an integrated defense architecture that separates trust boundaries within the local context, performs explanation-behaviorconsistency verification and continuous monitoring, and applies clear instruction prompts. The proposed defense architecture providesa fundamental structure to mitigate state interpretation distortion and behavioral goal bias in Physical AI agents.

키워드