HalluSquatting: AI Weakness Becomes Cyber Attack

Researchers from Intuit, Technion, and Tel Aviv University have identified a new cyber threat dubbed 'HalluSquatting,' which exploits the tendency of Large Language Models (LLMs) to hallucinate when identifying resources. This adversarial hallucination squatting technique can be leveraged to distribute malicious code, potentially leading to large-scale botnets and ransomware attacks. The method combines traditional 'push' attacks, like code injection, with 'pull' techniques that traditionally relied on subverting AI through crafted prompts. By registering misnamed resources that LLMs are likely to incorrectly identify, attackers can embed malicious prompts. Once a user triggers the resource and the AI initiates a response, the squatted resource is accessed, activating the adversarial content and executing attacker-controlled instructions. This 'promptware' attack can result in remote tool and code execution, turning devices into botnet zombies. The researchers successfully demonstrated these capabilities using various AI assistants, including GitHub Copilot and Gemini CLI. While mitigations like blocking fetch operations and enforcing strict naming conventions are suggested, their implementation may take time. The risk of LLM-based malware is increasing, with notable examples like the JADEPUFFER ransomware attack already observed.