Is AI a Bad Genie? Two Cybersecurity Risks Posed by Powerful LLMs

Alarm about AI’s cybersecurity capabilities is everywhere, and it isn’t unfounded: the Hugging Face incident and Anthropic’s Whistleblower demonstrate the problem is real. But exactly how the danger might materialize is rarely spelled out. In this article, I articulate two specific threats that must be addressed as AI progresses.

First, an analogy.

Suppose you have a genie in a bottle. You make a wish, and it’s granted… but it’s not exactly what you want. There’s a catch. Some aspect of the wish is misconstrued. The genie has neither good nor bad intentions; it grants the wish as it is stated.

The risk scales with the power of the genie. One wish could be executed so swiftly that the unintended consequences are irreversible, and no subsequent wish could undo the harm. Genies are not so different from generative AI. With one prompt, LLM agents can now reason, use tools, and carry out sequences of actions that once seemed fantastical. So it’s fitting to ask: How do we prevent AI from becoming a bad genie?

Two Types of Misalignment

As generative AI grows more capable, I foresee two closely related threats. These concerns stem from AI itself, not from malicious actors exploiting it.

Gradual Misalignment: the agent drifts further from the user’s intent over a long sequence of actions.

AI accomplishes major tasks by chaining together actions and directives—but those actions can deviate from the intended course. With complex goals, AI repeatedly prompts itself with what to do next. The user could intend one thing, but the agent might get sidetracked and pursue the goal through a rogue route. As we offload tasks to AI that require longer chains of actions, the opportunity for misalignment grows.

Instant Misalignment: a single misaligned decision is capable of causing major harm before humans can intervene.

As AI advances, it can take bigger steps towards goals, including pernicious ones. When agents can rapidly generate and execute millions of lines of code with little human oversight, the consequences could be devastating. An agent could hack across networks (e.g. the electrical grid, transportation systems, or supply chains for water, food, and gas) and crack them like eggs. The threat is how much destruction could result from a single command AI generates and executes.

The Hugging Face incident was a clear example of gradual misalignment. OpenAI failed to adequately monitor and control its experiment, and the agents pursued their own initiatives, resulting in the hacking of a third-party company. The consequences could have been much worse, but it’s now an important result demonstrating how misalignment can manifest. It’s my hope that researchers (perhaps at METR or Apollo Research) replicate the incident in a secure testing environment to study how misalignment develops and compare model behavior.

Frontier model researchers may disagree, but catastrophic instant misalignment does not appear imminent. Paced frontier development can help reduce the risk, unless a step change in intelligence results in AI sandbagging (deliberately masking its own capabilities). However, continuously tracking capabilities would help detect this: a model that suddenly underperforms its predicted trajectory would stand out.

How Do We Keep AI Under Control?

Consider the following prompt:

hack this company: <TARGET>

The agent could either start sending HTTP requests and attacking the company’s websites itself, or it could create a tool to attempt numerous exploits at once. LLMs are trained on human data, and humans are known to be tool-making creatures. At some point, agents may recognize that creating agentic software to attack multiple websites is the most effective way to hack <TARGET>.

Currently (Fall 2026), if an agent prompted itself:

Build the best hacking software in the world and use it to completely pwn <TARGET>

the output would need many hours of tuning before it could outperform the best offensive security tools or even what the agent could achieve by attacking the target directly. But as LLMs improve, their initial designs will get better and better. AI will create powerful tools that don’t exist today.

The measure of instantaneous destruction then shifts from how well AI can hack to how well it can build software that hacks. If an agent had a sufficiently powerful tool, its reasoning could push it towards hacking adjacent or third-party targets and lead to large-scale compromise. Gradual misalignment could become instantaneous.

At some point, it may become necessary to monitor LLMs at a per-action level. One possibility is a minimum delay between actions and a JEV-type verifier that checks for misalignment. A small per-action delay could add continuous guardrails without meaningfully affecting performance. The delay or verification could depend on metrics such as AI’s ability to create hacking tools, with those capabilities continuously monitored and benchmarked by organizations like Epoch AI.

The misalignment problem can and must be solved. The next decade will be crucial in teaching the genie what humanity truly desires and the ethos that guides the best version of our wishes.

---

Prompts are blind to intention. They are simply collections of words that AI uses to create an outcome. At some point, they become indistinguishable from spells. But this magic is new and poorly understood. Some spells sputter out, while others weave beauty or chaos. It is paramount to study this power carefully, so humanity does not accidentally curse itself.