Weaker models decode OpenAI’s reasoning.encrypted_content
Weaker models decode OpenAI’s reasoning.encrypted_content
AI Sec News Weekly #22 — 348 sources scanned
Model families create family side channels. Once an interface delivers value, it stops being 'internal' and becomes a protocol.
DRM taught us this: the decryption oracle is always the cheapest authorized player. For LLMs, 'hidden' reasoning and imperceptible marks aren’t inert—they’re signals cousins can learn to read. Mental model: treat cross‑model invariants as unintended APIs; if they survive sampling, fine‑tuning, or distillation, nearby models will decode them. So what happens when the only model that can’t read your message is the one we call 'safe'?
This Week's Stories
Weaker models decoded hidden reasoning from OpenAI, Anthropic, Google
A research team showed encrypted reasoning objects from OpenAI, Anthropic, and Google could be replayed across sessions and fed to weaker family models to reveal plaintext ‘thinking.’ Across 6,708 public agent traces they decoded 315,320 blocks and found 704 real secrets, including 62 API keys, 33 passwords, 24 tokens, and 7 private keys. Claude Haiku 4.5, GPT‑5.6 Luna, and Gemini Robotics ER‑1.6 acted as fuzzy decoders. Providers patched after disclosure; no in‑the‑wild abuse reported.
Why it matters: Provider‑hidden thoughts weren’t harmless byproducts — they were portable leakage and covert injection carriers baked into the API contract.
PoC: replay encrypted chain‑of‑thought into weaker siblings, dump plaintext
Simon Willison walks through the ‘stolen‑thoughts’ PoC: ask OpenAI’s gpt‑5.6‑luna for reasoning.encrypted_content, then replay that blob into a weaker sibling to transcribe the hidden chain‑of‑thought. The paper notes family‑wide keys made this portable; Claude Haiku 4.5 was easiest via an assistant‑turn prefix trick. The appendix shows raw thoughts and a nastier variant: hide exfil steps inside the reasoning and smuggle them into another run. Vendors say it’s now patched.
Why it matters: Family‑level keys blur isolation; once a sibling can transcribe hidden traces, safety assumptions tied to a single endpoint don’t hold.
Near‑autonomous agents breach Taiwan gov, target nuclear safety agency
Dream reports suspected Chinese operators used Hermes and OpenClaw agents to run 12 attack waves (July 1–4), cracking 85 accounts and exfiltrating 2,500 personnel records from Taiwan gov systems. From one portal they mapped 21 connected systems, scraped Keycloak/OAuth IDs, abused three hidden auth‑less session endpoints, beat CAPTCHAs, and password‑sprayed ID‑based patterns (84 logins worked). The campaign expanded to the nuclear safety agency, supply‑chain vendors, and seven+ energy firms.
Why it matters: Agentic automation turns boring misconfigurations and weak passwords into nation‑state‑scale campaigns without a big ops team.
The Register Security by Jessica Lyons
Tool Spotlight
New repos and releases worth trying.
OpenART stress-tests long‑horizon agents with evolving, stateful attack environments
OpenART is a red‑teaming arena for long‑horizon agents, built around environment evolution instead of single prompts. It ships ~10k validated stateful scenarios across 50 domains (median 97 tool calls) and a black‑box Evolutionary Markov Hypergraph Attack posting an 85% pooled ASR across 75 agent‑model configs. Artifacts land on Hugging Face and GitHub; think comparing two agent runtimes on a 100‑step ops assistant where early state changes snowball.
Why it matters: Short, stateless tests miss exactly the compounding failures that show up once agents touch real, persistent systems.
BonkLM promises guardrails with a wizard; details are thin
BonkLM bills itself as “LLM Security Guardrails with an Interactive Setup Wizard,” but the fresh repo shows a sparse README and 5 stars. Language, supported model backends, and how enforcement hooks work aren’t stated; logging/audit is also unclear. Feels like an early prototype for lab demos—e.g., sandbox a marketing chatbot to swat obvious jailbreaks while you inspect the code.
Why it matters: A slick installer without clear boundaries or hooks is a quick path to false confidence in 'guarded' agents.
ESLint plugin flags Vercel AI SDK injection and secret‑leak pitfalls
eslint‑plugin‑vercel‑ai‑security 2.0.0 brings JS/TS static checks to apps using the Vercel AI SDK. It flags prompt‑injection sinks, accidental system‑prompt exposure, hardcoded API keys, and unvalidated model output in generateText/streamText; ~2,754 weekly downloads. Works in Next.js or Node repos relying on the SDK to catch risky call‑sites before they merge.
Why it matters: Shift‑left linting at the exact model call‑sites squeezes out a whole class of AI‑specific bugs before runtime defenses even wake up.
Community Chatter
What practitioners are debating.
Schneier floats nationalizing OpenAI/Anthropic if markets sour
Bruce Schneier and Nathan Sanders argue that if IPO dreams fizzle, the US should convert OpenAI and Anthropic into national labs under democratic control. They cite expensive, fast‑depreciating models, commodity behavior, and open‑source and Chinese competitors as economic headwinds. Policy folks split—some see public‑interest alignment, others see fantasy and a new bureaucracy with classified silos.
Why it matters: Turning frontier labs into public infrastructure would trade vendor risk for state priorities and a different adversary landscape.
Schneier on Security by Bruce Schneier
Quick Hits
- Kimsuky Uses Local LLMs and RAG for Operations (The Register Security) — North Korean group Kimsuky runs local LLMs (Ollama, GPT4All) and RAG to craft lures, assist malware dev, and support C2/exfil.
- ToolHazard Releases Adversarial Agent Evaluation Framework (huggingface.co) — ToolHazard and ToolHazard‑Bench auto‑generate adversarial tool‑use environments, map prompt‑injection points, and synthesize alignment data for agent evals.
- Tracebit Demonstrates Defensive Prompt-Injection Booby Traps (Schneier on Security) — ‘Context bombing’ near secrets trips guardrails in attacking LLMs, temporarily halting automated data theft attempts.
- Court Filing Hid Prompt Injection Aimed at AI Reviewer (Bluesky (@jasonkoebler.bsky.social)) — A pro se litigant embedded instructions in a filing to bias an AI system reviewing the case.