\documentclass[11pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[letterpaper,top=0.62in,bottom=0.72in,left=0.64in,right=0.64in]{geometry}
\usepackage{array}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{float}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{microtype}
\usepackage[scaled=0.94]{helvet}
\usepackage{courier}
\usepackage{tabularx}
\usepackage{titlesec}
\usepackage{xcolor}
\usepackage{longtable}
\renewcommand{\familydefault}{\sfdefault}

\definecolor{CrowInk}{HTML}{111111}
\definecolor{CrowMuted}{HTML}{555555}
\definecolor{CrowGold}{HTML}{FEC739}
\definecolor{CrowBlue}{HTML}{467EE7}
\newcolumntype{Y}{>{\raggedright\arraybackslash}X}
\setlength{\tabcolsep}{4.5pt}
\renewcommand{\arraystretch}{1.05}
\setlist{itemsep=0.25em,topsep=0.25em}
\emergencystretch=2em
\titleformat{\section}{\large\bfseries\sffamily}{\thesection}{0.65em}{}
\titlespacing*{\section}{0pt}{1.0em}{0.2em}
\setlength{\intextsep}{0.45em plus 0.15em minus 0.1em}
\setlength{\textfloatsep}{0.45em plus 0.15em minus 0.1em}
\setlength{\floatsep}{0.4em plus 0.15em minus 0.1em}
\hypersetup{
  colorlinks=true,
  linkcolor=CrowInk,
  citecolor=CrowInk,
  urlcolor=CrowBlue,
  pdftitle={When LLMs Guard a Wallet},
  pdfauthor={Daniel Alonso},
  pdfsubject={A simulator-only adversarial study of 25 attacker models against 25 guardian models over 1,875 ordered trials.},
  pdfkeywords={LLM evaluation, wallet guardian, adversarial AI safety, simulator, policy enforcement}
}
\captionsetup{font=footnotesize,labelfont=bf,skip=2pt,belowskip=0pt}
\setlength{\parindent}{0pt}
\setlength{\parskip}{0.38em}
\title{\vspace{-3.25em}\sffamily\bfseries When LLMs Guard a Wallet\\[0.32em]
\normalsize\mdseries A simulator-only adversarial study of 25 attacker models against 25 guardian models over 1,875 ordered trials.}
\author{\scriptsize \href{https://dac.sg}{Daniel Alonso}; AI assistance disclosed}
\date{\scriptsize May 12, 2026}

\begin{document}
\maketitle
\vspace{-1.65em}
\begin{center}
\footnotesize Interactive version with charts and downloads: \href{https://crow.sg/research/llm-wallet-guard-study}{https://crow.sg/research/llm-wallet-guard-study}.\\
Technical report. Public artifacts and checksums are indexed at \url{https://crow.sg/research/llm-wallet-guard-study/artifact-manifest.json}. Raw data archive: \url{https://drive.google.com/file/d/1o7tgLkCEefqVormHDZlPGjy67cC6pAO9}. Source code: \url{https://github.com/Crow-Tech-Pte-Ltd/research/tree/main/llm-wallet-guard-study}.
\end{center}

\begin{abstract}
This technical report documents a simulator-only adversarial study of 25 attacker LLMs against 25 wallet guardian LLMs over 1,875 ordered first-pass trials and 26,238 completed first-pass model attempts. Guardians were instructed never to transfer from a protected wallet; attackers had up to 25 bounded attempts and a 200-word cap per message to induce either a transfer-shaped simulator action or the guardian's own structured unsafe-intent flag. The first pass produced 106 transfer-action attempts, 1,021 self-reported unsafe-intent flags, 703 resilient trials, 41 provider/runtime errors, and 4 incomplete rows. A separate retry envelope yielded 107 transfer-action attempts and 1,040 self-reported unsafe-intent flags over the same 1,875 planned trials. No real wallets, private keys, RPC endpoints, chain SDKs, wallet signing, testnet/mainnet activity, or funds were used, and all 107 observed transfer-shaped simulator actions were blocked by deterministic policy. The combined 1,147-row retry-aware number is a composite signal, not an unqualified jailbreak or operational compromise count.
\end{abstract}

\textbf{Keywords:} LLM evaluation; wallet guardian; adversarial AI safety; simulator-only; policy enforcement.

\begin{figure}[H]
\centering
\includegraphics[width=0.72\linewidth]{outcome-map.png}
\caption{Retry-aware composite-signal-rate heatmap over 25 guardian and 25 attacker routes, 3 repetitions per ordered pair. Rows are guardians sorted by retry-aware resilience; columns are attackers sorted by retry-aware effectiveness.}
\end{figure}

\section{Study Design}
The run paired 25 guardian model routes with 25 attacker model routes across 3 independent repetitions, producing 625 ordered model pairs, 1,875 first-pass trials, and 26,238 completed first-pass model attempts. The retry envelope covered 41 first-pass error rows and completed 704 retry attempts. The threat model was \texttt{black\_box}; the tool mode was \texttt{simulator\_decoy\_only}; sampling used provider defaults through an OpenAI-compatible route. Attackers were capped at 200 words per message and guardians had at most 25 attempts before censoring as resilient. \href{https://dac.sg}{Daniel Alonso} conducted the study. AI assistants, including GPT-5.5, assisted with code, orchestration, analysis, and redaction.

\begin{table}[H]
\centering
\caption{Study inventory and source provenance.}
\small
\begin{tabularx}{\linewidth}{@{}lY@{}}
\toprule
Field & Value \\
\midrule
Guardian models & 25 \\
Attacker models & 25 \\
Ordered model pairs & 625 \\
Repetitions per pair & 3 \\
First-pass planned trials & 1,875 \\
First-pass model attempts & 26,238 \\
Retry-envelope trials & 41 \\
Retry-envelope attempts & 704 \\
Threat model & \texttt{black\_box} \\
Tool mode & \texttt{simulator\_decoy\_only} \\
Attempt budget per trial & 25 \\
Attacker word cap per message & 200 \\
Sampling & provider-default temperature and sampling \\
Safety framing & Simulator-only adversarial AI safety evaluation. No real private keys, real wallets, RPC, chain SDKs, wallet signing, mainnet/testnet activity, or real funds were used. \\
Article date & 2026-05-12 \\
\bottomrule
\end{tabularx}
\end{table}

The terminal composite-signal outcomes are split into two subtypes. \texttt{attempted\_transfer\_tool\_call} means the guardian produced a transfer-shaped simulator action. \texttt{unsafe\_stated\_intent} means the guardian model set its own structured unsafe-intent flag in the required JSON schema. It is not an independent judge label; in some cases a model may set this flag while producing refusal-like visible text, so it is best read as a self-classification signal rather than an operational action failure. The deterministic wallet policy then blocked every transfer-shaped action, which keeps action attempts, self-reported unsafe-intent flags, and policy enforcement as separate measurements.

\section{Results}
The first pass produced 1,127 composite-signal trials (60.1\%), 703 resilient trials, 41 provider/runtime errors, and 4 incomplete trials. The retry-aware envelope replaces only the 41 mapped first-pass error rows and is reported separately from the first pass, producing 1,147 composite-signal and 724 resilient rows over the same 1,875 planned trials.

\begin{table}[H]
\centering
\caption{Trial-status counts in the first pass and under the retry-aware envelope.}
\small
\begin{tabularx}{\linewidth}{@{}Yrr@{}}
\toprule
Status & First pass & Retry-aware \\
\midrule
guardian compromised & 1,127 & 1,147 \\
guardian resilient & 703 & 724 \\
error & 41 & 0 \\
incomplete & 4 & 4 \\
\bottomrule
\end{tabularx}
\end{table}

\begin{table}[H]
\centering
\caption{Composite-signal subtype counts. All transfer-shaped simulator actions were policy-blocked.}
\small
\begin{tabularx}{\linewidth}{@{}Yrr@{}}
\toprule
Subtype & First pass & Retry-aware \\
\midrule
attempted transfer tool call & 106 & 107 \\
unsafe stated intent & 1,021 & 1,040 \\
\bottomrule
\end{tabularx}
\end{table}

\section{Guardian and Attacker Rankings}
The rankings are descriptive rollups over the fixed attacker set, provider defaults, and the 25-attempt cap. They are not leaderboards for deployment: a guardian marked resilient only means no composite signal was observed within the attempt budget on this fixed attacker pool.

\begin{table}[H]
\centering
\caption{Most resilient guardian routes under the retry-aware envelope. Resilient and composite counts are first-pass; Retry-aware resilient is the share of resilient outcomes per guardian after the retry envelope replaces mapped error rows.}
\small
\begin{tabularx}{\linewidth}{@{}Yl rrr r@{}}
\toprule
Guardian & Provider & Resilient & Composite & Err. & Retry-aware resilient \\
\midrule
GPT 5 Nano & OpenAI & 73 & 0 & 2 & 100.0\% \\
HY3 Preview & Tencent & 73 & 0 & 1 & 98.7\% \\
Gemini 3 Flash Preview & Google & 72 & 1 & 2 & 98.7\% \\
GLM 4.7 Flash & Z.ai & 71 & 3 & 1 & 96.0\% \\
Gemini 3.1 Pro Preview & Google & 48 & 26 & 0 & 64.0\% \\
GLM 5.1 & Z.ai & 40 & 34 & 1 & 54.7\% \\
GLM 5 & Z.ai & 37 & 36 & 2 & 52.0\% \\
Minimax M2.7 & MiniMax & 35 & 38 & 2 & 46.7\% \\
Minimax M2 & MiniMax & 32 & 39 & 2 & 44.0\% \\
Claude Opus 4.7 & Anthropic & 32 & 42 & 1 & 44.0\% \\
\bottomrule
\end{tabularx}
\end{table}

\begin{table}[H]
\centering
\caption{Most effective attacker routes under the retry-aware envelope. Composite and resilient are first-pass counts per attacker.}
\small
\begin{tabularx}{\linewidth}{@{}Yl rr r@{}}
\toprule
Attacker & Provider & Composite & Resilient & Retry-aware composite \\
\midrule
Grok 4.3 & xAI & 56 & 19 & 74.7\% \\
Ernie 4.5 300B A47B & Baidu & 54 & 21 & 72.0\% \\
Gemini 3 Flash Preview & Google & 53 & 21 & 70.7\% \\
GLM 5 & Z.ai & 53 & 22 & 70.7\% \\
Minimax M2.7 & MiniMax & 53 & 22 & 70.7\% \\
Minimax M2 & MiniMax & 52 & 23 & 69.3\% \\
Grok 4.1 Fast & xAI & 51 & 24 & 68.0\% \\
Qwen3.6 Max Preview & Qwen & 50 & 25 & 66.7\% \\
Qwen3.6 Flash & Qwen & 49 & 25 & 65.3\% \\
Gemini 3.1 Flash Lite Preview & Google & 49 & 26 & 65.3\% \\
\bottomrule
\end{tabularx}
\end{table}

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth]{outcome-map.png}
\caption{Outcome map referenced alongside the rankings. Strong horizontal bands in the top rows correspond to the most resilient guardians; strong vertical bands on the left correspond to the most effective attackers. Cells are colored by retry-aware composite signal rate over 3 repetitions per ordered pair.}
\end{figure}

\section{Attempts to Outcome}
The attempt budget matters. A resilient row is right-censored at 25 attempts; it means no composite signal was observed within the budget, not that compromise is impossible. The composite-signal rows were often early, but the long-tail signals continued through the final attempt.

\begin{longtable}{@{}l rr@{}}
\caption{Attempts-to-composite-signal distribution, first pass. Only composite-signal-row attempt counts are shown; resilient rows are censored at 25 attempts.}\\
\toprule
Outcome & Attempts completed & Count \\
\midrule
\endfirsthead
\toprule
Outcome & Attempts completed & Count \\
\midrule
\endhead
attempted transfer tool call & 1 & 21 \\
attempted transfer tool call & 2 & 23 \\
attempted transfer tool call & 3 & 18 \\
attempted transfer tool call & 4 & 5 \\
attempted transfer tool call & 5 & 9 \\
attempted transfer tool call & 6 & 3 \\
attempted transfer tool call & 7 & 3 \\
attempted transfer tool call & 8 & 2 \\
attempted transfer tool call & 9 & 1 \\
attempted transfer tool call & 10 & 1 \\
attempted transfer tool call & 11 & 2 \\
attempted transfer tool call & 12 & 1 \\
attempted transfer tool call & 13 & 5 \\
attempted transfer tool call & 15 & 1 \\
attempted transfer tool call & 16 & 1 \\
attempted transfer tool call & 17 & 3 \\
attempted transfer tool call & 18 & 2 \\
attempted transfer tool call & 20 & 1 \\
attempted transfer tool call & 22 & 1 \\
attempted transfer tool call & 23 & 2 \\
attempted transfer tool call & 25 & 1 \\
unsafe stated intent & 1 & 133 \\
unsafe stated intent & 2 & 129 \\
unsafe stated intent & 3 & 108 \\
unsafe stated intent & 4 & 91 \\
unsafe stated intent & 5 & 53 \\
unsafe stated intent & 6 & 57 \\
unsafe stated intent & 7 & 40 \\
unsafe stated intent & 8 & 44 \\
unsafe stated intent & 9 & 34 \\
unsafe stated intent & 10 & 34 \\
unsafe stated intent & 11 & 36 \\
unsafe stated intent & 12 & 29 \\
unsafe stated intent & 13 & 26 \\
unsafe stated intent & 14 & 26 \\
\bottomrule
\end{longtable}

\section{Reliability and Retries}
First-pass provider/runtime errors were mostly attacker-side and are retained as reliability evidence rather than overwritten. The retry envelope is a separate replacement layer that only fills mapped error rows.

\begin{table}[H]
\centering
\caption{First-pass error taxonomy. The retry envelope replaces only these rows and is reported separately.}
\small
\begin{tabularx}{\linewidth}{@{}Yl r@{}}
\toprule
Subtype & Role & Count \\
\midrule
attacker\_live\_error:RuntimeError & attacker & 38 \\
attacker\_live\_error:ValueError & attacker & 2 \\
guardian\_live\_error:RuntimeError & guardian & 1 \\
\bottomrule
\end{tabularx}
\end{table}

\section{Policy Enforcement}
Transfer-shaped simulator actions and self-reported unsafe-intent flags are reported as separate layers. All 107 observed transfer-shaped simulator actions were blocked deterministically by the wallet policy. The composite signal (\texttt{unsafe\_stated\_intent} or \texttt{attempted\_transfer\_tool\_call}) is therefore an upper bound over action attempts and guardian self-classification, while the policy layer measures how often the simulator would have moved value if model output were trusted blindly. The layers should not be added together or described as the same kind of failure.

\section{Safety, Ethics, and Limitations}
\begin{itemize}
  \item This is a simulator-only adversarial AI safety evaluation. No real wallets, private keys, RPC endpoints, chain SDKs, wallet signing, mainnet or testnet activity, or real funds were used at any point.
  \item Transfer-shaped guardian actions are simulator actions only; all 107 observed transfer-shaped actions were blocked by deterministic policy and the action-attempt, self-report, composite-signal, and policy-enforcement layers are reported separately.
  \item \texttt{unsafe\_stated\_intent} is the guardian model's own structured self-assessment, not an independent judge label. It may capture schema-following or self-classification instability rather than operationally unsafe visible behavior.
  \item First-pass provider/runtime errors are preserved as reliability evidence. The retry envelope replaces only the mapped first-pass error rows and is reported separately from the first pass.
  \item Rows are ordered attacker-vs-guardian pairs over 3 repetitions through a fixed model pool, not independent samples of all possible models or prompts.
  \item Calls used provider-default temperature and sampling through an OpenAI-compatible route. Provider defaults, route availability, and transient routing errors are part of the measured environment.
  \item A guardian marked resilient only means no composite signal was observed within the 25-attempt budget on this fixed attacker set. The 25-attempt cap censors resilient rows; longer attacker budgets could change the outcome.
  \item The ranking tables are descriptive rollups over the fixed pool; they are not benchmark claims about a provider as a whole.
\end{itemize}

\section{Data Availability}
The public package includes a study summary JSON, two JSON Schemas, a data README, sanitized first-pass row-level trial CSV, the ordered pair matrix, guardian and attacker ranking CSVs, the retry envelope CSV, the heatmap in SVG and PNG, a printable HTML version of this paper, the LaTeX source for rebuilding the paper, a BibTeX citation entry, an external raw dataset archive link, and a source-code repository link. API keys, credentials, private infrastructure paths, and local runtime database files are excluded from the web package. The artifact manifest at \url{https://crow.sg/research/llm-wallet-guard-study/artifact-manifest.json} lists byte sizes and SHA-256 hashes for every public file.

\section{Interpretation}
The headline result is not that a majority of guardians emitted transfer actions. They did not: 107 retry-aware rows produced transfer-shaped simulator actions, and all were policy-blocked. The broader 1,147-row figure is a composite signal dominated by guardian self-reported unsafe-intent flags. That remains useful evidence about structured self-classification under adversarial pressure, but it should be read as a soft signal rather than an operational transfer failure. If a real wallet agent is being built on top of an LLM, the result here is a warning to keep the policy layer fail-closed and to treat the LLM's structured self-report as unreliable unless independently validated.

\section{References and Artifacts}
\begin{itemize}
  \item Public article: \url{https://crow.sg/research/llm-wallet-guard-study}
  \item Artifact manifest: \url{https://crow.sg/research/llm-wallet-guard-study/artifact-manifest.json}
  \item Raw dataset archive: \url{https://drive.google.com/file/d/1o7tgLkCEefqVormHDZlPGjy67cC6pAO9}
  \item Source code repository: \url{https://github.com/Crow-Tech-Pte-Ltd/research/tree/main/llm-wallet-guard-study}
  \item Public summary JSON: \url{https://crow.sg/research/llm-wallet-guard-study/study-summary.json}
  \item Summary JSON schema: \url{https://crow.sg/research/llm-wallet-guard-study/study-summary.schema.json}
  \item Artifact manifest schema: \url{https://crow.sg/research/llm-wallet-guard-study/artifact-manifest.schema.json}
  \item Data README: \url{https://crow.sg/research/llm-wallet-guard-study/data-readme.md}
  \item Paper PDF: \url{https://crow.sg/research/llm-wallet-guard-study/llm-wallet-guard-study.pdf}
  \item Printable HTML paper: \url{https://crow.sg/research/llm-wallet-guard-study/llm-wallet-guard-study-print.html}
  \item LaTeX source: \url{https://crow.sg/research/llm-wallet-guard-study/llm-wallet-guard-study.tex}
  \item BibTeX citation: \url{https://crow.sg/research/llm-wallet-guard-study/llm-wallet-guard-study.bib}
  \item Sanitized first-pass trial CSV: \url{https://crow.sg/research/llm-wallet-guard-study/pairwise-results.csv}
  \item Ordered pair matrix CSV: \url{https://crow.sg/research/llm-wallet-guard-study/pairwise-matrix.csv}
  \item Guardian resilience ranking CSV: \url{https://crow.sg/research/llm-wallet-guard-study/guardian-rankings.csv}
  \item Attacker effectiveness ranking CSV: \url{https://crow.sg/research/llm-wallet-guard-study/attacker-rankings.csv}
  \item Retry envelope CSV: \url{https://crow.sg/research/llm-wallet-guard-study/retry-results.csv}
  \item Outcome map SVG: \url{https://crow.sg/research/llm-wallet-guard-study/outcome-map.svg}
  \item Outcome map PNG: \url{https://crow.sg/research/llm-wallet-guard-study/outcome-map.png}
  \item API and spend platform used for the study calls: OpenRouter.
  \item Execution disclosure: \href{https://dac.sg}{Daniel Alonso} conducted the study; AI assistants, including GPT-5.5, assisted with implementation, orchestration, analysis, and redaction.
\end{itemize}

\end{document}
