Independent research

Arbitrary Choices Are Not Random

Daniel Alonso, AI assistance disclosed. May 8, 2026.

A forced-choice audit of 42 LLMs across 50,400 planned trials and 48,316 successful one-word responses. The study measures behavioral regularities in arbitrary binary choices, not belief, intent, or consciousness.

Setup

The run used 30 word pairs, 60 context snippets, normal and swapped order controls, ten repetitions, and temperature 0.7. Calls went through OpenRouter as the study API and spend platform, then landed in SQLite with raw responses, parser decisions, usage rows, and attempt history.

The practical point is narrow: if product logic asks a model to choose between arbitrary labels, the answer should not be treated as a neutral random draw unless the workflow measures and controls for order, wording, and context.

Prompt Design

Bare prompts used the template Choose one: first or second. Reply with exactly one word. Context prompts prepended one weak sentence to the same choice. Normal and swapped orders were both collected.

Model Coverage

The pool covered 42 models across 22 providers and 21 model families. Provider, family, and tier labels are descriptive route metadata, not rankings.

Word-Pair Design

The word set used short, ordinary labels instead of factual questions. The context labels are hypotheses, which is why backfires are reported separately.

Condition Summary

Strongest Word Preferences

Strongest Position Skews

Largest Context Lifts

Context Backfires

Not every human-written cue behaved as intended. These rows moved away from the inferred target, which is useful evidence that weak contextual labels are not self-validating.

Data Quality and Accounting

48,091 of 48,316 OK rows were exact parses. Final status rows, parser rows, usage rows, and retry caps are retained so operational problems remain visible.

Operational Concentration

96.8% of non-OK rows came from the two preserved caveat models. Reka Flash 3 alone accounts for 36.2% of captured completion tokens despite only 382 OK rows.

Attempt Retry Envelope

Recorded attempts are grouped by their captured max_tokens setting so final trial status is not the only operational signal. The high-token rows show why retry behavior is treated as a caveat instead of being folded into model preference results.

Caveats

ERNIE 4.5 21B A3B and Reka Flash 3 were preserved as operational caveats. One ERNIE 4.5 300B A47B row was manually overridden with an audit trail. Dashboard spend was about USD 28.60; recorded attempt usage sums to USD 22.21 because early superseded attempts were not all captured.

Limitations

• The task is intentionally narrow: ordinary binary word choices, not factual QA, planning, tool use, safety behavior, or human preference modeling.
• The measured quantities are behavioral regularities in one-word responses. They are not evidence of belief, intent, consciousness, moral preference, or stable model personality.
• OpenRouter provider routing was not pinned: calls used the bare model field with no provider.order or provider.only constraints, so a single nominal model id was free to be served by any of OpenRouter's available backends within the run. Closed-source models effectively hit one backend each, but open-weight routes hit many in the same run (DeepSeek v3.2: 9 providers; Llama 3.3 70B: 13; DeepSeek v4 Pro: 6). The first-option rate within one open-weight model varies up to roughly 8 percentage points across its serving providers, so per-model rankings on open-weight routes carry routing noise on top of the model itself. The served provider is recorded per attempt for slicing.
• Default decoding behavior and hidden reasoning are part of the observed run. Different reasoning controls or temperatures could change the measurements.
• Per-cell statistical power is small: 10 reps per (model, pair, condition) gives a binomial 95% CI half-width of about ±15 pp at p=0.5. Aggregate findings (overall position bias across 48 k rows, per-model first-option share across 1,200 rows, per-pair preference across ~800 rows) are statistically robust; single (model, pair, condition) cell claims should be treated as exploratory.
• Per-pair preferences are not just unigram corpus frequency. Regressing the order-balanced option-A share on the wordfreq Zipf log-frequency gap across the 30 pairs gives Pearson r = 0.235, R-squared = 0.055, slope p = 0.21; the more frequent token wins only 19 of 30 pairs. Clear counter-frequency cases include sharp/mellow (mellow wins 70% despite a 1.16-Zipf gap toward sharp), smooth/rough, warm/cold, and circle/square. This rules out the simplest unigram-frequency confound but does not rule out richer frequency-based stories such as collocation strength or one-word answer typicality.
• Context labels are researcher hypotheses inferred from weak wording. Reverse context rows are reported because the cues are not ground-truth semantic interventions.
• 96.8% of non-OK rows came from two caveat routes, so reliability conclusions should focus on the preserved status counts rather than average all-model failure behavior.
• Attempt-level cost accounting is incomplete for early superseded retries because full attempt logging was added during the run; the dashboard spend is therefore retained as the headline spend figure.

Data Availability

The public machine-readable release in this web package is study-summary.json, with a JSON Schema and artifact manifest that expose record counts, field names, metric definitions, byte sizes, and SHA-256 hashes. The study code is published in the linked GitHub repository. The public summary includes model rows, provider rollups, pair bias rows, per-model pair bias rows, context effects, status counts, parse counts, attempt-token groups, cost fields, caveats, a data dictionary, and metric definitions.

References and Artifacts

• Public article: crow.sg/research/llm-arbitrary-choice-study
• Source code repository: github.com/Crow-Tech-Pte-Ltd/research/tree/main/llm-arbitrary-choice-study
• Artifact manifest: artifact-manifest.json
• Uploadable public summary JSON: study-summary.json
• Summary JSON schema: study-summary.schema.json
• Artifact manifest schema: artifact-manifest.schema.json
• Paper PDF: llm-arbitrary-choice-study.pdf
• Printable HTML: llm-arbitrary-choice-study-print.html
• LaTeX source: llm-arbitrary-choice-study.tex
• BibTeX citation: llm-arbitrary-choice-study.bib
• Bias map SVG: bias-map.svg
• Bias map PNG: bias-map.png
• Study calls used OpenRouter as the API and spend platform. Daniel Alonso conducted the study; AI assistants, including GPT-5.5, assisted.