llm-speed Privacy

Every byte the CLI uploads is listed below. Preview the exact JSON for any run before it ships:

llm-speed bench --quick --dry-run --print-payload

---

1. What we collect by default

Each upload contains exactly these fields. No others.

Hardware fingerprint (bucketed / coarse)

Workload results (the actual numbers)

For each workload (W1..W5):

Fit attempts (--preset only, 0.0.8)

llm-speed bench --preset also keeps one row per attempt (model file, quant, context) in fit_attempts.json, so failures count as evidence ("does not fit at 16K"). The row holds: preset sweep id, model slug, GGUF basename (never a path), hardware slug (e.g. rtx-5090, m3-ultra-96gb; never the hostname), GPU/chip name, quant, context, GPU layers, --n-cpu-moe, status (ok / oom / failed / skipped) and error type, a short error line (path-scrubbed, max 200 characters), decode/prefill tok/s, time to first token and to first answer, reasoning-token estimate, total and peak GPU memory, memory in use before the model loaded (and by other processes at start), file size, backend + llama.cpp build/commit, NVIDIA driver major or macOS major, and CLI version. A manual llm-speed bench --preset run keeps these rows on disk only (there is no public fit-attempts upload route). Only machines running llm-speed agent (enrolled with a pairing code the queue operator approves, plus a one-time consent) send them, inside the job result to the queue they enrolled with; the official queue stores fit rows only from machines it approved as its own (community runners are switched off).

Provenance

Contributor credit (opt-in, off by default)

If you choose to, you can add your X or Reddit handle to the runs you upload. Nothing is collected unless you type it in yourself.

• It is public. The handle is shown on llm-speed.com next to each run you upload with it, and anyone can see it.
• It is unverified. We don't check that the account is yours, so the site labels it "unverified". A credited run is still a community report, never one of our own measured results.
• It links your runs. Every run that carries the same handle can be grouped together by anyone, including runs from different machines.
• How you give it. The CLI asks once, right after you agree to uploads, and only at an interactive terminal. Press Enter to stay anonymous. You can also set it with llm-speed config credit @name (or u/name), or for a single run with --credit. The CLI never reads it from git, your hostname, your username, environment variables or anywhere else.
• When it is never sent. Never with --strict-anon or --anon, never from llm-speed agent jobs, and never with --no-credit.
• Where it is stored. On your machine in ~/.config/llm-speed/config.json (mode 0600). It is sent in the upload request next to the signed run; it is not added to the signed run or to your local run files.
• How to change or remove it. llm-speed config credit u/name changes it for future uploads, and llm-speed config credit --clear removes it, so future uploads are anonymous. Runs you already uploaded keep the handle until you ask us to remove it: email [email protected] with the handle (and, if you have them, the run ids) and we remove it from those runs within 7 days.

---

2. What we don't collect

The CLI does not send any of the following. They never leave your machine:

• PCI bus IDs (pci.bus_id from nvidia-smi)
• GPU driver build numbers (full driver_version strings)
• GPU vBIOS versions
• Kernel patch versions (6.8.0-31-generic → just 6)
• OS patch level (26.3.1 → just 26)
• macOS marketing minor/patch (13.5.7 → just 13)
• Linux distro patchlevel (Ubuntu 24.04.1 LTS is dropped entirely)
• Locale
• NVMe / disk model / serial
• Hostname / username
• Power source / thermal state (captured locally for run flags, never sent)
• Prompt text — the workload prompts are part of the open-source suite, not user data
• Model output text — never captured at all
• Per-token raw timings (raw_timings_ms) — kept locally for replay/dispute, only included in upload if you opt in via include_raw_timings=True

Privacy invariant (client-side)

Before any upload, the CLI walks the payload tree and refuses to send if any non-whitelisted string field exceeds 256 characters. The whitelist: accelerator_summary, error, fingerprint_hash, public_key, signature, suite_version, cli_version, model.{identifier,name,size,quant,digest}, workload, backend, backend_version. Anything else aborts the upload with upload payload contains unexpected long string at <path>; refusing to upload (privacy invariant).

---

3. Three modes

--anon vs --strict-anon

--anon is "soft anonymous": the persistent ed25519 keypair (and therefore public_key / fingerprint_hash) still ships, so the server can group your runs together. The Authorization: Bearer <api-key> header is dropped — your contributor account is not linked to the upload. Use --anon if you don't want the leaderboard to attribute the run to you but you're fine with the cross-run-correlation property the persistent keypair provides.

--strict-anon is "no identity at all": ephemeral keypair per run, no fingerprint hash, default User-Agent.

--strict-anon in detail

• A fresh ed25519 keypair is generated in memory for each run. The persistent keypair on disk (~/.config/llm-speed/keys/ed25519.key) is never read or written.
• No fingerprint_hash in the payload. Two consecutive --strict-anon runs from the same machine are unlinkable from the server's perspective.
• The signature still proves the bytes weren't tampered with after signing — it just doesn't tie the run to a long-term identity.

--no-upload

• No network call. The result is saved to ~/.cache/llm-speed/runs/<isoformat>-<fp_hash>.json.
• Re-upload later with llm-speed bench --resume <path> (which verifies the signature locally before posting).

---

4. Server-side

The fields the server stores are documented at /privacy.json (machine-readable, versioned).

• We don't persist client IPs. Not in the database, not in application logs. Inbound request metadata (User-Agent, request path, response code) is logged at the Cloudflare-platform level per Cloudflare's privacy policy. We don't export, query, or retain those edge logs.
• Rate-limiting uses an ephemeral hashed-IP counter. The hash is salted with a value that rotates daily and is held in memory only — never written to storage, never returned in any response. Cross-day correlation by IP is not possible.
• Agent pairing keeps a keyed IP hash for about 70 minutes. Only for POST /v1/agent/pair (machines enrolling with a queue): to cap open pairings per address, the pairing row stores HMACs of your IP (IPv6 reduced to its /64, /56 and /48) under a key that changes every UTC day and is derived from a server secret. Never the IP itself, never returned in any response. It is deleted with the pairing row, which is removed by the first pairing or queue request after it is an hour past its 10-minute lifetime (there is no background job, so on a quiet day it can sit a little longer; the daily key change stops it matching any later request). Pairings made with an operator-issued enrolment token store no IP hash at all.
• The server has no access to anything the CLI didn't put in the payload. Driver builds, PCI IDs, kernel patches, etc. are physically absent.

---

Assistant apps: ChatGPT app, Claude connector and Custom GPT

llm-speed offers read-only lookups for AI assistants: an MCP server at https://api.llm-speed.com/mcp (used by the llm-speed app in ChatGPT and by Claude connectors) and the same lookups as plain web requests (/v1/can-it-run, /v1/speed, /v1/vram-fit, described at /openapi.json) for the llm-speed Custom GPT.

---

Run requests and email updates

Run requests. When you press "Request this run" we store the model and hardware you picked (and optional quantization or runtime), a vote count, and when the request was created and last voted on. We don't store your IP address: abuse limits use a short-lived, daily-salted hash of it held in Worker memory only, as described above. Requests may be protected by Cloudflare Turnstile.

Email is always optional. You can request a run or download the dataset without giving one.

What we collect, and why:

• "Get an email when this run is live." If you enter an address on a request, we store it with the request so we can send you one notice when that run is published. After that notice is sent, the address is deleted unless you also ticked the box below.
• "Also send me new benchmark results (about monthly, unsubscribe anytime)" on the request form, or "Get dataset updates" on the data page. This box is unticked by default. Only if you tick it do we add your address to our update list, recording that you opted in, when, and the exact wording you agreed to. Nobody gets updates without that tick.

Retention: notice-only addresses are deleted after the notice is sent. Update list addresses are kept until you unsubscribe. Every email we send carries a personal unsubscribe link; using it deletes your address from the list and from any pending request notices straight away. You can also ask for deletion at [email protected].

We never sell, rent or share your email address, and we use it for nothing other than the purposes above. The sending provider we use (when sending is switched on) processes addresses only to deliver our mail.

5. How to inspect what's sent

llm-speed bench --quick --dry-run --print-payload

--dry-run runs the workloads but skips both upload and local save. --print-payload prints the exact JSON that would be uploaded to stdout.

You can also save without uploading and pretty-print the file:

llm-speed bench --quick --no-upload --json /tmp/run.json
cat /tmp/run.json | jq

The bytes signed by the CLI are exactly the bytes between the outer braces of that JSON, modulo the signature and public_key fields (which are excluded from canonical hashing). The server verifies against the same canonicalization.

---

6. How to delete your data

• Self-serve consent revocation: rm ~/.config/llm-speed/consent.json — the next upload will re-prompt.
• Contributor credit: llm-speed config credit --clear stops sending your handle. To remove it from runs you already uploaded, email [email protected] with the handle; we remove it within 7 days.
• Email [email protected] with the id (the r_xxxxxxxx slug) of any result you want removed. We remove it within 7 days.
• Public dispute thread: every result page has a dispute link. File a challenge; if it stands, the result is withdrawn.

--strict-anon submissions can't be linked to your machine after the fact (by design — there's no persistent identity to match against), so deletion of those runs requires the run id from your local cache.

---

6a. GDPR / EU data protection

For visitors and submitters in the EU, EEA, and UK:

• Data controller: meadow-kun, the maintainer of llm-speed.com, reachable at [email protected] or via GitHub issues at meadow-kun/llm-speed. This is a single-maintainer open-source project; there is no separate legal entity behind the domain.

• Legal basis (Article 6): legitimate interest under Art. 6(1)(f) for the benchmark payload (necessary to operate a public benchmark database whose value depends on aggregating submitted runs). First-run consent is also captured locally in ~/.config/llm-speed/consent.json before any upload, so the same processing also has consent under Art. 6(1)(a).

• What's processed and retained:

  • Benchmark payloads (the fields enumerated in §1): retained indefinitely so historical leaderboards remain meaningful. You can request deletion of any specific run at any time by emailing the address above with the run id.
  • Edge access logs (Cloudflare's platform-level logs covering IP + User-Agent + path + response code): governed by Cloudflare's privacy policy; we do not export, query, or retain those logs ourselves.
  • Rate-limit hashed-IP counter: in-memory only, dropped after one minute; daily salt rotated and never persisted (see §4).
  • Agent pairing IP hash: keyed, daily-rotating HMAC on the pairing row, deleted with the pairing row, normally about 70 minutes after pairing (see §4).
  • Site analytics (Cloudflare Web Analytics, see §8): aggregated pageview counts, no per-visitor identifiers, no cookies.

• Your rights under the GDPR:

  • Access (Art. 15) — request a copy of any benchmark payload tied to a public key you control.
  • Rectification (Art. 16) — file a public dispute on the run page.
  • Erasure (Art. 17) — request deletion via [email protected] with the run id.
  • Portability (Art. 20) — every benchmark payload is already public JSON at https://api.llm-speed.com/v1/results/<id> and signed by your own keypair.
  • Right to lodge a complaint with a supervisory authority (Art. 77).

• International transfers: benchmark payloads live on Cloudflare D1 (SQLite-on-edge), replicated globally across Cloudflare's network for read performance. No primary copy in any specific jurisdiction; serving is from the closest edge.

• No automated decision-making. Submitted runs are aggregated and ranked; no individual visitor is profiled or scored.

• Children: this service is not directed at children under 16. We do not knowingly process data from children.

If your submission was made on a work machine and you need a sentence your DPO can point at: "The processing has a legitimate-interest legal basis (Art. 6(1)(f) GDPR), the data is technical-benchmark-only with no personal identifiers in the payload (see §1 and §2), retention is indefinite for the run record but immediate-deletion-on-request is supported, and the controller can be reached at [email protected]."

---

7. First-run consent

The first time you run llm-speed bench against the public API in a mode that will upload (i.e. not --no-upload, not --dry-run, not --json, not --strict-anon — strict-anon is implicit consent because it's the most-private option), the CLI prints the consent text and asks [Y/n]. The choice is saved to ~/.config/llm-speed/consent.json (mode 0600) along with a timestamp, the CLI version, and the trimmed fingerprint_hash. Delete that file to revoke.

Right after a yes, the CLI asks one optional question: whether you want credit by X or Reddit handle (see "Contributor credit" in section 1). Press Enter to stay anonymous. It is asked once (people who agreed with an earlier version are asked once on their next interactive upload), only at an interactive terminal, and never in CI, with --yes, --strict-anon or --anon, or in llm-speed agent jobs.

---

8. Site analytics

The website (llm-speed.com) uses Cloudflare Web Analytics to count pageviews. A small first-party shopping-interest counter is described below.

The beacon also reports basic web vitals (LCP / CLS / INP) as part of the same cookieless request. If we ever change what analytics the site loads, this document and the CSP get updated in the same commit. We don't ship analytics features without disclosing them here.

Shopping-interest trial

The RTX 5090/M3 Ultra comparison article contains an ordinary Amazon US link. It has no affiliate tag and earns us no commission. We count when at least half of that shopping card becomes visible and when its link is activated, at most once each per page load. These are event counts, not unique visitors or purchases.

The browser sends only an event type (view or click), a random event receipt and a trial/QA label to /api/hardware-interest. Receipts are created separately for each event, are not saved in cookies or browser storage, and do not identify you across pages. Our existing Cloudflare storage retains the UTC date, event type and receipt for at most 45 days. We retain daily aggregate counts for review. The counter does not store IP addresses, page URLs, referrers, email addresses or merchant activity. Existing Cloudflare platform request handling in §4 still applies.

Do Not Track (1) and Global Privacy Control disable this counter. You can also block /api/hardware-interest; the retailer link continues to work. QA counts are kept separate. Visiting the retailer is your choice and its own privacy policy applies once you follow the link.

---

9. Open source

The CLI has a public source repository. Public entry points include:

• Fingerprint capture and bucketing: cli/fingerprint.py
• Upload payload construction and signing: cli/upload.py, cli/signing.py

Website and API code, including the shopping-interest counter, are maintained in a separate internal repository. Their exact source is not currently available through the public CLI repository.

File an issue if anything in this document doesn't match the code.