# Vansa API (closed beta)

Vansa is a **System One decision model** (current version: `vansa-3`). You send a *state* (text or JSON) and a set of *typed questions*;
it returns a calibrated answer for every one of them. No prompt engineering, no generated
text to parse, no invented options: every answer is one of the options you defined, with a probability
distribution over them.

| | |
|---|---|
| API base URL (API calls only) | `https://api.vansa.org` |
| Authentication | `Authorization: Bearer vsk_…` |
| Model | `vansa-3`: about 4B parameters, 16,384 tokens of context per question (aliases `vansa`, `vansa-latest`; `vansa-2` and `vansa-1` also resolve to vansa-3) |
| Endpoint | `POST /v1/systemone` |
| Documentation | https://docs.vansa.org (playground: https://docs.vansa.org/playground) |
| For AI coding agents | https://docs.vansa.org/llms.txt |
| Website | https://vansa.org |

The request and response format is the same "state + questions" format used by TypeSafe's Jev
(`/v1/systemone`). Existing Jev client code works by changing the base URL, the key and `model`.

## New: Vansa-3 (4B parameters, 16k context)

Since 24 September 2026 Vansa-3 answers every request. It is a new generation: about 4 billion parameters
(Vansa-2 had about 0.4 billion) and a 16,384-token context per question, eight times Vansa-2's 2,048. It is built on
[JevK5](https://huggingface.co/alibiserikbay/JevK5), a fine-tune of Qwen3.5-4B, plus an email skill trained by Vansa.

| | Vansa-3 | Vansa-2 |
|---|---:|---:|
| Parameters | about 4B | about 0.4B |
| Context per question | 16,384 tokens | 2,048 tokens |
| 106 never-trained questions | 0.871 (estimate, see Benchmarks) | 0.712 |
| JevBench public items, intelligence 0-100 | 80.1 | 44.3 |

- **Same API.** The request and response format is unchanged. What clients notice: responses say `"model": "vansa-3"`, a request over 16,384 tokens per question gets `400` instead of being cut, and the time a request takes grows with the number of questions (see below), so check your client timeouts.
- **Every model name works.** `vansa-3` is the default; `vansa`, `vansa-latest`, `vansa-2` and `vansa-1` keep working and are answered by Vansa-3. Responses say `"model": "vansa-3"`.
- **Longer inputs.** Up to 16,384 tokens per question (the whole prompt: state, question, options and about 100 tokens of fixed instructions). A longer request is refused with `400` instead of being cut, so `usage.truncated` is always `false`.
- **Better than Vansa-2 on** questions it was never trained on, reasoning (JevBench), prompt-injection detection, reworded email questions and email calibration. **Lower on** jailbreak detection (0.915 vs 0.975), email accuracy (96.1% vs 96.4% on all 52 email questions) and 11 of the 13 public classification tasks Vansa-2 was trained on (mean 0.709 vs 0.805). See Benchmarks.
- **Time grows with the number of questions.** Each question is read in its own pass, so a request takes longer with more questions, a longer state or questions with more than 16 options. Set client timeouts to at least 60 seconds.

## Use cases

Anything that is a classification, scoring or yes/no decision over some input: routing support tickets,
triaging email, detecting intent (up to 128 options per question), moderating content, scoring urgency or
sentiment on a scale, guarding LLM inputs, picking the next action in an agent loop. The model never generates
text (it reads its own probability for each of your options), so it is deterministic for a given request and
cannot answer outside your option set. Accuracy differs by task: see Benchmarks.

- **choice**: pick one option out of N (2 to 128), with a probability for every option.
- **score**: place the state on an ordered scale you describe (urgency levels, 1 to 5 stars); you get the expected value and the per-level distribution.
- **noul**: a yes/no question; you get the probability that the statement holds.

## Authentication

Every request needs your key in the `Authorization` header. Keep it server-side; never ship it in a browser or
mobile app.

```
Authorization: Bearer vsk_your_key_here
```

`GET /v1/me` shows the key's limits and how much of today's quota is left.

## Quickstart

```bash
curl https://api.vansa.org/v1/systemone \
  -H "Authorization: Bearer $VANSA_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vansa-3",
    "state": {
      "subject": "Duplicate charge on invoice #4411",
      "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
    },
    "questions": {
      "department": {"type": "choice", "instructions": "Which department should handle this request?",
                     "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors",
                                  "sales": "pricing, new contracts", "other": "everything else"}},
      "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                     "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel or leave?"},
      "refund_requested": {"type": "noul", "instructions": "Does the customer explicitly ask for a refund?"}
    }
  }'
```

Python:

```python
import requests  # pip install requests

VANSA_KEY = "vsk_..."

r = requests.post(
    "https://api.vansa.org/v1/systemone",
    headers={"Authorization": f"Bearer {VANSA_KEY}"},
    json={
        "model": "vansa-3",
        "state": {
            "subject": "Duplicate charge on invoice #4411",
            "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",
        },
        "questions": {
            "department": {"type": "choice", "instructions": "Which department should handle this request?",
                           "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors",
                                        "sales": "pricing, new contracts", "other": "everything else"}},
            "urgency": {"type": "score", "instructions": "How urgent is this request?",
                        "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
            "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel or leave?"},
            "refund_requested": {"type": "noul", "instructions": "Does the customer explicitly ask for a refund?"},
        },
    },
    timeout=60,
)
r.raise_for_status()
answers = r.json()["answers"]
print(answers["department"]["choice"])   # billing
print(answers["urgency"]["score"])       # 0.0 .. 2.0, expected level
print(answers["churn_risk"]["noul"])     # probability of "yes"
```

JavaScript (Node 18+):

```js
const res = await fetch("https://api.vansa.org/v1/systemone", {
  method: "POST",
  headers: { "Authorization": `Bearer ${process.env.VANSA_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "vansa-3", state: { /* ... */ }, questions: { /* ... */ } }),
});
if (!res.ok) throw new Error((await res.json()).error.message);
const { answers, usage } = await res.json();
```

Real response for the request above:

```json
{
  "id": "req_bfe3ef4ff5f38256",
  "object": "systemone.result",
  "model": "vansa-3",
  "created": 1790309369,
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {"billing": 0.9922, "technical": 0.0052, "sales": 0.0009, "other": 0.0017},
      "confidence": 0.9624
    },
    "urgency": {
      "type": "score",
      "score": 1.703,
      "legend": {"0": "not urgent", "1": "soon", "2": "critical deadline or blocking issue"},
      "probabilities": {"0": 0.0578, "1": 0.1815, "2": 0.7607},
      "confidence": 0.3788
    },
    "churn_risk":       {"type": "noul", "noul": 0.8568, "probabilities": {"false": 0.1432, "true": 0.8568}, "confidence": 0.8568},
    "refund_requested": {"type": "noul", "noul": 0.8794, "probabilities": {"false": 0.1206, "true": 0.8794}, "confidence": 0.8794}
  },
  "usage": {"input_tokens": 357, "output_tokens": 0, "state_tokens": 40, "truncated": false},
  "latency_ms": 711.5
}
```

## Reference

### `POST /v1/systemone`

Answers every question in `questions` about `state`. Each question is answered in its own pass over the state, so
the time grows with the number of questions and the length of the state.

Request body:

| Field | Type | Description |
|---|---|---|
| `model` | string, optional | `vansa-3` (aliases `vansa`, `vansa-latest`; the earlier ids `vansa-2` and `vansa-1` also work and are answered by vansa-3). Omit for the default model. |
| `state` | string, object or array | What the questions are about. A JSON object is recommended: keys become part of the context, so you can refer to them in instructions (`` `body` ``, `` `message` ``). Arrays work for conversation turns. Up to 16,384 tokens per question (the whole prompt: state, question, options and about 100 tokens of fixed instructions); a longer request is refused (see Limits). |
| `questions` | object | Map of question id → question definition, 1 to 32 per request. Ids are yours and come back unchanged in `answers`. |

Question definition:

| Field | Type | Description |
|---|---|---|
| `type` | string | `choice`, `score` or `noul` |
| `instructions` | string | The question in plain English. Refer to state fields by name. Max 4000 characters. |
| `criteria` | depends on type | **choice**: object `{"option": "description or null", …}`, 2 to 128 options (a plain array of option names also works). **score**: array of level descriptions, lowest first, 2 to 32 levels. **noul**: optional `{"true": "…", "false": "…"}`. |

Response body:

| Field | Description |
|---|---|
| `id` | Request id (also the `X-Request-Id` header). Quote it when reporting a problem. |
| `model` | The model that answered. |
| `answers` | One entry per question id; shape depends on the type (below). |
| `usage.input_tokens` | Prompt tokens read: your state and every question, with the part they share counted once. |
| `usage.state_tokens` | Tokens in your state alone. |
| `usage.truncated` | Always `false` with Vansa-3: a request that does not fit in 16,384 tokens per question is refused with `400` instead of being cut. Kept for compatibility. |
| `latency_ms` | Server-side time, including time spent waiting for the model. |

### `GET /v1/models`

Lists the available models with their context size. No key needed.

### `GET /v1/me`

Your key's name, limits and usage today. Requires the key.

```json
{
  "key": {"id": 3, "name": "Tomas – email triage", "prefix": "vsk_9Xk2mQ1p", "created_at": "2026-09-21T10:00:00Z", "expires_at": null},
  "limits": {"requests_per_minute": 60, "requests_per_day": 2000},
  "usage": {"today_requests": 17, "today_remaining": 1983, "quota_resets_at": "2026-09-22T00:00:00Z",
            "total_requests": 231, "total_input_tokens": 88410},
  "model": "vansa-3"
}
```

### `GET /health`

`200` with `"status": "ok"` when the model is ready. Otherwise `503`, with `"status"` `loading` (starting), `upstream_unreachable` (the model server cannot be reached or has no model loaded yet, e.g. while it restarts) or `error` (the API failed to start its model). No key needed.

## Question types

### choice: pick one option

```json
"intent": {
  "type": "choice",
  "instructions": "What does the customer want in `message`?",
  "criteria": {
    "refund": "money back or a charge reversed",
    "technical_help": "a bug, outage or integration problem",
    "cancellation": "wants to cancel or downgrade",
    "other": null
  }
}
```

Answer:

```json
"intent": {
  "type": "choice",
  "choice": "refund",
  "probabilities": {"refund": 0.91, "technical_help": 0.03, "cancellation": 0.04, "other": 0.02},
  "confidence": 0.71
}
```

Descriptions are optional (`null`); without one the model sees only the option name, so describe options whose names are terse or ambiguous. Up to 128 options per
question; with more than 16 the model answers in two rounds (see Limits). For `choice` and `score`, `confidence` is 1 minus
the normalized entropy of the distribution (1 = certain, 0 = uniform), so it is comparable across questions with different
option counts. For `noul` it is the larger of the two probabilities (0.5 = undecided, 1 = certain).

### score: position on an ordered scale

```json
"frustration": {
  "type": "score",
  "instructions": "How frustrated does the customer sound in `message`?",
  "criteria": ["calm and neutral", "concerned but civil", "clearly annoyed", "very angry or using strong language"]
}
```

Answer (`score` is the expected level, 0 to N-1):

```json
"frustration": {
  "type": "score",
  "score": 2.31,
  "legend": {"0": "calm and neutral", "1": "concerned but civil", "2": "clearly annoyed", "3": "very angry or using strong language"},
  "probabilities": {"0": 0.02, "1": 0.12, "2": 0.39, "3": 0.47},
  "confidence": 0.24
}
```

Use `score` whenever the options are ordered (severity, stars, priority): the expected value gives you a
continuous number to threshold or sort on. Round `score` for a single level, or take the argmax of `probabilities`.

### noul: yes or no

```json
"is_phishing": {
  "type": "noul",
  "instructions": "Is this email a phishing or scam attempt?",
  "criteria": {"true": "phishing, scam, or fraud", "false": "a legitimate email"}
}
```

Answer (`noul` is the probability the statement holds):

```json
"is_phishing": {"type": "noul", "noul": 0.08, "probabilities": {"false": 0.92, "true": 0.08}, "confidence": 0.92}
```

Threshold at 0.5 for a plain yes/no, or higher when a false positive is expensive. Probabilities are temperature-calibrated
on held-out data. Measured calibration error is 0.009 on held-out email threads and 0.066 on average over
13 public tasks (0.129 on the worst, emotion), so read 0.8 as roughly 80%, most exactly on email.

## Writing questions that work

- **Send the state as JSON with named fields.** The model receives the state as JSON, keys included, so instructions can name a field: "Which team should handle the request in `` `body` ``?".
- **Describe options**, especially when names are short or overlap.
- **Include an escape hatch** such as `other` in choice questions; the model must pick one of your options.
- **Batch related questions** in one request: one round trip for all of them. Each question is still its own pass, so ask what you need.
- **Long states are accepted up to 16,384 tokens** per question (the whole prompt, see Limits). Accuracy is validated on states up to about 4,000 tokens, and the email preset on threads up to about 1,800; longer states also take longer.
- **Send what the questions need.** Nothing is cut any more, but signatures, footers and quoted history are noise: leave them out.
- **Write in English.** Vansa-3's base model (JevK5) is documented by its author as English only, and its email skill was trained on threads that were 84% English (then Spanish and Portuguese).
- **Same request, same answers.** Inference is deterministic; there is no sampling and no temperature to set.

## Errors

```json
{"error": {"type": "invalid_request_error", "message": "`questions.intent.criteria` must list at least 2 options, as {\"option\": \"description or null\"}.", "param": "questions.intent.criteria"}}
```

| Status | type / code | Meaning |
|---|---|---|
| 400 | `invalid_request_error` | Malformed JSON, missing fields, bad question definition, or a question whose prompt is over 16,384 tokens. `param`, when present, names the field; for a request over the token limit the message names the question. |
| 401 | `authentication_error` | Missing, invalid, revoked or expired API key. |
| 404 | `not_found_error` / `model_not_found` | Unknown endpoint or model name. |
| 413 | `invalid_request_error` / `request_too_large` | Body over 512 KB. |
| 429 | `rate_limit_error` / `requests_per_minute` | Per-minute limit hit. Wait `Retry-After` seconds. |
| 429 | `rate_limit_error` / `daily_quota` | Daily quota used up; resets at 00:00 UTC. |
| 503 | `server_error` / `model_loading`, `server_busy` | Model starting, or too many requests queued. Retry after `Retry-After` seconds. |
| 502 | `server_error` | The model server is unreachable or restarting, did not answer in time, or failed on this request. Retry with backoff. |
| 500 | `server_error` | Something broke on our side. Send us the request id. |

Retry 429, 502 and 503 with backoff; do not retry 400, 401, 404 or 413. API responses carry `X-Request-Id`; an
unexpected 500 gives the id in its error message instead.

A **403 with the plain-text body "error code: 1010"** does not come from the API but from Cloudflare's browser
integrity check: it rejects the default `Python-urllib` User-Agent. Send any
User-Agent header (for example `my-app/1.0`); `requests`, `curl` and `fetch` already do.

## Rate limits and request limits

| Limit | Default | Notes |
|---|---|---|
| Requests per minute | 60 per key | Sliding window; shown in `GET /v1/me` (`null` = no limit on your key). |
| Requests per day | 2,000 per key | Only successful requests count. Resets at 00:00 UTC. `null` in `/v1/me` = no daily limit. |
| Questions per request | 32 | Each question is its own pass, so more questions take longer. |
| Options per choice | 128 | Up to 16 options are answered in one pass; with more, groups of up to 16 and a final between the group winners: slower and, on the 18-, 60- and 77-option benchmarks, less accurate than Vansa-2. Keep descriptions to one short sentence. |
| Levels per score | 32 | |
| Context | 16,384 tokens per question | Counted on the whole prompt: state, question, its options and about 100 tokens of fixed instructions. A longer request is refused with `400`; nothing is cut. |
| Request body | 512 KB | |
| Latency | depends on the request | Each question is its own pass: time grows with the number of questions, the length of the state and options beyond 16. Set client timeouts to at least 60 s. Requests may wait in a short queue during bursts. |

Closed beta, by private invitation only: no SLA, no billing. Keys may be rate-limited or rotated. Please do not send personal data you would not
send to a third-party API.

Request bodies are not stored. Each request is logged with its time, key, model, question count, token count,
latency, status and client IP, plus a short error message when a request is rejected (it can name the question
id or model at fault), for quotas and support. Requests pass through Cloudflare and are answered on
Vansa's own GPU server; no third-party AI provider receives them.

<!-- email-preset:start -->
## Email qualification preset (link-building outreach)

Vansa-3 has one domain Vansa trained a skill for: **guest-post and link-building outreach**. Vansa-3's email skill learned 52 questions about outreach threads (every email between an outreach team and one website about a paid post): the 26 in this preset and 26 extras, from 76,584 labelled decisions on 23,894 threads. Send a thread in the trained format and ask those questions exactly as published: that is how the agreement below was measured, on held-out threads from the same outreach data the skill was trained on. Reword them and it drops (0.956 to 0.939 in our test).

| | |
|---|---|
| Agreement, 26 questions | **96.8%** (always the most common answer: 81.8%) |
| Extra questions (26 more) | 95.4% (baseline 74.2%) |
| Requests per thread | 1 (2 with the extras) |
| Preset file | https://docs.vansa.org/presets/email-qualification.json |
| State builder (Python, stdlib) | https://docs.vansa.org/presets/email_state.py |
| Guide for AI coding agents | https://docs.vansa.org/llms.txt |

Measured on 600 held-out threads the model never saw (15,585 answers): their website, or for free-mail contacts their address, does not appear in the training data. Reference labels: Gemini 3.8 Flash as the teacher, corrected by code rules (facts from the mail headers, plus consistency rules that can override labels such as outcome, fail_reason and our_handling). An automated referee re-labelled 16 of the 26 questions on 80 audit threads and agreed on 95-98% of the answers on the 75 it scored, but the labelling rules were tuned on these same threads. Not yet human-verified: read the numbers as agreement with our labelling rules.

### Input: one thread

```json
{
  "site": "example-garden-blog.com",
  "subject": "Guest post on example-garden-blog.com",
  "roles": "us = our outreach team buying a sponsored guest post; them = the website owner or editor",
  "counts": {
    "ours": 2,
    "theirs": 1,
    "auto_replies": 0,
    "bounces": 0,
    "days_since_last": 112,
    "last_from": "us"
  },
  "messages": [
    {
      "n": 1,
      "from": "us",
      "date": "2026-06-02",
      "text": "Hi, we would like to publish a sponsored guest post on your site for one of our clients. Do you accept guest posts, and what is your price for one article with one do-follow link?"
    },
    {
      "n": 2,
      "from": "them",
      "date": "2026-06-03",
      "text": "Hello, our price is 150 USD per post, do-follow and permanent. Payment in advance by PayPal."
    },
    {
      "n": 3,
      "from": "us",
      "date": "2026-06-04",
      "text": "Thanks, we will check with our client and get back to you."
    }
  ]
}
```

- `us` is always the outreach side (the buyer of the post), `them` the website. Keep `roles` exactly as shown.
- `counts` are computed by your code from the mail headers and today's date (`theirs` excludes auto-replies and bounces; `auto_replies` counts every message with an auto-reply signal, bounces that carry one included).
- Messages oldest first, `date` as `YYYY-MM-DD`, new text only (quoted replies, `>` lines, mail footers and anything after a line of only 2 or 3 dashes or a 'Sent from my iPhone/iPad/Samsung/Android' line removed; at most its first 150 words, shortened to the first 120 plus up to 3 later sentences (35 words at most) with an amount or a yes/no answer; fewer words when a long thread must be trimmed), `"kind": "auto_reply" | "bounce"` when it applies.
- Long threads keep the first message, their first real reply and the last 4, with `{"n": "...", "omitted": k}` for the rest; under about 1,250 tokens.
- Send the state as a JSON object in this key order, not as a string. `email_state.py` builds all of this.

### Call it

```python
import os
import requests
from email_state import build_state   # https://docs.vansa.org/presets/email_state.py

VANSA_KEY = os.environ["VANSA_KEY"]   # your API key (vsk_...)
preset = requests.get("https://docs.vansa.org/presets/email-qualification.json", timeout=30).json()
state = build_state(site="example-garden-blog.com", subject="Guest post on example-garden-blog.com", emails=[
    {"from": "us",   "date": "2026-06-02T09:00:00+00:00", "text": "Hi, we would like to publish a sponsored guest post ..."},
    {"from": "them", "date": "2026-06-03T10:00:00+00:00", "text": "Hello, our price is 150 USD per post, do-follow and permanent. ..."},
    {"from": "us",   "date": "2026-06-04T09:00:00+00:00", "text": "Thanks, we will check with our client and get back to you."},
])
r = requests.post("https://api.vansa.org/v1/systemone", headers={"Authorization": f"Bearer {VANSA_KEY}"},
                  json={"model": preset["model"], "state": state, "questions": preset["questions"]}, timeout=60)
answers = r.json()["answers"]
```

Real vansa-3 answers for this example (the site quoted, we promised to come back and never did):

| Question | Answer | Probability |
|---|---|---:|
| `reply` | `answered` | 1.00 |
| `outcome` | `ghosted_by_us` | 1.00 |
| `negotiation` | `price_named_no_decision` | 1.00 |
| `fail_reason` | `we_dropped` | 1.00 |
| `they_named_price` | `true` | 1.00 |
| `rounds` | `0 (0, we never sent a number)` | 1.00 |
| `prepayment_demanded` | `yes` | 0.99 |
| `our_handling` | `ok` | 0.98 |
| `mistake_gave_up_too_early` | `true` | 0.93 |

### The 26 questions

| Id | Type | Answers | What it tells you | Agreement |
|---|---|---|---|---:|
| `reply` | choice | no_reply · auto_reply · bounce · answered | did the site answer at all, automatically, or bounce | 99.8% |
| `outcome` | choice | published · agreed_not_published · declined_by_them · declined_by_us · ghosted_by_them · ghosted_by_us · still_open · no_response · unclear | where the deal ended: published, agreed, declined, ghosted (either side), still open | 95.2% |
| `negotiation` | choice | no_price_talk · price_named_no_decision · accepted_first_price · negotiated_success · negotiated_fail · firm_price · they_accepted_ours · countered_no_decision | how the price talk went | 90.8% |
| `fail_reason` | choice | not_applicable · none_yet · price_too_high · niche_rejected · no_dofollow · sponsored_tag_only · not_permanent · off_script_terms · no_response · wrong_contact · we_dropped · payment_problem · other | the main reason it was not published | 91.3% |
| `they_named_price` | noul | true / false | did they quote a per-post price | 97.3% |
| `price_for_igaming` | choice | not_mentioned · same_price · different_price | a separate price for betting / casino content | 93.7% |
| `rounds` | score | 0: 0, we never sent a number · 1: 1 number · 2: 2 numbers · 3: 3 or more numbers | how many different amounts we proposed | 98.0% |
| `dofollow` | choice | yes · no · unknown | do-follow links confirmed by them | 96.7% |
| `permanent` | choice | yes · no · unknown | the post stays up for good | 97.0% |
| `sponsored_tag_required` | choice | yes · no · unknown | a sponsored / partnership label is required | 98.3% |
| `prepayment_demanded` | choice | yes · no · unknown | payment before publishing | 98.5% |
| `article_by_them_only` | choice | yes · no · unknown | they insist on writing the article | 87.0% |
| `links_allowed` | choice | not_said · one · two · three_or_more | links allowed in one post | 96.0% |
| `our_handling` | choice | good · ok · mistake | how well our side handled the thread | 87.8% |
| `mistake_answered_twice` | noul | true / false | our mistake: answered twice | 99.2% |
| `mistake_ignored_their_question` | noul | true / false | our mistake: ignored their question | 98.8% |
| `mistake_wrong_price` | noul | true / false | our mistake: wrong price | 99.7% |
| `mistake_too_pushy` | noul | true / false | our mistake: too pushy | 100.0% |
| `mistake_gave_up_too_early` | noul | true / false | our mistake: gave up too early | 93.3% |
| `mistake_accepted_too_fast` | noul | true / false | our mistake: accepted too fast | 100.0% |
| `mistake_wrong_language` | noul | true / false | our mistake: wrong language | 99.5% |
| `mistake_revealed_client_early` | noul | true / false | our mistake: revealed client early | 100.0% |
| `mistake_promised_payment` | noul | true / false | our mistake: promised payment | 100.0% |
| `mistake_sent_wrong_details` | noul | true / false | our mistake: sent wrong details | 99.8% |
| `mistake_slow_to_answer` | noul | true / false | our mistake: slow to answer | 99.7% |
| `mistake_other` | noul | true / false | our mistake: other | 99.2% |

### Extra questions (second request)

Where the Trained column gives a condition (`extra_questions_only_when` in the preset), ignore the answer when it does not hold. `payment_method` was trained and measured only on threads where the reference labels had a value; nothing in the state shows which threads those are, so its agreement covers only those threads.

| Id | Type | Answers | Trained | Agreement |
|---|---|---|---|---:|
| `last_from` | choice | us · them | always | 100.0% |
| `replies_bucket` | score | 0: none · 1: one · 2: two or three · 3: four or more | always | 100.0% |
| `thread_age` | choice | under_2_weeks · 2_to_8_weeks · over_8_weeks | always | 100.0% |
| `payment_method` | choice | none · paypal · bank_transfer · card · crypto · wise · other | where the reference labels had a value, "none" included (378 of the 600 test threads) | 99.2% |
| `invoice_mentioned` | noul | true / false | always | 99.2% |
| `deadline_mentioned` | noul | true / false | always | 97.7% |
| `word_count_bucket` | choice | not_said · under_500 · 500_to_999 · 1000_or_more | only when counts.theirs > 0 | 99.1% |
| `niche_restrictions_mentioned` | noul | true / false | always | 93.7% |
| `tone_of_them` | choice | friendly · neutral · cold · hostile | only when counts.theirs > 0 | 83.2% |
| `tone_of_us` | choice | friendly · neutral · pushy | always | 83.2% |
| `they_asked_question_last` | noul | true / false | always | 98.5% |
| `we_promised_to_return` | noul | true / false | always | 99.0% |
| `discount_offered` | noul | true / false | always | 95.5% |
| `package_offer` | noul | true / false | always | 93.3% |
| `language_mismatch` | noul | true / false | always | 98.8% |
| `their_last_topic` | choice | price · terms · article_requirements · scheduling · rejection · question · confirmation · payment · other | only when counts.theirs > 0 | 80.4% |
| `they_requested_something` | noul | true / false | always | 93.3% |
| `we_sent_article` | noul | true / false | always | 99.8% |
| `published_url_shared` | noul | true / false | always | 99.8% |
| `they_asked_payment_proof` | noul | true / false | always | 99.7% |
| `distinct_prices_bucket` | score | 0: none · 1: one · 2: two · 3: three or more | always | 87.3% |
| `price_band` | score | 0: no price named · 1: under 50 · 2: 50 to 99 · 3: 100 to 199 · 4: 200 to 499 · 5: 500 or more | only when counts.theirs > 0 | 95.4% |
| `currency` | choice | none · EUR · USD · GBP · other | only when counts.theirs > 0 | 97.1% |
| `price_came_down` | noul | true / false | only when they named a price and a final price was agreed | 96.1% |
| `they_lowered_price` | noul | true / false | only when they named a price | 95.7% |
| `igaming_costs_more` | noul | true / false | only when they named both a normal and an iGaming price | 96.4% |

### Limits of the preset

- No amounts or text: prices, counters, a suggested reply and notes need code or an LLM.
- Weakest questions: of the 26, `article_by_them_only`, `our_handling`, `negotiation`, `fail_reason` (87-91%); of the extras, `their_last_topic`, `tone_of_them`, `tone_of_us`, `distinct_prices_bucket` (80-87%); use `confidence`.
- Threads from the last two weeks were rare in training; check low-confidence answers on fresh threads.
- 84% of the labelled threads were English (then Spanish, Portuguese); on the held-out threads English (514) scored 96.0%, Spanish (39) 96.3% and Portuguese (11) 96.7% (all 52 questions, unquantised run).
- Asked from the buyer's side; the publisher's side (swapped roles) was not trained.
<!-- email-preset:end -->

## Benchmarks

All Vansa-3 numbers were measured over HTTP at the API's model server on 24 and 25 September 2026 with 8-bit weights, except
the 106-question figure, which is an estimate (see (1)). The unquantised model scores the same on email (0.961).

### Beyond the training data

| Test | Vansa-3 | Vansa-2 | Jev |
|---|---:|---:|---:|
| 106 new questions (2,689 decisions) (1) | 0.871 (estimate) | 0.712 | **0.949** |
| JevBench v1.3, public items, intelligence 0-100 (2) | 80.1 | 44.3 | **82.3** |
| prompt injections (662 cases) (3) | 0.819 | 0.699 | **0.858** |
| 20 Newsgroups topics (1,000 cases) (3) | 0.646 | 0.460 | **0.724** |
| toxic chat (1,000 cases) (3) | 0.953 | 0.943 | **0.966** |
| jailbreak prompts (1,000 cases) (3) | 0.915 | **0.975** | 0.941 |
| reworded email questions (4) | **0.939** | 0.915 | 0.851 |

(1) Questions written by us with answers computed by code: 49 on real held-out outreach threads, 57 on generated
JSON (guessing 0.41, always the most common answer 0.45). The set is internal and was used to compare model variants
during development; Vansa-2's figure includes a serving layer tuned on it. Vansa-3's 0.871 is an estimate, not one measured run: through the model server it scored 0.879, but a routing bug sent the 57 JSON questions to the email skill. 0.871 keeps the served run on the email-thread questions and replaces the JSON questions with a separate 8-bit run that answers them the way the fixed routing does. The fixed routing is live since 25 September 2026; the set has not been re-measured on it yet.
(2) The 231 public items only (the judge tier is not public), scored by us with JevBench's own scorer; not the
leaderboard score. Jev from the per-item outcomes of Jev 1.13.0 published with JevBench.
(3) Not in the training data of Vansa-1, Vansa-2 or Vansa-3's email skill; Vansa-3's base model (JevK5) was trained by its
author on data that is only partly published, and Jev's training data is unknown, so these tasks may not be new to
either. Jev measured over the TypeSafe API (jev-latest) on 25 September 2026 with the same cases and prompts. On 1,000 cases, differences under about 2 to 3 points are within
sampling noise.
(4) 36 hand-written rewordings of 31 trained email questions on held-out threads. Vansa-3: 0.956 on the same questions in their trained wording, 0.939 reworded (100 threads, 3,532 answers). Vansa-2: 0.957 on all trained questions, 0.915 reworded (the first 50 of those threads, 1,767 answers). Jev (never trained on them): 0.865 on the same questions in their original wording, 0.851 reworded (the same 100 threads, 3,532 answers).

### Email qualification (600 held-out outreach threads)

| | Vansa-3 | Vansa-2 | Most common answer |
|---|---:|---:|---:|
| 26 preset questions (15,585 answers) | 96.8% | **97.2%** | 81.8% |
| 26 extra questions (13,875 answers) | 95.4% | **95.5%** | 74.2% |
| all 52 (29,460 answers) | 96.1% | **96.4%** | 78.2% |
| calibration error (lower is better) | **0.009** | 0.050 | 0.218 |

Agreement with our reference labels (Gemini 3.8 Flash corrected by code rules; not yet human-verified). Vansa-3 is
within half a point of Vansa-2 on accuracy and better calibrated (calibration error 0.009 against 0.050).

### Public test sets

The same cases for every model (up to 1,000 per task, fewer where the test set is smaller), same prompts. Vansa-2 and
Vansa-1 trained on the train splits of these tasks. Vansa-3's own training was email only, but its base model (JevK5) was
fine-tuned by its author with items from BoolQ, banking77 and MultiNLI (close to XNLI), among others, so it is not
zero-shot on those. Jev's training data is unknown (Jev measured live over the TypeSafe API, September 2026). Treat
this as indicative, not as a controlled comparison.

| Task | Vansa-3 | Vansa-2 | Vansa-1 | Jev | Laya (base of Vansa-1/2) |
|---|---:|---:|---:|---:|---:|
| banking77 (77 intents) | 0.733 | **0.888** | 0.854 | 0.805 | 0.418 |
| MASSIVE intent (60 options) | 0.701 | **0.854** | 0.851 | 0.800 | 0.460 |
| MASSIVE scenario (18 options) | 0.603 | 0.895 | **0.904** | 0.714 | 0.565 |
| AG News (4 topics) | 0.874 | **0.927** | 0.926 | 0.886 | 0.921 |
| emotion (6 classes) | 0.568 | **0.895** | 0.862 | 0.577 | 0.598 |
| tweet offensive | 0.758 | **0.862** | 0.853 | 0.763 | 0.772 |
| tweet emotion | 0.814 | 0.831 | **0.839** | 0.832 | 0.787 |
| XNLI-en (entailment) | 0.789 | **0.888** | 0.868 | 0.875 | 0.865 |
| typed-decisions (workflows) | 0.640 | 0.723 | 0.707 | **0.735** | 0.361 |
| BoolQ (reading comprehension) | 0.878 | 0.846 | 0.830 | **0.899** | 0.737 |
| SST-5 (5-level score) | 0.509 | 0.581 | 0.558 | **0.595** | 0.365 |
| tweet irony | 0.659 | 0.730 | 0.749 | **0.815** | 0.753 |
| tweet hate | 0.696 | 0.541 | 0.516 | **0.732** | 0.649 |
| **mean accuracy** | 0.709 | **0.805** | 0.794 | 0.771 | 0.635 |
| **mean calibration error** (lower is better) | **0.066** | 0.072 | 0.090 | 0.090 | 0.200 |

On these 13 tasks Vansa-3 is weaker: lower than Vansa-2 on 11 of 13; the largest gaps are emotion, MASSIVE scenario,
banking77 and MASSIVE intent. If your use case is one of these trained classification tasks, test on your own data
first.

---

Vansa-3 is built on [JevK5](https://huggingface.co/alibiserikbay/JevK5) (Apache-2.0, alibiserikbay), a fine-tune of
[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) (Apache-2.0, Qwen team), with the one-pass prompt and letter
readout of SemIf (TheoLeeCJ, MIT). The email skill and its training data are
Vansa's. Vansa-3 is not endorsed by the upstream authors. Vansa-1 and Vansa-2 were fine-tunes of
[Laya](https://huggingface.co/convaiinnovations/laya) (Apache-2.0) on
[ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) (Apache-2.0).