vansaCLOSED BETA
On this page

VANSA / DEVELOPER DOCUMENTATION

Build with Vansa.

Everything you need to turn context into decisions.

Vansa is a System One decision model (current version: vansa-3). You send a state (text or JSON) and a set of typed questions; it returns a calibrated answer for every one of them. No prompt engineering, no generated text to parse, no hallucinated options: every answer is one of the options you defined, with a probability distribution over them.

Base URL
https://api.vansa.org
Authentication
Authorization: Bearer vsk_…
Model
vansa-3
Endpoint
POST /v1/systemone
The request and response format is the same "state + questions" format used by TypeSafe's Jev (/v1/systemone). Existing Jev client code works by changing the base URL, the key, and model to vansa-3.

NEW MODEL · 24 SEPTEMBER 2026

Vansa-3: 4B parameters, 16k context

Vansa-3 answers every request. It is a new generation: about 4 billion parameters (Vansa-2 had about 0.4 billion) and a 16,384-token context per question, eight times Vansa-2's 2,048. It is built on JevK5, a fine-tune of Qwen3.5-4B, plus an email skill trained by Vansa.

Parameters
about 4B
Context per question
16,384 tokens
106 new questions
0.871, estimated (Vansa-2: 0.712)
JevBench, public items
80.1 (Vansa-2: 44.3)

Use cases

Anything that is a classification, scoring or yes/no decision over some input: routing support tickets, triaging email, detecting intent (up to 128 options per question), moderating content, scoring urgency or sentiment on a scale, guarding LLM inputs, picking the next action in an agent loop. Because the model never generates text (it reads its own probability for each of your options), it is deterministic for a given request and cannot answer outside your option set. Accuracy differs by task: see Benchmarks.

Authentication

Every request needs your API key in the Authorization header. Keys look like vsk_… and are handed out by the Vansa team with a private invitation during the closed beta. Keep it server-side: never ship it in a browser or mobile app.

Authorization: Bearer vsk_your_key_here

Check the key and its remaining quota at any time with GET /v1/me.

Quickstart

curl https://api.vansa.org/v1/systemone \
  -H "Authorization: Bearer $VANSA_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vansa-3",
    "state": {
      "subject": "Duplicate charge on invoice #4411",
      "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
    },
    "questions": {
      "department": {"type": "choice", "instructions": "Which department should handle this request?",
                     "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors",
                                  "sales": "pricing, new contracts", "other": "everything else"}},
      "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                     "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel or leave?"},
      "refund_requested": {"type": "noul", "instructions": "Does the customer explicitly ask for a refund?"}
    }
  }'

Response

{
  "id": "req_bfe3ef4ff5f38256",
  "object": "systemone.result",
  "model": "vansa-3",
  "created": 1790309369,
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {"billing": 0.9922, "technical": 0.0052, "sales": 0.0009, "other": 0.0017},
      "confidence": 0.9624
    },
    "urgency": {
      "type": "score",
      "score": 1.703,
      "legend": {"0": "not urgent", "1": "soon", "2": "critical deadline or blocking issue"},
      "probabilities": {"0": 0.0578, "1": 0.1815, "2": 0.7607},
      "confidence": 0.3788
    },
    "churn_risk":       {"type": "noul", "noul": 0.8568, "probabilities": {"false": 0.1432, "true": 0.8568}, "confidence": 0.8568},
    "refund_requested": {"type": "noul", "noul": 0.8794, "probabilities": {"false": 0.1206, "true": 0.8794}, "confidence": 0.8794}
  },
  "usage": {"input_tokens": 357, "output_tokens": 0, "state_tokens": 40, "truncated": false},
  "latency_ms": 711.5
}

This is real vansa-3 output for the request above. Run it yourself in the playground.

Reference

POST /v1/systemone

Answers every question in questions about state. Each question is answered in its own pass over the state, so the time grows with the number of questions and the length of the state.

Request body

FieldTypeDescription
modelstring, optionalvansa-3 (aliases vansa, vansa-latest; the earlier ids vansa-2 and vansa-1 also work and are answered by vansa-3). Omit for the default model.
statestring · object · arrayWhat the questions are about. A JSON object is recommended: keys become part of the context, so you can refer to them in instructions (`body`, `message`). Arrays work for conversation turns. Up to 16,384 tokens per question (the whole prompt: state, question, options and about 100 tokens of fixed instructions); a longer request is refused, see Limits.
questionsobjectMap of question id → question definition (1 to 32 per request). Ids are yours; they come back unchanged in answers.

Question definition

FieldTypeDescription
typestringchoice, score or noul.
instructionsstringThe question, in plain English. Refer to state fields by name. Max 4000 characters.
criteriadepends on type choice: object {"option": "description or null", …}, 2 to 128 options. A plain array of option names also works.
score: array of level descriptions, lowest first, 2 to 32 levels.
noul: optional {"true": "…", "false": "…"} to spell out what yes and no mean.

Response body

FieldDescription
idRequest id (also sent as the X-Request-Id header). Quote it when reporting a problem.
modelThe model that answered.
answersOne entry per question id, shape depends on the type, see Question types.
usage.input_tokensPrompt tokens read: your state and every question, with the part they share counted once.
usage.state_tokensTokens in your state alone.
usage.truncatedAlways false with Vansa-3: a request that does not fit in 16,384 tokens per question is refused with 400 instead of being cut. Kept for compatibility.
latency_msServer-side time, including time spent waiting for the model.
GET /v1/models

Lists the available models with their context size. No key needed. The earlier ids vansa-2 and vansa-1 are listed too, with routed_to naming the model that answers them.

GET /v1/me

Your key's name, limits and usage today. Requires the key.

{
  "key": {"id": 3, "name": "Tomas – email triage", "prefix": "vsk_9Xk2mQ1p", "created_at": "2026-09-21T10:00:00Z", "expires_at": null},
  "limits": {"requests_per_minute": 60, "requests_per_day": 2000},
  "usage": {"today_requests": 17, "today_remaining": 1983, "quota_resets_at": "2026-09-22T00:00:00Z",
            "total_requests": 231, "total_input_tokens": 88410},
  "model": "vansa-3"
}
GET /health

Returns 200 with "status": "ok" when the model is ready. Otherwise 503, with "status" loading (starting), upstream_unreachable (the model server cannot be reached or has no model loaded yet, e.g. while it restarts) or error (the API failed to start its model). No key needed.

Question types

choice: pick one option

// definition
"intent": {
  "type": "choice",
  "instructions": "What does the customer want in `message`?",
  "criteria": {
    "refund": "money back or a charge reversed",
    "technical_help": "a bug, outage or integration problem",
    "cancellation": "wants to cancel or downgrade",
    "other": null
  }
}
// answer
"intent": {
  "type": "choice",
  "choice": "refund",
  "probabilities": {
    "refund": 0.91, "technical_help": 0.03,
    "cancellation": 0.04, "other": 0.02
  },
  "confidence": 0.71
}

Descriptions are optional (null); without one the model sees only the option name, so describe options whose names are terse or ambiguous. Up to 128 options per question; with more than 16 the model answers in two rounds (see Limits). For choice and score, confidence is 1 minus the normalized entropy of the distribution (1 = certain, 0 = uniform), so it stays comparable across questions with different option counts. For noul it is the larger of the two probabilities (0.5 = undecided, 1 = certain).

score: position on an ordered scale

// definition: levels from lowest to highest
"frustration": {
  "type": "score",
  "instructions": "How frustrated does the customer sound in `message`?",
  "criteria": [
    "calm and neutral",
    "concerned but civil",
    "clearly annoyed",
    "very angry or using strong language"
  ]
}
// answer: score is the expected level (0 .. N-1)
"frustration": {
  "type": "score",
  "score": 2.31,
  "legend": {"0": "calm and neutral", "1": "concerned but civil",
             "2": "clearly annoyed", "3": "very angry or using strong language"},
  "probabilities": {"0": 0.02, "1": 0.12, "2": 0.39, "3": 0.47},
  "confidence": 0.24
}

Use score whenever the options are ordered (severity, stars, priority): the expected value gives you a continuous number to threshold or sort on. Round score if you need a single level, or take the argmax of probabilities.

noul: yes or no

// definition (criteria optional)
"is_phishing": {
  "type": "noul",
  "instructions": "Is this email a phishing or scam attempt?",
  "criteria": {
    "true": "phishing, scam, or fraud",
    "false": "a legitimate email"
  }
}
// answer: noul = probability the statement holds
"is_phishing": {
  "type": "noul",
  "noul": 0.08,
  "probabilities": {"false": 0.92, "true": 0.08},
  "confidence": 0.92
}

Threshold noul at 0.5 for a plain yes/no, or higher when a false positive is expensive. Probabilities are temperature-calibrated on held-out data. Measured calibration error is 0.009 on held-out email threads and 0.066 on average over 13 public tasks (0.129 on the worst, emotion), so read 0.8 as roughly 80%, most exactly on email.

Writing questions that work

Errors

Errors use HTTP status codes and a JSON body:

{"error": {"type": "invalid_request_error", "message": "`questions.intent.criteria` must list at least 2 options, as {\"option\": \"description or null\"}.", "param": "questions.intent.criteria"}}
Statustype / codeMeaning
400invalid_request_errorMalformed JSON, missing fields, bad question definition, or a question whose prompt is over 16,384 tokens. param, when present, names the field; for a request over the token limit the message names the question.
401authentication_errorMissing, invalid, revoked or expired API key.
404not_found_error / model_not_foundUnknown endpoint or model name.
413invalid_request_error / request_too_largeBody over 512 KB.
429rate_limit_error / requests_per_minutePer-minute limit hit. Wait Retry-After seconds.
429rate_limit_error / daily_quotaDaily quota used up; resets at 00:00 UTC.
503server_error / model_loading, server_busyModel starting up, or too many requests queued. Retry after Retry-After seconds.
502server_errorThe model server is unreachable or restarting, did not answer in time, or failed on this request. Retry with backoff.
500server_errorSomething broke on our side. Send us the request id.

Retry 429, 502 and 503 with backoff; do not retry 400, 401, 404 or 413. API responses carry X-Request-Id; an unexpected 500 gives the id in its error message instead.

A 403 with the plain-text body "error code: 1010" does not come from the API but from Cloudflare's browser integrity check: it rejects the default Python-urllib User-Agent. Send any User-Agent header (for example my-app/1.0); requests, curl and fetch already do.

Rate limits and request limits

LimitDefaultNotes
Requests per minute60 per keySliding window. Shown in GET /v1/me (null = no limit on your key). Ask us if you need more for a test.
Requests per day2,000 per keyOnly successful requests count. Resets at 00:00 UTC. null in /v1/me = no daily limit.
Questions per request32Each question is its own pass, so more questions take longer.
Options per choice128Up to 16 options are answered in one pass. With more, the options are split into groups of up to 16 and the group winners meet in a final: slower and, on the 18-, 60- and 77-option benchmarks, less accurate than Vansa-2. Keep descriptions to one short sentence.
Levels per score32
Context16,384 tokens per questionCounted on the whole prompt: state, question, its options and about 100 tokens of fixed instructions. A longer request is refused with 400 and a message naming the question; nothing is cut.
Request body512 KB
Latencydepends on the requestEach question is its own pass: time grows with the number of questions, the length of the state and options beyond 16. Set client timeouts to at least 60 s. Requests may wait in a short queue during bursts.
Closed beta, by private invitation only: no SLA, no billing. Keys may be rate-limited or rotated. Please do not send personal data you would not send to a third-party API.

Request bodies are not stored. Each request is logged with its time, key, model, question count, token count, latency, status and client IP, plus a short error message when a request is rejected (it can name the question id or model at fault), for quotas and support. Requests pass through Cloudflare and are answered on Vansa's own GPU server; no third-party AI provider receives them.

Email qualification preset (link-building outreach)

Vansa-3 has one domain Vansa trained a skill for: guest-post and link-building outreach. Vansa-3's email skill learned 52 questions about outreach threads (every email between an outreach team and one website about a paid post): the 26 in this preset and 26 extras, from 76,584 labelled decisions on 23,894 threads. Send a thread in the trained format and ask those questions exactly as published: that is how the agreement below was measured, on held-out threads from the same outreach data the skill was trained on. Reword them and it drops (0.956 to 0.939 in our test), which is why the questions are published as a file.

Agreement, 26 questions
96.8%
Always the most common answer
81.8%
Extra questions (26 more)
95.4%
Requests per thread
1 (2 with the extras)

Measured on 600 held-out threads the model never saw (15,585 answers): their website, or for free-mail contacts their address, does not appear in the training data. The reference labels come from Gemini 3.8 Flash as the teacher, corrected by code rules (facts from the mail headers, plus consistency rules that can override labels such as outcome, fail_reason and our_handling). An automated referee re-labelled 16 of the 26 questions on 80 audit threads and agreed on 95–98% of the answers on the 75 it scored, but the labelling rules were tuned on these same threads, so it is not an independent check. They are not yet human-verified, so read the numbers as agreement with our labelling rules, which Vansa-3's email skill learned from the same teacher. Measured with Vansa-3 through the API's model server with 8-bit weights; the unquantised model scores the same on these 600 threads (0.961 over all 52 questions).

Files

FileWhat it is
/presets/email-qualification.jsonThe preset: questions (send as is), extra_questions, the state format, an example state, usage rules and the measured agreement per question.
/presets/email_state.pyReference state builder, Python standard library only: the same cleaning, counts and trimming the training data went through. Its trimming gives the trained state for all 3,123 held-out threads when given the tokenizer the training data was built with, and for 90.7% with its built-in token estimate (the rest are trimmed differently; the largest comes to 1,575 tokens, over the 1,250 budget); cleaning and counts are checked on test cases.
/llms.txtThe same guidance for AI coding agents, in plain text.

Input: one thread

{
  "site": "example-garden-blog.com",
  "subject": "Guest post on example-garden-blog.com",
  "roles": "us = our outreach team buying a sponsored guest post; them = the website owner or editor",
  "counts": {
    "ours": 2,
    "theirs": 1,
    "auto_replies": 0,
    "bounces": 0,
    "days_since_last": 112,
    "last_from": "us"
  },
  "messages": [
    {
      "n": 1,
      "from": "us",
      "date": "2026-06-02",
      "text": "Hi, we would like to publish a sponsored guest post on your site for one of our clients. Do you accept guest posts, and what is your price for one article with one do-follow link?"
    },
    {
      "n": 2,
      "from": "them",
      "date": "2026-06-03",
      "text": "Hello, our price is 150 USD per post, do-follow and permanent. Payment in advance by PayPal."
    },
    {
      "n": 3,
      "from": "us",
      "date": "2026-06-04",
      "text": "Thanks, we will check with our client and get back to you."
    }
  ]
}

Call it

# pip install requests; email_state.py from https://docs.vansa.org/presets/email_state.py
import os
import requests
from email_state import build_state

VANSA_KEY = os.environ["VANSA_KEY"]  # your API key (vsk_...)
preset = requests.get("https://docs.vansa.org/presets/email-qualification.json", timeout=30).json()
state = build_state(
    site="example-garden-blog.com",
    subject="Guest post on example-garden-blog.com",
    emails=[  # every email with this site; "us" = your outreach team
        {"from": "us",   "date": "2026-06-02T09:00:00+00:00", "text": "Hi, we would like to publish a sponsored guest post ..."},
        {"from": "them", "date": "2026-06-03T10:00:00+00:00", "text": "Hello, our price is 150 USD per post, do-follow and permanent. ..."},
        {"from": "us",   "date": "2026-06-04T09:00:00+00:00", "text": "Thanks, we will check with our client and get back to you."},
    ],
)
r = requests.post(
    "https://api.vansa.org/v1/systemone",
    headers={"Authorization": f"Bearer {VANSA_KEY}"},
    json={"model": preset["model"], "state": state, "questions": preset["questions"]},
    timeout=60,
)
r.raise_for_status()
answers = r.json()["answers"]
print(answers["outcome"]["choice"], answers["mistake_gave_up_too_early"]["noul"])

Real vansa-3 answers for the example thread above (the site quoted, we promised to come back and never did):

QuestionAnswerProbability
replyanswered1.00
outcomeghosted_by_us1.00
negotiationprice_named_no_decision1.00
fail_reasonwe_dropped1.00
they_named_pricetrue1.00
rounds0 (0, we never sent a number)1.00
prepayment_demandedyes0.99
our_handlingok0.98
mistake_gave_up_too_earlytrue0.93

This is the kind of thread the preset is built to catch: nothing failed on their side, we dropped a live deal.

The 26 questions

Exact wording and option descriptions are in the preset file; send them unchanged.

IdTypeAnswersWhat it tells youAgreement
replychoiceno_reply · auto_reply · bounce · answereddid the site answer at all, automatically, or bounce99.8%
outcomechoicepublished · agreed_not_published · declined_by_them · declined_by_us · ghosted_by_them · ghosted_by_us · still_open · no_response · unclearwhere the deal ended: published, agreed, declined, ghosted (either side), still open95.2%
negotiationchoiceno_price_talk · price_named_no_decision · accepted_first_price · negotiated_success · negotiated_fail · firm_price · they_accepted_ours · countered_no_decisionhow the price talk went90.8%
fail_reasonchoicenot_applicable · none_yet · price_too_high · niche_rejected · no_dofollow · sponsored_tag_only · not_permanent · off_script_terms · no_response · wrong_contact · we_dropped · payment_problem · otherthe main reason it was not published91.3%
they_named_pricenoultrue / falsedid they quote a per-post price97.3%
price_for_igamingchoicenot_mentioned · same_price · different_pricea separate price for betting / casino content93.7%
roundsscore0: 0, we never sent a number · 1: 1 number · 2: 2 numbers · 3: 3 or more numbershow many different amounts we proposed98.0%
dofollowchoiceyes · no · unknowndo-follow links confirmed by them96.7%
permanentchoiceyes · no · unknownthe post stays up for good97.0%
sponsored_tag_requiredchoiceyes · no · unknowna sponsored / partnership label is required98.3%
prepayment_demandedchoiceyes · no · unknownpayment before publishing98.5%
article_by_them_onlychoiceyes · no · unknownthey insist on writing the article87.0%
links_allowedchoicenot_said · one · two · three_or_morelinks allowed in one post96.0%
our_handlingchoicegood · ok · mistakehow well our side handled the thread87.8%
mistake_answered_twicenoultrue / falseour mistake: answered twice99.2%
mistake_ignored_their_questionnoultrue / falseour mistake: ignored their question98.8%
mistake_wrong_pricenoultrue / falseour mistake: wrong price99.7%
mistake_too_pushynoultrue / falseour mistake: too pushy100.0%
mistake_gave_up_too_earlynoultrue / falseour mistake: gave up too early93.3%
mistake_accepted_too_fastnoultrue / falseour mistake: accepted too fast100.0%
mistake_wrong_languagenoultrue / falseour mistake: wrong language99.5%
mistake_revealed_client_earlynoultrue / falseour mistake: revealed client early100.0%
mistake_promised_paymentnoultrue / falseour mistake: promised payment100.0%
mistake_sent_wrong_detailsnoultrue / falseour mistake: sent wrong details99.8%
mistake_slow_to_answernoultrue / falseour mistake: slow to answer99.7%
mistake_othernoultrue / falseour mistake: other99.2%

Extra questions

26 more facts the model was trained on, in extra_questions. Ask them in a second request with the same state (26 + 26 is over the 32-question limit). The "Trained" column says when a question was trained and measured. Where it gives a condition (extra_questions_only_when in the preset), ignore the answer when the condition does not hold. payment_method was trained and measured only on threads where the reference labels had a value; nothing in the state shows which threads those are, so its agreement covers only those threads.

IdTypeAnswersTrainedAgreement
last_fromchoiceus · themalways100.0%
replies_bucketscore0: none · 1: one · 2: two or three · 3: four or morealways100.0%
thread_agechoiceunder_2_weeks · 2_to_8_weeks · over_8_weeksalways100.0%
payment_methodchoicenone · paypal · bank_transfer · card · crypto · wise · otherwhere the reference labels had a value, "none" included (378 of the 600 test threads)99.2%
invoice_mentionednoultrue / falsealways99.2%
deadline_mentionednoultrue / falsealways97.7%
word_count_bucketchoicenot_said · under_500 · 500_to_999 · 1000_or_moreonly when counts.theirs > 099.1%
niche_restrictions_mentionednoultrue / falsealways93.7%
tone_of_themchoicefriendly · neutral · cold · hostileonly when counts.theirs > 083.2%
tone_of_uschoicefriendly · neutral · pushyalways83.2%
they_asked_question_lastnoultrue / falsealways98.5%
we_promised_to_returnnoultrue / falsealways99.0%
discount_offerednoultrue / falsealways95.5%
package_offernoultrue / falsealways93.3%
language_mismatchnoultrue / falsealways98.8%
their_last_topicchoiceprice · terms · article_requirements · scheduling · rejection · question · confirmation · payment · otheronly when counts.theirs > 080.4%
they_requested_somethingnoultrue / falsealways93.3%
we_sent_articlenoultrue / falsealways99.8%
published_url_sharednoultrue / falsealways99.8%
they_asked_payment_proofnoultrue / falsealways99.7%
distinct_prices_bucketscore0: none · 1: one · 2: two · 3: three or morealways87.3%
price_bandscore0: no price named · 1: under 50 · 2: 50 to 99 · 3: 100 to 199 · 4: 200 to 499 · 5: 500 or moreonly when counts.theirs > 095.4%
currencychoicenone · EUR · USD · GBP · otheronly when counts.theirs > 097.1%
price_came_downnoultrue / falseonly when they named a price and a final price was agreed96.1%
they_lowered_pricenoultrue / falseonly when they named a price95.7%
igaming_costs_morenoultrue / falseonly when they named both a normal and an iGaming price96.4%

Limits of the preset

Benchmarks

All Vansa-3 numbers were measured over HTTP at the API's model server on 24 and 25 September 2026 with 8-bit weights, except the 106-question figure, which is an estimate (see ¹). The unquantised model scores the same on email (0.961). Bold = best value in the row.

Beyond the training data

TestVansa-3Vansa-2Jev
106 new questions (2,689 decisions)¹0.871 (estimate)0.7120.949
JevBench v1.3, public items, intelligence 0-100²80.144.382.3
prompt injections (662 cases)³0.8190.6990.858
20 Newsgroups topics (1,000 cases)³0.6460.4600.724
toxic chat (1,000 cases)³0.9530.9430.966
jailbreak prompts (1,000 cases)³0.9150.9750.941
reworded email questions⁴0.9390.9150.851

¹ Questions written by us with answers computed by code: 49 on real held-out outreach threads, 57 on generated JSON (guessing 0.41, always the most common answer 0.45). The set is internal and was used to compare model variants during development; Vansa-2's figure includes a serving layer tuned on it. Vansa-3's 0.871 is an estimate, not one measured run: through the model server it scored 0.879, but a routing bug sent the 57 JSON questions to the email skill. 0.871 keeps the served run on the email-thread questions and replaces the JSON questions with a separate 8-bit run that answers them the way the fixed routing does. The fixed routing is live since 25 September 2026; the set has not been re-measured on it yet. ² The 231 public items only (the judge tier is not public), scored by us with JevBench's own scorer; not the leaderboard score. Jev from the per-item outcomes of Jev 1.13.0 published with JevBench. ³ Not in the training data of Vansa-1, Vansa-2 or Vansa-3's email skill; Vansa-3's base model (JevK5) was trained by its author on data that is only partly published, and Jev's training data is unknown, so these tasks may not be new to either. Jev measured over the TypeSafe API (jev-latest) on 25 September 2026 with the same cases and prompts. On 1,000 cases, differences under about 2 to 3 points are within sampling noise. ⁴ 36 hand-written rewordings of 31 trained email questions on held-out threads. Vansa-3: 0.956 on the same questions in their trained wording, 0.939 reworded (100 threads, 3,532 answers). Vansa-2: 0.957 on all trained questions, 0.915 reworded (the first 50 of those threads, 1,767 answers). Jev (never trained on them): 0.865 on the same questions in their original wording, 0.851 reworded (the same 100 threads, 3,532 answers).

Email qualification (600 held-out outreach threads)

Vansa-3Vansa-2Most common answer
26 preset questions (15,585 answers)96.8%97.2%81.8%
26 extra questions (13,875 answers)95.4%95.5%74.2%
all 52 (29,460 answers)96.1%96.4%78.2%
calibration error (lower is better)0.0090.0500.218

Agreement with our reference labels (Gemini 3.8 Flash corrected by code rules; not yet human-verified). Vansa-3 is within half a point of Vansa-2 on accuracy and better calibrated (calibration error 0.009 against 0.050).

Public test sets

The same cases for every model (up to 1,000 per task, fewer where the test set is smaller), same prompts. Vansa-2 and Vansa-1 trained on the train splits of these tasks. Vansa-3's own training was email only, but its base model (JevK5) was fine-tuned by its author with items from BoolQ, banking77 and MultiNLI (close to XNLI), among others, so it is not zero-shot on those. Jev's training data is unknown (Jev measured live over the TypeSafe API, September 2026). Treat this as indicative, not as a controlled comparison.

TaskVansa-3Vansa-2Vansa-1JevLaya (base of Vansa-1/2)
banking77 (77 intents)0.7330.8880.8540.8050.418
MASSIVE intent (60 options)0.7010.8540.8510.8000.460
MASSIVE scenario (18 options)0.6030.8950.9040.7140.565
AG News (4 topics)0.8740.9270.9260.8860.921
emotion (6 classes)0.5680.8950.8620.5770.598
tweet offensive0.7580.8620.8530.7630.772
tweet emotion0.8140.8310.8390.8320.787
XNLI-en (entailment)0.7890.8880.8680.8750.865
typed-decisions (workflows)0.6400.7230.7070.7350.361
BoolQ (reading comprehension)0.8780.8460.8300.8990.737
SST-5 (5-level score)0.5090.5810.5580.5950.365
tweet irony0.6590.7300.7490.8150.753
tweet hate0.6960.5410.5160.7320.649
mean accuracy0.7090.8050.7940.7710.635
mean calibration error (lower is better)0.0660.0720.0900.0900.200

On these 13 tasks Vansa-3 is weaker: lower than Vansa-2 on 11 of 13; the largest gaps are emotion, MASSIVE scenario, banking77 and MASSIVE intent. If your use case is one of these trained classification tasks, test on your own data first. Mean = unweighted average of the 13 tasks; calibration error = expected calibration error (15 bins) of the chosen answer's probability.