Intern-Decision

How inference works

1. Preserve the question and option order, and map each question's options to single-token symbols A, B, …, Z, a, …, z, 0, …, 9.
2. Render the original system prompt, state, decision schema, and a complete assistant JSON skeleton with one `` placeholder per field. Preserve the checkpoint's chat template and empty thinking block.
3. Run one causal Hugging Face forward pass. For the masked-next-token decision objective, read logits at the position immediately before each placeholder.
4. Take a softmax over only that field's allowed candidate-symbol logits, then apply the checkpoint's probability calibration.
5. Map symbols back to the original option values and return typed JSON answers.

This API performs structured candidate scoring. It does not call generate() or sample free-form text. A request can contain multiple fields; no gold answers are inserted into the prompt. The inference compiler uses only state, questions, and optional images.

Benchmark results

Inference latency

Measured on a single RTX 4090 with the local HF inference path. Values are per-query end-to-end latency; they are workload and hardware dependent.

Known-distribution calibration pilot

This separate 96-case diagnostic uses exact reference distributions rather than sampled hard labels. Lower is better. The pilot was not used to fit or select the published temperature; the 4B model used its separately fitted T=1.992418.

Quick start

Use Python 3.12+. Install requirements.txt in a suitable PyTorch/CUDA environment, then import DecisionEngine from the downloaded model directory:

pip install -r requirements.txt

from inference import DecisionEngine

engine = DecisionEngine(device="cuda")  # Load once; reuse for subsequent requests.
request = {
    "state": "The customer was charged twice and asks for the extra payment back.",
    "questions": {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "billing": "Payments and refunds",
                "delivery": "Shipping and delivery",
            },
        },
        "urgency": {
            "type": "score",
            "instructions": "Rate the priority.",
            "criteria": ["Low", "Medium", "High"],
        },
        "refund_requested": {
            "type": "noul",
            "instructions": "Is the customer asking for a refund?",
        },
    },
}
response = engine.predict(request)  # One Python dict in, one response dict out.
print(response["answers"])

predict(request) accepts one request dictionary per call and returns a JSON-serializable Jev-compatible response. It does not read request files or mutate the supplied dictionary. Reuse the engine for each subsequent request.

The engine defaults to the checkpoint next to inference.py. To load another local copy of this same model, use DecisionEngine(checkpoint="./model-copy"). Use the inference module shipped with the selected size so its default calibration matches. backend="hf" is the default and the only implemented backend. The optional request model field does not switch checkpoints; the response model identifies the weights actually loaded by this module.

Request format

{
  "state": "The customer was charged twice and asks for the extra payment back.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle