Work/01

Pramana

Open-source decision models with calibrated confidence

Small open models that answer a structured question with a probability for every option, and say how much to trust that answer.

PythonLoRAHugging FaceA100

01What it is

Pramana takes one piece of unstructured state and a typed question, and returns a probability over every option rather than a single label. Each answer carries two outputs from the same logits: a calibrated probability, and a separate confidence score that tracks whether the top answer is actually right.

Two models ship: Pramana-2B and Pramana-0.8B, both trained from Qwen bases and published on Hugging Face with their calibration files and predictions.

02How it works

  • Soft gold training. Fine-tuned with LoRA against soft labels and a Brier term, so the probabilities come out calibrated without a post-hoc temperature. The fitted temperature lands between 1.00 and 1.05.
  • Decoupled calibration. One set of logits, two temperatures: one fitted to the soft gold for KL and Brier, one fitted to the hard label for expected calibration error and deferral.
  • A 0.8B to 2B cascade. The small model answers whenever its top probability clears 0.68, chosen on holdout; the rest go to the 2B. One decision in three never reaches the larger model and accuracy holds.

03What it measured

ModelParamsAccuracyKLBrierConfidence ECE
Pramana-2B2.3B0.80050.07550.04150.0379
Pramana-0.8B0.8B0.79700.08190.04460.0249
Board, 3-4B entries3-4B0.808-0.82050.077-0.3010.042-0.132-

Typed Decisions benchmark: 400 test cases, 2,000 decisions, four workflows. Pramana is trained on the train split, so it is compared with other fine-tuned entries, not zero-shot rows.

04What it does not claim

The 3-4B board entries still edge out Pramana on raw accuracy. The claim here is calibration and cost, not a state-of-the-art score: the lowest KL and Brier of any fine-tuned entry listed, with a cascade that keeps accuracy while leaving the larger model idle for a third of decisions.

Email copied