IO LABResearch program ↗
ENRU
← All directionsWORKING PAPER / 003
IO / LEARNINGv0.1 · 08 SEP 2026Working edition · not peer reviewed

Distillation & Local Models

Learning from verified action without discarding the control system

IO Lab · Independent AI systems researchResearch proposal · training not demonstrated
Read the paper ↓Explore the mechanism ↗

00 / ABSTRACT

The training unit is an attributable episode.

The long-term IO Lab program asks which repeatable behaviors of an effective agent system can be transferred into a locally served model. The proposed learning unit is a verified episode connecting task state, context, a decision, an action, an observed outcome and an evaluation. Raw conversations mix successful policies with luck, unverified claims and sensitive data; they are not automatically suitable supervision.

This working paper specifies an episode contract, curation boundaries, a training objective and a promotion protocol. It distinguishes policy behavior that weights may internalize from fresh knowledge, permissions, tools and recovery that remain outside the model. No trained IO Lab model, dataset release or frontier-equivalence result is introduced.

Contributions of this edition

  1. An episode schema connecting model behavior to observable outcomes and provenance.
  2. A curation process separating training eligibility from task success.
  3. A promotion gate covering held-out quality, false acceptance, privacy and serving constraints.
FIGURE 01 · MECHANISMConceptual model
The verified-episode distillation pipelineObserved episodes pass curation before training. A frozen held-out evaluation decides whether a candidate can be promoted to local serving. Rejected candidates do not change the active model; the control runtime remains in place. 01Episode02Curate03Train04Evaluate05Promote
Observed episodes pass curation before training. A frozen held-out evaluation decides whether a candidate can be promoted to local serving. Rejected candidates do not change the active model; the control runtime remains in place.
  1. 01

    Record state, action and evidence.

  2. 02

    Check rights, provenance and quality.

  3. 03

    Fit a declared behavioral target.

  4. 04

    Test on isolated held-out tasks.

  5. 05

    Release only within passed gates.

01 / PROBLEM

A successful transcript can teach the wrong lesson.

A task can succeed because a tool compensated for a model error, a human repaired the output, or the environment happened to match an unjustified assumption. Imitating the entire conversation can encode those errors together with the useful behavior. An episode must therefore identify what the model actually decided, what the environment supplied and what verified the result.

The deployment question is also specific. A locally served model may reduce external calls for repeated task families, but its usefulness depends on end-to-end quality and serving constraints. Model size alone does not establish that the system will meet its workload.

02 / HYPOTHESIS

Internalize repeatable decisions; retain external checks.

Candidate behaviors include selecting relevant context, producing a valid action schema, asking a material clarification and choosing a repair after a specific failure. Fresh facts, changing permissions and external state remain runtime inputs. The model is not trained to bypass the verifier or infer authority from a successful precedent.

03 / EPISODE CONTRACT

What exactly is being learned?

Table 1. Minimum episode fields
FieldPurpose
Task and criterionFix what counted as success before the action.
State / contextRecord the available observations, source revisions and permitted scope.
Decision / actionCapture the observable plan choice, tool call or structured response.
Receipt / deltaRecord what actually happened, linked to the invocation.
EvaluationStore pass, fail or unknown per criterion, including evaluator identity and rule.
AttributionMark teacher intervention, human repair and environmental assistance.
EligibilityTrack data rights, privacy review, sanitization and allowed uses.

Use observable action traces, concise decision summaries and evaluation evidence. Hidden model reasoning is neither required nor assumed available. Retain unsuccessful and uncertain episodes with their labels: they can inform evaluation or a separately designed preference task, but should not be silently treated as positive imitation targets.

04 / MODEL

Define the target before choosing the training recipe.

D₊ = {e : eligible(e) ∧ verified(e) ∧ attributable(e)}(1)
Proposed positive-supervision subset. Each predicate requires a documented rule. A passed task does not, by itself, make every action in its episode eligible.
L(θ) = − Σₑ∈D₊ wₑ · log πθ(aₑ | sₑ, cₑ)(2)
Illustrative behavior-cloning objective over the selected observable action target. Weights w are fixed from a declared curation rule. This equation does not imply access to teacher logits or hidden reasoning.

Distinguish action imitation from classical probability-distribution distillation. If only API outputs are available, the training target is a sampled output or action, not the teacher’s full distribution. Include provenance and failure types so a model does not learn that verbosity or certainty is a substitute for evidence.

FIGURE 02 · PROMOTIONIllustrative gate

What does it take to promote a local model?

Four independent conditions. Quality alone does not authorize release. These switches illustrate the process, not the readiness of an actual model.

HOLD · 2 of 4 conditions satisfied

05 / CURATION

Quality, permission and leakage are separate filters.

  • Confirm the origin and allowed training use of each source. Successful execution does not imply permission to train.
  • Remove secrets and unnecessary personal data while preserving the decision-relevant structure. Re-evaluate transformed examples if sanitization changes meaning.
  • Deduplicate by task lineage, repository and template, not just by exact text.
  • Partition related tasks before generating synthetic variants or teacher corrections.
  • Quarantine contradictory or unresolvable evaluations. Keep an audit trail for exclusions.
  • Freeze the evaluation set before iterative training; never feed its corrections into the training pool.

Split by project or task family and, when appropriate, by time. A random row split can put near-identical incidents on both sides. The effective sample size is closer to the number of independent task situations than the number of logged messages.

06 / RUNTIME BOUNDARY

A better policy still needs a world model outside its weights.

Table 2. Candidate learned behavior vs retained runtime obligations
Potentially learnedRemains external
Selecting a context requestCurrent source content, access rules and provenance checks.
Decomposing a recurring taskCanonical goal state, budgets and stopping obligations.
Choosing a tool or repairActual tool execution, grants and side-effect reconciliation.
Producing a calibrated responseIndependent criterion checks and final acceptance.

Keep the same external evaluation and authority contracts when comparing models. Otherwise a gain may reflect looser checks rather than better behavior. A trained policy should also support abstention or escalation when the current task falls outside its demonstrated scope.

08 / NEXT EXPERIMENT

Compare the student under the same control conditions.

Table 3. Proposed conditions
ConditionQuestion
Base model, unchanged runtimeWhat is the starting end-to-end performance?
Curated episode training, same runtimeDoes training improve quality or reduce cost?
Output-only training baselineDoes episode structure add value beyond successful final answers?
Training without attributionDoes filtering assistance and accidental success matter?
Student with bounded escalationWhat quality is achieved at the measured teacher-call rate?

Keep base checkpoints, task splits, inference settings and tool access explicit. Count teacher generation, training and evaluation costs separately from serving costs. Evaluate novel task families, stale-context cases and recovery after an induced error. Report failure modes rather than only an aggregate score.

Promotion requires a practical quality threshold, a false-accept upper bound and a serving envelope specified before testing. Estimate uncertainty at the task level. A model that passes one workflow is promoted only for that workflow; it is not described as a general substitute for its teacher.

09 / LOCAL INFERENCE

Evaluate the complete serving configuration.

Record the model revision, weight precision, context length, cache configuration, concurrency, hardware and inference engine. Measure time to first useful output, end-to-end task latency, peak memory and quality under the intended context and tool workload. Quantized and full-precision configurations should be evaluated separately.

A local-only deployment can keep task content on the serving machine when tools, logs and integrations also respect that boundary. “Local model” alone does not establish privacy if a retrieval connector or telemetry path transmits the same data elsewhere. Hardware selection should follow a measured workload, not a paper’s parameter-count target.

10 / EVIDENCE SCOPE

This is a proposed learning path.

This edition introduces no completed distillation run, public episode dataset or trained local IO Lab model. The program’s ≥90% frontier-quality figure is a target for a defined system on its own task distribution. It is not a current score, a universal benchmark claim or a hardware recommendation.

The interactive gate illustrates how promotion criteria combine. Its switch states are chosen by the reader and do not describe a real model’s readiness.

11 / LIMITATIONS

Transfer can fail even when supervision looks clean.

  • The teacher and evaluator may share systematic blind spots.
  • Filtering only successful tasks can remove important recovery behavior and bias the training distribution.
  • Imitation may underperform when the student reaches states absent from the dataset.
  • Privacy transformations can remove necessary context or leave residual identifying patterns.
  • Serving precision and context constraints can change behavior after training.
  • No gain should be attributed to distillation without a matched base-model control and held-out evaluation.

SOURCES

References & intellectual context

These works provide intellectual context and comparison methods. Their results are not experimental evidence for IO Lab’s hypotheses.

  1. Hinton, G., Vinyals, O. & Dean, J.Distilling the Knowledge in a Neural Network 2015 · arXiv preprint
  2. Zelikman, E., Wu, Y., Mu, J. & Goodman, N. D.STaR: Bootstrapping Reasoning With Reasoning 2022 · arXiv preprint
  3. Ross, S., Gordon, G. J. & Bagnell, J. A.A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning 2011 · AISTATS

CITE THIS WORK

Citation & edition

Version 0.1 specifies a proposed training and evaluation pipeline. No dataset, trained checkpoint, measured transfer gain or serving recommendation is released with this paper.

IO Lab. “Distillation and Local Models: Learning from Verified Action Without Discarding the Control System.” IO Lab Working Paper 003, version 0.1, September 8, 2026. https://io-lab.nglain.com/works/distillation-local-models
BibTeX
@techreport{iolab2026distillation,
  author = {{IO Lab}},
  title = {Distillation and Local Models: Learning from Verified Action Without Discarding the Control System},
  institution = {IO Lab},
  type = {Working Paper},
  number = {003},
  year = {2026},
  month = {September},
  note = {Version 0.1; not peer reviewed},
  url = {https://io-lab.nglain.com/works/distillation-local-models}
}

Availability: the full text of this edition is public. The internal repository, raw traces and datasets are not published. No DOI has been assigned.