TypedLMv0.1.0

Architecture

The parts of TypedLM, how a call flows through them, and where each one lives in the code.

Crates

Crate Content
typedlm the library: signatures, programs, providers, validation, evaluation
typedlm-macros #[derive(TypedLm)], used through typedlm
typedlm-cli Evals harness, snapshot store, the typedlm binary for stored reports

Modules in crates/typedlm/src/:

Module Responsibility
signature.rs the Signature trait
program.rs Program: strategy choice, instructions, worked examples, repair loop, tracing
optimize.rs few-shot optimizer on pathwise’s budgeted local search, reflective instruction optimizer (feature optimize)
tools.rs action enums as tools, Action, Policy, ToolProgram
compiled.rs CompiledProgram: configuration as JSON artefact, signature hash, checks on load
provider.rs Provider, DynProvider, Request, Response, Capabilities
dialect.rs schema_for_dialect: schema rewriting per provider
output.rs parse_output: the validation pipeline
json.rs tolerant JSON extraction
schema.rs schema validator, Violation
error.rs Error, repair feedback
http.rs OpenAiCompatible (feature http)
eval.rs datasets, metrics, evaluate, Report (feature eval)
eval/regression.rs Baseline, compare, Report::require_score
sha256.rs dataset and request fingerprints
testing.rs TestProvider, FnProvider, Recorder, Replay

Signature

Signature is implemented on the input type. It names the output type, which must implement DeserializeOwned + JsonSchema, and carries a name and a description as constants. The derive fills them from #[lm(output = …)] and the doc comment; implementing the trait by hand is supported. An optional validate function holds domain rules.

A call, step by step

Focus
One program run
One program runProgram::execute builds a request, the provider calls the model, and the answer is validated. A repairable mistake goes back to the model with feedback until the repairs are used up.Input → Build requestBuild request → ProviderProvider → Answer valid?Provider → Error::Provider: failsAnswer valid? → Typed output: yesAnswer valid? → Repairs left?: repairableAnswer valid? → Error::Refusal: refused, cut offRepairs left? → Repair feedback: yesRepairs left? → RetryLimitExceeded: noRepair feedback → Build request: next attemptfailsyesrepairablerefused, cut offyesnonext attemptInput – your structInputyour structBuild request – instructions, schema, examplesBuild requestinstructions, schema, examplesProvider – retries 429 and 5xxProviderretries 429 and 5xxAnswer valid? – parse_outputAnswer valid?parse_outputTyped outputTyped outputRepairs left?Repairs left?Repair feedback – every violationRepair feedbackevery violationError::ProviderError::ProviderError::Refusal – or TruncatedError::Refusalor TruncatedRetryLimitExceeded – every attempt's errorRetryLimitExceededevery attempt's error

Source: crates/typedlm/src/program.rs

Show chart data
Data for One program run
StepKindLeads to
InputstartBuild request
Build requeststepProvider
Providerinput or outputAnswer valid?, Error::Provider (fails)
Answer valid?decisionTyped output (yes), Repairs left? (repairable), Error::Refusal (refused, cut off)
Error::Providerend
Typed outputend
Repairs left?decisionRepair feedback (yes), RetryLimitExceeded (no)
Error::Refusalend
RetryLimitExceededend
Repair feedbackstepBuild request (next attempt)
Focus
One program run
One program runProgram::execute builds a request, the provider calls the model, and the answer is validated. A repairable mistake goes back to the model with feedback until the repairs are used up.Input → Build requestBuild request → ProviderProvider → Answer valid?Provider → Error::Provider: failsAnswer valid? → Typed output: yesAnswer valid? → Repairs left?: repairableAnswer valid? → Error::Refusal: refused, cut offRepairs left? → Repair feedback: yesRepairs left? → RetryLimitExceeded: noRepair feedback → Build request: next attemptfailsyesrepairablerefused, cut offyesnonext attemptInput – your structInputyour structBuild request – instructions, schema, examplesBuild requestinstructions, schema, examplesProvider – retries 429 and 5xxProviderretries 429 and 5xxAnswer valid? – parse_outputAnswer valid?parse_outputTyped outputTyped outputRepairs left?Repairs left?Repair feedback – every violationRepair feedbackevery violationError::ProviderError::ProviderError::Refusal – or TruncatedError::Refusalor TruncatedRetryLimitExceeded – every attempt's errorRetryLimitExceededevery attempt's error

Source: crates/typedlm/src/program.rs

Show chart data
Data for One program run
StepKindLeads to
InputstartBuild request
Build requeststepProvider
Providerinput or outputAnswer valid?, Error::Provider (fails)
Answer valid?decisionTyped output (yes), Repairs left? (repairable), Error::Refusal (refused, cut off)
Error::Providerend
Typed outputend
Repairs left?decisionRepair feedback (yes), RetryLimitExceeded (no)
Error::Refusalend
RetryLimitExceededend
Repair feedbackstepBuild request (next attempt)
  1. Program::execute serialises the input to JSON.
  2. It asks the provider for its Capabilities and chooses the strategy: the one set on the program, otherwise the strongest the provider supports.
  3. It rewrites the output schema into the provider’s dialect and builds the instructions: the description, plus the schema when the strategy or the provider requires it.
  4. The provider sends the request and returns the raw answer, the model, the finish reason and token usage.
  5. parse_output turns the answer into the output type or an error (see Validation and repair).
  6. A repairable error adds the answer and its feedback to the next request, until the answer is valid or the attempts are used up.
  7. The result is an Execution (output, attempts, usage, model, strategy) or a Failed with the error, attempts and usage.

All of it runs inside one tracing span.

Providers

Provider uses native async fn in traits, so a program over a concrete provider has no boxing. DynProvider is the object-safe counterpart, implemented for every provider; Box<dyn DynProvider> is a provider again. Transport retries live in the provider, repairs in the program.

Evaluation

evaluate runs every example through Program::execute with a concurrency limit (futures-util, no runtime binding) and builds a Report from the outcomes. It reuses the schema validator to check dataset labels when loading. The report keeps each example’s score and a SHA-256 of the dataset, so eval/regression.rs can compare two runs pairwise and refuse runs on different examples.

Edit this page on GitHub · Docs for v0.1.0