TypedLM

Changelog

Generated from CHANGELOG.md in the project repository. Never maintained twice.

Unreleased

  • addedSignature trait and #[derive(TypedLm)] with #[lm(output, name, description, validate)].
  • addedProgram with strategy selection, repair of invalid answers, run and execute.
  • addedValidation pipeline: tolerant JSON extraction, schema validation with paths, custom rules.
  • addedOpenAiCompatible provider with four strategies, schema dialects and transport retries.
  • addedEvaluation: datasets with partial labels, ExactMatch, FieldAccuracy, reports with confidence intervals.
  • addedTracing spans with OpenTelemetry GenAI field names.
  • addedRegression tests: Baseline stores a report and compares later runs example by example, compare for paired comparisons, Report::require_score against the interval. Reports carry per-example scores and a dataset fingerprint and can be read back.
  • addedevaluate_with and EvalOptions: several runs per example, with a count of examples whose runs agree.
  • addedtypedlm::testing: TestProvider (fixed or schema-valid answers, recorded requests), FnProvider, Recorder and Replay for offline tests.
  • addedProvider for &P and Arc<P>; Request, Response and Capabilities serialise.
  • addedtypedlm-cli: the Evals harness (run across models with a matrix, compare, history, snapshots under .typedlm/evaluations) and the typedlm binary for stored reports.
  • addedReport::label for the run's configuration, e.g. the requested model.
  • addedWorked examples: Program::demonstration, sent before the input as user/assistant turns.
  • addedCompiled programs: Program::compile, CompiledProgram (JSON artefact with signature hash and optional provenance), loading with checks of signature and demonstrations.
  • addedProgram implements Debug without showing the provider.
  • addedOptimizer (feature optimize): optimize_few_shot chooses demonstrations by budgeted local search on a validation set; candidates from demonstrations_from_labels or bootstrap_demonstrations; result as CompiledProgram with a paired comparison.
  • addedoptimize_instructions: a teacher model proposes instructions from the program's mistakes on a validation set; better proposals are kept. The teacher is a TypedLM program itself (ProposeInstructions).
  • fixedReports, baselines and artefacts read back numbers exactly (serde_json with float_roundtrip).