Skip to content

Track 13

Optimization, Regularization, and PEFT

This track is for the stage after a basic training loop works: schedule shape, clipping, checkpoint choice, freezing depth, and minimal adapter-style updates under one honest validation rule.

Open Optimization Topic Open Transfer Topic Use The Study Plan

Primary Goal

Stabilize The Recipe Before The Backbone

The question is not whether you can add more model. The question is whether schedule, clipping, checkpointing, freezing, or a small adapter is the smallest fix that survives validation.

You Will Practice

Schedules, Checkpoints, Adaptation Depth

AdamW, scheduler choice, gradient clipping, best-checkpoint selection, frozen versus partial updates, and minimal PEFT-style adaptation in native torch.

Best First Move

Run The Recipe Demo Before The Full Lab

Use the short example to see whether the instability is already visible. Then use the full lab only after the recipe comparison makes sense line by line.

Use This Track When

  • your loop runs, but the curve is unstable or the best checkpoint arrives long before the final epoch
  • you do not yet know whether the bottleneck is learning-rate shape, regularization, or adaptation depth
  • full fine-tuning feels expensive, but frozen reuse is not clearly enough
  • you want one defensible answer to: schedule fix, regularization fix, or smaller adaptation surface

What This Track Is Training

  • choosing the smallest recipe change that actually improves the validation story
  • separating optimization trouble from regularization trouble and from adaptation trouble
  • comparing frozen, partial, and adapter-style updates under the same split and checkpoint rule
  • reading histories and checkpoints as evidence instead of decoration

First Session

Start from the repo root.

Run the short example first:

academy/.venv/bin/python academy/examples/optimization-peft-recipes/optimizer_schedule_demo.py

Then enter the full lab:

academy/.venv/bin/python -m pip install -r academy/labs/optimization-regularization-and-peft/requirements.txt
academy/.venv/bin/python academy/labs/optimization-regularization-and-peft/src/peft_workflow.py

Then open the matching exercises:

  • exercises directory: academy/exercises/optimization-regularization-and-peft/
  • exercises file: academy/exercises/optimization-regularization-and-peft/README.md

On the first pass, do not try to improve everything. Make one call: was the first visible problem schedule, regularization, or adaptation depth?

Full Track Loop

  1. Read Optimizers and Regularization and Transfer and Fine-Tuning if the recipe-versus-adaptation boundary is still blurry.
  2. Run the example command from repo root and identify whether clipping, pacing, or checkpoint timing is already the visible problem.
  3. Run the lab from repo root and keep the split, checkpoint rule, and comparison order fixed.
  4. Inspect the written artifacts before making any new change.
  5. Work through academy/exercises/optimization-regularization-and-peft/README.md and defend one target strategy.
  6. End with one short decision: the main bottleneck was optimization, regularization, or missing adaptation capacity.

What To Inspect

Look at the lab outputs in this order:

  • peft_summary.csv
  • pretrain_history.csv
  • frozen_head_history.csv
  • partial_unfreeze_history.csv
  • adapter_history.csv
  • peft_curves.png
  • source_pretrained_best_state.pt
  • adapter_best_state.pt

What to decide from them:

  • whether the best checkpoint appears before the final epoch
  • whether schedule and clipping already solve the instability before deeper adaptation
  • whether the frozen or partially unfrozen path is already acceptable
  • whether the adapter buys enough validation gain to justify the extra moving parts
  • whether the next change should stay in the recipe or move into adaptation depth

Common Failure Modes

  • changing schedule, clipping, and adaptation path all at once
  • treating the final epoch as the default winner
  • using one learning rate when head and backbone clearly need different speeds
  • calling it a PEFT gain when the real gain was checkpoint choice or weight decay
  • escalating to a larger adaptation surface before proving the recipe is stable
  • reading one pretty curve without checking which checkpoint you would actually keep

Exit Standard

Before leaving this track, a learner should be able to:

  • name the winning target strategy and point to the checkpoint or curve evidence that supports it
  • explain whether the main issue was optimization, regularization, or adaptation depth
  • defend why frozen, partial, or adapter-style updates were enough or not enough
  • point to the matching exercises directory and finish the comparison there: academy/exercises/optimization-regularization-and-peft/
  • state the smallest next change worth testing under the same validation rule