@agency-lang/lora
Train a LoRA adapter for a local SDXL image model from a folder of pictures, from Agency. A LoRA adapter is a small file, tens of megabytes, that teaches a model a style or a character from a few dozen examples. Everything runs on this machine, with Hugging Face offline.
Installation
npm install @agency-lang/lora
Training needs the Python agency local serve uses, with the image
server's packages installed (torch, diffusers, and safetensors come with
them). The base model must be downloaded already:
agency local download diffusers:Laxhar/noobai-XL-1.1
Usage
import { trainLora, loraInfo, STYLE_TAGS } from "pkg::@agency-lang/lora"
node main() {
const r = trainLora(
"./drawings",
"pen and ink",
"diffusers:Laxhar/noobai-XL-1.1",
"./adapters/sketch.safetensors",
steps: 1000,
flip: true,
samplePrompts: ["pen and ink, a cat on a chair"],
)
if (isFailure(r)) { print("failed: ${r.error}"); return }
print("adapter at ${r.value.path} after ${r.value.minutes} minutes")
}
trainLora raises lora::train before it starts, with the folder, the
base model, the output path, the steps, and the estimated minutes, so
you approve the whole run with the numbers in front of you. Nothing runs,
and nothing in the folder is read, until you answer. Write the adapter into your client.adaptersDir and a running
image server loads it by name the first time a request asks:
generateImageLocal("pen and ink, a cat on a chair", "diffusers:Laxhar/noobai-XL-1.1", lora: "sketch")
The two functions and the constant
| Export | Job |
|---|---|
trainLora(imagesDir, trigger, base, outPath, ...) | Train one adapter. Minutes. Raises lora::train. |
loraInfo(path) | Read an adapter file's header: base model, trigger, rank, steps, size. Raises lora::info. |
STYLE_TAGS | The tags a caption should drop so the style binds to the trigger word. |
There is no caption loop and no preview function here on purpose. Both
are a few lines of your own code over tools you already have, and the
examples/ folder has them: captionFolder.agency (glob, tagImage, a
filter against STYLE_TAGS, write) and previewAdapter.agency (two
generateImageLocal calls with one seed, writeBinary, pasteImages).
The knobs
Eight settings are parameters, because a person judging the sample grids
would change them. The rest are fixed at the values the first runs used,
in one dictionary at the top of trainer/rules.py, with a comment each.
| Parameter | Default | Why you would change it |
|---|---|---|
trigger | required | The word the adapter answers to. A real phrase learns faster and keeps the base model's idea of it; a nonsense word owns the token. |
steps | 1000 | The main dial. Too few, the style is faint; too many, every output is a training image. Judge by the grids. |
rank | 16 | How much the adapter can hold: 8 for a style, 16 for a character, 32 for a character with a wardrobe. Doubling the rank doubles the file. |
learningRate | 1e-4 | Halve it if the grids get worse after getting better. |
resolution | 1024 | 768 trains twice as fast for a first look. |
flip | false | Doubles a small set with mirror images. Off for an asymmetric character. |
seed | 1 | Repeatability, and a second seed is a cheap second opinion. |
samplePrompts, sampleEvery | none, 250 | What to render before and after at each checkpoint. The grids are how a run is judged. |
Captions
Each image may have a caption beside it, a.txt next to a.png, with
comma-separated booru tags: what is in the picture, not how it is drawn.
The trigger word goes in front of every caption. If every caption says
monochrome, sketch, the model learns the style belongs to those words
and the trigger learns nothing; that is what STYLE_TAGS is for.
std::vision's tagImage writes tags in this vocabulary.
Numbers
On an M5 Ultra against NoobAI-XL 1.1 at 1024×1024, rank 16: about 0.8 s a step, so a thousand steps with sample grids is about 15 minutes, and the adapter is 93 MB. Memory stays under 30 GB.