PolyMigrate

The i18n-first static-site migrator — the only site migrator that pairs your multilingual pages automatically.

Project write-up · NuGet · 繁體中文說明 · MIT

PolyMigrate turns legacy dynamic sites (old PHP sites and the like) into clean, static-site-ready Markdown — and it treats multilingual content as a first-class concern, not an afterthought. Config-driven, fully offline-rerunnable, built on .NET.

Status: 3.0. The extraction pipeline, pairing, verification, thumbnails and orphan-page recovery are complete and validated against a real full-site migration (see below). The CLI surface, exit codes, Phase output contracts and YAML config are frozen within a major version: new features bump the minor version, fixes bump the patch, and anything that changes one of those four surfaces waits for a major. 3.0 is such a release — it changes exit codes in a few cases so they follow the 0 / 1 / 2 / 130 contract (an orphan probe that got no usable answer, a failed npm launcher and an interrupted command no longer look like success), gives each section its own orphan slug list (orphan_slugs.<section>.txt), keeps animated GIF/WebP thumbnails animated, and adds probe/fetch entry points and cancellable overloads to the library. Config fields, CSV columns and frontmatter fields are unchanged, but a config with an invalid value (a CSS selector, a regex, a path that isn't a single segment, a malformed URL or language tag, a key left without a value…) is now rejected when it is loaded, by every command, instead of when some page or command happens to use it, and a probe in which only some candidates got no answer exits 1 (the list is written, but may be incomplete). Read the 3.0 upgrade checklist in the CHANGELOG before upgrading. Earlier history: 2.0 was an engineering release — it narrowed the Cornhsu.PolyMigrate.Core public .NET API to its intended entry points and removed unused config fields; 2.1 fixed up the release pipeline (arm64 npm packages, symbol packages) — though the npm launcher could only start the Windows/Linux arm64 builds from 2.3.1 on. 2.2 closed a blind spot in verify — pages nothing links to were reported as fine, because a link-following audit never reaches them — and added slugs for keeping up with a source site that is still being updated. Upgrading to 2.2 can turn a passing verify into exit 1. 2.3 reads the source site's own hreflang declarations (see below) and reports them in a new hreflang_map.csv. See the CHANGELOG for the migration notes.

Why

Every multilingual institution site (governments, universities, NGOs, religious organizations) faces the same painful migration step: matching up the language versions of every page by hand. No existing tool solves this — general-purpose scrapers extract pages one at a time and leave the pairing to you. PolyMigrate:

• pairs automatically where filenames are symmetric (/ch/news/x ↔ /en/news/x share a translation_key),
• reads the site's own hreflang declarations — <link rel="alternate" hreflang> is the one pairing signal the site's authors stated rather than something a tool inferred, so it is tried first and can pair across sections (/ch/news/ ↔ /en/press/),
• suggests pairs heuristically where neither applies — slugs that differ only in separators (2025-light-offering vs 2025_light_offering), shared photo albums, normalized dates hidden in slugs (20240121 vs 01212024), title similarity,
• honestly reports what it cannot pair, producing a review-ready gap inventory instead of guessing wrong.

Note what PolyMigrate deliberately does not do: match language versions by URL string similarity. /products/, /produkte/ and /產品/ have a similarity of zero and are the same page; symmetric-path pairing is an exact match after the language prefix is stripped, never a fuzzy one.

Languages are not limited to two: declare any number in lang_map and every output (frontmatter, inventories, pairing) expands accordingly. All locale output is standard BCP-47.

About hreflang

Legacy sites are exactly the sites whose hreflang is missing or wrong — usually bolted on by an SEO contractor years later. So PolyMigrate treats it as strong evidence, not as gospel. A declaration is only used for pairing if it:

• isn't x-default (that's "where to go when nothing matches", not a language version),
• doesn't point at the page itself,
• resolves to a page actually in your mirror — same host included, since plenty of sites put the English version on en.example.org, where a path-only comparison would make a page its own translation,
• has a content page at both ends — neither end is a site-level page (a language picker and the like, which isn't anyone's translation),
• and isn't one of many same-language pages claiming the same target. This is the classic failure: one <link> pasted into the site-wide template, so every Chinese page declares the English homepage as its English version. One page cannot be the translation of three others, so all of those claims are dropped rather than turned into a confident-looking wrong pair.

Everything declared — including the rejected ones — is written to hreflang_map.csv with in_mirror / reciprocal / usable and a reject_reason naming which rule it failed, because "can I trust this site's hreflang?" is a question you want answered with data, not a total.

Pairs are still only ever suggested: hreflang raises the evidence quality, it does not switch the tool into merging pages on its own. That is a deliberate call, not an unfinished one — the measurement behind it, and what would have to change to revisit it, are recorded in docs/hreflang_量測與決策.md.

Battle-tested defaults

The extraction pipeline bakes in fixes for real problems found during a real migration — things generic tools and LLM extractors silently get wrong:

Case study: full temple-site migration

PolyMigrate's pipeline is the productized version of a completed real migration (a bilingual Buddhist temple site, Chinese/English):

• 516 pages mirrored and extracted, 4.6 GB of media
• 281 translation keys; 231 bilingual articles paired automatically by symmetric paths
• built-in verifier: 1,269 internal links + 4,116 media references checked — 0 errors
• 13 orphaned articles recovered via date probing; 141 EXIF-rotated photos fixed in thumbnails

The original Python prototype's output was used as the golden baseline while porting: 466/516 extracted bodies are byte-identical after whitespace normalization, and the remainder are render-equivalent or strictly more faithful.

Two more numbers from that run:

• Re-runs are ~7× faster: media hashes are cached by (size, mtime), taking a full re-run of the 4.6 GB site from 30.1 s down to 4.6 s.
• The output is deployable as-is: redirect_map is auto-filled with the new paths, and PolyMigrate emits both an nginx conf and a Netlify _redirects file — turning a half-day of hand-written 301s into copying one file.

Install & use

Two channels — pick whichever runtime you already have. Same tool, same behaviour.

npx cornhsu-polymigrate extract site.yaml   # Node — no .NET install needed
dotnet tool install -g Cornhsu.PolyMigrate  # .NET 10

The npm build ships a self-contained native binary and only downloads the one matching your platform (win32 / linux / darwin, each in x64 and arm64). One exception: thumbs doesn't run on Intel Macs (darwin-x64), because Magick.NET 15, the image library behind it, no longer ships a build for them; it exits 2 with an explanation. Every other command works there; for thumbnails on an Intel Mac, use 3.0.x. Migrating a site is usually a one-off job — installing a whole SDK for a single run is friction most people won't accept, so the tool meets you where you are.

dotnet tool install -g Cornhsu.PolyMigrate
polymigrate extract site.yaml               # mirror HTML -> frontmatter Markdown + inventories
polymigrate verify .                        # link/media/frontmatter/unlinked-page audit, CI exit codes
polymigrate thumbs site.yaml                # EXIF-corrected, width-capped thumbnails
polymigrate probe-orphans site.yaml --section news --years 2021-2023   # -> orphan_slugs.news.txt
polymigrate fetch-orphans site.yaml --section news                      # reads orphan_slugs.news.txt
polymigrate slugs . --section news          # slugs already mirrored under ./raw (pipeable)

Animated GIF/WebP thumbnails keep every frame — within limits. Scaling an animation frame by frame costs memory and time in proportion to frames × width × height (measured: a 400-frame 1300×900 GIF took 39 s and 3.2 GB). So an animation over 50 megapixels in total, or one whose animated thumbnail would be larger than the original, gets a first-frame thumbnail instead — the same result as before 3.0 (an animation that needs no scaling is copied as-is instead). At the limit it takes about 6 s and a 250–400 MB memory peak, depending on the content (two 100-frame 1000×500 GIFs measured 250 MB and 384 MB). The thumbs summary lists which files got a first frame and why; neither case is a failure or changes the exit code. Still images get a limit too: one over 179 megapixels (Pillow's decompression-bomb threshold — a 270 KB GIF can declare 20000×20000, and decoding it took 14.8 s and 1.6 GB) is not decoded at all and counts as a failed thumbnail (exit 1). 100-megapixel camera originals are well within it.

raw/ is the one thing you cannot regenerate. Re-crawling a site takes hours at a polite request rate, and the source site may be switched off the moment you finish migrating — at which point your mirror is the only copy left. No PolyMigrate command ever deletes or overwrites raw/ or media/; keep them until the migration is signed off, and back them up if the source is going away.

No .NET? Every command above also runs with zero install via npm — just prefix it: npx cornhsu-polymigrate extract site.yaml, npx cornhsu-polymigrate verify ., and so on.

One YAML config per site describes everything site-specific — see examples/ibps-austin.yaml for a fully-annotated real example.

config_version: 1
site:
  base_url: https://legacy.example.org
url_pattern:
  lang_map: { ch: zh-Hant, en: en }     # any number of languages
  default_lang: zh-Hant
  strip_extensions: [.php]
extract:
  content: "section[id]:not(#header):not(#footer)"
pairing:
  fallback: [hreflang, slug_normalized, shared_media, date, title_similarity]

Use as a library

The CLI is a thin shell over Cornhsu.PolyMigrate.Core. To drive a migration from your own .NET code, use the PolyMigrator facade — the single documented entry point:

dotnet add package Cornhsu.PolyMigrate.Core

using PolyMigrate.Core;

var migrator = PolyMigrator.FromConfigFile("site.yaml");
var report = migrator.Extract("out/");        // out/raw, out/media -> out/content + inventories
if (report.HasErrors) { /* unsafe paths were skipped; see path_issues.csv */ }

var verify = PolyMigrator.Verify("out/");      // no config needed; reads Phase 2 output only
Console.WriteLine($"{verify.Errors} errors, {verify.Warnings} warnings");

// Orphan pages (probe-orphans / fetch-orphans). Nothing is written until you decide to.
var probe = await migrator.ProbeOrphansAsync("out/", "news", 2021, 2023);
if (probe.NoUsableAnswers) { /* the site never answered: an empty list means nothing */ }
else { await migrator.FetchOrphansAsync("out/", "news", probe.Found); }

Extract, Verify and GenerateThumbnails also have overloads that take a CancellationToken (the probe/fetch methods take one as their last parameter).

Layout

Development

dotnet build
dotnet test
dotnet run --project src/PolyMigrate.Cli -- --help

See also

多語言網站搬遷最容易搬丟的東西 (in Traditional Chinese) — background on the category rather than this tool: the five pairing signals in priority order, why "just read hreflang" often misses on real legacy sites (measured on one: 528 pages, 0 declarations, sitemap 404), the five gates a declaration has to pass before it is usable at all, and why URL string similarity is the wrong tool for this job.

License: MIT. All dependencies are MIT/BSD/Apache-2.0 (imaging via Magick.NET; ImageSharp was dropped when its 4.x line began requiring a license key at build time). See THIRD-PARTY-NOTICES.

PolyMigrate fetches pages from the site you point it at, and can be configured to send cookies past a bot challenge. It assumes the site is yours, or that you are migrating it with the owner's permission — that is what it is for. Check the source site's terms before pointing it at anything else.