Goals and modes
What a hop is for, what each command's modes do on each side, and what the cutover adds when a hop turns it on.
Goals
A hop says what it is for in one word, its goal. What each does on the source and on the target, and what it guarantees, is what migkit init, migkit assess and every dry run say for the hop at hand.
| Goal | What it is for | On the source | On the target | Guaranteed |
|---|---|---|---|---|
copy (the default) | move the data once and prove every row arrived | read, never written: each table read as of one moment where the engine gives one; a row the application changes while it is read is not carried, and the check names it | the tables it lacks made, and filled with the source's rows, indexes built after them; a table that already holds rows is refused unless the hop says what to do with them (target_rows: refuse, replace or keep) | every row compared after the copy (migkit check): equal, or each row that differs named |
cutover | move, follow the source's changes, prove both equal, and switch with the least downtime - a stream back ready from the moment of the switch | read, and its log of changes followed from a position taken before the copy - the stream's own slot and publication on PostgreSQL, its binary log read on MySQL, a change stream on MongoDB - made by the stream and dropped when it is torn down; nothing else is written to it unless the hop turns cutover: on, and then only during the cutover | filled as a copy fills it, then kept level with the source change by change; kept from the application's writes until the cutover where the engine allows it (protect_target); after it, a stream back from it to the old source (reverse: at_cutover) | no change lost between the copy and the stream (the position is taken before the copy); each cycle's changes compared (watch --verify --delta); the stream torn down only once nothing writes to the source and the stream has passed where it stands |
verify | prove two databases hold the same; nothing is written to either | read only | read only | equal, or every table and row that differs named; nothing written to either side |
two-way | both sides written, each side's changes carried to the other | written: the target's changes carried to it by key, and its own told apart from what migkit wrote there, so nothing goes round | written the same way: the source's changes carried to it | a change made on either side arrives on the other once; where both sides changed one row, what only one side changed is kept and a column both changed is settled by its rule, alike on both sides (two_way) |
keep-in-sync | a copy that follows the source for as long as it runs, the application kept off it | read, and its log of changes followed for as long as it runs; nothing else is written to it | a copy kept level with the source, the application kept from writing to it (protect_target) | each cycle's changes compared, the stream's lag said, a difference named |
rehearse | the whole path the real run will take - copy, follow, prove, cut over - on a copy or a sample of the source into a target of its own, every phase timed, so the hop it rehearses (rehearses:) is forecast from what it measured | a copy or a sample of the real run's source, read and followed as a cutover follows it; written only by a cutover the hop turns on, as the real run's would be | a target of its own, filled, followed and cut over to as the real run's will be; its own writes are timed there, rolled back | every phase measured and kept for the real run's forecast; the two sides proved equal before the real run is allowed to start |
upgrade | the same engine at a newer version, a higher edition (SQL Server Web to Enterprise), or both, with the least downtime: what the new version or edition changes checked first, then the data moved into it and followed, or a copy of its pages taken to the target where the hop allows it - upgraded there where the version moves - and switched over | read and followed as a cutover follows it; never upgraded where it runs - its downtime is the cutover's alone | a server of the new version or edition filled and followed, or a copy of the source's pages taken there - SQL Server's own backup restored, MySQL's clone, PostgreSQL's pages - upgraded by the new version's own upgrade where the version moves, then followed from the copy's position | what the new version removes or changes, and what the new edition lacks of what the source uses, said before anything is written; the source's own statements planned, timed and held to the same answers on both versions |
subset | a copy for testing: the rows chosen and every row they refer to, masked where the hop says, into a staging or developer database, proved row by row | read, never written: the rows chosen and those they refer to | its tables in scope emptied and filled with the subset, masked on the way where the hop says | every reference in the copy resolves inside it; every row as the source's, its masked columns as the rule writes them, none beyond the subset |
consolidate | many databases or tenants into one: each source's rows under a column the hop names, filled with its own value, keys kept apart by the hop's rule, proved source by source | read, never written: each table through its row filter, and its changes followed with every other source's in one loop; frozen with every other source at the cutover only where the hop turns cutover: on | one database the sources share: this source's rows there replaced by its own, under its value, keys shifted where the rule shifts them; the other sources' rows left as they are; each change after the copy shifted and under its value the same way; the counters raised past every source's keys | keys two sources hold found before anything is written and refused, or kept apart by the rule (tenant, offset, remap); a key arrived since that would meet another source's stops the follow before it; every source's rows proved in the target under its value; no row under a value no source has; every source cut over at once |
split | one database into many - a service extracted, a database split by region: each part's rows by its rule, every row placed exactly once, proved, and followed until every part is cut over at once | read, never written: every row counted against every part's rule before the move, its changes followed for every part in one loop; frozen at the cutover only where the hop turns cutover: on | this part's tables (the hop's scope) filled with the rows its rule takes (mapping.where), and kept so: a row a change moves out of its rule removed here before it is written on the part it moves to | rows no part takes and rows two parts take found before the move and refused (rest: stays, shared); each key in exactly one part's target, the targets together the source's; no row on two parts at once while followed; references across the split counted and said |
reshard | N sources onto M targets by one shard function - shards split or merged, a new shard key, a cluster's own distribution - every row placed exactly once, proved, followed until every source and target is cut over at once | read, never written: every row placed by the function before the move, every change of every source followed in one loop; frozen together at the cutover only where the hop turns cutover: on | the rows the function places there: hash, range, list or lookup of the shard key, or the target cluster's own distribution - each batch sent to the node that owns its rows where migkit reaches it; a row whose key moves it removed from the target it left before it is written on the one it reaches | rows no target takes, keys two sources hold and tables with no key found before the move and refused; each key on exactly the target its key places it, as its source has it, the targets together the sources - per target and across them, before the copy ends and inside the cutover; no row on two targets at once while followed; each target's batch one numbered transaction |
archive | old rows to cheaper storage: the rows each rule picks added to an archive and proved there, then removed from the source only once proved and only where the hop allows it | read; written only with archive: {remove: true} and the source's own answer by name (source_changes: {archived_rows: yes}) behind the hop's approvals: each archived row removed once the archive holds it as the source has it now, a row other rows still name never | the archive: the rows each rule picks added, what it holds already kept, never overwritten | every row the rule picks in the archive as the source has it; every row removed from the source in the archive with the digest it had, in the ledger; put back from the archive by archive: {restore: true} |
The commands each goal runs, in order:
| Goal | Commands, in order |
|---|---|
copy | migkit move HOP --mode full; migkit move HOP --mode full --go; migkit check HOP |
cutover | migkit sync HOP --mode migrate; migkit sync HOP --mode migrate --go; migkit watch HOP |
verify | migkit check HOP |
two-way | migkit move HOP --mode full+cdc; migkit move HOP --mode full+cdc --go |
keep-in-sync | migkit sync HOP --mode migrate; migkit sync HOP --mode migrate --go --serve |
rehearse | migkit move HOP --mode full+cdc --go; migkit check HOP; migkit move HOP --mode cdc --drop --go; migkit assess HOP |
upgrade | migkit assess HOP; migkit move HOP --mode full+cdc; migkit move HOP --mode full+cdc --go; migkit check HOP; migkit move HOP --mode cdc --drop --go |
subset | migkit move HOP --mode full; migkit move HOP --mode full --go; migkit check HOP |
consolidate | migkit assess HOP; migkit move HOP --mode full; migkit move HOP --mode full+cdc --go; migkit check HOP; migkit move HOP --mode cdc --drop --go |
split | migkit assess HOP; migkit move HOP --mode full; migkit move HOP --mode full+cdc --go; migkit check HOP; migkit move HOP --mode cdc --drop --go |
reshard | migkit assess HOP; migkit move HOP --mode full; migkit move HOP --mode full+cdc --go; migkit check HOP; migkit move HOP --mode cdc --drop --go |
archive | migkit assess HOP; migkit move HOP --mode full; migkit move HOP --mode full --go; migkit check HOP |
goal: cutover does not turn cutover: on. The goal sets up the stream, the stream back and the protection of the target; the cutover migkit drives itself runs only where the hop's options say cutover:.
Modes
| Command | What it does |
|---|---|
move --mode full | one copy of every table the hop carries: the source read, the target written - kept from the application's writes before the first row and again after the last where the hop asks (protect_target): its rights taken, its access policy changed on an answer of yes, or its writes named, the way the target allows |
move --mode cdc | the stream of changes started, or resumed from where it stood: the source's log read, the target written change by change |
move --mode full+cdc | the copy, then the stream from a position taken before the copy began, so nothing changed during it is lost |
move --mode cdc --drop | the cutover: the stream torn down, the target given back to the application where protect_target kept it, and what else happens on each side is what cutover: says (below); run again once a cutover of the hop has flipped, its point of no return |
sync --mode reconcile | the last check's differences shown; --apply writes the target to match the source |
sync --mode verify | read only: every table compared as of one moment |
sync --mode seed | the schema aligned, the copy, the differences repaired and a consistent check, one after another |
sync --mode stream | the stream started, and each cycle's changes compared |
sync --mode migrate | seed, then stream |
Without --go (or --apply) every one of them only shows its plan, starting with what each side will see.
Verification modes: check vs deep vs delta vs scan
Every verification mode is 100% read-only on both source and target, throttles itself automatically when server load or replication lag rises, and writes verdict.json, summary.json and report.html:
| Layer / Mode | Command | Cost / Complexity | What it verifies | When to use |
|---|---|---|---|---|
check (Full battery) | migkit check shop | O(all rows) in resumable key ranges | Runs schema, counts, autoinc (sequences), data (injective per-row hash + adaptive Merkle / key-range drilldown), deep, and params. Row counts ride inside the checksum query so tables are not scanned twice. | Post-load certification, pre-cutover sign-off, or standalone database audit (goal: verify). |
deep (Semantic & catalog audit) | migkit check shop --deep or migkit check shop --only deep | O(1 fused scan) + catalog queries | Audits 20+ silent failure classes beyond row hashes: NULL vs '', float/NaN precision drift, charset/double-encoding, timezone shift, unvalidated (NOT VALID / NOT ENFORCED) constraints, invalid indexes, disabled triggers, FK orphans, RLS, matviews, grants, and ownership. | Catching subtle cross-engine or dialect/collation corruption that row counts and naive checksums miss. |
scan (Fused aggregate read plan) | Automatic inside migkit check shop (scanplan) | 1 pass per key range (up to 512 expressions/read) | Instead of scanning a billion-row table once per column for each deep/value check, scanplan fuses every aggregate expression (SUM, MIN, MAX, digest) into one shared range scan per side, falling back to isolated column reads only if an expression fails. | Always active automatically during check and deep to keep database I/O and wall-clock time flat. |
delta (Change-driven continuous verify) | migkit watch shop --verify --delta or migkit sync shop --mode stream | O(changed rows since last mark) | Reads the source change stream (WAL slot, binlog, oplog, SCN/redo, or change table) and re-verifies only the primary keys touched since the last verified checkpoint. Distinguishes in-flight replication lag from real drift and announces SAFE TO CUT OVER only when diffs=0, source is quiet, and lag=0. | Continuous live verification on multi-terabyte tables during CDC streaming right up to the cutover second. |
sample (Time-bounded quick probe) | migkit check shop --sample 10 | O(seconds budgeted) | Sizes and reads random keys on both sides within the given time budget (e.g. 10 seconds). Reports different immediately if any sampled row mismatches, or incomplete (never a false same) until full check runs. | Fast smoke check in seconds before running the full table scan. |
drill (Column-by-column row diff) | migkit check shop --db shop --table public.orders --drill | O(limit rows) | Pinpoints the exact primary keys and specific columns that differ inside one table, masking values with keyed HMAC-SHA256 when mask is configured. | Inspecting exact row/column differences before running migkit sync shop --apply. |
Ready-to-run verification commands (click any highlighted parameter to customize inline):
# 1. Full read-only verification (schema + counts + sequences + data + deep + params)
migkit check shop
# 2. Consistent snapshot verification across tables + deep semantic checks
migkit check shop --consistent --deep
# 3. Run only specific check layers (e.g. deep audit or data checksums only)
migkit check shop --only deep
migkit check shop --only schema,counts,data
# 4. Continuous O(changes) delta verification while the source is live
migkit watch shop --verify --delta
# 5. Drill into one table's differing columns, or export the signed evidence bundle
migkit check shop --db shop --table public.orders --drill
migkit report shop
The cutover, on or off
Off (no cutover: in the hop's options, whatever the goal): on the source, nothing written at cutover: the operator stops its writes and its jobs. In order:
- switch off on the source the jobs it runs (pg_cron jobs, MySQL events), so they write in one place
- stop the application's writes to the source: stop it, or revoke its write privileges and end its sessions - ending them is what rolls back what they had not committed
- migkit move HOP --mode cdc --drop --go - migkit proves nothing writes to the source and nothing is prepared there, waits for the stream to pass where the source stands, and only then tears it down (and starts the stream back, with
reverse: at_cutover) - migkit check HOP - the final comparison
- migkit sync HOP --go --kind sequences --tag pre-cutover - the target's counters past the source's
- switch the jobs on on the target
- point the application at the target
- once the application is on the target for good, the point of no return: drop the stream back, if one runs, and give the old source back its writes
On (cutover: true, or its settings): migkit does all of that itself, and adds:
- a timed path: the write outage clocked from the pooler's pause or the freeze to the flip, against the hop's budget - past it, all of it is undone
- the freeze: migkit stops the source's writes itself, the way chosen from what the source allows - the whole server made read-only where it may, or the application's accounts kept from writing and their sessions ended - and reads it back; PostgreSQL, MySQL, TiDB, CockroachDB, MongoDB, Redis, SQL Server, Oracle, Db2, SAP ASE, Teradata, ClickHouse, Cassandra, Couchbase, Kafka, OpenSearch, Redshift and Snowflake sources alike
- the drain: the source's position taken after the freeze, and the hop's own stream waited for until it passes it - or, where the stream keeps no position migkit can read, a mark written on the source after the freeze and waited for on the target; a streaming pipeline and a leg run from here drained by their own offsets and positions
- the fenced check: the rows changed since the last verified pass compared inside the outage, and the schema compared with the preflight's
- the target's hold read back first (protect_target): the application's accounts write nothing there, or, where nothing holds a writer, the writes that are not migkit's named - one named is drift, and stops the cutover - and given back at the flip, put on again if the cutover is undone
- the counters: the target's sequences and auto-increments raised past every key the source handed out
- the stream back: from the new primary to the old source (reverse: at_cutover), made and proven before the freeze where its engine lets it wait idle, so the flip only starts it, and proven to reach the old source
- the undo: every step recorded before it acts and put back, the last first, on any failure, on the budget passing, or on the next run after one killed half way
- the outage measured: each run's outage and each step's time recorded, and the next cutover refused where the last one's was past its budget
- the point of no return: a step of its own behind an approval of its own - the forward stream ended for good, the old source given back its writes, the stream back dropped, what the cutover left taken away - after which going back is refused
With it on, the source is written during the cutover only: its jobs switched off, its writes frozen, each step reversible and behind the hop's approvals; after the flip it stays frozen until migkit rollback HOP --db DB --apply, or until the point of no return gives it back.
Limits the move keeps to
Each is a hard limit: the plan keeps to it wherever some way can, and where none can it says what binds and what would help. A hop that sets none is planned for the soonest end.
- deadline: the whole move - copy, index builds and check - planned to end within it; where no plan can, what binds is said and what would help
- source_load: the share of the source's reads the move may take: the source is planned as if it read only that fast, the ways that read it less are preferred where that ends sooner, and while it runs the pace holds the move's reads at that share
- replica_lag: the seconds the source's replica may fall behind while it is read: the pace holds the move to it
- money: what the move may spend on what it carries (at the hop's transfer_price_per_gb) and on what a warehouse bills: the ways that bill less are taken first where the budget would be passed
- bytes_billed: the bytes a warehouse may bill the move for: read as money at its price
- outage: the cutover's write outage, seconds: the plan says whether its predicted outage keeps within it (with cutover: on, the cutover's own budget is this limit)
- target_memory: the target's memory the move may take: as many writers at once as it holds
- target_connections: the target's connections the move may take: as many writers at once as they carry