Telemetry

What migkit keeps about how it behaved, what it never keeps, the default, and how to turn it off.

What is collected

how migkit behaved: the command, the engines, the ways chosen and why (as codes), timings and rates in buckets, workers allocated and used, progress and how far the estimate was off, error categories, retries and recoveries, how much was verified and to which guarantee, how a run ended and what the operator stepped in to do.

What is never collected

rows, values, samples, keys or digests of them, statements, schemas, the names of databases, tables, columns or accounts, addresses, paths, connection strings, credentials, logs, stack traces and error messages. Every field is a closed code, a bucket or a bounded number; anything else is refused where the event is made.

Why

to tell which ways work on which engines and shapes, how good the estimates are, and where runs fail or need a hand.

Where it goes

this machine first: events are kept under the state directory in sealed, append-only segments of a fixed size, each chained to the one before, the oldest dropped past the number or age the organisation's policy keeps (telemetry_keep, telemetry_days). Nothing leaves on its own: migkit telemetry push <bucket> sends the sealed segments, once each, to an S3-compatible bucket you run and name, signed in with that store's own credentials, never a key migkit carries; migkit telemetry receive <bucket> reads them back and checks every event again. With MIGKIT_EGRESS=none, or a policy saying telemetry_push: off, nothing is sent and the push says why.

Default

on: events are kept locally; a push happens only when you run it. migkit telemetry show prints exactly what is kept.

How to turn it off

migkit telemetry off, MIGKIT_TELEMETRY=off, MIGKIT_EGRESS=none, or an organisation policy file (MIGKIT_TELEMETRY_POLICY, /etc/migkit/policy.yaml by default) saying telemetry: off. Off stops keeping events and deletes those already kept, the sealed segments not yet pushed included.

Every field an event may have

An event is one of these kinds, each field a closed code, a bucket or a bounded number; a field not listed here, or a value outside its codes, is refused where the event is made.

EventFieldKindWhat it holdsCodes
runversionone ofmigkit's version0.2.0
runcommandone ofthe command
runengineone ofthe hop's engine
runtarget_engineone ofthe target's engine
runcompletionone ofhow it endeddone, same, different, could not verify, incomplete, usage, failed, refused, interrupted
runerrorone ofwhat stopped it, by kindnone, timeout, connection, permission, resource, not found, refused, usage, interrupted, data, stopped, internal
runtookone ofhow long it took<1s, 1s+, 2s+, 4s+, 8s+, 16s+, 32s+, 64s+, 128s+, 256s+, 512s+, 1024s+, 2048s+, 4096s+, 8192s+, 16384s+, 32768s+, 65536s+, 131072s+, 262144s+, 524288s+, >=12d
runsecondsnumberhow long it took0 to 10000000
runwritesflagit was asked to write (--go, --apply)
runscheduledflagits schedule started it
rundatabasesone ofdatabases in the hop's scope0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
rundecisionswholeways chosen0 to 1000000
runfallbackswholeways given up for another0 to 1000000
runoptions_setwholeoptions the operator set0 to 1000
rundroppedwholeevents dropped for a full queue0 to 1000000000
decisionstrategyone ofthe way chosen
decisionreasonssome ofwhy, as codescarried, left alone, row filter, column mapping, resume by chunk, resume by key, keyless, renamed, apart, exact, byte exact, consistent as of, online, masked on the way, access history, certificate checked, source position, into an older server, written in pieces, whole server, indexes built, pages proved, first choice, fell back, none holds, other
decisionguaranteessome ofthe guarantees it reachescorrect:unproved, correct:counted, correct:proved, correct:byte exact, consistent:none, consistent:per range, consistent:as of one moment, idempotent:no, idempotent:converges, exactly once:at least once, exactly once:exactly once, resumable:starts over, resumable:goes on
decisionrankedwholeways that held0 to 1000
decisionpassedwholeways passed over0 to 1000
decisionfallbackflagit replaced a way chosen before
decisiontrialflagit ran on a bounded trial
progressphaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
progressrowsone ofrows done0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
progressbytesone ofbytes done0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
progressrateone ofrows a second0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
progresstookone ofhow long it has run<1s, 1s+, 2s+, 4s+, 8s+, 16s+, 32s+, 64s+, 128s+, 256s+, 512s+, 1024s+, 2048s+, 4096s+, 8192s+, 16384s+, 32768s+, 65536s+, 131072s+, 262144s+, 524288s+, >=12d
progresseta_offnumberactual time over the estimate0 to 1000
progressdoneone ofhow much of it is done0, <50%, <90%, <99%, <100%, 100%
resourcesphaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
resourcesworkers_allocatedwholeworkers allocated0 to 100000
resourcesworkers_usedwholeworkers that ran0 to 100000
resourcesconnectionswholeconnections held0 to 100000
resourcesmemoryone ofbytes of memory the workers hold0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
resourcescpuswholeprocessors on this machine0 to 10000
recoveryphaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
recoveryerrorone ofthe failure, by kindnone, timeout, connection, permission, resource, not found, refused, usage, interrupted, data, stopped, internal
recoveryfaultone ofthe class of failureconnectivity, auth, permission, expired_credential, quota, cpu_pressure, memory_pressure, disk_pressure, lock, packet, statement_timeout, orphaned, duplicate, uncertain_commit, interruption, failover, stale_checkpoint, inconsistent_view, lost_history, schema_change, lossy_conversion, malformed_data, corrupted_evidence, dependency, external_impact, unknown
recoverypathone ofthe action the decision engine chosereconcile, proceed, pause, throttle, retry, replan, isolate, probe, stop
recoveryactionone ofwhat was doneretry, resume, fallback, autotune, reconcile, gave up
recoveryoutcomeone ofhow it endedrecovered, failed, blocked
recoveryattemptswholeattempts0 to 10000
coveragephaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
coverageverifiedone ofthe scope verified0, <50%, <90%, <99%, <100%, 100%
coverageguaranteessome ofthe guarantees reachedcorrect:unproved, correct:counted, correct:proved, correct:byte exact, consistent:none, consistent:per range, consistent:as of one moment, idempotent:no, idempotent:converges, exactly once:at least once, exactly once:exactly once, resumable:starts over, resumable:goes on
coveragecompletionone ofthe verdictdone, same, different, could not verify, incomplete, usage, failed, refused, interrupted
coverageunitsone ofunits compared0, 1-9, 1e1-1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5, 1e5-1e6, 1e6-1e7, 1e7-1e8, 1e8-1e9, 1e9-1e10, 1e10-1e11, 1e11-1e12, >=1e12
unitphaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
unitscopeone ofwhat kind of unit it istable, database, range
unitunitopaquethe unit, as a salted hash
unitoutcomeone ofhow the unit comparedok, skip, warn, diff, error
unitguaranteessome ofthe guarantees it reachedcorrect:unproved, correct:counted, correct:proved, correct:byte exact, consistent:none, consistent:per range, consistent:as of one moment, idempotent:no, idempotent:converges, exactly once:at least once, exactly once:exactly once, resumable:starts over, resumable:goes on
interventionphaseone ofthe phasediscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
interventiondidone ofwhat they didpaused, resumed, cancelled, approved, overrode, retried by hand, repaired by hand, asked, answered, bundled
interventionquestionone ofthe question migkit put to themsource, target, password, goal, write, target_rows, grant, switch
crashcommandone ofthe command
crashphaseone ofthe phase it worked indiscover, assess, configure, plan, schema, move, follow, verify, repair, cutover, rollback, report
crasherrorone ofwhat stopped it, by kindnone, timeout, connection, permission, resource, not found, refused, usage, interrupted, data, stopped, internal