Sync

The sync skeleton is a push-then-pull loop over one resource, for an app that keeps a local store consistent with a server across many devices.

• Introduction
• The Wire Protocol
• Writing a Feed
• Registering the Ledger's Table
• The Two-Clock Rule
• The Savepoint Note
• What Stays App-Side

Introduction

SyncFeed runs one resource's sync: it pushes everything this device wrote since its last push, then, only when that push had no failure, pulls everything the server has past this device's last pull. It never throws, and answers a SyncReport (pushed, adopted, failure), logged through Log.error on the way out rather than propagated, because a sync is a convenience layered on top of a local store that already works without it.

final SyncReport report = await ItemsSyncFeed().run(scope: 'team-42', account: userId);

if (!report.complete) {
  Log.warning('items sync did not finish: ${report.failure}');
}

SyncLedger is the bookkeeping SyncFeed reads and writes between runs, backed by the sync_cursors table CreateSyncCursorsTable creates.

The Wire Protocol

Push. POST '$resource/sync' with {envelopeKey: [row.data, ...]}, batched at SyncFeed.batchSize (default 500) rows per request. The push response is read the same way a pull page is ({"data": [...]}), so a write this device lost to a conflicting write elsewhere adopts the winner immediately, without waiting for the next pull.

Pull. GET resource with query {scope, cursor}, cursor omitted when there is none yet. The pull response is {"data": [...], "cursor": , "has_more": bool}. cursor is taken from the body rather than derived from the last row: the pair it encodes is the server's business, and a client that built one would be guessing at a tie-break it cannot see. A run walks pages until has_more is false or SyncFeed.maxPages (default 100) is reached.

Push runs first, and the pull runs only after a push with no failure: a failed push returns before the pull ever starts. The push makes the server's answer to the pull already reflect this device's own writes, so one round leaves the device consistent. Pulling first would leave a device that had just resolved a conflict holding a stale view until the next run.

Writing a Feed

A feed subclasses SyncFeed and supplies three things: where its bookmarks live (feed), which endpoint it talks to (resource, envelopeKey), and how to read and write its own rows (pending, adoptRow).

import 'package:magic/magic.dart';

class ItemsSyncFeed extends SyncFeed {
  @override
  String get feed => 'items';

  @override
  String get resource => 'items';

  @override
  String get envelopeKey => 'items';

  @override
  Future> pending({
    required String account,
    required int sinceMillis,
    required String scope,
  }) async {
    final List> rows = DB.select(
      'SELECT * FROM items WHERE scope = ? AND updated_at_ms > ? ORDER BY updated_at_ms ASC',
      [scope, sinceMillis],
    );

    return [
      for (final row in rows)
        (data: row, mark: row['updated_at_ms'] as int),
    ];
  }

  @override
  Future adoptRow({
    required String account,
    required Map row,
  }) async {
    final int? remoteMark = Cast.intOrNull(row['updated_at_ms']);
    if (remoteMark == null) return false;

    final List> existing = DB.select(
      'SELECT updated_at_ms FROM items WHERE id = ?',
      [row['id']],
    );

    if (existing.isNotEmpty && (existing.single['updated_at_ms'] as int) >= remoteMark) {
      return false;
    }

    DB.statement(
      'INSERT OR REPLACE INTO items (id, scope, updated_at_ms, payload) VALUES (?, ?, ?, ?)',
      [row['id'], row['scope'], remoteMark, row['payload']],
    );

    return true;
  }
}

pending answers rows oldest first: the push advances its mark to the last row of each batch it sends, so a batch that did not end on its newest row would move the mark past rows it never sent. adoptRow answers false for a row this client cannot read (a malformed payload) rather than throwing, so one bad row does not stop the rest of a page from landing. Use Cast.intOrNull/doubleOrNull/boolOrNull inside adoptRow when the wire value's exact numeric type (int vs double on web) is not guaranteed.

Registering the Ledger's Table

CreateSyncCursorsTable creates the sync_cursors table SyncLedger reads and writes by default. Magic has no migration discovery, so list it explicitly alongside the app's own migrations:

await Migrator().run([
  CreateUsersTable(),
  CreateSyncCursorsTable(),
]);

The table carries five columns (scope, feed, account, pull_cursor, push_mark) with no id(), no timestamps(), and no unique index: SyncLedger upserts by delete-then-insert, so an app that already keeps a table of this shape can point a custom SyncLedger(table: '...') at it and skip this migration.

The Two-Clock Rule

A SyncFeed run tracks two clocks, and only one of them advances locally. The push mark is this device's own updated_at, epoch millis, deciding which rows are worth sending next time. The pull cursor is the server's, opaque text handed straight back unread.

Only the push mark advances here; the pull cursor never substitutes for it. A row adopted from the server carries the ORIGINATING device's clock, so advancing this device's push mark to that value would step over a local row written earlier and never sent. The cost is small: an adopted row is echoed back to the server exactly once, on the next run, where the server's own >= comparison rejects it and the mark then covers it.

The Savepoint Note

SyncLedger.write issues its delete-then-insert inside a SAVEPOINT/RELEASE pair rather than DB.transaction, because a bare BEGIN does not nest and a feed's own sync run may already be executing inside a caller's outer transaction. A savepoint does nest: opened with no transaction open it behaves like one, and opened inside an existing transaction it joins that transaction and rolls back with it.

That means a caller whose outer transaction fails after this savepoint released loses the bookmark write too. The next run simply resends the tail the server's own >= check already absorbs: one extra round trip, never a duplicate write.

What Stays App-Side

The skeleton is deliberately narrow. Three things stay the app's own responsibility:

• Scope derivation. scope is an opaque string SyncFeed.run passes straight through to the ledger and the wire calls; how an app derives it (a team id, a workspace id) is not this package's concern.
• Salt. Any value mixed into a feed's own local identifiers or cache keys beyond scope and account is the app's own.
• Orchestration. When and how often a feed runs (on a timer, on reconnect, on app resume) is left to the app; SyncFeed.run answers one call, it does not schedule itself.