PRODUCT

Versioned Data Sync Spec

📄 Engineering spec for Halal Maps — Designed for scale, speed, and reliability

🔧 Purpose

To keep users’ restaurant data updated without wasting bandwidth. The sync system enables:

• Offline functionality
• Diff-based updates between versions
• Full updates when necessary or when it is most bandwidth-efficient option
• Hash validation to ensure data integrity
• Mobile-friendly performance

🧩 The Problem

Restaurant data changes frequently — status updates, business details, etc. Fetching the full dataset every time leads to:

• High and wasteful data usage for unchanged data
• Inefficient approach to scale app to many users

The solution needed to:

• Detect version differences
• Deliver only what’s changed

🧠 Solution Overview

We implemented a multi-stage versioned sync system, optimised for bandwidth efficiency and safety.

3 Layers:

• metadata.json — ultra-lightweight (~150B), includes current version + hash
• manifest.json — lightweight (~1-3KB) guides the client on most effective update path
• Dataset files — will download and update specified file paths

🔄 Sync Logic (Client-Side)

1. On launch, app checks last time a version check was performed
2. If more than 24 hours → fetch live metadata.json
3. If app version/hash mismatch live metadata → fetch manifest.json
4. Based on manifest, download either a full or diff update path
5. Validate update via SHA-256 hash to confirm download integrity
6. Apply and save locally

🔁 Version Flow Logic

The sync system manages data updates through a version chain, where each dataset file is either a full version or a diff(incremental) update. Each file includes:

• A version (e.g. 1.00, 1.01, 1.02)
• A type: "full" or "update"
• A toVersion, which tells the app what comes next
• An entrypoint , which tells the app where to start for new users

There are two main update paths the system supports:

1. Diff-Based Update Chain (for active users)

Users who started from the first full version (1.00) are updated progressively using small diff files:

1.00 (full) → 1.01 (diff) → 1.02 (diff) → 1.04 (diff)

Each diff contains only the data changes from the previous version. ✅ This path is ideal for users who open the app regularly and stay up to date. ✅ It saves bandwidth because most diffs are just a few kilobytes.

2. Full Update Entry Point (for new or out-of-sync users)

At a certain point, we create a new full snapshot (e.g. at diff update 1.02 we simultaneously created a full checkpoint update 1.03) that becomes the new entry point:

1.03 (full) → 1.04 (diff)

This path is used when:

• A user installs the app for the first time
• A user’s current version isn’t recognized
• The local dataset hash doesn’t match (e.g. corruption or manual wipe)

✅ This keeps syncing reliable without forcing a long diff chain ✅ Users still get to the latest version (1.04) in just two steps

🌍 Why This Matters

This hybrid system is designed to minimise bandwidth usage while keeping all users in sync:

• Regular users download only small diffs (e.g. 5–15KB vs. full file sizes of 1–2MB)
• Infrequent or new users skip the diff chain and enter through the latest full snapshot
• No unnecessary downloads — the system always chooses the most efficient path
• Robust fallback — full versions ensure users can recover from any sync mismatch

📁 Example of a version chain

🛠 Debugging & Recovery

The sync system includes built-in safeguards for debugging and rollback without disrupting users

🧯 Handling Faulty Updates

If an update is later found to be faulty:

• We can overwrite the file on the server to correct it
• Users who haven’t downloaded it yet will get the corrected version
• The update chain remains intact — no reversion of other diffs or full versions is required

🔐 Hash Validation on Device

Each file is hashed using SHA-256, and the client always verifies it before applying:

• If a user already downloaded a faulty file, the app will detect a mismatch between the faulty update's hash and the expected fixed update's hash in the manifest
• When this happens, the client:
  1. Abandons the bad update
  2. Follows the manifest’s fallback path
  3. Downloads from a newer full update entry point

🔐 Example

• If version 1.04 is faulty, it can be silently overwritten on the server.
• Users who already downloaded the bad file will fail hash validation on the next metadata check and will automatically fallback to a full update path (e.g. 1.03 → 1.04).
• This allows fixes without breaking the update chain or requiring an app update. From the user's perspective, the fix is silent — the system simply delivers the right file on the next sync attempt.
• This allows developers to correct issues without invalidating or rewriting the whole update chain.

✅ Why This Matters

• No app update is needed to fix a broken data file
• Users with broken versions self-heal through manifest logic
• This makes the system safe to iterate, even in production
• It supports fast patching, without disrupting unaffected users

🛠 Backend Dashboard

• Controls creation and publishing of new dataset versions
• Includes a test mode for safe simulation of updates before production push
• Enables non-technical contributors to QA data changes before they go live

📊 Results

• Can lead up to ~95% reduction in update payload
• Scales easily as the number of users or restaurants grows
• Keeps bandwidth associated costs low

🧱 Possible Future Improvements

• Add compression to updates
• Enable push updates to client with faulty data
• Real-time database connectivity with offline caching

👥 Stakeholder Benefits

• Users: Seamless experience with fast, low-bandwidth updates
• Engineering: Deterministic, easy-to-debug system
• Product/Ops: Controlled rollout via versioning

✅ Summary

This sync system is lightweight, scalable, and robust — keeping user data fresh without compromising performance. Built for Halal Maps, it’s equally applicable to any product handling structured, evolving datasets that requires offline functionality.