A “Billable Usage Alert” in your morning inbox is a rude awakening. An optimist might hope this is a sign of overnight success, driven by an uptick in legitimate usage. Anyone with production infrastructure experience immediately wonders, “How much money was burned while I was asleep?”

I woke up to a message like this a few months ago. I’m the founder and lead developer of Plot, an app that organizes and prioritizes your work and messages across your team chat, email, apps, and more. My experience building products, both in the chaotic early days of startups and at the massive scale of Google, showed me that we’re moving beyond our ability to stay on top of everything. After the initial shock, I was pleased to see that Plot surfaced this message as my top priority.

Time for action

Plot uses a local-first sync model popularized by the product management app Linear. Each app, whether desktop, mobile, or in a browser, has a full copy of everything it needs. This ensures the app is fast and reliable, even if the server is slow or the user is offline. Apps sync with the server when they can. In theory, this reduces server load from clients because instead of loading all the data for each page, only the actual changes are transferred.

The alert was concerning because it was for CPU usage by a sync worker. The usage graph showed a drastic increase starting late the day before. Unless many more users were suddenly much more active, sync usage shouldn’t increase this dramatically. 

Logs showed the increase came from continuous sync traffic. Looking closer, all that traffic was from apps that had updated to the latest version. Something was causing those apps to keep syncing the same data over and over again. This was a distributed systems bug that could only be solved by understanding the complex interactions of the whole system.

Time was of the essence. Each hour the CPU usage remained high was costing us. Without shutting off sync for everyone, fixing it would likely require pushing out an app update that could take time to roll outWe would soon learn this had far more to do with time than we guessed.

Having a bad time

The good news about noisy bugs like this is that they’re usually fairly easy to nail down. You have a continuous stream of reproductions, and making an effective fix leads to instant feedback. Often, looking at your recent changes leads to a forehead slap and a quick fix. This was not one of those times.

The obvious culprit was an infrastructure migration completed the day before. We moved our core PostgreSQL database from Supabase to Google Cloud. While this sounds as simple as updating a few database connection strings, it also involved switching from Supabase’s database library to another, generic PostgreSQL query builder. All of our unit tests continued to pass after the migration, as did end-to-end testing. We were confident making the switch, and thought everything looked fine after we made it.

Digging into those high-traffic logs revealed that clients were repeatedly refetching the same data. They’d ask, “What’s new since this timestamp?” The server would provide items updated after that timestamp. The client would then immediately ask for the same thing again.

Clients were supposed to update their last-synced timestamp. None of that code had changed. Why were they suddenly forgetting?

Looking at the SQLite app DB and API traffic, they weren’t forgetting. For each request, the latest timestamp returned by the API matched the latest timestamp in the app DB. But something stood out clearly. All the values looked like 1772366482123000 and 1772366467321000. There was a distinct lack of randomness in the final three digits.

PostgreSQL timestamps have microsecond precision, which is also supported in Dart and SQLite. But something was clearly truncating our timestamps to millisecond precision.

For a good time, call our API

JavaScript! (Excuse the bad language.) The new APIs foolishly created JavaScript Date objects from PostgreSQL timestamps before sending them as ISO-8601 strings to the client. (For context: ISO-8601 is the international standard for representing dates and times, designed to eliminate ambiguity across different regions.) There’s a reason the entire date system in JavaScript is being replaced. Date objects only store millisecond precision. Time was lost.

You might wonder how a few silly microseconds could have such an outsized impact. Gaining the same amount might never have been discovered, as it would have created a tiny race condition in which new items landing in that impossibly small window might be missed in the next sync. But losing them? It means that every time the client asked for items newer than its timestamp, the API happily returned items that were a few microseconds newer than the timestamp it gave the client.

The happy ending here is that the fix could be made in the API rather than requiring all clients to update. We simply had to return timestamps with microsecond precision. Luckily, we were using ISO-8601 strings rather than integer milliseconds, so it was simply a matter of returning values like “2026-03-01T12:00:00.123456Z”.

It’s about time

While this resolved this particular incident, any seasoned sync system expert has been chuckling the whole time. There was a more fundamental issue that was bound to bite us no matter how precise our timestamps were.

The presentation of the bug in this case couldn’t have been more different. It wasn’t noticed for months. It happened very infrequently. We couldn’t spot it or reproduce it. But occasionally, we were sure something was missing from a sync. 

Sure enough, we found a case where the app DB was missing an item that should have been included according to its sync watermark. Looking at the missing item in the PostgreSQL DB, we saw that its timestamp was very close to that of the last synced item. In this case, it was just a little less. Why wasn’t it included?

In PostgreSQL, now() represents the current time. More precisely, it’s the time at the start of a transaction. That means an earlier now can be committed after a later now. For example, imagine Mandy adds a note for Alex while Alex’s calendar is syncing. Mandy’s note might arrive after a much larger calendar-related transaction has started. Because it’s small and isolated, Mandy’s note commits before the other transaction finishes. Mandy’s note gets a later timestamp but is available sooner.

If a client responds to a sync broadcast quickly enough, it fetches Mandy’s note and updates its sync watermark to the note’s timestamp. Then, when the calendar transaction completes, the client will sync again, requesting everything since Mandy’s note, which doesn’t include the calendar items because they have earlier timestamps.

The solution is to use strictly increasing sequences rather than timestamps. This is the same system used for primary keys in PostgreSQL, so it is very fast and reliable.

Timely lessons

For anyone building sync services, a clear takeaway is to avoid timestamps entirely for pagination, ordering, and filters. Sequence numbers neatly sidestep all the issues highlighted above and are readily available in modern DBs.

Of course, the technicalities of incidents rarely repeat. Zooming out, there are lessons we learned that apply to whole classes of issues in this critical system.

This could be a poster child for the value of properly tuned monitoring and alerting. Catching an inter-service interaction like this would be difficult even with integration testing, as the user-visible behaviour was correct. The volume of activity was the issue. Since adding this monitoring, it has flagged issues with slow queries and retry swarms before they were noticed by users or our accountant.

I’m happy to share that we rarely think about the sync service anymore. As with all good infrastructure, we simply assume that data in one place will become available everywhere, giving us time to focus on bigger things.