The most useful part of a postmortem is sometimes the correction to the first explanation. UltraFit360’s notification audit initially looked like a simple double-send story. On closer inspection, some repeated type labels represented different intended messages. Other counts did reveal over-firing, and a separate product problem remained: too many individually valid reminders could still feel like spam. That separation changed the investigation. This article explains the reasoning and documented partial fixes with fictional examples, without publishing user records, notification text or operational endpoints.
First establish what one count means
A group labelled “meal reminder” might contain lunch and dinner subtypes. Two rows under that label do not automatically mean the same message was delivered twice. Before diagnosing duplication, establish the unit: queued event, subtype, attempt, provider acceptance, device delivery or opened notification.
The corrected audit matters because it went back to the underlying records and separated expected subtype activity from actual over-firing. I would preserve that correction in the public story rather than use the initial alarming count as a headline statistic. A misleading denominator can send a team toward the wrong code path for days.
A read-then-write throttle can race
Imagine two fictional workers checking whether today’s reminder was sent. Both read “not yet,” both queue the event and both update the count afterwards. Each worker followed the local rule. The combined system violated it because the permission to send was not claimed atomically.
A process-local lock may help within one process while doing nothing across separate workers or instances. The protection must live at a boundary shared by the contenders. A database uniqueness or atomic-update pattern can turn “check then later record” into a single decision. The exact design should fit the event identity and failure semantics.
INSERT INTO delivery_claims (event_key, calendar_day)
VALUES (:event_key, :day)
ON CONFLICT (event_key, calendar_day) DO NOTHING
RETURNING event_key;
-- Queue only if a claim was returned.
-- External delivery still needs its own failure policy.Two workers can both pass a read-before-write check
This ordering explains why a non-atomic throttle can allow duplicate work.
Worker A
- 1 · Read eligibility
No existing claim is visible.
- 3 · Decide to send
Its earlier read still looked clear.
- 5 · Write the marker
The marker arrives after the decision.
Worker B
- 2 · Read eligibility
A has not written its marker yet.
- 4 · Decide to send
B also considers the event eligible.
- 6 · Write the marker
Both decisions have already passed.
Read the numbered steps across both workers. A later marker cannot retroactively make those decisions exclusive.
Claiming once is not the same as delivering once
An atomic claim prevents some duplicate queuing, but external delivery has its own uncertainty. A worker can send a message and crash before marking it complete. Retrying may then repeat the external side effect. A provider response or receipt can establish a different stage from actual user-visible delivery.
This is why I avoid a casual “exactly once” promise. Define the guarantees at each stage, use stable event identities where supported and reconcile ambiguous outcomes. The article’s lesson is not that one SQL pattern solves the entire delivery pipeline. It is that concurrency and external effects require separate reasoning.
An atomic claim protects the decision boundary
The database resolves competing claims; external delivery remains a separate failure boundary.
Claim outcome
- Workers attempt the same unique claim
The event identity and calendar scope must match the intended policy.
- Only a returned claim may enqueue
A conflicting attempt does not create another queued send.
- Worker calls the delivery provider
Track attempts and decide what recovery means after an ambiguous response.
One successful claim is not proof of exactly-once delivery. A crash after provider acceptance still needs an explicit recovery policy.
Local and server reminders can overlap
The audit also identified a server workout reminder that overlapped an on-device local reminder. Those messages came through different channels, so examining only the server queue could miss why a person saw two prompts about the same task.
A useful inventory lists which component owns each reminder. The mobile device may schedule a personal local alert, while the backend sends a contextual message. Both can be valid features, but their overlap should be intentional. Removing an unnecessary duplicate channel is often simpler than adding another throttle around two systems that should not both send the same nudge.
Per-type limits do not create a global product limit
A meal reminder, step reminder and weekly check-in can each respect their own rule while adding up to a tiring day. That is a product coordination problem, not necessarily a concurrency defect. A person experiences the total interruption load, not the neat organisation of the job registry.
The historical audit explicitly leaves some cross-type ceiling and locking work unfinished. The later engagement review also distinguishes active per-subtype protection from the absence of a fully enforced global cap. I would not claim that the team solved notification fatigue merely because one race was addressed. The remaining product policy deserves its own measurement and decision.
Use the user’s calendar deliberately
A “per day” rule needs a timezone definition. Counting a UTC calendar day while scheduling messages around local dinner can produce surprising boundaries. Daylight-saving changes and travel can add more edge cases. The right policy depends on the product, but it should not be an accidental consequence of the server’s timezone.
For a test, use fictional accounts near a day boundary and run overlapping scheduler windows. Check eligibility, claim identity and the resulting queued events. Do not infer delivery from the presence of a row. A useful test names the stage it has established and leaves the rest to integration or device evidence.
A reminder needs an outcome beyond being sent
Delivery volume is easy to count. Helpfulness is harder. Did the person complete the intended action? Did they disable notifications? Was the reminder relevant to an actual missing action, or did onboarding-generated data trick the eligibility rule?
I would review message targeting alongside the first meaningful task in the product. If returning users cannot find the logging action, another reminder may simply send them back into the same confusion. The frontend and lifecycle strategy should be considered together. A quieter, clearer flow may be more useful than a better-timed interruption.
Preserve the partial nature of a fix
The recorded implementation includes atomic-claim changes and removal of overlapping reminders. That is a legitimate engineering improvement. It does not prove the remaining global policy was deployed, that all users received fewer pushes or that retention improved as a result.
A good follow-up records which invariants now hold, which remain open and how they will be observed. Use a synthetic event ledger in public explanations rather than raw customer records. Correcting the first diagnosis makes the story more credible: the team learned something important by checking the actual meaning of its data.
Questions worth asking
Can I solve duplicates with a boolean flag? Only if the transition is protected against concurrent readers and its failure behaviour is defined. A flag alone is not an atomic operation.
Should every reminder share one daily cap? That is a product decision. Transactional alerts and optional nudges may need different policies, but exemptions should be explicit rather than accidental.
What would I investigate first? Trace one reported experience through subtype, scheduling, queue and device channel. A person saying “too many notifications” may be describing a duplicate, a channel overlap or a valid but exhausting product policy.
Sources
Technical references checked .


