BUILD NOTES / FAILURE MECHANISMS

The useful part
of getting it wrong.

Three reminders that a green indicator is only as good as the thing it actually checks.

Published
Last updated ·

Illustration of a leafy workshop with two compact computers and a monitor on a wooden bench.
Workshop illustration.

01 / PERMISSIONS

Writing a file is not owning its timestamps.

Obsidian Sync failed with EPERM while trying to preserve a file’s modification time with utimes. Group-write access was not enough.

On Linux, setting an arbitrary timestamp requires ownership or appropriate privilege. Write permission alone can allow setting timestamps to the current time, but does not grant the same authority to restore a specific saved time.

The check to carry forward: test both content writes and timestamp preservation under the exact account that runs the sync.

02 / BACKUP HEALTH

The heartbeat was not the backup.

During intermittent DNS trouble, a nightly backup failed while a separate liveness heartbeat still succeeded.

A successful request to a monitoring endpoint did not prove that the backup’s own dependencies were reachable, or that a backup had completed. The signal and the result were not equivalent.

The check to carry forward: tie the success signal to a verified job outcome, monitor the age of the last successful backup, and test restoration separately.

03 / ALERT TESTING

The alarm needs to stay armed.

During a manual browser test, an alert check ended up paused. I resumed it. While paused, it would not alert on a missed heartbeat.

The incident log records the state change, but does not retain the exact browser request. This is a lesson about checking the monitor’s state after testing, not a claim that an ordinary test ping always pauses a check.

The check to carry forward: verify the monitor is active after every test, then confirm that a controlled failure produces an alert.

Based on my build ledger’s August 2026 incident accounts, including the sync repair recorded on 5 August and the DNS disruption on 4–5 August. These are retrospective accounts, not newly reproduced tests. The date above is this note’s update date. Private system identifiers are omitted.

Mechanism reference: Linux manual: file timestamps.

Related project: Memory & learning. Keeping useful corrections so the next attempt is better.

Back to build notes ↑