For thirty days my content pipeline reported a perfect success rate while publishing absolutely nothing.
Four YouTube channels run off one server here. Every scheduled run did what it was told: pulled trends, researched, wrote a script, generated a voiceover, rendered 1080p video, called the upload endpoint, wrote status=done to SQLite, and posted a green tick to Discord. Nine times a week, for a month. Meanwhile the channels sat frozen.
Nothing alerted, because nothing was checking the outcome. Everything was checking the process.
The bug is four lines long
The upload step looked roughly like this:
try:
response = youtube.videos().insert(
part="snippet,status",
body=body,
media_body=MediaFileUpload(video_path),
).execute()
log.info("uploaded %s", title)
return True
except Exception as e:
log.error("upload failed: %s", e)
return True # <- the whole outage, right here
That return True was not a typo I can wave away. It was a deliberate decision I made months earlier, for a reason that seemed sound at the time: a single channel failing should not abort the sequential run for the other three. So I swallowed the exception, logged it, and let the runner continue.
What I never did was distinguish "this channel is done" from "this channel succeeded." Both collapsed into the same boolean, and that boolean was what got written to the database and what the notifier read to decide the message colour.
The underlying failure was mundane. The OAuth refresh token expired. Every subsequent insert() raised invalid_grant. Thirty consecutive runs, thirty identical stack traces in the log, thirty green ticks in Discord.
How I actually found it
Not from a dashboard. I opened one of the channels on my phone to check a thumbnail and the upload list looked short. I thought I had mis-remembered the schedule. Then I checked the second channel, and it was short too.
One grep settled it:
$ grep -c "invalid_grant" logs/pipeline.log
30
Thirty. One per run, going back exactly a month, sitting in a log file I had stopped reading because the notifications told me everything was fine. That is the part that stung: the data was there the whole time. I had built a system that generated evidence of its own failure and then talked over it.
Process checks versus outcome checks
Here is the distinction I did not have language for before this:
- A process check asks did my code finish?
- An outcome check asks did the thing I wanted to happen actually happen?
Almost all monitoring people ship is process checks, because they are trivial to write — you already have the exit code, the try/except, the "job completed" line. They tell you your program ran. They tell you nothing about whether the world changed.
An outcome check has to leave the process and go look. It costs an API call. It is annoying to write. It is the only kind that would have caught this.
What I built instead
Three jobs, all of which query reality rather than trusting the pipeline's own opinion of itself.
**check_uploads.py**, daily at 11:00. Reads the video IDs the pipeline claims it published in the last seven days, then asks the YouTube API whether each one is actually there and public:
recent = db.query(
"SELECT video_id, channel, title FROM runs "
"WHERE status='done' AND published_at > date('now','-7 days')"
)
ids = [r["video_id"] for r in recent]
found = youtube.videos().list(part="status", id=",".join(ids)).execute()
live = {item["id"] for item in found["items"]}
missing = [r for r in recent if r["video_id"] not in live]
if missing:
alert(f"{len(missing)} of {len(recent)} uploads are not live", missing)
If the pipeline says it published seven videos this week and YouTube can only find five, I hear about it that morning. A row claiming success with a null or unresolvable video_id is itself the alarm.
**check_tokens.py**, daily at 10:00. Reads the credential expiry for every channel and warns two days out, so the token gets refreshed on a Tuesday afternoon instead of failing silently on a Saturday night.
**disk_guard.sh**, every five minutes. Each render writes roughly 400 MB of intermediate files. This purges anything older than six hours, and alerts if the disk crosses a threshold — the failure that would otherwise take out all four channels at once.
And the return True became this:
except Exception as e:
log.error("upload failed for %s: %s", channel, e)
db.mark(run_id, status="failed", error=str(e))
notify_failure(channel, e)
return False # this channel failed; the runner continues regardless
Per-channel isolation was a good instinct. Implementing it by lying about the result was not. The runner still continues to the next channel — it just does so knowing that one of them failed.
The alert chain
The other half of the fix: a failure now has to reach a human, and one dead notification path cannot swallow it. Discord first, Telegram if Discord fails, email if Telegram fails, and the log as the final fallback. Fanning out matters less than the ordering — each fallback only fires when the one before it errors, so a working day produces one message, not four.
What this changed about how I build
Every automation I have shipped since carries at least one outcome check, and I now write it before I write the happy path. For a report robot it is "did an email with a non-zero attachment actually arrive." For a document agent it is "did the row count in the destination go up." For a sync it is "do the two sides agree on a checksum."
The numbers on the front page of this site — 169 scheduled runs since 13 May, 162 published, 95.9% before retries — come from that same database. They are only worth showing because a separate job goes and verifies them against YouTube every morning. Before this outage I would have quoted you a 100% success rate with total confidence, and I would have been wrong for a month.
The lesson is not "add more monitoring." I had monitoring. The lesson is that a system reporting on itself will always report that it is fine, and the only cure is a second system that goes and looks.