Meet the platform
A US community impact platform publishing short stories about acts of kindness, city by city: 989 people, 1,078 stories, hundreds of embedded videos from Instagram, Facebook, TikTok and YouTube. The site had grown fast, and its quality control had not kept up. The owner kept finding broken stories himself, one by one, which is the most expensive way to run quality control there is.
The challenge
A video embed can pass every cheap check and still be broken for a visitor. The link returns 200 while the video was deleted at the source. The post scrapes fine while its embed endpoint serves a removal notice. The recorded dimensions describe the cover photo, not the video stream, so the player renders in the wrong shape. A photo post wears a video embed that throws the visitor off the site on the first click. None of this shows up in an HTTP check. It only shows up when you gather real evidence per story and refuse to ship on anything less.
The solution
A gate with seven evidence feeds
We built a technical gate that scores every story from evidence, not impressions: recorded browser play test verdicts, stream dimensions pulled from the actual video files, YouTube embed permission and country availability fields, media type and carousel first slide checks, a one post one story duplicate check, and a failed once blocklist. Each story gets one verdict: PASS, CONDITIONAL for never play tested and may not ship, or FAIL.
Fix every failing class, then run it again
Round one came back 683 PASS, 285 CONDITIONAL, 110 FAIL. The fixes went class by class. Aspect ratios were rebuilt from the real streams on 110 Instagram stories, 15 of which were recorded as portrait and are actually landscape, and all 47 Facebook embeds got their true stream ratios. Twenty seven photo posts and photo first carousels were stripped to text and added to the blocklist. Twenty eight YouTube embeds whose owners had disabled embedding were stripped. Three duplicate posts shared by two stories each were deduped to one story each.
Round two came back 710 PASS, 289 CONDITIONAL, 79 FAIL, and every one of the 79 was a single class: text stories with no cover photo anywhere in their sources. A cover wave pulled candidates from each story own source article and eyeballed every one: 34 stories got real covers, and 45 were deliberately left text only because their sources held only logos, ads, game action shots and unrelated images. A wrong cover is worse than no cover.
Round three: 743 PASS, 289 CONDITIONAL, 46 FAIL, of which 45 were those same honest text stories and one was a profile photo that existed in the build but had never reached the server, uploaded and verified the same pass.
The incident, left in
Midway through the fix wave, a patch to the page script dropped a closing parenthesis and the story viewer went dead site wide for about 15 minutes. It was caught by our own live verification, fixed, and the live script syntax checked. The standing rule that came out of it: after any edit to that script, extract the built block, syntax check it, deploy, then syntax check the live block too.
The results
- 1,078 stories audited, 989 people, 12 cities
- 110 Instagram aspect ratios corrected from real stream dimensions
- 27 photo posts stripped and blocklisted, 28 unembeddable YouTube videos stripped
- 34 covers backfilled from the stories own sources, 45 left text only on purpose
- 0 video defect classes remaining at the end of the loop
- $1.98 total cost for the entire audit and fix loop, full run time about 13 minutes
Honest note: 289 stories sit at CONDITIONAL, never play tested. They are a watchlist, not a defect list, and the gate treats them as unproven rather than passed. That distinction is the whole point of the gate: it says what it knows and labels what it does not.
The return: the owner stopped being the quality control department. A full site audit that used to mean a human clicking through a thousand stories now runs in about 13 minutes for under two dollars, and every story on the site carries a verdict earned from evidence.
FAQ
Why is a link returning 200 not proof a video works?
Because the page can be alive while the video is deleted, region blocked, embedding disabled by its owner, or served in the wrong shape. Only the embed endpoint, the stream itself and a real play test can settle it.
What does CONDITIONAL mean?
The story has never been play tested in a browser, so the gate has no direct evidence it plays. Conditional stories may not ship as new work. They are the unproven watchlist, kept separate from proven failures.
Why leave 45 stories without covers?
Their own source articles held no usable photo: logos, ads, stock graphics and unrelated images. Publishing a wrong face on a real person story is a worse failure than a clean text card.
What does the gate cost to run?
The full 1,078 story loop cost $1.98 in total model usage and runs in about 13 minutes, after the evidence phase was threaded and the stream probes cached.
