The dedup rebuild

A few months into running the content pipeline, it started repeating itself. Not identically — that would have been easy. It published a video about mortgage rates, and eleven days later published a different video about mortgage rates, with different wording, a different structure, and the same three stock clips of a house behind it.

I had a deduplication check. It was passing. The problem was that I had written one check for what turned out to be two entirely different problems.

Why hashing does not work

The original dedup was the thing everybody writes first: normalise the topic string, hash it, keep a set of hashes, reject on collision.

key = hashlib.sha256(topic.lower().strip().encode()).hexdigest()
if key in used_topics:
    skip()

This catches exactly one case — the identical headline arriving twice — and that case is rare. What actually arrives is:

• "Fed holds rates steady for third straight meeting"
• "Federal Reserve leaves interest rates unchanged again"

Two different strings. Two different hashes. One story. The hash check waves both through, and a week later the channel has published the same segment twice with different nouns.

String similarity does not save you either. I tried token overlap and Levenshtein distance before accepting that the headlines share almost no vocabulary. The similarity I needed was semantic, and lexical tools cannot see it.

Problem one: topic dedup

The rebuild embeds the candidate topic and compares it against the embeddings of everything published recently, in vector space rather than string space.

def is_duplicate_topic(candidate, profile):
    vec = embed(candidate)
    recent = db.recent_topics(
        channel=profile["name"],
        limit=profile["dedup"]["topic_window"],   # 40
    )
    for row in recent:
        if cosine(vec, row["embedding"]) > profile["dedup"]["similarity_threshold"]:
            return True, row["title"]
    return False, None

Two parameters here mattered more than the algorithm.

The threshold. At 0.95 nothing was ever a duplicate. At 0.70 the channel could not cover the same broad subject twice in a quarter, which for a niche channel is fatal — a property channel is supposed to talk about interest rates repeatedly, just not about the same announcement. 0.82 was where it rejected re-reporting while still allowing recurring subjects.

The window. Comparing against all history is both slow and wrong. A topic from eight months ago is legitimately fresh again. Forty recent topics per channel, scoped per channel so one channel's coverage never blocks another's.

The candidate set is generated wide and filtered hard on purpose: a typical run pulls fourteen candidates from RSS and the model, and thirteen get rejected against roughly thirty-three recent topics. Discovery is cheap; publishing a repeat is expensive. Over-generating and throwing most of it away is the correct trade.

Problem two: visual dedup

This is where I had genuinely misunderstood the problem. Even after topic dedup worked, the videos still felt repetitive, because the b-roll was.

Stock footage selection is a search against a library, and searching "house exterior" returns the same top results every time. The pipeline was dutifully picking the best match for each content card, and the best match for a given query is stable. Different scripts, same query terms, same clips. Four videos in a row opened on the identical drone shot.

Topic dedup cannot help with this at all. The topics were legitimately distinct; the rendering of them was not. Different similarity space, different store, different window:

def pick_clip(query, profile, used_assets):
    results = stock.search(query, per_page=25)
    for clip in results:
        if clip["id"] in used_assets:
            continue
        used_assets.add(clip["id"])
        db.record_asset(profile["name"], clip["id"])
        return clip
    return results[0]      # exhausted: reuse rather than ship a gap

Asset IDs used in the last N videos per channel are excluded from selection. The fallback matters — if every candidate has been used recently, reusing a clip beats failing the render or shipping a black frame. A slightly repetitive video is a much smaller problem than no video.

Deeper down, per-content-card rendering varies the treatment as well as the asset: b-roll on some cards, Ken Burns on stills, animated stat slides where the script cites a number. Two videos on adjacent subjects no longer look like the same template with the words swapped.

The actual lesson

"Duplicate" is not one predicate. I had a function called is_duplicate() and assumed the noun was well-defined. It was hiding two questions with different answers:

They do not even fail in the same direction. Topic dedup should be strict — refusing a topic costs nothing, there are thirteen more. Visual dedup has to be lenient, because refusing every clip means no video at all.

Collapsing both into one check meant tuning a single threshold against two objectives that pull opposite ways, which is why no value of it ever worked. The rebuild was not a better algorithm. It was noticing there were two problems.