Automate Content Deduplication for High-Volume AEO Pipelines

Standard approaches to deduplication fail inside automated AEO pipelines. Simple keyword or n-gram overlap scripts are too rigid. They cannot distinguish between necessary template boilerplate and genuinely overlapping content, leading to a high rate of false positives that require manual intervention and break the automation.

Consider an AEO system generating pages based on a template like "[Service] in [City]". The pages for "Roof Repair in Houston" and "Roof Repair in Dallas" will share 80% of their text due to the template structure. A naive check flags them as duplicates, even though they serve distinct user intents and need to coexist. This forces a choice: either make templates so generic they lose value or manually approve every page.

Third-party plagiarism APIs are not a viable solution at scale. They are designed to catch human plagiarism, not validate programmatic uniqueness, and their per-page API call pricing becomes cost-prohibitive when generating 100+ pages per day. These services are also external dependencies that introduce latency and an additional point of failure into a high-throughput publishing pipeline. The core problem is that these tools lack the context of your content strategy and cannot be tuned to your specific definition of a 'duplicate'.

We built a vector-based deduplication checker as a core component of our own four-stage AEO pipeline. The approach treats content uniqueness as a data problem, not a text-matching problem. It runs as the second check in our 8-point validation stage, right after a rendering safety check and before data accuracy verification.

The system uses the `pg_trgm` extension in a Supabase Postgres database, which creates a trigram-based vector for every piece of content. When a new page is generated, its content is vectorized. A single, indexed SQL query then compares this new vector against the vectors of all 10,000+ pages already in our database using pgvector. The query calculates the Jaccard similarity and returns the highest score in under 500 milliseconds.

If the similarity score is 0.72 or higher, the page fails validation and is sent back to Stage 2 for regeneration. The regeneration prompt is automatically appended with feedback, like 'Failed validation: Jaccard score 0.79, too similar to page_id 8192'. This automated loop, managed by a GitHub Action, allows our pipeline to safely generate and publish up to 200 unique pages per day with zero manual checks and no API fees.