Three slug-collision strategies I evaluated for a HuggingFace model directory ETL

My AI model directory runs an ETL that fetches the top models from HuggingFace Hub and stores them in a Turso libSQL database. Each model gets a slug — a normalized, URL-safe version of its ID — that becomes the page URL at aiappdex.com/models/<slug>/.

The first time I ran the ETL on 500+ models, it threw:

SQLITE_CONSTRAINT_UNIQUE: UNIQUE constraint failed: models.slug

The cause: HuggingFace model IDs like meta-llama/Llama-3.1-8B and meta-llama/Llama-3.1-8b (capitalization only) normalize to the same slug llama-3-1-8b after lowercasing and punctuation collapse. I had a few hundred more pairs like that. I needed a collision strategy that kept URLs stable across ETL runs.

Here are the three strategies I considered.

Strategy 1: Last writer wins

The simplest approach: if a slug already exists, overwrite it. Use ON CONFLICT(slug) DO UPDATE SET id = excluded.id and let the ETL replace the row.

I discarded this immediately. The problem isn't just that one model disappears — it's that the disappearing model's URL now points to a different model's content. A user who bookmarked aiappdex.com/models/llama-3-1-8b/ expecting Llama-3.1-8B would get Llama-3.1-8b after the next ETL run, or the other way around depending on sort order. For a directory site where the goal is stable, linkable model pages, silent redirects of this kind are worse than a 404.

There's also a harder problem: the pairwise comparison pages reference model slugs to build slug_a--vs--slug_b URLs. A model that silently takes over another model's slug corrupts the comparison pages without any error in the ETL log.

Strategy 2: Sequential suffix counter

The second approach: detect the collision at insert time, then retry with slug-2, slug-3, and so on until the insert succeeds.

This handles the collision correctly — both models get unique slugs, and neither overwrites the other. The problem is stability. If Llama-3.1-8B inserts first and claims llama-3-1-8b, then Llama-3.1-8b gets llama-3-1-8b-2. Fine. But if the ETL table gets rebuilt from scratch (during a migration or a data refresh), the insertion order might flip: now Llama-3.1-8b claims llama-3-1-8b and Llama-3.1-8B gets llama-3-1-8b-2. Every inbound link to the second model's page is now pointing at the wrong model.

Sequential counters are insertion-order-dependent, and ETL order is not guaranteed to be stable across runs.

Strategy 3: SHA-1 hash suffix (what I use)

The third approach is what the ETL now uses: when the base slug collides, compute a 6-character SHA-1 hash of the original model ID and append it.

export function collisionSlug(id: string): string {
  const hash = createHash("sha1").update(id).digest("hex").slice(0, 6);
  return `${slugify(id).slice(0, 93)}-${hash}`;
}

The hash is computed from id — the raw HuggingFace model ID, not the slug. Two models with different IDs that happen to normalize to the same slug will produce different hashes, so they get different collision slugs. And because the hash depends only on the ID (deterministic), the same model gets the same collision slug every time the ETL runs, regardless of insertion order.

For meta-llama/Llama-3.1-8b the hash is stable at a3f2c1 (or whatever SHA-1 produces for that specific string). That slug doesn't change between ETL runs or database resets.

The 93-character truncation on the base slug keeps the full slug + "-" + hash within 100 characters, which is the URL length I want to stay under for readability.

Detecting the collision

The ETL catches the collision at insert time using libSQL's unique constraint error. One catch: the error shape changed between @libsql/client 0.14 and 0.17:

function isUniqueViolation(err: unknown): boolean {
  if (typeof err !== "object" || err === null) return false;
  const e = err as { code?: string; extendedCode?: string; rawCode?: number };
  return (
    e.code === "SQLITE_CONSTRAINT_UNIQUE" ||
    e.extendedCode === "SQLITE_CONSTRAINT_UNIQUE" ||
    e.rawCode === 2067
  );
}

Version 0.14 sets code = "SQLITE_CONSTRAINT_UNIQUE". Version 0.17 moves that to extendedCode and sets code = "SQLITE_CONSTRAINT". The raw SQLite extended result code 2067 is stable across both versions and is what I now rely on as the primary check.

The ETL flow:

try {
  await upsert(baseSlug);
} catch (err) {
  if (isUniqueViolation(err)) {
    await upsert(collisionSlug(id));
  } else {
    throw err;
  }
}

Non-collision errors re-throw immediately. Collision errors get one retry with the hash-suffixed slug. If that also fails (extremely unlikely, since the SHA-1 hash space for 6 hex chars is 16 million values), the error propagates and the ETL surfaces it.

What I haven't solved

The hash suffix isn't human-readable. A model page at aiappdex.com/models/llama-3-1-8b-a3f2c1/ is less discoverable than aiappdex.com/models/llama-3-1-8b/. I don't know yet whether that matters for the directory's search ranking. The collision rate is low — maybe 30 models out of 1,000 — so the majority of pages have clean slugs. The question I can't answer until month 4 or 5 is whether Google treats the hash-suffixed URLs differently than the clean ones.

The two approaches I'm watching: a custom disambiguation based on the model author (so llama-3-1-8b-meta-llama vs llama-3-1-8b-community) instead of a hash, and a pre-insert normalization pass that resolves collisions before they hit the database. Both are more work. The SHA-1 approach ships correctly; the others ship better. I'll reconsider when I have data.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.