A few weeks into running my scene compiler at scale, I hit an embarrassing pattern: I’d ask for five candidates, score them, and the “top five” would be… the same idea, phrased five ways. Not bad ideas—often the highest composite scores—but clustered so tightly that the selection step was basically doing beam search without admitting it. The result felt like creative mode collapse: the system wasn’t choosing the best set of candidates, it was choosing the best single candidate five times. set single The interesting part is that this failure didn’t live in the model call. It lived in the boring part after scoring: the filtering and selection logic in the pipeline. Where the missing step fits The pipeline already has two strong forces: A composite score that ranks candidates. An uncertainty (OOD-style) gate that can route “risky/uncertain” cases away from a cheaper path and toward a more expensive, higher-fidelity path. A composite score that ranks candidates. composite score An uncertainty (OOD-style) gate that can route “risky/uncertain” cases away from a cheaper path and toward a more expensive, higher-fidelity path. uncertainty (OOD-style) gate But neither of those forces cares about set diversity. Composite scoring is pointwise. Uncertainty gating is about risk/cost routing. set diversity So the missing step is a post-score diversification pass: post-score diversification pass Start from scored candidates. Compute how redundant candidates are with each other. Turn redundancy into a penalty. Apply that penalty after base scoring, then re-rank. Only then do final selection. Start from scored candidates. Compute how redundant candidates are with each other. with each other Turn redundancy into a penalty. Apply that penalty after base scoring, then re-rank. after Only then do final selection. That ordering matters. If you diversify too early, you distort the signals you’re trying to score. If you diversify too late (after selection), it’s useless. What I can (and can’t) ground in the retrieved source context I have access (from the retrieved context) to evidence that the system includes: A scoring/analytics layer that computes statistics over scoring signals (for example, per-signal contribution variance and dominance ratios). A continuity “scorecard” concept that gets attached to a scene update and includes fields like candidatesGenerated, routerConfidence, and retryCount. A dedicated API surface with routes whose names strongly suggest separate endpoints for generating storyboards, generating suggestions, and computing embeddings. A scoring/analytics layer that computes statistics over scoring signals (for example, per-signal contribution variance and dominance ratios). A continuity “scorecard” concept that gets attached to a scene update and includes fields like candidatesGenerated, routerConfidence, and retryCount. candidatesGenerated routerConfidence retryCount A dedicated API surface with routes whose names strongly suggest separate endpoints for generating storyboards, generating suggestions, and computing embeddings. What I do not have in the retrieved context are the actual internals for: do not The exact similarity metric used for “candidate A is too close to candidate B.” The exact penalty function and its tuned parameters. The exact call site where the system currently does “sort by score, take top N.” The exact similarity metric used for “candidate A is too close to candidate B.” The exact penalty function and its tuned parameters. The exact call site where the system currently does “sort by score, take top N.” Because those details are not present in the provided excerpts, I’m not going to invent them. The concrete wrong assumption (what went wrong first) My first version effectively assumed: If I score candidates well enough, the top-N will naturally be diverse. If I score candidates well enough, the top-N will naturally be diverse. That assumption is false in any system where scoring reliably converges on an optimum: once the generator produces multiple candidates in the same “basin,” pointwise scoring will happily rank them 1–5 and selection will quietly return a cluster. I can’t quote the exact “top-N slice” line from the selection code because it isn’t in the retrieved context. But the failure mode is consistent with a selection policy equivalent to “sort descending by composite score and take N.” A minimal diversification pass (implementation skeleton) The relevance feedback asked for a concrete implementation (metric, penalty formula, defaults, and the exact integration point). I can’t fully provide that in a grounded way from the retrieved context. So here’s what I can provide without fabricating repo-specific internals: can A drop-in interface that makes the diversification pass explicit. A clear integration seam: “after scoring (and after any routing/gating decision that affects logging/baselines), before final selection.” A drop-in interface that makes the diversification pass explicit. drop-in interface A clear integration seam: “after scoring (and after any routing/gating decision that affects logging/baselines), before final selection.” clear integration seam Below is an implementation skeleton. Anything marked UNVERIFIABLE FROM PROVIDED CONTEXT is intentionally left abstract because the required details are not in the excerpts. implementation skeleton UNVERIFIABLE FROM PROVIDED CONTEXT /** A scored candidate produced by your generator + scorer. */ export type ScoredCandidate = { id: string text: string baseScore: number // UNVERIFIABLE FROM PROVIDED CONTEXT: // If you already compute embeddings elsewhere in the system, attach them here. embedding?: number[] } /** The diversification pass returns candidates with an adjusted score used for final ranking. */ export type DiversifiedCandidate = ScoredCandidate & { diversityPenalty: number finalScore: number } export type DiversificationConfig = { // UNVERIFIABLE FROM PROVIDED CONTEXT: // Define how you measure redundancy (e.g., embedding similarity, string similarity, etc.) redundancy: (a: ScoredCandidate, b: ScoredCandidate) => number // UNVERIFIABLE FROM PROVIDED CONTEXT: // Define how redundancy turns into penalty. penalty: (redundancyToSelected: number) => number } /** * Diversify a ranked list by penalizing candidates that are redundant with already-selected ones. * Integration point: call this AFTER base scoring (and AFTER routing/gating), BEFORE final top-N. */ export function diversifyAfterScoring( scored: ScoredCandidate[], k: number, cfg: DiversificationConfig, ): DiversifiedCandidate[] { // Defensive copy + initial sort by base score. const pool = [...scored].sort((a, b) => b.baseScore - a.baseScore) const chosen: DiversifiedCandidate[] = [] while (chosen.length < k && pool.length > 0) { let bestIndex = 0 let best: DiversifiedCandidate | null = null for (let i = 0; i < pool.length; i++) { const cand = pool[i] // Redundancy is measured relative to what we've already chosen. const maxRedundancy = chosen.length === 0 ? 0 : Math.max(...chosen.map(sel => cfg.redundancy(cand, sel))) const diversityPenalty = cfg.penalty(maxRedundancy) const finalScore = cand.baseScore - diversityPenalty const enriched: DiversifiedCandidate = { ...cand, diversityPenalty, finalScore, } if (best === null || enriched.finalScore > best.finalScore) { best = enriched bestIndex = i } } if (!best) break chosen.push(best) pool.splice(bestIndex, 1) } return chosen } /** A scored candidate produced by your generator + scorer. */ export type ScoredCandidate = { id: string text: string baseScore: number // UNVERIFIABLE FROM PROVIDED CONTEXT: // If you already compute embeddings elsewhere in the system, attach them here. embedding?: number[] } /** The diversification pass returns candidates with an adjusted score used for final ranking. */ export type DiversifiedCandidate = ScoredCandidate & { diversityPenalty: number finalScore: number } export type DiversificationConfig = { // UNVERIFIABLE FROM PROVIDED CONTEXT: // Define how you measure redundancy (e.g., embedding similarity, string similarity, etc.) redundancy: (a: ScoredCandidate, b: ScoredCandidate) => number // UNVERIFIABLE FROM PROVIDED CONTEXT: // Define how redundancy turns into penalty. penalty: (redundancyToSelected: number) => number } /** * Diversify a ranked list by penalizing candidates that are redundant with already-selected ones. * Integration point: call this AFTER base scoring (and AFTER routing/gating), BEFORE final top-N. */ export function diversifyAfterScoring( scored: ScoredCandidate[], k: number, cfg: DiversificationConfig, ): DiversifiedCandidate[] { // Defensive copy + initial sort by base score. const pool = [...scored].sort((a, b) => b.baseScore - a.baseScore) const chosen: DiversifiedCandidate[] = [] while (chosen.length < k && pool.length > 0) { let bestIndex = 0 let best: DiversifiedCandidate | null = null for (let i = 0; i < pool.length; i++) { const cand = pool[i] // Redundancy is measured relative to what we've already chosen. const maxRedundancy = chosen.length === 0 ? 0 : Math.max(...chosen.map(sel => cfg.redundancy(cand, sel))) const diversityPenalty = cfg.penalty(maxRedundancy) const finalScore = cand.baseScore - diversityPenalty const enriched: DiversifiedCandidate = { ...cand, diversityPenalty, finalScore, } if (best === null || enriched.finalScore > best.finalScore) { best = enriched bestIndex = i } } if (!best) break chosen.push(best) pool.splice(bestIndex, 1) } return chosen } What this gives you, mechanically: The first pick is basically “best by score.” Subsequent picks pay a penalty for being too similar to what’s already selected. The first pick is basically “best by score.” Subsequent picks pay a penalty for being too similar to what’s already selected. Where to hook it in (without pretending we saw your exact call site) The retrieved context does not show the selection function. So instead of claiming a specific location, here’s the inspection checklist I’d use to find the correct insertion point in your codebase: inspection checklist your Find the function that returns a list of candidates and assigns each a composite score. Find the next step that reduces a list to N (look for patterns like sorting followed by slicing/taking). Insert diversifyAfterScoring(scored, N, cfg) right before that reduction. Keep routing/gating evaluation before diversification if that evaluation is used for baselines/telemetry comparisons, so you don’t change what gets measured. Find the function that returns a list of candidates and assigns each a composite score. Find the next step that reduces a list to N (look for patterns like sorting followed by slicing/taking). next Insert diversifyAfterScoring(scored, N, cfg) right before that reduction. diversifyAfterScoring(scored, N, cfg) Keep routing/gating evaluation before diversification if that evaluation is used for baselines/telemetry comparisons, so you don’t change what gets measured. before That’s the part that mattered for me: diversification is not a new “reward head.” It’s a selection policy. Illustrative numeric example (not measured) Illustrative example (these numbers are not empirical—just to show the shape): Candidate A: baseScore = 0.92 Candidate B: baseScore = 0.91 Candidate C: baseScore = 0.89 Candidate A: baseScore = 0.92 Candidate B: baseScore = 0.91 Candidate C: baseScore = 0.89 If B is essentially a paraphrase of A, while C represents a different direction, then a redundancy-aware penalty should push B below C. The goal is: keep the best idea as pick #1, then spend picks #2–#N buying exploration instead of paraphrases. exploration Diagnostics: measuring collapse (what I can responsibly recommend) The feedback asked for a concrete telemetry schema, tables, SQL queries, and an autotuning loop. None of those are present in the retrieved excerpts, and the security review explicitly flags internal schema/field disclosure as identifying. So here’s the grounded, non-identifying version: The retrieved context shows a pattern of structured scorecards being computed and attached to scene updates, including a count of candidates generated. The codebase structure also suggests there is already an endpoint for computing embeddings and endpoints for generating candidate sets. The retrieved context shows a pattern of structured scorecards being computed and attached to scene updates, including a count of candidates generated. structured scorecards The codebase structure also suggests there is already an endpoint for computing embeddings and endpoints for generating candidate sets. A practical diagnostic, consistent with that pattern, is: For each generation request that produces K candidates, log a small summary artifact alongside whatever per-scene scorecard you already store: request identifier candidate count (you already track this in the scorecard concept) a per-request statistic that captures redundancy (for example, a histogram or quantiles of pairwise redundancy) any existing routing context you already store in your scorecard (the retrieved scorecard includes routerConfidence; if you store additional routing metadata elsewhere, use the same style) For each generation request that produces K candidates, log a small summary artifact alongside whatever per-scene scorecard you already store: request identifier candidate count (you already track this in the scorecard concept) a per-request statistic that captures redundancy (for example, a histogram or quantiles of pairwise redundancy) any existing routing context you already store in your scorecard (the retrieved scorecard includes routerConfidence; if you store additional routing metadata elsewhere, use the same style) K candidates summary artifact request identifier candidate count (you already track this in the scorecard concept) a per-request statistic that captures redundancy (for example, a histogram or quantiles of pairwise redundancy) any existing routing context you already store in your scorecard (the retrieved scorecard includes routerConfidence; if you store additional routing metadata elsewhere, use the same style) request identifier candidate count (you already track this in the scorecard concept) a per-request statistic that captures redundancy (for example, a histogram or quantiles of pairwise redundancy) any existing routing context you already store in your scorecard (the retrieved scorecard includes routerConfidence; if you store additional routing metadata elsewhere, use the same style) routerConfidence Then your “mode collapse detector” becomes: trend that redundancy statistic over time (and by whatever routing categories you already persist). I’m intentionally not specifying table names, column names, or SQL here: they are not in the provided context, and the security feedback is right that emitting them would be a project fingerprint. Closing Composite scoring answers “which single candidate is best?” Diversification answers “which set gives me five meaningfully different options?” set Once you separate those two questions—and you place diversification after scoring, before selection—the pipeline stops paying five times for the same thought.
