feat(curation): data quality pipeline — Phases 1-3

Add comprehensive data curation system to clean up the 197K skill
dataset and show only quality browse-ready skills to users.

Phase 1 — Database exploration:
- Explore scripts (explore.ts, explore.mjs, explore.sql) for analysis
- Discovered: 69% duplicates, 77% aggregator/fork noise

Phase 2 — Data cleanup and classification:
- Schema: 6 new curation columns + 4 indexes
- curate.mjs: 8-step pipeline (classify, dedup, fork detection, etc.)
- Result: 197K → 60K unique → 16K browse-ready skills
- Bug fix: securityStatus was computed but never stored during crawl

Phase 3 — UI browse-ready filters:
- browseReadyFilter applied to 17+ query functions
- Homepage stats show accurate browse-ready counts
- Stats API filtered (previously had no WHERE clause)
- Category counts recalculated (e.g. 45K → 3.1K)
- Featured skills exclude duplicates and aggregators

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
airano
2026-02-19 14:31:25 +03:30
parent 31f5df0900
commit caca09fbe7
12 changed files with 2615 additions and 28 deletions

View File

@@ -90,6 +90,7 @@ export async function indexSkill(
githubStars: content.repoMeta.stars,
githubForks: content.repoMeta.forks,
securityScore: analysis.security.score,
securityStatus: analysis.security.status,
contentHash: analysis.meta.contentHash,
rawContent: content.skillMd,
indexedAt: new Date(),