feat(curation): data quality pipeline — Phases 1-3
Add comprehensive data curation system to clean up the 197K skill dataset and show only quality browse-ready skills to users. Phase 1 — Database exploration: - Explore scripts (explore.ts, explore.mjs, explore.sql) for analysis - Discovered: 69% duplicates, 77% aggregator/fork noise Phase 2 — Data cleanup and classification: - Schema: 6 new curation columns + 4 indexes - curate.mjs: 8-step pipeline (classify, dedup, fork detection, etc.) - Result: 197K → 60K unique → 16K browse-ready skills - Bug fix: securityStatus was computed but never stored during crawl Phase 3 — UI browse-ready filters: - browseReadyFilter applied to 17+ query functions - Homepage stats show accurate browse-ready counts - Stats API filtered (previously had no WHERE clause) - Category counts recalculated (e.g. 45K → 3.1K) - Featured skills exclude duplicates and aggregators Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -90,6 +90,7 @@ export async function indexSkill(
|
||||
githubStars: content.repoMeta.stars,
|
||||
githubForks: content.repoMeta.forks,
|
||||
securityScore: analysis.security.score,
|
||||
securityStatus: analysis.security.status,
|
||||
contentHash: analysis.meta.contentHash,
|
||||
rawContent: content.skillMd,
|
||||
indexedAt: new Date(),
|
||||
|
||||
Reference in New Issue
Block a user