feat(curation): data quality pipeline — Phases 1-3
Add comprehensive data curation system to clean up the 197K skill dataset and show only quality browse-ready skills to users. Phase 1 — Database exploration: - Explore scripts (explore.ts, explore.mjs, explore.sql) for analysis - Discovered: 69% duplicates, 77% aggregator/fork noise Phase 2 — Data cleanup and classification: - Schema: 6 new curation columns + 4 indexes - curate.mjs: 8-step pipeline (classify, dedup, fork detection, etc.) - Result: 197K → 60K unique → 16K browse-ready skills - Bug fix: securityStatus was computed but never stored during crawl Phase 3 — UI browse-ready filters: - browseReadyFilter applied to 17+ query functions - Homepage stats show accurate browse-ready counts - Stats API filtered (previously had no WHERE clause) - Category counts recalculated (e.g. 45K → 3.1K) - Featured skills exclude duplicates and aggregators Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -18,17 +18,26 @@ vi.mock('@/lib/cache', () => ({
|
||||
|
||||
// Mock the db module - must be before imports that use it
|
||||
vi.mock('@skillhub/db', () => {
|
||||
const mockStatsRow = [{ totalSkills: 100, totalDownloads: 5000, totalContributors: 50 }];
|
||||
const mockCategoryRow = [{ count: 8 }];
|
||||
return {
|
||||
createDb: vi.fn(() => ({
|
||||
select: vi.fn().mockReturnValue({
|
||||
from: vi.fn().mockResolvedValue([
|
||||
{ totalSkills: 100, totalDownloads: 5000, totalContributors: 50 },
|
||||
]),
|
||||
from: vi.fn((table: unknown) => {
|
||||
// categories table query returns count directly (no .where())
|
||||
if (table === 'categories-table') {
|
||||
return Promise.resolve(mockCategoryRow);
|
||||
}
|
||||
// skills table query has .where() chained
|
||||
return {
|
||||
where: vi.fn().mockResolvedValue(mockStatsRow),
|
||||
};
|
||||
}),
|
||||
}),
|
||||
})),
|
||||
skills: { downloadCount: 'download_count', githubOwner: 'github_owner' },
|
||||
categories: {},
|
||||
sql: vi.fn(() => 'mock-sql'),
|
||||
skills: { downloadCount: 'download_count', githubOwner: 'github_owner', isDuplicate: 'is_duplicate', skillType: 'skill_type' },
|
||||
categories: 'categories-table',
|
||||
sql: vi.fn((..._args: unknown[]) => 'mock-sql'),
|
||||
};
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user