/nemo-curator
為 LLM 訓練做 GPU 加速的資料整理。支援文字、影像、影片、音訊。功能含模糊去重、品質過濾、語意去重、PII 遮蔽、NSFW 偵測。
- 安裝次數
- 982
- 專案星數
- 1.4 萬 ★
- 授權
- MIT
- 專案更新
- 2026-10-11
怎麼裝
npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill nemo-curator安裝前請先看過原文內容。中文說明是 AI 翻譯，不保證安全。
原文
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.