第五轮:锁/续跑/超时/自动分支/R4 探针与 encode-once/控制台加固;阶段内断点续跑(gen 按条、prep 按 shard、训练与 APG 定时原子存档+体面停止);分歧以实现方意见为准
This commit is contained in:
@@ -74,5 +74,7 @@ writeNpy(path.join(OUTDIR, base + '.Y.npy'), cat(Y, H * 6), [n, H, 6]);
|
||||
writeNpy(path.join(OUTDIR, base + '.V.npy'), Float32Array.from(V), [n]);
|
||||
writeNpy(path.join(OUTDIR, base + '.SG.npy'), cat(SG, 13), [n, 13]);
|
||||
writeNpy(path.join(OUTDIR, base + '.TID.npy'), TID, [n], 'int32');
|
||||
fs.writeFileSync(path.join(OUTDIR, base + '.tasks.jsonl'), tasks.map(t => JSON.stringify(t)).join('\n') + '\n');
|
||||
// 2026-10-18 断点续跑:tasks.jsonl 最后写、先写 .tmp 再改名 —— 它存在就代表这个 shard 的 5 个 npy 都已写完,run_prep.sh 据此跳过
|
||||
const tf = path.join(OUTDIR, base + '.tasks.jsonl');
|
||||
fs.writeFileSync(tf + '.tmp', tasks.map(t => JSON.stringify(t)).join('\n') + '\n'); fs.renameSync(tf + '.tmp', tf);
|
||||
console.log(JSON.stringify({ shard: base, tasks: tasks.length, samples: n, aug: nAug, augRejected: nAugRej, gapDrop: nGapDrop, featVer: POL.FEAT_VER }));
|
||||
|
||||
+17
-3
@@ -1,8 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# 并行打包样本:bash ml/prep/run_prep.sh [进程数=CPU 核数]
|
||||
# 2026-10-18 断点续跑:samples/<shard>.tasks.jsonl 存在且比 shard 新 → 该 shard 已打包完,跳过;
|
||||
# shard 已不存在的旧打包结果删掉(不让上一批数据混进来)。中断后重跑只补没打完的 shard。
|
||||
set -e
|
||||
cd "$(dirname "$0")/../.."
|
||||
P=${1:-$(nproc)}
|
||||
mkdir -p ml/data/samples
|
||||
ls ml/data/shard_*.jsonl | xargs -P "$P" -I{} node ml/prep/build_dataset.cjs {} ml/data/samples
|
||||
${PYTHON:-python} -m ml.data --index ml/data/samples
|
||||
D=ml/data/samples
|
||||
mkdir -p $D
|
||||
for t in $D/*.tasks.jsonl; do
|
||||
[ -e "$t" ] || continue; b=$(basename "$t" .tasks.jsonl)
|
||||
[ -f ml/data/$b.jsonl ] || { echo "[prep] 删除旧打包 $b(对应 shard 已不存在)"; rm -f $D/$b.*; }
|
||||
done
|
||||
todo=()
|
||||
for f in ml/data/shard_*.jsonl; do
|
||||
b=$(basename "$f" .jsonl)
|
||||
if [ -f $D/$b.tasks.jsonl ] && [ $D/$b.tasks.jsonl -nt "$f" ]; then continue; fi
|
||||
todo+=("$f")
|
||||
done
|
||||
echo "[prep] 共 $(ls ml/data/shard_*.jsonl | wc -l) 个 shard,待打包 ${#todo[@]} 个"
|
||||
[ ${#todo[@]} = 0 ] || printf '%s\n' "${todo[@]}" | xargs -P "$P" -I{} node ml/prep/build_dataset.cjs {} $D
|
||||
${PYTHON:-python} -m ml.data --index $D
|
||||
|
||||
Reference in New Issue
Block a user