ml: 第二轮审查 B-1~B-8 处置 + 一键训练控制台
- B-1 运行时投影护栏恢复 5 mm；老师轨迹按认证最小间隙 < 3 mm 剔除（stats/build_dataset，gapDrop） - B-2 APG 端点+中点 hinge 取均值；B-3 check_r2 空匹配保护、5000×50000、分块 256，pass=false 时 prep 停下 - B-4 R7 按全部 B/C 题统计；B-5 r8 起点贴柱；B-7 pipeline r4（export_onnx --random，训练前测延迟） - B-8 起步失败 ≤0.2%、column_gap 分块、pipefail、门禁结果先写 .tmp - 训练控制台：ml/console.bat|sh → ml/monitor.py（开始/停止/进度/日志接口）+ monitor.html； pipeline.sh all 支持 FROM/TO，阶段状态写 ml/runs/pipeline.json（ml/pstate.py）；训练/APG 日志加 total、t - 文档：README 一键运行与训练控制台；审查文档追加第三轮处置；方案余量决策与 R4 时点
This commit is contained in:
Hark
+66
-15
@@ -3,25 +3,28 @@
|
||||
目标：在你的机器上从零跑出 Transformer 避障策略，并在查看器里直接用上。
|
||||
方案依据见 `docs/Transformer轨迹拟合训练方案.md`。所有命令在**仓库根目录**执行。
|
||||
|
||||
按阶段执行 `bash ml/pipeline.sh <阶段>`，每一阶段完成后看本手册对应小节的「检查」再进下一步。
|
||||
**一键运行**：Windows 双击 `ml/console.bat`（Linux/WSL 运行 `bash ml/console.sh`），浏览器自动打开训练控制台 `http://localhost:8100/`，点「开始训练」选好起止阶段即从对拍一路跑到验收，全程进度、曲线、终端输出都在网页上，可随时停止、从停下的那一步继续（见「训练控制台」）。
|
||||
也可以在终端按阶段执行 `bash ml/pipeline.sh <阶段>`，或 `bash ml/pipeline.sh all` 一次跑完；每一阶段完成后看本手册对应小节的「检查」。R2（第 5 步）或 M5（第 7 步）不通过时一键流程会停下，按对应小节处理。
|
||||
|
||||
| 阶段 | 命令 | 预计耗时（28 核 CPU + RTX 3090） | 产出 |
|
||||
|---|---|---|---|
|
||||
| 1 环境 | `bash ml/pipeline.sh setup` | 10 分钟 | 依赖 |
|
||||
| 2 对拍 | `bash ml/pipeline.sh kin` | 1 分钟（实测 7.6 s 对拍） | `ml/assets/kin.json`、对拍 PASS |
|
||||
| 2 对拍 | `bash ml/pipeline.sh kin` | 约 1 分钟（估算：300 条对拍实测 7.6 s，流程跑 2000 条） | `ml/assets/kin.json`、对拍 PASS |
|
||||
| 2′ R4 延迟 | `bash ml/pipeline.sh r4` → 浏览器测 | 10 分钟 | 浏览器 `__POLICY.latency()` 输出 |
|
||||
| 3 老师门禁 | `bash ml/pipeline.sh gate1` | 25–30 分钟（单核） | `ml/data/consistency.json` |
|
||||
| 4 生成数据 | `bash ml/pipeline.sh gen` | 不共识约 1 小时；共识约 4–5 小时 | `ml/data/shard_*.jsonl`、`stats.json` |
|
||||
| 5 打包样本 | `bash ml/pipeline.sh prep` | 15–30 分钟 | `ml/data/samples/`、`ml/data/r2.json` |
|
||||
| 6 预训练 | `bash ml/pipeline.sh train` | 测速 2000 步约 6 分钟，正式约 3 小时 | `ml/runs/base/best.pt` |
|
||||
| 7 M5 门禁 | `bash ml/pipeline.sh m5` | 10–20 分钟 | `ml/runs/m5_gate_base.json` |
|
||||
| 8 APG 微调 | `bash ml/pipeline.sh apg` | 以前 100 步速度为准（预计 2–4 小时） | `ml/runs/apg/best.pt` |
|
||||
| 9 挖错循环 | `bash ml/pipeline.sh mine` | 每轮约 1–2 小时，3–6 轮 | `ml/runs/current.pt` |
|
||||
| 7 M5 门禁 | `bash ml/pipeline.sh m5` | 估算 10–20 分钟 | `ml/runs/m5_gate_base.json` |
|
||||
| 8 APG 微调 | `bash ml/pipeline.sh apg` | 未实测：展开长度 60–150 次推理随批变化，以监控页前 50 步换算的 ETA 为准 | `ml/runs/apg/best.pt` |
|
||||
| 9 挖错循环 | `bash ml/pipeline.sh mine` | 未实测：每轮 = 5 万题作答 + APG，以第 1 轮实际耗时 × 轮数为准 | `ml/runs/current.pt` |
|
||||
| 10 导出 | `bash ml/pipeline.sh export` | 1 分钟 | `webapp/models/traj_policy.onnx`（fp32） |
|
||||
| 11 部署评测 | `bash ml/pipeline.sh bench` | 约 1 小时（Node CPU，1120 个任务） | `ml/runs/policy_bench.json` |
|
||||
| 11 部署评测 | `bash ml/pipeline.sh bench` | 估算约 1 小时（Node CPU，1120 个任务；R8 出题约 2 秒/题） | `ml/runs/policy_bench.json` |
|
||||
| 12 验收汇总 | `bash ml/pipeline.sh accept` | 10–20 分钟 | `ml/runs/acceptance.json` |
|
||||
| 13 使用 | `python ml/serve.py` → 浏览器 `?policy=1` | — | — |
|
||||
|
||||
第 2/4/5/6 步耗时为 RTX 3090 + 28 核实测换算，其余为估算，以每步首次输出的速度为准。
|
||||
第 4/5/6 步耗时为 RTX 3090 + 28 核实测换算；标「估算」「未实测」的不要当预算，以每步首次输出的速度为准。
|
||||
第 1 步（安装依赖）和第 2′ 步（R4，需要在浏览器里看延迟）建议先手动做；一键流程默认从第 2 步开始。
|
||||
|
||||
---
|
||||
|
||||
@@ -46,6 +49,16 @@ bash ml/pipeline.sh kin
|
||||
检查：最后打印的 JSON 里 `"PASS": true`（fk、姿态、组对间隙、83 维特征、间隙梯度与 JS 一致）。
|
||||
`docs/collide.json` 以后只要重新拟合或重生成 `pair2d`/`roadmap`，这一步、数据、模型都要重来（指纹不符时各脚本会拒绝运行）。
|
||||
|
||||
**R4 延迟摸底（训练前做）**：推理延迟只取决于模型结构，与权重无关，所以用随机初始化模型先测，不过就在训练前改结构，不必等训完。
|
||||
|
||||
```bash
|
||||
bash ml/pipeline.sh r4 # 随机初始化 4.92 M 模型导出 fp32 ONNX 到 webapp/models/
|
||||
python ml/serve.py # 浏览器打开 http://localhost:8000/?policy=1 ，来回拖动黄球约 1 分钟
|
||||
```
|
||||
|
||||
控制台运行 `__POLICY.latency()`：推理 batch=8 P99 < 15 ms、安全层 P99 < 15 ms 为通过（随机权重的动作是乱的，手臂会乱动或频繁刹停，这一步只看延迟）。
|
||||
不通过：把 `ml/configs/base.json` 的 `model.d` 从 192 改为 128（`ff` 改 512）后重跑 `r4`；仍不通过再加 `&ep=webgpu` 对比。把输出发我。
|
||||
|
||||
## 3. 老师一致性门禁
|
||||
|
||||
```bash
|
||||
@@ -71,7 +84,8 @@ bash ml/pipeline.sh gen # 默认 5 万条、进程数 = CPU
|
||||
| `samplesPerTask` | ≈ 36.5 | ≈ 36.5 | — |
|
||||
| `samplesEvery3` | ≥ 170 万（自动补量后） | ≥ 170 万 | 补量后仍不足：发我 |
|
||||
| `consensusDrop` | 0 | 约任务数的 10–20% | — |
|
||||
| `vertexMinGapMm` | p50 ≈ 4.4、p90 ≈ 5.9 mm（p10 预计约 3） | 同 | p10 < 2.5 mm：发我（APG 与投影的 3 mm 余量要随之下调） |
|
||||
| `vertexMinGapMm` | p50 ≈ 4.4、p90 ≈ 5.9 mm | 同 | — |
|
||||
| `gapDrop` | 认证最小间隙 < 3 mm 被剔除的轨迹数（未实测，预计约 10%） | 同 | > 25%：发我（自动补量会按剔除后的样本数补，但老师整体太贴柱） |
|
||||
| `byCatMode` | C 类（贴立柱）各模式都有数千条 | 同 | — |
|
||||
|
||||
## 5. 打包训练样本 + R2 检查
|
||||
@@ -80,9 +94,10 @@ bash ml/pipeline.sh gen # 默认 5 万条、进程数 = CPU
|
||||
bash ml/pipeline.sh prep
|
||||
```
|
||||
|
||||
每条轨迹按 50 Hz 时间律重放，每 3 步取一个样本（特征与部署同一份 JS 代码计算），30% 样本加 σ = 1° 噪声增广。
|
||||
每条轨迹按 50 Hz 时间律重放，每 3 步取一个样本（特征与部署同一份 JS 代码计算），30% 样本加 σ = 1° 噪声增广（加噪后间隙 < 3 mm 的不用）。
|
||||
认证最小间隙 < 3 mm 的老师轨迹不进训练集（`MIN_GAP_MM=3`，与 `stats.json` 的 `gapDrop` 同一口径）：模型只学离障碍 ≥ 3 mm 的走法，运行时投影护栏仍保持 5 mm。
|
||||
检查 `ml/data/samples/index.json`：`train_samples` 约 170 万、`val_samples` 约 9 万（5% 任务划为 val）。
|
||||
随后自动跑 R2 检查（`ml/data/r2.json`）：近邻样本前 3 步标签差 `label_std_mean < 0.10`（10% 步长）为通过；不通过先发我，不要直接训练。
|
||||
随后自动跑 R2 检查（`ml/data/r2.json`）：近邻样本前 3 步标签差 `label_std_mean < 0.10`（10% 步长）为通过；不通过时这一步以返回码 4 结束、一键流程停下，先把 `r2.json` 发我。小样本时近邻对不足，`pass` 为 null，不拦。
|
||||
|
||||
## 6. 预训练
|
||||
|
||||
@@ -132,7 +147,7 @@ bash ml/pipeline.sh apg
|
||||
|
||||
- **展开长度**：每批覆盖该批老师时长 p90 × 1.2（无老师题按 6 s），在 60–150 次推理（3.6–9 s）之间；日志 `plans` 字段是本批实际次数
|
||||
- **到达项**：每 10 次推理段末的 TCP 距离与姿态误差，没有「到达即停」。这是有意的：段末项在整条展开上持续给出接近目标的梯度
|
||||
- **碰撞项**：每步端点 + 中点的逐胶囊对铰链 `ReLU(3 mm − 间隙)`。3 mm ≈ 老师轨迹最小间隙的 p10（p50 4.4 mm），避免碰撞项与 BC 标签对抗；运行时投影余量同为 3 mm
|
||||
- **碰撞项**：每步端点与中点的逐胶囊对铰链 `ReLU(3 mm − 间隙)`，两处取均值（与只查端点同量纲，`w_col` 不变）。3 mm 与训练数据的过滤线一致，碰撞项不与 BC 标签对抗；运行时投影护栏是独立的 5 mm
|
||||
- 显存不够把 `configs/apg.json` 的 `batch` 降到 64，或把 `seg_plans` 从 10 降到 5
|
||||
|
||||
## 9. 挖错循环
|
||||
@@ -175,7 +190,7 @@ Node 里的推理/安全层耗时只作参考，浏览器延迟用第 13 步实
|
||||
bash ml/pipeline.sh accept
|
||||
```
|
||||
|
||||
自己补跑 val 全部任务的闭环（开仿真安全层）：可达任务到达率 ≥ 98%、D 类停靠距离 ≤ 老师 + 5 mm、起步失败 = 0；再读第 5–11 步产物核对 R2、R7、R8、R10、M5 与部署形态各项。逐项打印「通过 / 未过」，结果在 `ml/runs/acceptance.json`。
|
||||
自己补跑 val 全部任务的闭环（开仿真安全层）：可达任务到达率 ≥ 98%、D 类停靠距离 ≤ 老师 + 5 mm、起步失败率 ≤ 0.2%；再读第 5–11 步产物核对 R2、R7、R8、R10、M5 与部署形态各项。逐项打印「通过 / 未过」，结果在 `ml/runs/acceptance.json`。
|
||||
|
||||
## 13. 在查看器里使用
|
||||
|
||||
@@ -187,6 +202,7 @@ python ml/serve.py # 带跨源隔离头的本地服务，wasm
|
||||
浏览器打开 `http://localhost:8000/?policy=1`：右上角徽标显示「策略模式就绪」后，拖动黄球或松手，手臂由策略闭环执行（不调用任何规划器）。
|
||||
- `&ep=webgpu` 改用 WebGPU 后端对比延迟
|
||||
- 控制台 `__POLICY.latency()` 查看推理与安全层的 P50/P99；`__POLICY.stats()` 查看刹停、换向、精修次数
|
||||
- 运行时投影护栏 5 mm：候选动作离障碍 < 5 mm 时沿间隙梯度外推（训练数据的过滤线是 3 mm，护栏比数据更保守）
|
||||
- 工具偏移 ≠ 0 时自动交回原规划器
|
||||
- 离线环境：把 onnxruntime-web 的 `ort.min.js` 与 wasm 文件放到 `webapp/vendor/ort/`，地址加 `&ortLocal=1`
|
||||
|
||||
@@ -194,19 +210,51 @@ python ml/serve.py # 带跨源隔离头的本地服务，wasm
|
||||
|
||||
| 步骤 | 文件 |
|
||||
|---|---|
|
||||
| 2 | 对拍输出（终端 JSON） |
|
||||
| 2 | 对拍输出（终端 JSON）、R4 的 `__POLICY.latency()` 输出 |
|
||||
| 3–5 | `ml/data/consistency.json`、`ml/data/stats.json`、`ml/data/samples/index.json`、`ml/data/r2.json` |
|
||||
| 6 | `ml/runs/bench_base/log.jsonl`（测速）、`ml/runs/base/log.jsonl` |
|
||||
| 7 | `ml/runs/m5_gate_*.json` |
|
||||
| 8–9 | `ml/runs/apg/log.jsonl`、`ml/runs/mine/r*/report.json` 与 `gate.json` |
|
||||
| 11–13 | `ml/runs/policy_bench.json`、`ml/runs/acceptance.json`、浏览器 `__POLICY.latency()` 输出 |
|
||||
| 任一步出错 | `ml/runs/pipeline.json`、`ml/runs/console.log`（控制台启动时）或终端输出 |
|
||||
|
||||
## 训练控制台
|
||||
|
||||
```text
|
||||
Windows：双击 ml/console.bat Linux / WSL：bash ml/console.sh
|
||||
（等价于 python -m ml.monitor --open，需要训练用的 Python 环境；Windows 另需 Git for Windows 提供的 Git Bash）
|
||||
```
|
||||
|
||||
浏览器自动打开 `http://localhost:8100/`，零第三方依赖、不连外网，每 5 秒刷新。
|
||||
|
||||
**开始**：点右上角「▶ 开始训练」→ 选起始/结束阶段（默认从第一个还没成功完成的阶段到「验收汇总」）、动作头、共识过滤、任务数、进程数、目标样本数、挖错轮数 → 「开始」。控制台在后台执行 `bash ml/pipeline.sh all`（`FROM`/`TO` 即起止阶段），用的是启动控制台的同一个 Python。
|
||||
**停止**：「■ 停止」结束整个进程树，这一步记为「已停止」，已完成的步骤保留；再点「开始」默认从这一步继续。
|
||||
**网页可以随时关**，训练不受影响，重新打开即可接着看；**关闭控制台窗口会结束训练**，要关终端继续跑，Linux 用 `nohup bash ml/console.sh &`。
|
||||
终端里直接跑的 `bash ml/pipeline.sh …` 同样会显示进度（只是不能在网页上停止）；控制台检测到终端里有阶段在跑时不会重复开始。
|
||||
|
||||
页面内容，从上到下：
|
||||
|
||||
| 区域 | 内容 | 数据来源 |
|
||||
|---|---|---|
|
||||
| 流水线 | 11 个阶段的状态（完成 / 运行中 / 失败 / 已停止 / 未开始）与各步用时 | `ml/runs/pipeline.json`（每个阶段开始结束时自动写） |
|
||||
| 全流程 | 全流程百分比（按各步粗估耗时加权）、当前步骤与它的进度、总用时、本步用时、剩余粗估、开始/停止按钮 | 同上 + 各阶段产物 |
|
||||
| GPU | 利用率、显存、温度、功耗 | `nvidia-smi` |
|
||||
| 当前运行 | 正在跑的训练（自动跟随；顶部下拉框可切换 `ml/runs` 下任一次运行）：步数、已用/剩余时间、预计完成时刻、评估标记 | `ml/runs/*/log.jsonl` |
|
||||
| 指标卡与曲线 | 预训练：总损失、各损失分量、吞吐（R10 线 5000 样本/s）、闭环评估（到达率 / 碰撞率 / ≤老师时长×1.3 / 起步失败）；APG：到达误差、损失分量、展开长度 | 同上 |
|
||||
| 各阶段结果 | 数据统计（含 `gapDrop`）、R2、M5 两种口径、挖错各轮与门禁、部署评测、验收逐项 | `ml/data/*.json`、`ml/runs/*.json` |
|
||||
| 终端输出 | 控制台启动的流水线的命令行输出，报错标红 | `ml/runs/console.log` |
|
||||
|
||||
当前步骤的进度怎么算：生成数据 = 已写出任务条数 / 目标条数（含自动补量）；打包 = 已打包分片 / 分片数；预训练 = (测速 2000 步 + 正式步数) / (2000 + 总步数)；APG = 步数；挖错 = 已完成轮数 + 本轮 APG 步数（共最多 `ROUNDS` 轮，提升不足会提前停）；M5 = 两遍评估；对拍、老师门禁、导出、部署评测、验收无法计量，显示用时与粗估。全流程百分比与剩余时间按手册表格的粗估耗时加权，只作参考。
|
||||
|
||||
同一局域网的手机或其他电脑要看：`python -m ml.monitor --host 0.0.0.0`，用本机 IP 访问；其他机器只能看，开始/停止只接受本机操作（`--remote-control` 放开）。
|
||||
|
||||
## 常见问题
|
||||
|
||||
- **指纹不符**：`collide.json` 变了。依次重跑第 2、4、5 步并重训。
|
||||
- **训练发散**（loss 出现 nan/inf，自动停在 `diverged.pt`）：先把 `lr` 降到 5e-4 重试一次，仍发散就走 WTA 回退（第 7 步）。
|
||||
- **显存/内存不足**：降 `batch`；APG 还可把 `seg_plans` 降到 5。出题与间隙计算已分块（每块 512 个位形）。
|
||||
- **小样本摸底**：几百条任务时 val 可能为空，训练会提示并跳过闭环验证。
|
||||
- **小样本摸底**：几百条任务时 val 可能为空，训练会提示并跳过闭环验证；R2 检查近邻对不足时 `pass` 为 null，不拦后续步骤。
|
||||
- **`all` 在第 7 步中止**：M5 门禁不通过时有意退出（返回码 3），按第 7 步换 WTA 头。
|
||||
- **浏览器只有 1 线程**：没走 `ml/serve.py`（缺 COOP/COEP 头）。
|
||||
- **查看器徽标「策略模式不可用」**：看控制台报错，多为模型文件缺失、指纹或 TCP 不符。
|
||||
|
||||
@@ -214,7 +262,10 @@ python ml/serve.py # 带跨源隔离头的本地服务，wasm
|
||||
|
||||
```
|
||||
ml/
|
||||
pipeline.sh 一条龙阶段脚本（环境变量见文件头）
|
||||
pipeline.sh 一条龙阶段脚本（环境变量见文件头）；每阶段开始/结束经 pstate.py 写 runs/pipeline.json
|
||||
console.bat / console.sh 训练控制台一键启动（Windows / Linux）
|
||||
monitor.py monitor.html 训练控制台：本地 HTTP 服务（开始/停止/进度/日志接口）+ 零依赖网页（地址带 ?demo=1 看示例数据）
|
||||
pstate.py 流水线状态文件读写
|
||||
gen/ 老师数据：sampler / teacher（确定性 PRM A*）/ cert.cjs（距离下界认证，实现在 ik-policy.js）/
|
||||
pathdev.cjs（同伦类判定，共识过滤与一致性门禁用）/ timelaw / gen_dataset / consistency / stats / run_parallel.sh
|
||||
prep/ export_kin.cjs（→ assets/kin.json）、build_dataset.cjs（→ samples/*.npy）、npy.cjs（.npy 写出）、run_prep.sh
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
损失：段末到达项 + Σ碰撞铰链 ReLU(margin_train − 间隙)（逐胶囊对，端点+中点）+ 平滑（速度²+加速度²）+ 行程 + BC 锚定（MeanFlow 损失）。
|
||||
2026-10-18：展开长度不再定长 60 次推理（3.6 s，会截掉老师时长 > 3.6 s 的绕远题），改为覆盖本批 exec_s p90×1.2，上限 plans_max；
|
||||
"到达"仍是每 seg_plans 段末的距离项（无到达即停），属有意设计：段末项给出持续的接近梯度。
|
||||
margin_train 5 → 3 mm：老师轨迹自身最小间隙 p50 4.4 / p90 5.9 mm，5 mm 会让约一半 BC 标签与碰撞项对抗；3 mm ≈ 老师 p10。"""
|
||||
margin_train 3 mm：与训练数据过滤线一致（认证最小间隙 < 3 mm 的老师轨迹不进训练集），碰撞项不与 BC 标签对抗；
|
||||
运行时投影护栏独立保持 5 mm。端点与中点两个铰链取均值，与只查端点同量纲，w_col 无需重标。"""
|
||||
import argparse, json, math, os, random, time
|
||||
import torch
|
||||
|
||||
@@ -101,7 +102,7 @@ def main():
|
||||
# 碰撞铰链：端点 + 中点（与仿真/运行时的离散检查一致）
|
||||
G = kin.gaps(torch.cat([q, 0.5 * (q + qp)], 0), M)
|
||||
hinge = torch.relu(M - G['gap'])
|
||||
col = torch.zeros(2 * B, device=dev).index_add(0, G['b'], hinge).view(2, B).sum(0)
|
||||
col = torch.zeros(2 * B, device=dev).index_add(0, G['b'], hinge).view(2, B).mean(0) # 端点与中点取均值：与只查端点时同量纲，w_col 无需重标
|
||||
sm = ((d / step) ** 2).sum(-1) + (((d - prev_d) / step) ** 2).sum(-1)
|
||||
prev_d = d
|
||||
seg_loss = seg_loss + C['w_col'] * col.mean() + C['w_smooth'] * sm.mean() + C['w_len'] * d.abs().sum(-1).mean()
|
||||
@@ -129,13 +130,13 @@ def main():
|
||||
opt.step()
|
||||
with torch.no_grad():
|
||||
e, oe = goal_err(kin, q, T, R)
|
||||
rec = {'it': it, 'plans': n_plans, 'err_mm_mean': e.mean().item(), 'reach_frac': ((e < 1) & (oe < 0.5)).float().mean().item(), 'bc': l_bc.item(),
|
||||
rec = {'it': it, 'total': C['steps'], 't': time.time(), 'plans': n_plans, 'err_mm_mean': e.mean().item(), 'reach_frac': ((e < 1) & (oe < 0.5)).float().mean().item(), 'bc': l_bc.item(),
|
||||
'gnorm': gn.item(), **stats, 'min': (time.time() - t0) / 60}
|
||||
if it % 10 == 0: print(json.dumps(rec), flush=True)
|
||||
log.write(json.dumps(rec) + '\n'); log.flush()
|
||||
if (it + 1) % C['eval_every'] == 0 or it == C['steps'] - 1:
|
||||
res, _ = evaluate(model.eval(), kin, val_tasks, dev)
|
||||
res['it'] = it + 1
|
||||
res['it'] = it + 1; res['t'] = time.time()
|
||||
print('EVAL', json.dumps(res), flush=True); log.write(json.dumps({'eval': res}) + '\n'); log.flush()
|
||||
sd = {'model': model.state_dict(), 'cfg': mcfg, 'step': it + 1, 'fp': kin.fp, 'eval': res}
|
||||
torch.save(sd, os.path.join(a.out, 'last.pt'))
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
@echo off
|
||||
rem 训练控制台一键启动（Windows）：双击本文件 → 自动打开 http://localhost:8100 ，在网页上点「开始训练」。
|
||||
rem 需要 Python（与训练同一个环境）和 Git for Windows（提供 Git Bash）。关闭本窗口会结束控制台和正在跑的训练。
|
||||
chcp 65001 >nul
|
||||
cd /d "%~dp0\.."
|
||||
where python >nul 2>nul || (echo 找不到 python，请先激活训练用的 Python 环境 & pause & exit /b 1)
|
||||
python -m ml.monitor --open %*
|
||||
pause
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# 训练控制台一键启动（Linux / WSL / Git Bash）：bash ml/console.sh → 打开 http://localhost:8100 ，在网页上点「开始训练」。
|
||||
# 关闭本终端会结束控制台和正在跑的训练；要关终端继续跑，用 nohup bash ml/console.sh & 。
|
||||
cd "$(dirname "$0")/.."
|
||||
PY=${PYTHON:-python}; command -v "$PY" >/dev/null 2>&1 || PY=python3
|
||||
exec "$PY" -m ml.monitor --open "$@"
|
||||
@@ -41,7 +41,7 @@ def main():
|
||||
if reach:
|
||||
r, _ = evaluate(model, kin, reach, a.device, safety=True)
|
||||
add('held-out 可达任务到达率', r['success'], r['success'] >= 0.98, '≥ 0.98（仿真安全层）')
|
||||
add('起步失败率（R2）', r['start_fail'], r['start_fail'] == 0, '= 0')
|
||||
add('起步失败率（R2）', r['start_fail'], r['start_fail'] <= 0.002, '≤ 0.2%（约 2500 题容许 ≤5 题，0 题判据过脆）')
|
||||
else:
|
||||
add('held-out 可达任务到达率', None, False, 'val 中无可达任务')
|
||||
if dock:
|
||||
|
||||
+9
-6
@@ -3,7 +3,7 @@
|
||||
做法：特征逐维标准化；随机取 Q 个查询样本，在 P 个样本池里找最近邻（排除同一任务，避免同轨迹相邻帧冒充"近邻"）；
|
||||
取距离最近的 5% 近邻对，比较前 3 步（实际提交的动作）标签：std ≈ RMS(ΔY)/√2，单位为每步最大步长 STEP 的比例。
|
||||
判据：label_std < 0.10（即 < 10% 步长）。起步失败率在闭环仿真结果的 start_fail 字段（acceptance 汇总）。
|
||||
2026-10-18 新增（审查：R2 无执行点）。GPU 上约 1 分钟；CPU 上把 --pool 降到 50000。"""
|
||||
2026-10-18 新增（审查：R2 无执行点）。默认 5000 查询 × 50000 池，分块 256：GPU 峰值 < 0.5 GB，CPU 约 1–2 分钟；近邻对为空时输出 pass=null 而不报错。"""
|
||||
import argparse, json, math
|
||||
import torch
|
||||
|
||||
@@ -13,7 +13,7 @@ from ml.data import Samples
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--samples', nargs='+', default=['ml/data/samples']); ap.add_argument('--out', default='ml/data/r2.json')
|
||||
ap.add_argument('--queries', type=int, default=20000); ap.add_argument('--pool', type=int, default=200000)
|
||||
ap.add_argument('--queries', type=int, default=5000); ap.add_argument('--pool', type=int, default=50000)
|
||||
ap.add_argument('--frac', type=float, default=0.05); ap.add_argument('--seed', type=int, default=0)
|
||||
ap.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu')
|
||||
a = ap.parse_args()
|
||||
@@ -25,11 +25,14 @@ def main():
|
||||
Xp, Xq = (X[pi] - mu) / sd, (X[qi] - mu) / sd
|
||||
tp, tq = d.TID[pi], d.TID[qi]
|
||||
best_d = torch.empty(Q, device=a.device); best_j = torch.empty(Q, dtype=torch.long, device=a.device)
|
||||
for s in range(0, Q, 1024):
|
||||
D = torch.cdist(Xq[s:s + 1024], Xp)
|
||||
D = torch.where(tq[s:s + 1024, None] == tp[None], torch.full_like(D, float('inf')), D)
|
||||
v, j = D.min(1); best_d[s:s + 1024] = v; best_j[s:s + 1024] = j
|
||||
for s in range(0, Q, 256):
|
||||
D = torch.cdist(Xq[s:s + 256], Xp)
|
||||
D.masked_fill_(tq[s:s + 256, None] == tp[None], float('inf'))
|
||||
v, j = D.min(1); best_d[s:s + 256] = v; best_j[s:s + 256] = j
|
||||
fin = torch.isfinite(best_d)
|
||||
if fin.sum() < 20:
|
||||
out = {'pairs': 0, 'label_std_mean': None, 'pass': None, 'rule': 'label_std_mean < 0.10', 'note': '近邻对不足（样本太少），未判定'}
|
||||
print(json.dumps(out)); json.dump(out, open(a.out, 'w'), indent=1); return
|
||||
thr = torch.quantile(best_d[fin], a.frac)
|
||||
m = fin & (best_d <= thr)
|
||||
Yq, Yp = d.Y[qi][m, :3].float(), d.Y[pi][best_j[m], :3].float()
|
||||
|
||||
+9
-4
@@ -21,11 +21,16 @@ class Wrap(torch.nn.Module):
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--ckpt', required=True); ap.add_argument('--out', default='webapp/models/traj_policy.onnx')
|
||||
ap.add_argument('--ckpt'); ap.add_argument('--random', metavar='CFG', help='R4 测延迟：按配置随机初始化导出，不需要训练'); ap.add_argument('--out', default='webapp/models/traj_policy.onnx')
|
||||
ap.add_argument('--fp16', action='store_true'); ap.add_argument('--opset', type=int, default=17)
|
||||
a = ap.parse_args()
|
||||
ck = torch.load(a.ckpt, map_location='cpu')
|
||||
m = build(ck['cfg']); m.load_state_dict(ck['model']); m.eval()
|
||||
if a.random: # 2026-10-18：R4（浏览器 batch=8 P99）提前到训练前测，延迟只取决于结构不取决于权重
|
||||
torch.manual_seed(0); ck = {'cfg': json.load(open(a.random)), 'random_init': True}
|
||||
m = build(ck['cfg']); m.eval()
|
||||
elif a.ckpt:
|
||||
ck = torch.load(a.ckpt, map_location='cpu')
|
||||
m = build(ck['cfg']); m.load_state_dict(ck['model']); m.eval()
|
||||
else: raise SystemExit('需要 --ckpt 或 --random <配置>')
|
||||
kin = Kin()
|
||||
if ck.get('fp') and ck['fp'] != kin.fp: raise SystemExit(f"checkpoint 指纹 {ck['fp']} ≠ 当前碰撞模型 {kin.fp}：须重训")
|
||||
os.makedirs(os.path.dirname(a.out) or '.', exist_ok=True)
|
||||
@@ -44,7 +49,7 @@ def main():
|
||||
with torch.no_grad(): t = m.sample(feat, z)
|
||||
err = max(float(np.abs(o[i] - t[i].numpy()).max()) for i in range(3))
|
||||
meta = {'fp': kin.fp, 'tcp0': kin.tcp0.tolist(), 'head': m.head, 'feat_ver': FEAT_VER, 'f_dim': F_DIM, 'h': H, 'step': STEP, 'fp16': a.fp16,
|
||||
'ckpt': os.path.abspath(a.ckpt), 'eval': ck.get('eval'), 'max_abs_err_vs_torch': err}
|
||||
'ckpt': os.path.abspath(a.ckpt) if a.ckpt else None, 'random_init': bool(a.random), 'eval': ck.get('eval'), 'max_abs_err_vs_torch': err}
|
||||
json.dump(meta, open(a.out.replace('.onnx', '.meta.json'), 'w'), indent=1)
|
||||
print(json.dumps({'onnx': a.out, 'MB': os.path.getsize(a.out) / 1e6, 'max_abs_err_vs_torch': err}))
|
||||
|
||||
|
||||
+7
-1
@@ -2,6 +2,10 @@
|
||||
'use strict';
|
||||
const fs = require('fs');
|
||||
const files = process.argv.slice(2);
|
||||
/* 2026-10-18：老师样本按认证最小间隙过滤（< MIN_GAP_MM 的轨迹不进训练集，build_dataset.cjs 同值）。
|
||||
安全护栏（运行时投影 5 mm）不迁就数据分布；样本数与补量只按过滤后计。 */
|
||||
const MIN_GAP_MM = +(process.env.MIN_GAP_MM || 3);
|
||||
let gapDrop = 0;
|
||||
const by = {}; let total = 0, keep = 0, steps = 0, samples = 0, padTail = 0;
|
||||
const q = a => { a.sort((x, y) => x - y); return a.length ? { p10: +a[Math.floor(a.length * 0.1)].toFixed(1), p50: +a[a.length >> 1].toFixed(1), p90: +a[Math.floor(a.length * 0.9)].toFixed(1), max: +a[a.length - 1].toFixed(1) } : null; };
|
||||
const plan = [], certMs = [], travel = [], exec = [], gaps = [];
|
||||
@@ -10,7 +14,8 @@ for (const f of files) for (const line of fs.readFileSync(f, 'utf8').split('\n')
|
||||
const k = r.cat + '/' + r.mode; const b = by[k] = by[k] || { n: 0, reached: 0, certFail: 0, keep: 0, demoted: 0, consensusDrop: 0, j6Frozen: 0 };
|
||||
b.n++; if (r.reached) b.reached++; if (r.cert && !r.cert.ok) b.certFail++; if (r.demoted) b.demoted++; if (r.why === 'consensus') b.consensusDrop++; if (r.j6Frozen) b.j6Frozen++;
|
||||
if (r.plan_ms != null) plan.push(r.plan_ms); if (r.cert) certMs.push(r.cert.ms); if (r.cert) gaps.push(r.cert.minGap);
|
||||
if (r.keep) { b.keep++; keep++; travel.push(r.travel_deg); exec.push(r.exec_s);
|
||||
if (r.keep && r.cert && r.cert.minGap != null && r.cert.minGap < MIN_GAP_MM) { gapDrop++; b.gapDrop = (b.gapDrop || 0) + 1; }
|
||||
else if (r.keep) { b.keep++; keep++; travel.push(r.travel_deg); exec.push(r.exec_s);
|
||||
const n = r.traj.length - 1; steps += n; const s = Math.floor(n / 3) + 1; samples += s;
|
||||
for (let t = 0; t <= n; t += 3) if (t + 16 > n) padTail++; }
|
||||
}
|
||||
@@ -18,5 +23,6 @@ const out = { tasks: total, keep, keepRate: +(keep / total).toFixed(3), byCatMod
|
||||
planMs: q(plan), certMs: q(certMs), vertexMinGapMm: q(gaps), travelDeg: q(travel), execS: q(exec),
|
||||
samplesEvery3: samples, samplesPerTask: +(samples / keep).toFixed(1), paddedSampleShare: +(padTail / samples).toFixed(3),
|
||||
projectedSamplesFor50k: Math.round(samples / total * 50000),
|
||||
gapDrop, minGapFilterMm: MIN_GAP_MM,
|
||||
consensusDrop: Object.values(by).reduce((a, b) => a + b.consensusDrop, 0) };
|
||||
console.log(JSON.stringify(out, null, 1));
|
||||
|
||||
+2
-1
@@ -44,7 +44,8 @@ def main():
|
||||
cat = [t['cat'] for t in tasks]
|
||||
so_side = torch.cat([p['side'] for _, p in po]); sn_side = torch.cat([p['side'] for _, p in pn])
|
||||
det = torch.tensor([c in ('B', 'C') for c in cat])
|
||||
left = lambda sd, ok: (((sd > 0) & det & ok).sum() / ((det & ok).sum().clamp(min=1))).item()
|
||||
# 2026-10-18：左右比例按全部 B/C 题的首次 J1 方向统计，不以到达为条件（APG 把一侧绕行全做失败时，到达子集比例可纹丝不动）
|
||||
left = lambda sd, ok: (((sd > 0) & det).sum() / det.sum().clamp(min=1)).item()
|
||||
ok_o, ok_n = torch.tensor(so, dtype=torch.bool), torch.tensor(sn, dtype=torch.bool)
|
||||
lr_shift = abs(left(sn_side, ok_n) - left(so_side, ok_o)) * 100
|
||||
both = ok_o & ok_n
|
||||
|
||||
+6
-3
@@ -33,9 +33,12 @@ def straight_blocked(kin, a, b, n=24):
|
||||
return (min_gap_chunked(kin, pts) <= 0).view(a.shape[0], n).any(-1)
|
||||
|
||||
|
||||
def column_gap(kin, q):
|
||||
gm = kin.gaps(q, 40)['gm']
|
||||
return torch.stack([gm[:, g] for g in (2, 3, 4, 5, 6)], -1).min(-1).values # 组 0（立柱/底座）× 组 2..6
|
||||
def column_gap(kin, q, chunk=256):
|
||||
out = []
|
||||
for s in range(0, q.shape[0], chunk): # 与 min_gap_chunked 一样分块防 OOM
|
||||
gm = kin.gaps(q[s:s + chunk], 40)['gm']
|
||||
out.append(torch.stack([gm[:, g] for g in (2, 3, 4, 5, 6)], -1).min(-1).values) # 组 0（立柱/底座）× 组 2..6
|
||||
return torch.cat(out)
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
|
||||
+795
@@ -0,0 +1,795 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>训练监控 · 机械臂避障策略</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0b0f17; --bg2: #0f1521; --panel: #131a27; --panel2: #18212f; --line: #222c3c; --line2: #2c3850;
|
||||
--text: #e8edf5; --dim: #8c99ad; --dim2: #5d6a7e;
|
||||
--accent: #5aa9ff; --accent2: #7c6cff; --ok: #34d399; --warn: #fbbf24; --bad: #f87171;
|
||||
--c1: #5aa9ff; --c2: #fbbf24; --c3: #34d399; --c4: #f87171; --c5: #a78bfa; --c6: #22d3ee; --c7: #fb923c;
|
||||
--mono: "JetBrains Mono", ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
--r: 14px;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { margin: 0; }
|
||||
body { background: radial-gradient(1200px 600px at 10% -10%, #16223a 0%, transparent 60%), radial-gradient(900px 500px at 100% 0%, #1c1838 0%, transparent 55%), var(--bg);
|
||||
background-attachment: fixed; color: var(--text); font: 14px/1.5 -apple-system, "Segoe UI", "PingFang SC", "Microsoft YaHei", sans-serif; min-height: 100vh; }
|
||||
|
||||
/* ---------- 顶栏 ---------- */
|
||||
header { position: sticky; top: 0; z-index: 20; display: flex; align-items: center; gap: 14px; flex-wrap: wrap; padding: 12px 24px;
|
||||
background: rgba(11,15,23,.78); backdrop-filter: blur(12px); border-bottom: 1px solid var(--line); }
|
||||
.brand { display: flex; align-items: center; gap: 10px; }
|
||||
.logo { width: 30px; height: 30px; border-radius: 9px; background: linear-gradient(135deg, var(--accent), var(--accent2)); display: grid; place-items: center; box-shadow: 0 4px 18px #5aa9ff44; }
|
||||
.logo svg { width: 18px; height: 18px; }
|
||||
.brand h1 { font-size: 15px; margin: 0; font-weight: 650; letter-spacing: .3px; }
|
||||
.brand .sub { font-size: 12px; color: var(--dim); margin-top: -2px; }
|
||||
header .sp { flex: 1; }
|
||||
.ctl { display: flex; align-items: center; gap: 8px; color: var(--dim); font-size: 13px; }
|
||||
select, button { background: var(--panel2); color: var(--text); border: 1px solid var(--line2); border-radius: 9px; padding: 6px 10px; font: inherit; font-size: 13px; outline: none; }
|
||||
select:hover, button:hover { border-color: #3d4c68; }
|
||||
button { cursor: pointer; display: inline-flex; align-items: center; gap: 6px; }
|
||||
button.on { border-color: var(--accent); color: var(--accent); background: #5aa9ff14; }
|
||||
input[type=range] { accent-color: var(--accent); width: 90px; }
|
||||
.pill { display: inline-flex; align-items: center; gap: 7px; padding: 4px 11px; border-radius: 99px; font-size: 12px; border: 1px solid var(--line2); color: var(--dim); background: var(--panel); }
|
||||
.pill .dot { width: 7px; height: 7px; border-radius: 50%; background: var(--dim2); }
|
||||
.pill.live { color: var(--ok); border-color: #34d39955; } .pill.live .dot { background: var(--ok); box-shadow: 0 0 0 0 #34d399aa; animation: pulse 1.6s infinite; }
|
||||
.pill.stale { color: var(--warn); border-color: #fbbf2455; } .pill.stale .dot { background: var(--warn); }
|
||||
.pill.dead { color: var(--bad); border-color: #f8717155; } .pill.dead .dot { background: var(--bad); }
|
||||
.pill.demo { color: var(--c5); border-color: #a78bfa55; } .pill.demo .dot { background: var(--c5); }
|
||||
@keyframes pulse { 0% { box-shadow: 0 0 0 0 #34d39999; } 70% { box-shadow: 0 0 0 7px #34d39900; } 100% { box-shadow: 0 0 0 0 #34d39900; } }
|
||||
|
||||
main { padding: 20px 24px 48px; max-width: 1680px; margin: 0 auto; display: flex; flex-direction: column; gap: 16px; }
|
||||
.card { background: linear-gradient(180deg, #151d2b 0%, #121926 100%); border: 1px solid var(--line); border-radius: var(--r); padding: 16px 18px; min-width: 0; box-shadow: 0 1px 0 #ffffff05 inset, 0 8px 24px #00000030; }
|
||||
.card h2 { font-size: 12.5px; font-weight: 600; color: var(--dim); margin: 0 0 12px; display: flex; justify-content: space-between; align-items: center; letter-spacing: .4px; }
|
||||
.card h2 .r { font-weight: 400; color: var(--dim2); font-size: 12px; letter-spacing: 0; }
|
||||
|
||||
/* ---------- 流水线阶段 ---------- */
|
||||
.steps { display: flex; gap: 0; overflow-x: auto; padding: 4px 2px 2px; scrollbar-width: thin; }
|
||||
.step { flex: 1 0 96px; display: flex; flex-direction: column; align-items: center; gap: 6px; position: relative; min-width: 96px; }
|
||||
.step::before { content: ""; position: absolute; top: 14px; left: -50%; width: 100%; height: 2px; background: var(--line2); z-index: 0; }
|
||||
.step:first-child::before { display: none; }
|
||||
.step.done::before, .step.active::before { background: linear-gradient(90deg, var(--ok), var(--ok)); }
|
||||
.step .ic { width: 28px; height: 28px; border-radius: 50%; display: grid; place-items: center; font: 600 12px var(--mono); background: var(--panel2); border: 2px solid var(--line2); color: var(--dim); z-index: 1; }
|
||||
.step.done .ic { background: #34d39922; border-color: var(--ok); color: var(--ok); }
|
||||
.step.active .ic { background: #5aa9ff22; border-color: var(--accent); color: var(--accent); box-shadow: 0 0 0 4px #5aa9ff1f; }
|
||||
.step.fail .ic { background: #f8717122; border-color: var(--bad); color: var(--bad); }
|
||||
.step .nm { font-size: 12.5px; color: var(--text); white-space: nowrap; } .step .ds { font-size: 11px; color: var(--dim2); white-space: nowrap; }
|
||||
.step.pending .nm { color: var(--dim); }
|
||||
.step.active .ds { color: var(--accent); }
|
||||
|
||||
/* ---------- 进度主卡 + GPU ---------- */
|
||||
.hero { display: grid; grid-template-columns: minmax(0, 2.2fr) minmax(300px, 1fr); gap: 16px; }
|
||||
.prog-top { display: flex; align-items: flex-end; justify-content: space-between; gap: 16px; flex-wrap: wrap; }
|
||||
.prog-top .ttl { font-size: 13px; color: var(--dim); } .prog-top .ttl b { color: var(--text); font-weight: 600; font-size: 16px; margin-right: 8px; }
|
||||
.tag { display: inline-block; font-size: 11px; padding: 1px 8px; border-radius: 6px; background: #5aa9ff1c; color: var(--accent); border: 1px solid #5aa9ff33; vertical-align: 2px; }
|
||||
.tag.apg { background: #a78bfa1c; color: var(--c5); border-color: #a78bfa33; } .tag.bench { background: #fbbf241c; color: var(--warn); border-color: #fbbf2433; }
|
||||
.bigpct { font: 700 44px/1 var(--mono); letter-spacing: -1px; background: linear-gradient(90deg, #fff, #b9d7ff); -webkit-background-clip: text; background-clip: text; color: transparent; }
|
||||
.bigpct small { font-size: 18px; margin-left: 2px; }
|
||||
.bar { position: relative; height: 12px; background: #0d131d; border: 1px solid var(--line); border-radius: 99px; overflow: visible; margin: 16px 0 10px; }
|
||||
.bar .fill { height: 100%; border-radius: 99px; background: linear-gradient(90deg, var(--accent), var(--c6)); box-shadow: 0 0 14px #5aa9ff66; transition: width .8s ease; position: relative; }
|
||||
.bar .fill::after { content: ""; position: absolute; inset: 0; border-radius: 99px; background: repeating-linear-gradient(45deg, #ffffff14 0 8px, transparent 8px 16px); animation: move 1.2s linear infinite; }
|
||||
.bar.idle .fill::after { animation: none; display: none; }
|
||||
@keyframes move { from { background-position: 0 0; } to { background-position: 22px 0; } }
|
||||
.bar .mk { position: absolute; top: -4px; width: 2px; height: 20px; border-radius: 1px; background: #ffffff30; }
|
||||
.bar .mk.best { background: var(--ok); }
|
||||
.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(130px, 1fr)); gap: 10px; margin-top: 14px; }
|
||||
.fact { background: #0f1520; border: 1px solid var(--line); border-radius: 10px; padding: 9px 12px; }
|
||||
.fact .k { font-size: 11.5px; color: var(--dim); } .fact .v { font: 600 15px/1.35 var(--mono); margin-top: 2px; white-space: nowrap; }
|
||||
.gpuwrap { display: flex; gap: 16px; align-items: center; }
|
||||
.ring { position: relative; width: 116px; height: 116px; flex: none; }
|
||||
.ring svg { transform: rotate(-90deg); }
|
||||
.ring .c { position: absolute; inset: 0; display: grid; place-items: center; text-align: center; }
|
||||
.ring .c b { font: 700 24px/1 var(--mono); display: block; } .ring .c span { font-size: 11px; color: var(--dim); }
|
||||
.gpurows { flex: 1; display: flex; flex-direction: column; gap: 10px; min-width: 0; }
|
||||
.gr .l { display: flex; justify-content: space-between; font-size: 12px; color: var(--dim); margin-bottom: 4px; } .gr .l b { color: var(--text); font: 500 12.5px var(--mono); }
|
||||
.mini { height: 6px; background: #0d131d; border-radius: 99px; overflow: hidden; } .mini > div { height: 100%; border-radius: 99px; transition: width .6s; }
|
||||
.gname { font-size: 12px; color: var(--dim2); margin-top: 12px; }
|
||||
|
||||
/* ---------- 指标卡 ---------- */
|
||||
.kpis { display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 16px; }
|
||||
.kpi { position: relative; overflow: hidden; padding-bottom: 12px; }
|
||||
.kpi .v { font: 650 26px/1.15 var(--mono); letter-spacing: -.5px; } .kpi .v small { font-size: 13px; color: var(--dim); margin-left: 4px; font-weight: 500; }
|
||||
.kpi .s { color: var(--dim); font-size: 12px; margin-top: 3px; } .kpi canvas { position: absolute; right: 0; bottom: 0; width: 55%; height: 46px; opacity: .9; }
|
||||
.delta { font: 500 12px var(--mono); margin-left: 6px; }
|
||||
|
||||
/* ---------- 图表 ---------- */
|
||||
.charts { display: grid; grid-template-columns: repeat(auto-fit, minmax(520px, 1fr)); gap: 16px; }
|
||||
.chart canvas.plot { width: 100%; height: 260px; display: block; cursor: crosshair; }
|
||||
.legend { display: flex; flex-wrap: wrap; gap: 6px 8px; margin-top: 10px; font-size: 12px; color: var(--dim); }
|
||||
.legend span { cursor: pointer; user-select: none; padding: 2px 9px; border-radius: 99px; border: 1px solid var(--line); background: #0f1520; display: inline-flex; align-items: center; gap: 6px; }
|
||||
.legend span b { font: 500 11.5px var(--mono); color: var(--text); }
|
||||
.legend span.off { opacity: .4; } .legend i { width: 8px; height: 8px; border-radius: 50%; display: inline-block; }
|
||||
|
||||
/* ---------- 底部面板 ---------- */
|
||||
.side { display: grid; grid-template-columns: repeat(auto-fit, minmax(380px, 1fr)); gap: 16px; }
|
||||
table { width: 100%; border-collapse: collapse; font-size: 13px; }
|
||||
th, td { text-align: left; padding: 7px 6px; border-bottom: 1px solid #1d2635; white-space: nowrap; }
|
||||
tr:last-child td { border-bottom: none; }
|
||||
th { color: var(--dim2); font-weight: 500; font-size: 12px; } td.n { font-family: var(--mono); text-align: right; } th.n { text-align: right; }
|
||||
.ok { color: var(--ok); } .bad { color: var(--bad); } .warn { color: var(--warn); } .dim { color: var(--dim); }
|
||||
.chip { display: inline-block; font-size: 11.5px; padding: 1px 8px; border-radius: 99px; border: 1px solid; }
|
||||
.chip.ok { border-color: #34d39955; background: #34d39914; } .chip.bad { border-color: #f8717155; background: #f8717114; } .chip.dim { border-color: var(--line2); }
|
||||
.chip.warn { border-color: #fbbf2455; background: #fbbf2414; }
|
||||
.empty { display: flex; align-items: center; gap: 10px; color: var(--dim2); font-size: 13px; padding: 18px 4px; }
|
||||
.empty svg { width: 18px; height: 18px; opacity: .6; }
|
||||
.accbanner { display: flex; align-items: center; gap: 10px; padding: 10px 12px; border-radius: 10px; margin-bottom: 8px; font-weight: 600; }
|
||||
.accbanner.ok { background: #34d39914; color: var(--ok); border: 1px solid #34d39944; } .accbanner.bad { background: #f8717114; color: var(--bad); border: 1px solid #f8717144; }
|
||||
.tip { position: fixed; pointer-events: none; background: #0a0e15f2; border: 1px solid var(--line2); border-radius: 10px; padding: 8px 10px; font: 12px/1.6 var(--mono); display: none; z-index: 30; white-space: nowrap; box-shadow: 0 10px 30px #0008; }
|
||||
.tip .h { color: var(--dim); margin-bottom: 2px; }
|
||||
footer { color: var(--dim2); font-size: 12px; text-align: center; padding-top: 6px; }
|
||||
/* ---------- 控制：开始/停止、对话框、终端 ---------- */
|
||||
button.primary { background: linear-gradient(135deg, var(--accent), var(--accent2)); border: none; color: #fff; font-weight: 600; padding: 7px 16px; box-shadow: 0 4px 16px #5aa9ff40; }
|
||||
button.primary:hover { filter: brightness(1.08); } button.primary:disabled { opacity: .45; cursor: not-allowed; filter: none; }
|
||||
button.danger { color: var(--bad); border-color: #f8717166; } button.danger:hover { background: #f8717114; }
|
||||
.overall { display: flex; align-items: flex-end; justify-content: space-between; gap: 16px; flex-wrap: wrap; }
|
||||
.overall .now { font-size: 13px; color: var(--dim); } .overall .now b { display: block; color: var(--text); font-size: 18px; font-weight: 650; margin: 2px 0 2px; }
|
||||
.sublabel { font-size: 12.5px; color: var(--dim); display: flex; justify-content: space-between; margin-top: 4px; }
|
||||
.bar.thin { height: 8px; margin: 8px 0 4px; } .bar.ind .fill { width: 30% !important; animation: ind 1.6s ease-in-out infinite; }
|
||||
@keyframes ind { 0% { margin-left: 0; } 50% { margin-left: 70%; } 100% { margin-left: 0; } }
|
||||
.ctlrow { display: flex; gap: 10px; margin-top: 14px; align-items: center; flex-wrap: wrap; }
|
||||
.ctlrow .hint { font-size: 12px; color: var(--dim2); }
|
||||
.term { background: #070a10; border: 1px solid var(--line); border-radius: 10px; padding: 10px 12px; height: 260px; overflow: auto; font: 12px/1.55 var(--mono); color: #c9d4e3; white-space: pre-wrap; word-break: break-all; }
|
||||
.term .e { color: var(--bad); } .term .c { color: var(--accent); } .term .g { color: var(--ok); }
|
||||
.modal { position: fixed; inset: 0; background: #0009; backdrop-filter: blur(3px); display: none; align-items: center; justify-content: center; z-index: 40; }
|
||||
.modal.show { display: flex; }
|
||||
.dlg { width: min(560px, 92vw); background: #131a27; border: 1px solid var(--line2); border-radius: 16px; padding: 20px 22px; box-shadow: 0 30px 80px #000a; }
|
||||
.dlg h3 { margin: 0 0 4px; font-size: 16px; } .dlg .d { color: var(--dim); font-size: 12.5px; margin-bottom: 14px; }
|
||||
.form { display: grid; grid-template-columns: 1fr 1fr; gap: 12px 14px; }
|
||||
.form label { display: flex; flex-direction: column; gap: 5px; font-size: 12px; color: var(--dim); }
|
||||
.form input, .form select { background: #0f1520; color: var(--text); border: 1px solid var(--line2); border-radius: 8px; padding: 7px 9px; font: 13px var(--mono); }
|
||||
.form .full { grid-column: 1 / -1; }
|
||||
.dlg .acts { display: flex; justify-content: flex-end; gap: 10px; margin-top: 18px; }
|
||||
.dlg .warnbox { margin-top: 12px; font-size: 12px; color: var(--warn); background: #fbbf2410; border: 1px solid #fbbf2433; border-radius: 8px; padding: 8px 10px; display: none; }
|
||||
.toast { position: fixed; bottom: 22px; left: 50%; transform: translateX(-50%); background: #1b2433; border: 1px solid var(--line2); padding: 9px 16px; border-radius: 10px; font-size: 13px; z-index: 50; display: none; box-shadow: 0 10px 30px #0008; }
|
||||
@media (max-width: 900px) { .hero { grid-template-columns: 1fr; } .charts { grid-template-columns: 1fr; } main { padding: 14px; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="brand">
|
||||
<div class="logo"><svg viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M4 20h6"/><path d="M7 20v-5l5-6"/><path d="M12 9l5 2"/><circle cx="12" cy="9" r="1.6"/><path d="M17 11l2-3"/></svg></div>
|
||||
<div><h1>训练监控</h1><div class="sub">机械臂避障策略 · Transformer</div></div>
|
||||
</div>
|
||||
<select id="run" title="选择运行（ml/runs/*）"></select>
|
||||
<select id="scenario" title="示例场景" style="display:none"><option value="train">示例：预训练进行中</option><option value="gen">示例：生成数据中</option><option value="late">示例：挖错第 3 轮</option><option value="idle">示例：还没开始</option></select>
|
||||
<span id="state" class="pill"><span class="dot"></span><span>连接中</span></span>
|
||||
<span class="sp"></span>
|
||||
<label class="ctl">平滑 <input id="smooth" type="range" min="0" max="0.99" step="0.01" value="0.85"><span id="smoothv" style="font-family:var(--mono);width:30px">0.85</span></label>
|
||||
<button id="logy" title="损失图纵轴取对数">对数轴</button>
|
||||
<button id="auto" class="on" title="每 5 秒刷新"><span>●</span>自动刷新</button>
|
||||
<button id="btnStart" class="primary" title="一键运行流水线">▶ 开始训练</button>
|
||||
<button id="btnStop" class="danger" style="display:none" title="停止正在运行的流水线">■ 停止</button>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<section class="card"><h2><span>流水线</span><span class="r" id="stepnote"></span></h2><div class="steps" id="steps"></div></section>
|
||||
|
||||
<section class="hero">
|
||||
<div class="card" id="pipe"></div>
|
||||
<div class="card"><h2><span>GPU</span><span class="r" id="gputime"></span></h2><div id="gpu"></div></div>
|
||||
</section>
|
||||
|
||||
<section class="card" id="prog"></section>
|
||||
<section class="kpis" id="kpis"></section>
|
||||
<section class="charts" id="charts"></section>
|
||||
|
||||
<section class="side">
|
||||
<div class="card"><h2><span>数据阶段</span><span class="r">ml/data</span></h2><div id="data"></div></div>
|
||||
<div class="card"><h2><span>M5 头门禁</span><span class="r">纯策略穿柱率 ≤ 2%</span></h2><div id="m5"></div></div>
|
||||
<div class="card"><h2><span>挖错循环</span><span class="r">ml/runs/mine</span></h2><div id="mine"></div></div>
|
||||
<div class="card"><h2><span>部署形态评测</span><span class="r">策略 + 安全层</span></h2><div id="bench"></div></div>
|
||||
<div class="card"><h2><span>验收汇总</span><span class="r">方案 §9.2</span></h2><div id="accept"></div></div>
|
||||
</section>
|
||||
<section class="card"><h2><span>终端输出</span><span class="r" id="termnote">ml/runs/console.log</span></h2><div class="term" id="term"></div></section>
|
||||
<footer id="foot"></footer>
|
||||
</main>
|
||||
<div class="tip" id="tip"></div>
|
||||
<div class="toast" id="toast"></div>
|
||||
<div class="modal" id="modal"><div class="dlg">
|
||||
<h3>开始训练</h3>
|
||||
<div class="d">后台执行 <code>bash ml/pipeline.sh all</code>，从起始阶段一路跑到结束阶段。网页可以关，训练不受影响；关闭控制台窗口会结束训练。</div>
|
||||
<div class="form">
|
||||
<label>起始阶段<select id="fFrom"></select></label>
|
||||
<label>结束阶段<select id="fTo"></select></label>
|
||||
<label>动作头 CFG<select id="fCfg"><option value="ml/configs/base.json">MeanFlow（base）</option><option value="ml/configs/wta.json">WTA 回退（wta）</option></select></label>
|
||||
<label>共识过滤 CONSENSUS<select id="fCons"><option value="">按老师门禁自动</option><option value="0">关</option><option value="1">开（生成约慢 5 倍）</option></select></label>
|
||||
<label>任务数 N_TASKS<input id="fN" value="50000"></label>
|
||||
<label>生成进程数 NPROC<input id="fP" placeholder="默认 = CPU 核数"></label>
|
||||
<label>目标样本 TARGET_SAMPLES<input id="fT" value="1700000"></label>
|
||||
<label>挖错轮数 ROUNDS<input id="fR" value="6"></label>
|
||||
</div>
|
||||
<div class="warnbox" id="fWarn"></div>
|
||||
<div class="acts"><button id="fCancel">取消</button><button id="fGo" class="primary">▶ 开始</button></div>
|
||||
</div></div>
|
||||
|
||||
<script>
|
||||
'use strict';
|
||||
/* =====================================================================
|
||||
数据层：python -m ml.monitor 提供 /api/runs、/api/log、/api/status。
|
||||
直接双击打开本文件、地址带 ?demo=1，或 FORCE_DEMO=true 时用内置示例数据（每次刷新模拟训练推进）。
|
||||
2026-10-18 新增（用户要求：训练中随时可视化查看进度）。
|
||||
===================================================================== */
|
||||
const FORCE_DEMO = false;
|
||||
const DEMO = FORCE_DEMO || new URLSearchParams(location.search).has('demo') || location.protocol === 'file:';
|
||||
const $ = id => document.getElementById(id);
|
||||
const isNum = v => typeof v === 'number' && isFinite(v);
|
||||
const fmt = (v, d = 3) => !isNum(v) ? '—' : (Math.abs(v) >= 1000 ? Math.round(v).toLocaleString() : Math.abs(v) >= 100 ? v.toFixed(1) : Math.abs(v) >= 1 ? v.toFixed(Math.min(d, 3)) : v === 0 ? '0' : v.toPrecision(3));
|
||||
const pct = (v, d = 1) => !isNum(v) ? '—' : (100 * v).toFixed(d) + '%';
|
||||
const hms = s => { if (!isNum(s)) return '—'; s = Math.max(0, s); const h = Math.floor(s / 3600), m = Math.floor(s % 3600 / 60); return h ? `${h} 小时 ${m} 分` : m ? `${m} 分 ${Math.floor(s % 60)} 秒` : `${Math.floor(s)} 秒`; };
|
||||
const hmsShort = s => !isNum(s) ? '—' : s >= 3600 ? `${(s / 3600).toFixed(1)} h` : s >= 60 ? `${Math.round(s / 60)} 分` : `${Math.round(s)} 秒`;
|
||||
const ago = t => { const s = Date.now() / 1000 - t; return s < 60 ? `${Math.max(0, Math.floor(s))} 秒前` : s < 3600 ? `${Math.floor(s / 60)} 分钟前` : `${(s / 3600).toFixed(1)} 小时前`; };
|
||||
const ICON_WAIT = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><circle cx="12" cy="12" r="9"/><path d="M12 7v5l3 2"/></svg>';
|
||||
const empty = t => `<div class="empty">${ICON_WAIT}<span>${t}</span></div>`;
|
||||
const KIND = { train: ['预训练', ''], apg: ['APG 微调', 'apg'], bench: ['测速', 'bench'] };
|
||||
|
||||
const api = DEMO ? demoApi() : async (path) => { const r = await fetch(path, { cache: 'no-store' }); if (!r.ok) throw new Error(r.status); return r.json(); };
|
||||
if (DEMO) $('scenario').style.display = '';
|
||||
|
||||
/* ===================== 图表：零依赖 canvas ===================== */
|
||||
const css = v => getComputedStyle(document.documentElement).getPropertyValue(v).trim();
|
||||
const COLORS = ['--c1', '--c2', '--c3', '--c4', '--c5', '--c6', '--c7'].map(css);
|
||||
const tip = $('tip');
|
||||
const hexA = (hex, a) => { const n = parseInt(hex.slice(1), 16); return `rgba(${n >> 16},${n >> 8 & 255},${n & 255},${a})`; };
|
||||
|
||||
class Chart {
|
||||
constructor(parent, title, opts = {}) {
|
||||
this.o = opts; this.hidden = new Set(opts.hidden || []);
|
||||
const card = document.createElement('div'); card.className = 'card chart';
|
||||
card.innerHTML = `<h2><span>${title}</span><span class="r">${opts.note || ''}</span></h2><canvas class="plot"></canvas><div class="legend"></div>`;
|
||||
parent.appendChild(card);
|
||||
this.cv = card.querySelector('canvas'); this.lg = card.querySelector('.legend'); this.series = [];
|
||||
this.cv.addEventListener('mousemove', e => this.hover(e));
|
||||
this.cv.addEventListener('mouseleave', () => { tip.style.display = 'none'; this.hx = null; this.draw(); });
|
||||
new ResizeObserver(() => this.draw()).observe(this.cv);
|
||||
}
|
||||
set(series) {
|
||||
this.series = series.map((s, i) => ({ ...s, color: s.color || COLORS[i % COLORS.length] }));
|
||||
const last = s => { for (let i = s.pts.length - 1; i >= 0; i--) if (isNum(s.pts[i][1])) return s.pts[i][1]; return null; };
|
||||
this.lg.innerHTML = this.series.filter(s => !s.noLegend).map(s => `<span data-k="${s.name}" class="${this.hidden.has(s.name) ? 'off' : ''}"><i style="background:${s.color}"></i>${s.label || s.name}<b>${this.o.percent ? pct(last(s)) : fmt(last(s))}</b></span>`).join('');
|
||||
this.lg.querySelectorAll('span').forEach(el => el.onclick = () => { const k = el.dataset.k; this.hidden.has(k) ? this.hidden.delete(k) : this.hidden.add(k); el.classList.toggle('off'); this.draw(); });
|
||||
this.draw();
|
||||
}
|
||||
vis() { return this.series.filter(s => !this.hidden.has(s.name) && s.pts.length); }
|
||||
draw() {
|
||||
const cv = this.cv, dpr = devicePixelRatio || 1, W = cv.clientWidth, H = cv.clientHeight;
|
||||
if (!W) return;
|
||||
cv.width = W * dpr; cv.height = H * dpr;
|
||||
const g = cv.getContext('2d'); g.scale(dpr, dpr); g.clearRect(0, 0, W, H);
|
||||
const S = this.vis(), L = 54, R = 14, T = 10, B = 24;
|
||||
g.font = '11px ui-monospace, Menlo, Consolas, monospace';
|
||||
if (!S.length) { g.fillStyle = css('--dim2'); g.textAlign = 'center'; g.fillText(this.o.emptyText || '暂无数据', W / 2, H / 2); this.map = null; return; }
|
||||
const logy = this.o.logy && this.o.logy();
|
||||
const ty = v => logy ? Math.log10(Math.max(v, 1e-12)) : v;
|
||||
let x0 = Infinity, x1 = -Infinity, y0 = Infinity, y1 = -Infinity;
|
||||
for (const s of S) for (const [x, y] of [...s.pts, ...(s.raw || [])]) { if (!isNum(y)) continue; x0 = Math.min(x0, x); x1 = Math.max(x1, x); if (s.noScale) continue; const v = ty(y); y0 = Math.min(y0, v); y1 = Math.max(y1, v); }
|
||||
if (this.o.xmax && this.o.xmax()) x1 = Math.max(x1, this.o.xmax());
|
||||
if (this.o.xmin0) x0 = Math.min(0, x0);
|
||||
if (this.o.range) [y0, y1] = this.o.range;
|
||||
if (this.o.zero && !logy) y0 = Math.min(0, y0);
|
||||
if (!isFinite(y0)) { y0 = 0; y1 = 1; }
|
||||
if (y1 - y0 < 1e-12) { y1 += 0.5; y0 -= 0.5; }
|
||||
if (!this.o.range) { const pad = (y1 - y0) * 0.08; y1 += pad; if (!(this.o.zero && !logy && y0 === 0)) y0 -= pad; }
|
||||
if (x1 === x0) x1 = x0 + 1;
|
||||
const X = x => L + (x - x0) / (x1 - x0) * (W - L - R), Y = v => T + (1 - (v - y0) / (y1 - y0)) * (H - T - B);
|
||||
this.map = { X, Y, x0, x1, L, R, W, ty };
|
||||
// 网格与刻度
|
||||
g.strokeStyle = '#1e2838'; g.lineWidth = 1; g.fillStyle = css('--dim2'); g.textAlign = 'right';
|
||||
for (let i = 0; i <= 4; i++) {
|
||||
const v = y0 + (y1 - y0) * i / 4, y = Math.round(Y(v)) + .5;
|
||||
g.beginPath(); g.moveTo(L, y); g.lineTo(W - R, y); g.stroke();
|
||||
g.fillText(logy ? fmt(10 ** v) : this.o.percent ? (100 * v).toFixed(0) + '%' : fmt(v), L - 8, y + 4);
|
||||
}
|
||||
g.textAlign = 'center';
|
||||
for (let i = 0; i <= 5; i++) { const x = x0 + (x1 - x0) * i / 5; const lab = x >= 1000 ? (x / 1000).toFixed(x1 - x0 < 5000 ? 1 : 0) + 'k' : x.toFixed(0); g.fillText(lab, Math.max(L + 10, Math.min(X(x), W - R - 12)), H - 6); }
|
||||
// 未到达区域（总步数之后）淡色
|
||||
const lastX = Math.max(...S.map(s => s.pts.length ? s.pts[s.pts.length - 1][0] : -Infinity));
|
||||
if (this.o.xmax && lastX < x1) { g.fillStyle = 'rgba(255,255,255,0.018)'; g.fillRect(X(lastX), T, X(x1) - X(lastX), H - T - B); }
|
||||
// 参考线
|
||||
for (const ref of this.o.refs || []) {
|
||||
const y = Y(ty(ref.y)); g.strokeStyle = hexA(ref.color, .7); g.setLineDash([4, 4]); g.beginPath(); g.moveTo(L, y); g.lineTo(W - R, y); g.stroke(); g.setLineDash([]);
|
||||
g.fillStyle = ref.color; g.textAlign = 'left'; g.fillText(ref.label, L + 6, y - 5);
|
||||
}
|
||||
const path = (pts) => { g.beginPath(); let f = true; for (const [x, y] of pts) { if (!isNum(y)) { f = true; continue; } const px = X(x), py = Y(ty(y)); f ? g.moveTo(px, py) : g.lineTo(px, py); f = false; } };
|
||||
for (const s of S) {
|
||||
if (s.raw) { path(s.raw); g.strokeStyle = hexA(s.color, .22); g.lineWidth = 1; g.stroke(); }
|
||||
if (s.area) {
|
||||
path(s.pts); const pts = s.pts.filter(p => isNum(p[1]));
|
||||
if (pts.length > 1) {
|
||||
g.lineTo(X(pts[pts.length - 1][0]), H - B); g.lineTo(X(pts[0][0]), H - B); g.closePath();
|
||||
const gr = g.createLinearGradient(0, T, 0, H - B); gr.addColorStop(0, hexA(s.color, .28)); gr.addColorStop(1, hexA(s.color, 0)); g.fillStyle = gr; g.fill();
|
||||
}
|
||||
}
|
||||
path(s.pts); g.strokeStyle = s.color; g.lineWidth = s.width || 2; g.lineJoin = 'round'; g.lineCap = 'round'; g.setLineDash(s.dash || []); g.stroke(); g.setLineDash([]);
|
||||
if (s.dots) for (const [x, y] of s.pts) if (isNum(y)) { g.beginPath(); g.arc(X(x), Y(ty(y)), 3, 0, 7); g.fillStyle = '#0b0f17'; g.fill(); g.lineWidth = 1.8; g.strokeStyle = s.color; g.stroke(); }
|
||||
if (s.best) { const [bx, by] = s.best; g.beginPath(); g.arc(X(bx), Y(ty(by)), 6, 0, 7); g.strokeStyle = s.color; g.lineWidth = 1.5; g.stroke(); g.fillStyle = s.color; g.textAlign = 'left'; g.fillText('最好', X(bx) + 9, Y(ty(by)) - 6); }
|
||||
}
|
||||
if (this.hx != null) { g.strokeStyle = '#8c99ad88'; g.setLineDash([3, 3]); g.beginPath(); g.moveTo(X(this.hx), T); g.lineTo(X(this.hx), H - B); g.stroke(); g.setLineDash([]);
|
||||
for (const s of S) { const p = nearest(s.pts, this.hx); if (p && isNum(p[1])) { g.beginPath(); g.arc(X(p[0]), Y(ty(p[1])), 4, 0, 7); g.fillStyle = s.color; g.fill(); } } }
|
||||
}
|
||||
hover(e) {
|
||||
if (!this.map) return;
|
||||
const r = this.cv.getBoundingClientRect(), { x0, x1, L, R, W } = this.map;
|
||||
const x = x0 + (e.clientX - r.left - L) / (W - L - R) * (x1 - x0);
|
||||
const rows = []; let hx = null;
|
||||
for (const s of this.vis()) { if (s.noLegend) continue; const p = nearest(s.pts, x); if (p) { hx = hx ?? p[0]; rows.push(`<span style="color:${s.color}">●</span> ${s.label || s.name} ${this.o.percent ? pct(p[1], 2) : fmt(p[1], 4)}`); } }
|
||||
this.hx = hx; this.draw();
|
||||
if (!rows.length) return;
|
||||
tip.innerHTML = `<div class="h">第 ${hx.toLocaleString()} 步</div>` + rows.join('<br>'); tip.style.display = 'block';
|
||||
tip.style.left = Math.min(e.clientX + 16, innerWidth - tip.offsetWidth - 10) + 'px'; tip.style.top = (e.clientY + 16) + 'px';
|
||||
}
|
||||
}
|
||||
function nearest(pts, x) { let lo = 0, hi = pts.length - 1; if (hi < 0) return null; while (hi - lo > 1) { const m = (lo + hi) >> 1; pts[m][0] < x ? lo = m : hi = m; } return Math.abs(pts[lo][0] - x) <= Math.abs(pts[hi][0] - x) ? pts[lo] : pts[hi]; }
|
||||
function spark(cv, pts, color) {
|
||||
const dpr = devicePixelRatio || 1, W = cv.clientWidth, H = cv.clientHeight; if (!W || pts.length < 2) return;
|
||||
cv.width = W * dpr; cv.height = H * dpr; const g = cv.getContext('2d'); g.scale(dpr, dpr);
|
||||
const ys = pts.map(p => p[1]).filter(isNum); const lo = Math.min(...ys), hi = Math.max(...ys) || 1, x0 = pts[0][0], x1 = pts[pts.length - 1][0] || 1;
|
||||
const X = x => (x - x0) / (x1 - x0 || 1) * W, Y = y => H - 4 - (y - lo) / ((hi - lo) || 1) * (H - 10);
|
||||
g.beginPath(); pts.forEach(([x, y], i) => i ? g.lineTo(X(x), Y(y)) : g.moveTo(X(x), Y(y)));
|
||||
g.strokeStyle = color; g.lineWidth = 1.6; g.stroke(); g.lineTo(W, H); g.lineTo(0, H); g.closePath();
|
||||
const gr = g.createLinearGradient(0, 0, 0, H); gr.addColorStop(0, hexA(color, .25)); gr.addColorStop(1, hexA(color, 0)); g.fillStyle = gr; g.fill();
|
||||
}
|
||||
const thin = (pts, n = 400) => { if (pts.length <= n) return pts; const k = pts.length / n, o = []; for (let i = 0; i < n; i++) o.push(pts[Math.floor(i * k)]); o.push(pts[pts.length - 1]); return o; };
|
||||
|
||||
/* ===================== 状态 ===================== */
|
||||
let cur = null, recs = [], evals = [], offset = 0, timer = null, charts = null, lastMtime = 0, lastStatus = null, runsCache = [];
|
||||
let logyOn = false, job = null, userPicked = false, conOff = 0;
|
||||
/* 阶段表：名称、说明、粗估耗时（小时，RTX 3090 + 28 核；只用于全流程百分比的权重） */
|
||||
const STAGE_META = [
|
||||
['setup', '安装依赖', 0.15], ['kin', '对拍', 0.02], ['gate1', '老师门禁', 0.45], ['gen', '生成数据', 1.0], ['prep', '打包样本', 0.4],
|
||||
['train', '预训练', 3.0], ['m5', 'M5 门禁', 0.25], ['apg', 'APG 微调', 3.0], ['mine', '挖错循环', 9.0], ['export', '导出 ONNX', 0.02],
|
||||
['bench', '部署评测', 1.0], ['accept', '验收汇总', 0.25]];
|
||||
const SM = Object.fromEntries(STAGE_META.map(([k, n, w]) => [k, { n, w }]));
|
||||
const smoothv = () => +$('smooth').value;
|
||||
const ema = (pts, a) => { if (!a) return pts; let m = null; return pts.map(([x, y]) => { if (!isNum(y)) return [x, y]; m = m == null ? y : a * m + (1 - a) * y; return [x, m]; }); };
|
||||
const xOf = r => r.step ?? r.it ?? 0;
|
||||
|
||||
function buildCharts(kind) {
|
||||
const c = $('charts'); c.innerHTML = '';
|
||||
const total = () => { const r = recs[recs.length - 1]; return r && r.total; };
|
||||
if (kind === 'apg') {
|
||||
charts = {
|
||||
a: new Chart(c, '到达误差（mm，展开末）', { logy: () => logyOn, xmax: total, xmin0: true, note: '越低越好 · 浅色为原始值' }),
|
||||
b: new Chart(c, '损失分量', { logy: () => logyOn, xmax: total, xmin0: true }),
|
||||
e: new Chart(c, '闭环评估（val，部署节奏）', { percent: true, range: [0, 1], xmax: total, xmin0: true, note: '每 500 步', emptyText: '第一次评估在第 500 步' }),
|
||||
t: new Chart(c, '展开长度与梯度', { xmax: total, xmin0: true, zero: true, note: '展开 60–150 次推理随批变化' }),
|
||||
};
|
||||
} else {
|
||||
charts = {
|
||||
a: new Chart(c, '总损失', { logy: () => logyOn, xmax: total, xmin0: true, note: '浅色为原始值' }),
|
||||
b: new Chart(c, '损失分量', { logy: () => logyOn, xmax: total, xmin0: true, hidden: ['smooth'] }),
|
||||
e: new Chart(c, '闭环评估（val，部署节奏，关安全层）', { percent: true, range: [0, 1], xmax: total, xmin0: true, note: kind === 'bench' ? '测速运行不评估' : '每 2000 步',
|
||||
emptyText: kind === 'bench' ? '测速运行不做闭环评估' : '第一次评估在第 2000 步' }),
|
||||
t: new Chart(c, '吞吐（样本/s）', { xmax: total, xmin0: true, zero: true, refs: [{ y: 5000, label: 'R10 通过线 5000', color: css('--bad') }] }),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function render() {
|
||||
if (!cur || !charts) return;
|
||||
const a = smoothv(), kind = cur.kind;
|
||||
const raw = k => thin(recs.filter(r => isNum(r[k])).map(r => [xOf(r), r[k]]));
|
||||
const S = (k, label, extra = {}) => ({ name: k, label: label || k, pts: thin(ema(recs.filter(r => isNum(r[k])).map(r => [xOf(r), r[k]]), a)), ...extra });
|
||||
if (kind === 'apg') {
|
||||
charts.a.set([S('err_mm_mean', '平均到达误差', { raw: raw('err_mm_mean'), area: true })]);
|
||||
charts.b.set([S('tgt', '到达项'), S('col', '碰撞铰链'), S('bc', 'BC 锚定'), S('smooth', '平滑 /10', { pts: thin(ema(recs.filter(r => isNum(r.smooth)).map(r => [xOf(r), r.smooth / 10]), a)) })]);
|
||||
charts.t.set([{ name: 'plans', label: '展开推理次数', pts: thin(recs.filter(r => r.plans).map(r => [xOf(r), r.plans])), width: 1.4 }, S('gnorm', '梯度范数 ×50', { pts: thin(ema(recs.filter(r => isNum(r.gnorm)).map(r => [xOf(r), r.gnorm * 50]), a)) })]);
|
||||
} else {
|
||||
charts.a.set([S('loss', '总损失', { raw: raw('loss'), area: true })]);
|
||||
charts.b.set([S('mf', cur.name.includes('wta') ? 'WTA' : 'MeanFlow'), S('fk_cm', 'FK（cm）'), S('value', '价值头'), S('sub', '子目标头'), S('smooth', '平滑')]);
|
||||
charts.t.set([S('samples_per_s', '样本/s', { area: true })]);
|
||||
}
|
||||
const E = (k, label, best) => { const pts = evals.filter(e => isNum(e[k])).map(e => [e.x, e[k]]); let b = null; if (best && pts.length) b = pts.reduce((m, p) => p[1] > m[1] ? p : m); return { name: k, label, dots: true, pts, best: b }; };
|
||||
charts.e.set([E('success', '到达率', true), E('collision_rate', '碰撞率'), E('time_ok_1p3', '≤ 老师时长×1.3'), E('start_fail', '起步失败')]);
|
||||
renderProgress(); renderKpis();
|
||||
}
|
||||
|
||||
function renderProgress() {
|
||||
const r = recs[recs.length - 1] || {}, kind = cur.kind;
|
||||
const step = xOf(r), total = r.total, done = total ? Math.min(1, step / total) : null;
|
||||
const t0 = recs.find(v => v.t)?.t, elapsed = r.t && t0 ? r.t - t0 : (isNum(r.min) ? r.min * 60 : null);
|
||||
let eta = isNum(r.eta_h) ? r.eta_h * 3600 : null;
|
||||
if (eta == null && elapsed && done) eta = elapsed / done * (1 - done);
|
||||
const running = !DEMO ? (Date.now() / 1000 - lastMtime < 120) : cur.live;
|
||||
const finished = done != null && done >= 0.999;
|
||||
const bestE = evals.length ? evals.reduce((m, e) => (e.success ?? -1) > (m.success ?? -1) ? e : m) : null;
|
||||
const marks = total ? evals.map(e => `<div class="mk ${bestE && e === bestE ? 'best' : ''}" style="left:${100 * e.x / total}%" title="评估 @${e.x}：到达 ${pct(e.success)}"></div>`).join('') : '';
|
||||
const [kl, kc] = KIND[kind];
|
||||
const fin = eta != null && running && !finished ? new Date(Date.now() + eta * 1000).toLocaleString('zh-CN', { hour12: false, month: 'numeric', day: 'numeric', hour: '2-digit', minute: '2-digit' }) : '—';
|
||||
const fact = (k, v, cls = '') => `<div class="fact"><div class="k">${k}</div><div class="v ${cls}">${v}</div></div>`;
|
||||
$('prog').innerHTML = `<h2><span>当前运行 · 训练曲线</span><span class="r">${finished ? '已完成' : running ? '运行中' : '未在运行'}</span></h2>
|
||||
<div class="prog-top"><div class="ttl"><b>${cur.name}</b><span class="tag ${kc}">${kl}</span>${cur.diverged ? ' <span class="chip bad">已发散 · diverged.pt</span>' : ''}</div>
|
||||
<div class="bigpct">${done != null ? (100 * done).toFixed(1) : '—'}<small>%</small></div></div>
|
||||
<div class="bar ${running && !finished ? '' : 'idle'}"><div class="fill" style="width:${done != null ? 100 * done : 0}%"></div>${marks}</div>
|
||||
<div class="facts">
|
||||
${fact('步数', `${step.toLocaleString()}${total ? ' / ' + total.toLocaleString() : ''}`)}
|
||||
${fact('已用时间', hms(elapsed))}
|
||||
${fact('剩余时间', finished ? '—' : hms(eta))}
|
||||
${fact('预计完成', fin)}
|
||||
${kind === 'apg' ? fact('本批展开', r.plans ? `${r.plans} 次推理` : '—') : fact('学习率', isNum(r.lr) ? r.lr.toExponential(2) : '—')}
|
||||
${fact('日志更新', lastMtime ? ago(lastMtime) : '—', running ? '' : 'warn')}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function renderKpis() {
|
||||
const r = recs[recs.length - 1] || {}, e = evals[evals.length - 1] || {}, pe = evals[evals.length - 2] || {}, kind = cur.kind;
|
||||
const best = evals.length ? Math.max(...evals.map(v => v.success ?? -1)) : null;
|
||||
const dlt = (k, good = 1) => { if (!isNum(e[k]) || !isNum(pe[k])) return ''; const d = (e[k] - pe[k]) * 100; if (Math.abs(d) < 0.05) return ''; const up = d > 0; return `<span class="delta ${(up ? 1 : -1) * good > 0 ? 'ok' : 'bad'}">${up ? '▲' : '▼'} ${Math.abs(d).toFixed(1)}pp</span>`; };
|
||||
const last = k => recs.filter(v => isNum(v[k])).slice(-300).map(v => [xOf(v), v[k]]);
|
||||
const evs = k => evals.filter(v => isNum(v[k])).map(v => [v.x, v[k]]);
|
||||
const tiles = [];
|
||||
const tile = (label, v, s, cls, sp, color) => tiles.push({ html: `<div class="card kpi"><h2><span>${label}</span></h2><div class="v ${cls || ''}">${v}</div><div class="s">${s || ' '}</div><canvas></canvas></div>`, sp, color });
|
||||
if (kind === 'apg') {
|
||||
tile('到达误差', `${fmt(r.err_mm_mean)}<small>mm</small>`, '展开末平均', '', last('err_mm_mean'), COLORS[0]);
|
||||
tile('展开内到达比例', pct(r.reach_frac), '误差 < 1 mm 且姿态 < 0.5°', '', last('reach_frac'), COLORS[2]);
|
||||
tile('闭环到达率', pct(e.success) + dlt('success'), `最好 ${pct(best)}`, '', evs('success'), COLORS[2]);
|
||||
tile('碰撞率', pct(e.collision_rate, 2) + dlt('collision_rate', -1), '关安全层', e.collision_rate > 0.03 ? 'warn' : '', evs('collision_rate'), COLORS[3]);
|
||||
} else {
|
||||
const sps = r.samples_per_s;
|
||||
tile('总损失', fmt(r.loss, 4), `MF ${fmt(r.mf)} · FK ${fmt(r.fk_cm)} cm`, '', ema(last('loss'), 0.9), COLORS[0]);
|
||||
tile('吞吐', sps ? `${(sps / 1000).toFixed(2)}<small>k 样本/s</small>` : '—', sps ? (sps >= 5000 ? 'R10 达标（≥ 5k/s）' : 'R10 未达标（< 5k/s）') : 'R10 通过线 5k/s', sps ? (sps >= 5000 ? 'ok' : 'bad') : '', last('samples_per_s'), COLORS[5]);
|
||||
tile('闭环到达率', pct(e.success) + dlt('success'), evals.length ? `最好 ${pct(best)} · @${e.x.toLocaleString()} 步` : '尚未评估', '', evs('success'), COLORS[2]);
|
||||
tile('碰撞率', pct(e.collision_rate, 2) + dlt('collision_rate', -1), '关安全层，部署另有护栏', e.collision_rate > 0.03 ? 'warn' : '', evs('collision_rate'), COLORS[3]);
|
||||
tile('梯度范数', fmt(r.gnorm), '裁剪阈值 1.0', r.gnorm > 5 ? 'warn' : '', ema(last('gnorm'), 0.9), COLORS[4]);
|
||||
}
|
||||
$('kpis').innerHTML = tiles.map(t => t.html).join('');
|
||||
[...$('kpis').querySelectorAll('canvas')].forEach((cv, i) => spark(cv, tiles[i].sp, tiles[i].color));
|
||||
}
|
||||
|
||||
/* ===================== 轮询 ===================== */
|
||||
async function loadRuns() {
|
||||
const runs = await api('/api/runs'); runsCache = runs;
|
||||
const sel = $('run'), prev = sel.value;
|
||||
sel.innerHTML = runs.map(r => `<option value="${r.name}">${r.name} · ${KIND[r.kind][0]}${r.diverged ? ' · 发散' : ''}</option>`).join('') || '<option value="">ml/runs 下还没有日志</option>';
|
||||
const follow = lastStatus && lastStatus.pipeline && lastStatus.pipeline.progress && lastStatus.pipeline.progress.run;
|
||||
if (!userPicked) sel.value = runs.some(r => r.name === follow) ? follow : (runs[0] ? runs[0].name : ''); // 自动跟随正在跑的那条
|
||||
else if (runs.some(r => r.name === prev)) sel.value = prev;
|
||||
const r = runs.find(v => v.name === sel.value) || runs[0];
|
||||
if (r && (!cur || cur.name !== r.name)) { cur = { ...r }; recs = []; evals = []; offset = 0; buildCharts(r.kind); }
|
||||
else if (r) Object.assign(cur, r);
|
||||
if (!r) { cur = null; $('charts').innerHTML = ''; $('kpis').innerHTML = ''; $('prog').innerHTML = '<h2><span>当前运行 · 训练曲线</span></h2>' + empty('还没有训练日志：预训练开始后这里出现损失、吞吐和评估曲线。'); }
|
||||
}
|
||||
|
||||
async function poll() {
|
||||
try {
|
||||
lastStatus = await api('/api/status');
|
||||
job = await api('/api/job').catch(() => null);
|
||||
await loadRuns();
|
||||
if (cur) {
|
||||
const d = await api(`/api/log?run=${encodeURIComponent(cur.name)}&offset=${offset}`);
|
||||
if (d.reset && offset) { recs = []; evals = []; }
|
||||
for (const r of d.records || []) {
|
||||
if (r.eval) { const e = r.eval; evals.push({ ...e, x: e.step ?? e.it ?? r.step ?? 0 }); } else recs.push(r);
|
||||
}
|
||||
offset = d.offset ?? offset; lastMtime = d.mtime || lastMtime; cur.live = d.live;
|
||||
render();
|
||||
}
|
||||
const P = lastStatus.pipeline, busy = (job && job.running) || (P && P.current);
|
||||
if (DEMO) setPill('demo', '示例数据');
|
||||
else if (busy) setPill('live', `运行中 · ${P && P.current ? SM[P.current]?.n || P.current : '启动中'}`);
|
||||
else if (cur && Date.now() / 1000 - lastMtime < 120) setPill('live', `运行中 · ${ago(lastMtime)}更新`);
|
||||
else setPill('', '空闲');
|
||||
renderStatus(lastStatus); renderPipe(lastStatus); renderJob();
|
||||
await pollConsole();
|
||||
} catch (err) {
|
||||
setPill('dead', '连不上控制台（双击 ml/console.bat 或 python -m ml.monitor）'); console.error(err);
|
||||
}
|
||||
}
|
||||
function setPill(cls, text) { const st = $('state'); st.className = 'pill ' + cls; st.lastElementChild.textContent = text; }
|
||||
|
||||
/* ===================== 流水线 / GPU / 各阶段面板 ===================== */
|
||||
function stages(s) {
|
||||
const P = s.pipeline;
|
||||
if (P) { // 有流水线记录：以它为准
|
||||
const plan = P.plan && P.plan.length ? P.plan : STAGE_META.map(x => x[0]).filter(k => k !== 'setup');
|
||||
const keys = STAGE_META.map(x => x[0]).filter(k => plan.includes(k) || P.stages[k]);
|
||||
return keys.map(k => {
|
||||
const r = P.stages[k], nm = SM[k].n;
|
||||
if (!r) return [nm, 'pending', '—'];
|
||||
if (P.current === k) return [nm, 'active', P.progress && P.progress.frac != null ? pct(P.progress.frac, 0) : '进行中'];
|
||||
if (r.rc === 0) return [nm, 'done', '用时 ' + hmsShort(r.end - r.start)];
|
||||
if (r.rc === -15 || r.rc === 143) return [nm, 'fail', '已停止'];
|
||||
if (r.rc != null) return [nm, 'fail', `失败（返回码 ${r.rc}）`];
|
||||
return [nm, 'fail', '中断'];
|
||||
});
|
||||
}
|
||||
const D = s.data || {}, has = n => runsCache.find(r => r.name === n || r.name.startsWith(n));
|
||||
const live = n => { const r = has(n); return r && Date.now() / 1000 - r.mtime < 120; };
|
||||
const anyTrain = runsCache.find(r => r.kind === 'train');
|
||||
const m5ok = s.m5 && s.m5.some(m => m.pass), m5fail = s.m5 && s.m5.length && !m5ok;
|
||||
const mineN = (s.mine || []).length, mineDone = (s.mine || []).filter(m => m.gate && m.gate.accept != null).length;
|
||||
const L = [
|
||||
['对拍', D.kin_pass != null ? (D.kin_pass ? 'done' : 'fail') : (D.consistency != null ? 'done' : 'pending'), '—'],
|
||||
['老师门禁', D.consistency != null ? 'done' : 'pending', D.consistency != null ? `一致率 ${pct(D.consistency, 0)}` : '—'],
|
||||
['生成数据', D.stats ? 'done' : D.gen_tasks_written ? 'active' : 'pending', D.stats ? `${(D.stats.tasks / 1000).toFixed(1)}k 条` : D.gen_tasks_written ? `${D.gen_tasks_written.toLocaleString()} 条…` : '—'],
|
||||
['打包样本', D.index ? 'done' : 'pending', D.index ? `${(D.index.train_samples / 1e6).toFixed(2)}M 样本` : '—'],
|
||||
['预训练', s.m5 && s.m5.length ? 'done' : anyTrain ? (live(anyTrain.name) ? 'active' : 'done') : 'pending', anyTrain ? anyTrain.name : '—'],
|
||||
['M5 门禁', m5ok ? 'done' : m5fail ? 'fail' : 'pending', m5ok ? '通过' : m5fail ? '未通过' : '—'],
|
||||
['APG 微调', has('apg') && !has('apg_r') ? (live('apg') ? 'active' : 'done') : has('apg_r') || mineN ? 'done' : 'pending', has('apg') ? 'apg' : '—'],
|
||||
['挖错循环', mineN ? (mineDone === mineN && !live('apg_r') ? 'done' : 'active') : 'pending', mineN ? `第 ${mineN} 轮` : '—'],
|
||||
['导出/评测', s.bench ? 'done' : 'pending', s.bench ? '已评测' : '—'],
|
||||
['验收汇总', s.acceptance ? (s.acceptance.all_pass ? 'done' : 'fail') : 'pending', s.acceptance ? (s.acceptance.all_pass ? '全部通过' : `${s.acceptance.failed.length} 项未过`) : '—'],
|
||||
];
|
||||
return L;
|
||||
}
|
||||
function renderStatus(s) {
|
||||
const L = stages(s);
|
||||
$('steps').innerHTML = L.map(([nm, st, ds], i) => `<div class="step ${st}"><div class="ic">${st === 'done' ? '✓' : st === 'fail' ? '!' : i + 1}</div><div class="nm">${nm}</div><div class="ds">${ds}</div></div>`).join('');
|
||||
const nDone = L.filter(x => x[1] === 'done').length; $('stepnote').textContent = `${nDone} / ${L.length} 完成`;
|
||||
|
||||
// GPU
|
||||
const G = s.gpu || [];
|
||||
$('gputime').textContent = G.length ? '实时' : '';
|
||||
$('gpu').innerHTML = G.length ? G.map(g => {
|
||||
const u = g.util ?? 0, C = 2 * Math.PI * 50, col = u > 85 ? css('--ok') : u > 50 ? css('--accent') : css('--warn');
|
||||
const memP = 100 * g.mem_used / g.mem_total, tc = g.temp > 83 ? css('--bad') : g.temp > 75 ? css('--warn') : css('--c6');
|
||||
const row = (k, v, p, c) => `<div class="gr"><div class="l"><span>${k}</span><b>${v}</b></div><div class="mini"><div style="width:${Math.min(100, p)}%;background:${c}"></div></div></div>`;
|
||||
return `<div class="gpuwrap"><div class="ring"><svg width="116" height="116"><circle cx="58" cy="58" r="50" stroke="#1b2433" stroke-width="10" fill="none"/>
|
||||
<circle cx="58" cy="58" r="50" stroke="${col}" stroke-width="10" fill="none" stroke-linecap="round" stroke-dasharray="${C}" stroke-dashoffset="${C * (1 - u / 100)}" style="transition:stroke-dashoffset .6s"/></svg>
|
||||
<div class="c"><div><b>${u}%</b><span>利用率</span></div></div></div>
|
||||
<div class="gpurows">${row('显存', `${(g.mem_used / 1024).toFixed(1)} / ${(g.mem_total / 1024).toFixed(0)} GB`, memP, css('--accent2'))}
|
||||
${row('温度', `${g.temp} °C`, g.temp, tc)}${row('功耗', g.power != null ? `${Math.round(g.power)} W` : '—', g.power ? g.power / 3.5 : 0, css('--c7'))}</div></div>
|
||||
<div class="gname">${g.name}</div>`;
|
||||
}).join('<hr style="border:none;border-top:1px solid var(--line);margin:14px 0">') : empty('未检测到 nvidia-smi');
|
||||
|
||||
// 数据
|
||||
const D = s.data || {}, st = D.stats, ix = D.index;
|
||||
const row = (k, v, cls = '') => `<tr><td>${k}</td><td class="n ${cls}">${v}</td></tr>`;
|
||||
if (D.consistency == null && !st && !ix && !D.gen_tasks_written) $('data').innerHTML = empty('还没开始生成数据');
|
||||
else {
|
||||
let h = '<table>';
|
||||
h += row('老师一致性', D.consistency == null ? '—' : pct(D.consistency), D.consistency == null ? '' : D.consistency >= 0.9 ? 'ok' : 'warn');
|
||||
h += row('已生成任务', D.gen_tasks_written ? `${D.gen_tasks_written.toLocaleString()} 条 · ${D.shards} 个分片` : '—');
|
||||
if (st) {
|
||||
h += row('保留率', fmt(st.keepRate), st.keepRate >= 0.75 ? 'ok' : 'bad');
|
||||
if (st.gapDrop != null) h += row(`间隙 < ${st.minGapFilterMm ?? 3} mm 剔除`, `${st.gapDrop.toLocaleString()}（${pct(st.gapDrop / Math.max(1, st.keep + st.gapDrop))}）`, st.gapDrop / Math.max(1, st.keep + st.gapDrop) > 0.25 ? 'bad' : '');
|
||||
h += row('样本数（每 3 步）', (st.samplesEvery3 || 0).toLocaleString(), st.samplesEvery3 >= 1.7e6 ? 'ok' : 'warn');
|
||||
if (st.consensusDrop != null) h += row('共识丢弃', st.consensusDrop.toLocaleString());
|
||||
if (st.vertexMinGapMm) h += row('最小间隙 p10 / p50 / p90', `${st.vertexMinGapMm.p10 ?? '—'} / ${st.vertexMinGapMm.p50} / ${st.vertexMinGapMm.p90} mm`);
|
||||
}
|
||||
if (ix) h += row('训练 / 验证样本', `${ix.train_samples.toLocaleString()} / ${ix.val_samples.toLocaleString()}`);
|
||||
if (D.r2) h += row('R2 近邻标签 std', D.r2.pass == null ? '未判定' : `${fmt(D.r2.label_std_mean)}（< 0.10）`, D.r2.pass == null ? 'dim' : D.r2.pass ? 'ok' : 'bad');
|
||||
$('data').innerHTML = h + '</table>';
|
||||
}
|
||||
|
||||
$('m5').innerHTML = s.m5 && s.m5.length ? '<table><tr><th>动作头</th><th class="n">纯策略穿柱</th><th class="n">到达</th><th class="n">加安全层穿柱</th><th class="n">结论</th></tr>' + s.m5.map(m =>
|
||||
`<tr><td>${m.head}</td><td class="n ${m.policy_only.collision_rate <= 0.02 ? 'ok' : 'bad'}">${pct(m.policy_only.collision_rate, 2)}</td><td class="n">${pct(m.policy_only.success)}</td><td class="n">${pct(m.with_sim_safety.collision_rate, 2)}</td><td class="n"><span class="chip ${m.pass ? 'ok' : 'bad'}">${m.pass ? '通过' : '不通过'}</span></td></tr>`).join('')
|
||||
+ `</table><div class="dim" style="margin-top:8px;font-size:12px">选定模型：<span style="font-family:var(--mono)">${s.selected || '—'}</span></div>` : empty('预训练完成后运行 m5');
|
||||
|
||||
$('mine').innerHTML = s.mine && s.mine.length ? '<table><tr><th>轮</th><th class="n">E1 撞</th><th class="n">E2 极小</th><th class="n">正确</th><th class="n">提升</th><th class="n">R7 左右</th><th class="n">门禁</th></tr>' + s.mine.map(m => {
|
||||
const r = m.report || {}, g = m.gate || {};
|
||||
const chip = g.accept == null ? '<span class="chip dim">进行中</span>' : g.accept ? '<span class="chip ok">接受</span>' : '<span class="chip bad">回滚</span>';
|
||||
return `<tr><td>${m.round}</td><td class="n">${pct(r.E1, 2)}</td><td class="n">${pct(r.E2, 2)}</td><td class="n">${pct(r.ok)}</td><td class="n">${isNum(g.gain_pp) ? '+' + g.gain_pp.toFixed(1) + 'pp' : '—'}</td><td class="n">${isNum(g.R7_left_share_shift_pp) ? g.R7_left_share_shift_pp.toFixed(1) + 'pp' : '—'}</td><td class="n">${chip}</td></tr>`;
|
||||
}).join('') + '</table>' : empty('APG 之后开始挖错');
|
||||
|
||||
$('bench').innerHTML = s.bench ? '<table><tr><th>集合</th><th class="n">通过</th><th class="n">穿模帧</th><th class="n">刹停率</th><th class="n">推理 P99</th></tr>' + Object.entries(s.bench).map(([k, v]) =>
|
||||
`<tr><td>${k}</td><td class="n ${v.ok === v.n ? 'ok' : 'bad'}">${v.ok}/${v.n}</td><td class="n ${v.collisionFrames ? 'bad' : 'ok'}">${v.collisionFrames}</td><td class="n">${pct(v.brakeRate, 2)}</td><td class="n">${v.inferMsP99 ?? '—'} ms</td></tr>`).join('') + '</table>' : empty('导出 ONNX 后运行 bench');
|
||||
|
||||
const A = s.acceptance;
|
||||
$('accept').innerHTML = A ? `<div class="accbanner ${A.all_pass ? 'ok' : 'bad'}">${A.all_pass ? '✓ 全部通过' : `✕ ${A.failed.length} 项未通过`}</div><table>` + A.items.map(i =>
|
||||
`<tr><td>${i.item}</td><td class="n">${typeof i.value === 'number' ? (/率|比例/.test(i.item) && i.value <= 1 ? pct(i.value, 1) : fmt(i.value)) : (Array.isArray(i.value) || i.value == null ? '' : i.value)}</td><td class="n"><span class="chip ${i.pass === true ? 'ok' : i.pass === false ? 'bad' : 'dim'}">${i.pass === true ? '通过' : i.pass === false ? '未过' : '—'}</span></td></tr>`).join('') + '</table>' : empty('最后一步 accept');
|
||||
|
||||
$('foot').textContent = `${DEMO ? '示例数据 · ' : ''}每 5 秒刷新 · 网页可随时关闭，训练不受影响 · ${new Date().toLocaleTimeString('zh-CN', { hour12: false })}`;
|
||||
}
|
||||
|
||||
/* ===================== 全流程卡片 / 开始停止 / 终端 ===================== */
|
||||
function renderPipe(s) {
|
||||
const P = s.pipeline, running = job && job.running;
|
||||
const btns = `<div class="ctlrow">${running || (P && P.current)
|
||||
? `<button class="danger" onclick="stopJob()">■ 停止</button><span class="hint">${running ? `PID ${job.pid} · 从「${SM[job.from]?.n}」到「${SM[job.to]?.n}」` : '由终端启动，需在终端里结束'}</span>`
|
||||
: `<button class="primary" onclick="openStart()">▶ ${P ? '继续 / 重新开始' : '开始训练'}</button><span class="hint">${job && job.rc != null ? (job.stopped ? '上次已手动停止' : job.rc === 0 ? '上次运行已全部完成' : `上次运行失败（返回码 ${job.rc}），看终端输出`) : '点击后选择起止阶段与参数'}</span>`}</div>`;
|
||||
if (!P) { $('pipe').innerHTML = `<h2><span>全流程</span><span class="r">尚未开始</span></h2>${empty('还没运行过流水线。点「开始训练」从对拍一直跑到验收；中途每一步的进度都显示在这里。')}${btns}`; return; }
|
||||
const plan = P.plan && P.plan.length ? P.plan : Object.keys(P.stages);
|
||||
let tot = 0, got = 0;
|
||||
for (const k of plan) { const w = SM[k]?.w || 0.1, r = P.stages[k]; tot += w; if (r && r.rc === 0 && P.current !== k) got += w; else if (P.current === k) got += w * ((P.progress && P.progress.frac) || 0); }
|
||||
const frac = tot ? got / tot : 0, now = Date.now() / 1000;
|
||||
const cur = P.current, cr = cur && P.stages[cur], pr = P.progress || {};
|
||||
const doneN = plan.filter(k => P.stages[k] && P.stages[k].rc === 0 && P.current !== k).length;
|
||||
const failed = plan.find(k => P.stages[k] && P.stages[k].rc != null && P.stages[k].rc !== 0);
|
||||
const allDone = !cur && doneN === plan.length;
|
||||
let remain = 0; for (const k of plan) { const r = P.stages[k]; if (!r || r.rc == null) remain += (SM[k]?.w || 0.1) * (k === cur ? 1 - (pr.frac || 0) : 1); }
|
||||
const head = cur ? `<div class="now">正在运行<b>${SM[cur]?.n || cur}</b>${pr.label || ''}</div>`
|
||||
: allDone ? `<div class="now">状态<b class="ok">全部完成</b>共 ${plan.length} 步</div>`
|
||||
: failed ? `<div class="now">状态<b class="bad">停在「${SM[failed].n}」</b>${P.stages[failed].rc === -15 || P.stages[failed].rc === 143 ? '已手动停止' : '失败，看终端输出'}</div>`
|
||||
: `<div class="now">状态<b>未在运行</b>已完成 ${doneN} / ${plan.length} 步</div>`;
|
||||
$('pipe').innerHTML = `<h2><span>全流程</span><span class="r">${doneN} / ${plan.length} 步完成</span></h2>
|
||||
<div class="overall">${head}<div class="bigpct">${(100 * frac).toFixed(1)}<small>%</small></div></div>
|
||||
<div class="bar ${cur ? '' : 'idle'}"><div class="fill" style="width:${100 * frac}%"></div></div>
|
||||
${cur ? `<div class="sublabel"><span>当前步骤 · ${SM[cur]?.n}</span><span>${pr.frac != null ? pct(pr.frac) : '无法计量，按用时估'}</span></div>
|
||||
<div class="bar thin ${pr.frac == null ? 'ind' : ''}"><div class="fill" style="width:${pr.frac != null ? 100 * pr.frac : 30}%"></div></div>` : ''}
|
||||
<div class="facts">
|
||||
<div class="fact"><div class="k">总用时</div><div class="v">${hms((cur ? now : Math.max(...Object.values(P.stages).map(r => r.end || r.start))) - P.started)}</div></div>
|
||||
<div class="fact"><div class="k">本步用时</div><div class="v">${cr ? hms(now - cr.start) : '—'}</div></div>
|
||||
<div class="fact"><div class="k">本步粗估</div><div class="v">${cur ? hmsShort((SM[cur]?.w || 0) * 3600) : '—'}</div></div>
|
||||
<div class="fact"><div class="k">全流程剩余（粗估）</div><div class="v">${allDone ? '—' : '约 ' + hmsShort(remain * 3600)}</div></div>
|
||||
</div>${btns}`;
|
||||
}
|
||||
function renderJob() {
|
||||
const busy = (job && job.running) || (lastStatus && lastStatus.pipeline && lastStatus.pipeline.current);
|
||||
$('btnStart').style.display = busy ? 'none' : ''; $('btnStop').style.display = busy ? '' : 'none';
|
||||
}
|
||||
async function pollConsole() {
|
||||
const d = await api(`/api/console?offset=${conOff}`).catch(() => null); if (!d) return;
|
||||
const t = $('term'), atBottom = t.scrollTop + t.clientHeight >= t.scrollHeight - 30;
|
||||
if (d.reset) t.innerHTML = '';
|
||||
if (d.text) {
|
||||
const esc = x => x.replace(/&/g, '&').replace(/</g, '<');
|
||||
t.insertAdjacentHTML('beforeend', d.text.split('\n').map(l => /error|Error|失败|Traceback|不通过|Terminated/.test(l) ? `<span class="e">${esc(l)}</span>` : /^== |^\$ /.test(l) ? `<span class="c">${esc(l)}</span>` : /PASS|通过|完成/.test(l) ? `<span class="g">${esc(l)}</span>` : esc(l)).join('\n'));
|
||||
if (t.textContent.length > 400000) t.textContent = t.textContent.slice(-200000);
|
||||
}
|
||||
if (!t.textContent) t.innerHTML = '<span style="color:var(--dim2)">点「开始训练」后，这里实时显示命令行输出。</span>';
|
||||
conOff = d.offset; if (atBottom || d.reset) t.scrollTop = t.scrollHeight;
|
||||
}
|
||||
function toast(msg) { const t = $('toast'); t.textContent = msg; t.style.display = 'block'; clearTimeout(toast.h); toast.h = setTimeout(() => t.style.display = 'none', 3500); }
|
||||
function openStart() {
|
||||
const P = lastStatus && lastStatus.pipeline, keys = STAGE_META.map(x => x[0]);
|
||||
const opt = keys.map(k => `<option value="${k}">${SM[k].n}（${k}）</option>`).join('');
|
||||
$('fFrom').innerHTML = opt; $('fTo').innerHTML = opt;
|
||||
// 默认从第一个还没成功完成的阶段开始，到验收结束
|
||||
let from = 'kin';
|
||||
if (P) { const k = keys.slice(1).find(k => !(P.stages[k] && P.stages[k].rc === 0)); if (k) from = k; }
|
||||
$('fFrom').value = from; $('fTo').value = 'accept';
|
||||
const w = $('fWarn'); const upd = () => { const f = $('fFrom').value; const msgs = [];
|
||||
if (P && P.stages[f] && P.stages[f].rc === 0) msgs.push(`「${SM[f].n}」上次已成功完成，重跑会覆盖它的产物。`);
|
||||
if (keys.indexOf(f) > keys.indexOf('gen') && !(lastStatus.data && lastStatus.data.stats)) msgs.push('还没有生成数据，建议从「老师门禁」或「生成数据」开始。');
|
||||
if (keys.indexOf(f) > keys.indexOf($('fTo').value)) msgs.push('起始阶段在结束阶段之后。');
|
||||
w.style.display = msgs.length ? 'block' : 'none'; w.innerHTML = msgs.join('<br>'); };
|
||||
$('fFrom').onchange = upd; $('fTo').onchange = upd; upd();
|
||||
$('modal').classList.add('show');
|
||||
}
|
||||
async function startJob() {
|
||||
const env = { CFG: $('fCfg').value, CONSENSUS: $('fCons').value, N_TASKS: $('fN').value.trim(), NPROC: $('fP').value.trim(), TARGET_SAMPLES: $('fT').value.trim(), ROUNDS: $('fR').value.trim() };
|
||||
const body = JSON.stringify({ from: $('fFrom').value, to: $('fTo').value, env });
|
||||
const r = DEMO ? await api('/api/start', body) : await fetch('/api/start', { method: 'POST', body }).then(x => x.json()).catch(e => ({ error: String(e) }));
|
||||
if (r.error) { toast('没能开始：' + r.error); return; }
|
||||
$('modal').classList.remove('show'); toast('已开始'); conOff = 0; userPicked = false; poll();
|
||||
}
|
||||
async function stopJob() {
|
||||
if (!confirm('确定停止？正在运行的这一步会中断，已完成的步骤保留，之后可以从这一步继续。')) return;
|
||||
const r = DEMO ? await api('/api/stop', '{}') : await fetch('/api/stop', { method: 'POST' }).then(x => x.json()).catch(e => ({ error: String(e) }));
|
||||
toast(r.error ? '没能停止：' + r.error : '已停止'); poll();
|
||||
}
|
||||
$('btnStart').onclick = openStart; $('btnStop').onclick = stopJob;
|
||||
$('fCancel').onclick = () => $('modal').classList.remove('show'); $('fGo').onclick = startJob;
|
||||
$('modal').onclick = e => { if (e.target === $('modal')) $('modal').classList.remove('show'); };
|
||||
|
||||
/* ===================== 控件 ===================== */
|
||||
$('smooth').oninput = () => { $('smoothv').textContent = smoothv().toFixed(2); render(); };
|
||||
$('logy').onclick = () => { logyOn = !logyOn; $('logy').classList.toggle('on', logyOn); render(); };
|
||||
$('run').onchange = () => { userPicked = true; cur = null; poll(); };
|
||||
$('scenario').onchange = () => { api.scenario($('scenario').value); cur = null; userPicked = false; conOff = 0; $('run').innerHTML = ''; poll(); };
|
||||
$('auto').onclick = () => { const on = !$('auto').classList.contains('on'); $('auto').classList.toggle('on', on); clearInterval(timer); if (on) timer = setInterval(poll, 5000); };
|
||||
timer = setInterval(poll, 5000); poll();
|
||||
|
||||
/* =====================================================================
|
||||
示例数据：两种场景，每次刷新让"运行中"的那条日志向前推进。
|
||||
===================================================================== */
|
||||
function demoApi() {
|
||||
let seed = 7; const rnd = () => (seed = (seed * 16807) % 2147483647) / 2147483647;
|
||||
const gauss = () => { let u = 0; for (let i = 0; i < 6; i++) u += rnd(); return (u - 3) * 1.41; };
|
||||
const now = () => Date.now() / 1000;
|
||||
const T0 = now() - 2.05 * 3600;
|
||||
function trainRec(st, total, sps, t0, wta) {
|
||||
const lr = 1e-3 * (st < 1000 ? st / 1000 : 0.5 * (1 + Math.cos(Math.PI * (st - 1000) / (total - 1000))));
|
||||
const mf = (wta ? 0.42 : 0.55) * Math.exp(-st / 9000) + 0.12 + 0.012 * gauss(), fk = 2.6 * Math.exp(-st / 6000) + 0.35 + 0.04 * gauss();
|
||||
const sm = 3.2 * Math.exp(-st / 4000) + 0.6 + 0.06 * gauss(), val = 0.4 * Math.exp(-st / 5000) + 0.05 + 0.006 * gauss(), sub = 0.25 * Math.exp(-st / 7000) + 0.04 + 0.004 * gauss();
|
||||
const sp = sps * (1 - 0.18 * Math.exp(-st / 250)) * (1 + 0.012 * gauss());
|
||||
return { step: st, total, t: t0 + st * 1024 / sps, loss: mf + 0.5 * fk + 0.05 * sm + 0.1 * val + 0.1 * sub, mf, fk_cm: fk, smooth: sm, value: val, sub,
|
||||
gnorm: Math.max(0.15, 0.55 + 2.4 * Math.exp(-st / 2500) + 0.12 * gauss()), lr, samples_per_s: sp, eta_h: (total - st) * 1024 / sps / 3600 };
|
||||
}
|
||||
function trainEval(x, t0, sps) {
|
||||
const q = 1 - Math.exp(-x / 14000);
|
||||
return { eval: { step: x, t: t0 + x * 1024 / sps, n: 512, n_ref: 512, success: Math.min(0.975, 0.08 + 0.87 * q + 0.012 * gauss()), collision_rate: Math.max(0.004, 0.31 * (1 - q) + 0.012 + 0.005 * gauss()),
|
||||
time_ok_1p3: Math.min(0.95, 0.05 + 0.8 * q + 0.012 * gauss()), start_fail: Math.max(0, 0.05 * (1 - q) + 0.003 * gauss()) } };
|
||||
}
|
||||
function trainLog(total, from, to, sps, t0, evalEvery = 2000, wta = false) {
|
||||
const L = [];
|
||||
for (let st = from; st <= to; st += 50) { L.push(trainRec(st, total, sps, t0, wta)); if (evalEvery && st > 0 && st % evalEvery === 0) L.push(trainEval(st, t0, sps)); }
|
||||
return L;
|
||||
}
|
||||
function apgRec(it, total, t0) {
|
||||
const q = 1 - Math.exp(-it / 900);
|
||||
return { it, total, t: t0 + it * 3.1, plans: 60 + Math.floor(rnd() * 45), err_mm_mean: Math.max(3, 140 * (1 - q) + 9 + 5 * gauss()), reach_frac: Math.min(1, 0.35 + 0.52 * q + 0.03 * gauss()),
|
||||
bc: 0.9 - 0.15 * q + 0.02 * gauss(), gnorm: Math.max(0.2, 1.2 + 0.25 * gauss()), col: Math.max(0, 2.4 * (1 - q) + 0.15 + 0.06 * gauss()), tgt: Math.max(0, 3.5 * (1 - q) + 0.2 + 0.08 * gauss()), smooth: 40 - 12 * q + 1.5 * gauss() };
|
||||
}
|
||||
function apgLog(total, from, to, t0, base = 0.93) {
|
||||
const L = [];
|
||||
for (let it = from; it <= to; it += 10) { L.push(apgRec(it, total, t0)); if (it > 0 && it % 500 === 0) { const q = 1 - Math.exp(-it / 900); L.push({ eval: { it, success: base + 0.035 * q + 0.004 * gauss(), collision_rate: 0.028 * (1 - q) + 0.006, time_ok_1p3: 0.86 + 0.06 * q, start_fail: 0.001 } }); } }
|
||||
return L;
|
||||
}
|
||||
let sc, runs, logs, pipe, jobS, con, genAt = 0;
|
||||
// 示例流水线：stages 里的 [阶段, 开始距今小时, 用时小时]
|
||||
const mkPipe = (done, cur, curStartH) => { const t = now(); const st = {}; let tt = t - curStartH * 3600;
|
||||
for (let i = done.length - 1; i >= 0; i--) { const [k, dur] = done[i]; st[k] = { start: tt - dur * 3600, end: tt, rc: 0, info: {} }; tt -= dur * 3600; }
|
||||
if (cur) st[cur] = { start: t - curStartH * 3600, end: null, rc: null, info: {} };
|
||||
return { started: tt, plan: ['kin', 'gate1', 'gen', 'prep', 'train', 'm5', 'apg', 'mine', 'export', 'bench', 'accept'], stages: st, current: cur }; };
|
||||
const CON = {
|
||||
train: '$ FROM=kin TO=accept bash ml/pipeline.sh all\n计划：kin gate1 gen prep train m5 apg mine export bench accept\n== node ml/prep/export_kin.cjs\n== node ml/eval/dump_parity.cjs 2000\n== python -m ml.eval.check_parity\n{"PASS": true, "fk_mm": 0.0004, "gap_mm": 0.0011}\n== node ml/gen/consistency.cjs 300 4242\n{"consistentByGateRule": 0.86}\n一致性门禁结果 → CONSENSUS=1\n== bash ml/gen/run_parallel.sh 1786 28 1 --consensus 4\n样本数 1702000 ≥ 目标，完成\n== bash ml/prep/run_prep.sh 28\n== python -m ml.eval.check_r2 --samples ml/data/samples --out ml/data/r2.json\n{"pairs": 250, "label_std_mean": 0.071, "pass": true}\n== python -m ml.train --samples ml/data/samples --out ml/runs/bench_base --steps 2000 --no-eval\nstep 2000 loss 1.912 6150 samples/s\n== python -m ml.train --samples ml/data/samples --out ml/runs/base\n',
|
||||
gen: '$ FROM=kin TO=accept bash ml/pipeline.sh all\n计划：kin gate1 gen prep train m5 apg mine export bench accept\n== node ml/prep/export_kin.cjs\n== python -m ml.eval.check_parity\n{"PASS": true}\n== node ml/gen/consistency.cjs 300 4242\n{"consistentByGateRule": 0.86}\n一致性门禁结果 → CONSENSUS=1\n== bash ml/gen/run_parallel.sh 1786 28 1 --consensus 4\n',
|
||||
late: '... 挖错第 2 轮\nround 2 gain 1.1 pp\n== python -m ml.mining.mine_round --ckpt ml/runs/current.pt --seed 1003 --n 50000 --out ml/runs/mine/r3\n{"n": 50000, "E1": 0.009, "E2": 0.019, "ok": 0.938}\n== python -m ml.apg.apg_finetune --init ml/runs/current.pt --out ml/runs/apg_r3 --steps 1000\n', idle: '' };
|
||||
const GPU = () => [{ name: 'NVIDIA GeForce RTX 3090', util: 93 + Math.round(5 * rnd()), mem_used: 15872 + Math.round(500 * rnd()), mem_total: 24576, temp: 71 + Math.round(3 * rnd()), power: 318 + Math.round(24 * rnd()) }];
|
||||
const DATA = { kin_pass: true, consistency: 0.86, gen_tasks_written: 56412, shards: 56,
|
||||
stats: { tasks: 56412, keep: 43120, gapDrop: 4830, minGapFilterMm: 3, keepRate: 0.85, samplesEvery3: 1702000, samplesPerTask: 36.5, consensusDrop: 7520, vertexMinGapMm: { p10: 3.1, p50: 4.4, p90: 5.9, max: 12.3 } },
|
||||
index: { train_samples: 1617000, val_samples: 85000 }, r2: { label_std_mean: 0.071, pass: true } };
|
||||
function setup(name) {
|
||||
sc = name; seed = 7; con = CON[name] || ''; jobS = { running: name !== 'idle', pid: 21844, from: 'kin', to: 'accept', started: now() - 5 * 3600, rc: null, bash: true }; genAt = 31200;
|
||||
if (name === 'idle') { runs = {}; logs = {}; pipe = null; jobS = { running: false, bash: true }; return; }
|
||||
if (name === 'gen') { runs = {}; logs = {}; pipe = mkPipe([['kin', 0.02], ['gate1', 0.47]], 'gen', 2.7); pipe.stages.gen.info.target_tasks = 50008; return; }
|
||||
if (name === 'train') { pipe = mkPipe([['kin', 0.02], ['gate1', 0.47], ['gen', 4.6], ['prep', 0.38]], 'train', 2.15);
|
||||
runs = { base: { kind: 'train', at: 36000, total: 60000, sps: 6208, t0: T0, live: true },
|
||||
bench_base: { kind: 'bench', at: 2000, total: 2000, sps: 6150, t0: T0 - 700 } };
|
||||
logs = { base: trainLog(60000, 0, 36000, 6208, T0), bench_base: trainLog(2000, 0, 2000, 6150, T0 - 700, 0) };
|
||||
} else { pipe = mkPipe([['kin', 0.02], ['gate1', 0.47], ['gen', 4.6], ['prep', 0.38], ['train', 2.9], ['m5', 0.2], ['apg', 2.6]], 'mine', 3.4); pipe.stages.mine.info.rounds = 6;
|
||||
pipe.stages.export = pipe.stages.bench = pipe.stages.accept = undefined; delete pipe.stages.export; delete pipe.stages.bench; delete pipe.stages.accept;
|
||||
const t1 = now() - 9 * 3600;
|
||||
runs = { base: { kind: 'train', at: 60000, total: 60000, sps: 6208, t0: t1 - 3 * 3600 },
|
||||
bench_base: { kind: 'bench', at: 2000, total: 2000, sps: 6150, t0: t1 - 3.2 * 3600 },
|
||||
apg: { kind: 'apg', at: 3000, total: 3000, t0: t1 },
|
||||
apg_r3: { kind: 'apg', at: 620, total: 1000, t0: now() - 620 * 3.1, live: true } };
|
||||
logs = { base: trainLog(60000, 0, 60000, 6208, t1 - 3 * 3600), bench_base: trainLog(2000, 0, 2000, 6150, t1 - 3.2 * 3600, 0),
|
||||
apg: apgLog(3000, 0, 3000, t1), apg_r3: apgLog(1000, 0, 620, now() - 620 * 3.1, 0.955) };
|
||||
}
|
||||
}
|
||||
setup('train');
|
||||
const mtime = name => { const r = runs[name]; const L = logs[name]; const last = L[L.length - 1]; return r.live ? now() - 2 : (last.t || last.eval?.t || now() - 3600); };
|
||||
const status = () => {
|
||||
let P = pipe && JSON.parse(JSON.stringify(pipe));
|
||||
if (P && P.current) {
|
||||
const r = sc === 'train' ? runs.base : runs.apg_r3;
|
||||
P.progress = !['gen', 'train', 'late'].includes(sc) || (sc !== 'gen' && !(sc === 'train' ? runs.base : runs.apg_r3)) ? { frac: null, label: '进行中（示例）' } : sc === 'gen' ? { frac: Math.min(1, (genAt += 180) / 50008), label: `已生成 ${genAt.toLocaleString()} / 50,008 条任务` }
|
||||
: sc === 'train' ? { frac: (2000 + r.at) / 62000, label: `正式训练 ${r.at.toLocaleString()} / 60,000 步`, run: 'base' }
|
||||
: { frac: (2 + 0.3 + 0.6 * r.at / r.total) / 6, label: `第 3 轮：APG ${r.at} / 1,000 步（共最多 6 轮，提升不足会提前停）`, run: 'apg_r3' };
|
||||
}
|
||||
const gpuIdle = [{ name: 'NVIDIA GeForce RTX 3090', util: 0, mem_used: 412, mem_total: 24576, temp: 38, power: 24 }];
|
||||
const base = { now: now(), gpu: sc === 'train' || sc === 'late' ? GPU() : gpuIdle, pipeline: P, data: DATA, m5: [], selected: null, mine: [], bench: null, acceptance: null };
|
||||
if (sc === 'idle') return { ...base, data: {} };
|
||||
if (sc === 'gen') return { ...base, data: { kin_pass: true, consistency: 0.86, gen_tasks_written: genAt, shards: 28, stats: null, index: null, r2: null } };
|
||||
if (sc === 'train') return base;
|
||||
return { ...base,
|
||||
m5: [{ head: 'base', pass: true, policy_only: { collision_rate: 0.012, success: 0.912 }, with_sim_safety: { collision_rate: 0.0, success: 0.948 } }],
|
||||
selected: 'ml/runs/base/best.pt',
|
||||
mine: [{ round: 'r1', report: { E1: 0.018, E2: 0.031, ok: 0.902 }, gate: { gain_pp: 2.4, R7_left_share_shift_pp: 1.2, accept: true } },
|
||||
{ round: 'r2', report: { E1: 0.011, E2: 0.022, ok: 0.931 }, gate: { gain_pp: 1.1, R7_left_share_shift_pp: 0.8, accept: true } },
|
||||
{ round: 'r3', report: { E1: 0.009, E2: 0.019, ok: 0.938 }, gate: null }],
|
||||
bench: { nc: { n: 30, ok: 30, collisionFrames: 0, brakeRate: 0.004, inferMsP99: 9.8 }, route: { n: 40, ok: 40, collisionFrames: 0, brakeRate: 0.002, inferMsP99: 9.6 },
|
||||
r8: { n: 50, ok: 48, collisionFrames: 0, brakeRate: 0.031, inferMsP99: 10.1 }, nc1000: { n: 1000, ok: 997, collisionFrames: 0, brakeRate: 0.006, inferMsP99: 9.9 } },
|
||||
acceptance: { all_pass: false, failed: ['R8 恢复率'], items: [
|
||||
{ item: 'held-out 可达任务到达率', value: 0.983, pass: true }, { item: '起步失败率（R2）', value: 0.0008, pass: true },
|
||||
{ item: 'D 类停靠距离 ≤ 老师 + 5 mm 的比例', value: 0.991, pass: true }, { item: '部署形态穿模帧', value: 0, pass: true },
|
||||
{ item: 'near_column 30/30', value: '30/30', pass: true }, { item: 'R7 左右比例变化', value: '1.2pp', pass: true },
|
||||
{ item: 'R8 恢复率', value: 0.96, pass: false }, { item: 'R10 训练吞吐（样本/s）', value: 6208, pass: true, rule: '样本/s' } ] } };
|
||||
};
|
||||
const fn = async (path, body) => {
|
||||
const u = new URL(path, 'http://x');
|
||||
if (u.pathname === '/api/runs') return Object.entries(runs).map(([name, r]) => ({ name, kind: r.kind, size: 0, mtime: mtime(name), has_best: true, diverged: false })).sort((a, b) => b.mtime - a.mtime);
|
||||
if (u.pathname === '/api/log') {
|
||||
const name = u.searchParams.get('run'), off = +u.searchParams.get('offset'), r = runs[name];
|
||||
if (!r) return { records: [], offset: 0, reset: true, mtime: now() };
|
||||
if (off === 0) return { records: logs[name], offset: logs[name].length, reset: true, mtime: mtime(name), live: !!r.live };
|
||||
if (r.live && r.at < r.total) { // 模拟推进
|
||||
const step = r.kind === 'apg' ? 10 : 250, from = r.at + (r.kind === 'apg' ? 10 : 50); r.at = Math.min(r.total, r.at + step);
|
||||
const add = r.kind === 'apg' ? apgLog(r.total, from, r.at, r.t0, 0.955) : trainLog(r.total, from, r.at, r.sps, r.t0);
|
||||
if (r.kind !== 'apg') add.forEach(v => { if (v.t) v.t = now(); });
|
||||
logs[name].push(...add); return { records: add, offset: off + add.length, reset: false, mtime: now(), live: true };
|
||||
}
|
||||
return { records: [], offset: off, reset: false, mtime: mtime(name), live: !!r.live };
|
||||
}
|
||||
if (u.pathname === '/api/status') return status();
|
||||
if (u.pathname === '/api/job') return jobS;
|
||||
if (u.pathname === '/api/console') { const off = +u.searchParams.get('offset'); return { text: con.slice(off), offset: con.length, reset: off === 0 || off > con.length }; }
|
||||
if (u.pathname === '/api/start') { const b = JSON.parse(body || '{}'); jobS = { running: true, pid: 23001, from: b.from, to: b.to, started: now(), rc: null, bash: true };
|
||||
con = `$ FROM=${b.from} TO=${b.to} bash ml/pipeline.sh all\n（示例数据：真实使用时这里是命令行输出）\n`;
|
||||
if (!pipe) pipe = mkPipe([], b.from, 0.001); else { pipe.current = b.from; pipe.stages[b.from] = { start: now(), end: null, rc: null, info: {} }; }
|
||||
return { ok: true, pid: 23001 }; }
|
||||
if (u.pathname === '/api/stop') { jobS = { ...jobS, running: false, rc: -15, stopped: true };
|
||||
if (pipe && pipe.current) { pipe.stages[pipe.current].end = now(); pipe.stages[pipe.current].rc = -15; pipe.current = null; }
|
||||
Object.values(runs).forEach(r => r.live = false); con += '\n[已在网页上停止]\n'; return { ok: true }; }
|
||||
};
|
||||
fn.scenario = setup;
|
||||
return fn;
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+315
@@ -0,0 +1,315 @@
|
||||
"""训练控制台（本地网页，零第三方依赖，离线可用）：一键开始全流程、看进度、可停止。
|
||||
双击 ml/console.bat（Windows）或运行 bash ml/console.sh（Linux/WSL）→ 自动打开 http://localhost:8100
|
||||
也可：python -m ml.monitor [--port 8100] [--host 0.0.0.0] [--open]
|
||||
网页上点「开始训练」→ 选起止阶段与参数 → 后台执行 bash ml/pipeline.sh all（FROM/TO 环境变量），
|
||||
输出写 ml/runs/console.log，网页实时显示；「停止」结束整个进程树。网页关掉不影响训练，重新打开即可继续看。
|
||||
只读数据来源：
|
||||
ml/runs/pipeline.json 各阶段开始/结束/返回码（pipeline.sh 自动写）→ 全流程进度
|
||||
ml/runs/*/log.jsonl 预训练（base / wta / bench_*）与 APG（apg / apg_r*）逐步日志与闭环评估
|
||||
ml/runs/m5_gate_*.json、mine/r*/report.json + gate.json、policy_bench.json、acceptance.json
|
||||
ml/data/stats.json / consistency.json / r2.json / samples/index.json、nvidia-smi
|
||||
安全：默认只监听 127.0.0.1；--host 0.0.0.0 时其他机器只能看，开始/停止只接受本机请求（--remote-control 放开）。
|
||||
2026-10-18 新增（用户要求：训练全程可视化，并在网页上点按钮一键开始）。"""
|
||||
import argparse, glob, json, os, re, shutil, signal, subprocess, sys, threading, time, webbrowser
|
||||
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
|
||||
from urllib.parse import urlparse, parse_qs
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
HTML = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'monitor.html')
|
||||
RUNS, DATA = 'ml/runs', 'ml/data'
|
||||
|
||||
|
||||
def jload(p):
|
||||
try:
|
||||
with open(p, encoding='utf-8') as f: return json.load(f)
|
||||
except Exception: return None
|
||||
|
||||
|
||||
def kind_of(name):
|
||||
if name.startswith('apg'): return 'apg'
|
||||
if name.startswith('bench'): return 'bench'
|
||||
return 'train'
|
||||
|
||||
|
||||
def list_runs():
|
||||
out = []
|
||||
for f in glob.glob(os.path.join(ROOT, RUNS, '*', 'log.jsonl')):
|
||||
d = os.path.basename(os.path.dirname(f)); st = os.stat(f)
|
||||
out.append({'name': d, 'kind': kind_of(d), 'size': st.st_size, 'mtime': st.st_mtime,
|
||||
'has_best': os.path.exists(os.path.join(os.path.dirname(f), 'best.pt')),
|
||||
'diverged': os.path.exists(os.path.join(os.path.dirname(f), 'diverged.pt'))})
|
||||
out.sort(key=lambda r: -r['mtime'])
|
||||
return out
|
||||
|
||||
|
||||
def read_log(name, offset):
|
||||
if not re.fullmatch(r'[\w.\-]+', name or ''): return {'error': 'bad run'}
|
||||
f = os.path.join(ROOT, RUNS, name, 'log.jsonl')
|
||||
if not os.path.exists(f): return {'error': 'no log'}
|
||||
size = os.path.getsize(f)
|
||||
if offset > size: offset = 0 # 日志被新训练覆盖 → 从头读
|
||||
recs = []
|
||||
with open(f, 'rb') as fh:
|
||||
fh.seek(offset); buf = fh.read(min(size - offset, 64 << 20))
|
||||
end = buf.rfind(b'\n') + 1 # 只取完整行，半行留到下次
|
||||
for line in buf[:end].decode('utf-8', 'replace').splitlines():
|
||||
if line.strip():
|
||||
try: recs.append(json.loads(line))
|
||||
except Exception: pass
|
||||
return {'records': recs, 'offset': offset + end, 'size': size, 'reset': offset == 0, 'mtime': os.path.getmtime(f)}
|
||||
|
||||
|
||||
def gpu():
|
||||
try:
|
||||
q = 'name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw'
|
||||
r = subprocess.run(['nvidia-smi', f'--query-gpu={q}', '--format=csv,noheader,nounits'], capture_output=True, text=True, timeout=3)
|
||||
out = []
|
||||
for line in r.stdout.strip().splitlines():
|
||||
n, u, mu, mt, t, p = [x.strip() for x in line.split(',')]
|
||||
num = lambda x: float(x) if re.fullmatch(r'[\d.]+', x) else None
|
||||
out.append({'name': n, 'util': num(u), 'mem_used': num(mu), 'mem_total': num(mt), 'temp': num(t), 'power': num(p)})
|
||||
return out
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def last_rec(name):
|
||||
"""读日志最后一条非评估记录（只读文件尾 16 KB）。"""
|
||||
f = os.path.join(ROOT, RUNS, name, 'log.jsonl')
|
||||
try:
|
||||
with open(f, 'rb') as fh:
|
||||
fh.seek(0, 2); n = fh.tell(); fh.seek(max(0, n - 16384)); tail = fh.read().decode('utf-8', 'replace').splitlines()
|
||||
for line in reversed(tail):
|
||||
try:
|
||||
r = json.loads(line)
|
||||
if 'eval' not in r: return r
|
||||
except Exception: pass
|
||||
except Exception: pass
|
||||
return None
|
||||
|
||||
|
||||
def newer(pattern, t0):
|
||||
return [f for f in glob.glob(os.path.join(ROOT, pattern)) if os.path.getmtime(f) >= t0 - 1]
|
||||
|
||||
|
||||
def stage_progress(st, s):
|
||||
"""当前阶段的进度：返回 {'frac': 0..1 或 None（无法计量）, 'label': 文字}。"""
|
||||
t0, info = s['start'], s.get('info') or {}
|
||||
if st == 'gen':
|
||||
n = 0
|
||||
for f in newer(f'{DATA}/shard_*.jsonl', t0):
|
||||
with open(f, 'rb') as fh: n += sum(1 for _ in fh)
|
||||
tgt = info.get('target_tasks') or 0
|
||||
return {'frac': min(1, n / tgt) if tgt else None, 'label': f'已生成 {n:,} / {tgt:,} 条任务'}
|
||||
if st == 'prep':
|
||||
nsh = len(glob.glob(os.path.join(ROOT, DATA, 'shard_*.jsonl')))
|
||||
done = len(newer(f'{DATA}/samples/*.X.npy', t0))
|
||||
if os.path.exists(os.path.join(ROOT, DATA, 'r2.json')) and newer(f'{DATA}/r2.json', t0): return {'frac': 1, 'label': '打包完成，R2 检查完成'}
|
||||
return {'frac': min(0.95, done / nsh) if nsh else None, 'label': f'已打包 {done} / {nsh} 个分片' + ('，R2 检查中' if nsh and done >= nsh else '')}
|
||||
if st == 'train':
|
||||
tag, bs = info.get('tag', 'base'), info.get('bench_steps', 2000)
|
||||
b, m = last_rec(f'bench_{tag}') if newer(f'{RUNS}/bench_{tag}/log.jsonl', t0) else None, last_rec(tag) if newer(f'{RUNS}/{tag}/log.jsonl', t0) else None
|
||||
if m and m.get('total'):
|
||||
return {'frac': (bs + m['step']) / (bs + m['total']), 'label': f'正式训练 {m["step"]:,} / {m["total"]:,} 步', 'run': tag}
|
||||
if b: return {'frac': b['step'] / (bs + 60000), 'label': f'测速 {b["step"]:,} / {bs:,} 步', 'run': f'bench_{tag}'}
|
||||
return {'frac': 0, 'label': '加载数据中'}
|
||||
if st == 'apg':
|
||||
r = last_rec('apg') if newer(f'{RUNS}/apg/log.jsonl', t0) else None
|
||||
return {'frac': r['it'] / r['total'] if r and r.get('total') else 0, 'label': f'APG {r["it"]:,} / {r["total"]:,} 步' if r else '准备中', 'run': 'apg'}
|
||||
if st == 'mine':
|
||||
R = info.get('rounds', 6)
|
||||
k = len(newer(f'{RUNS}/mine/r*/gate.json', t0))
|
||||
cur = sorted(newer(f'{RUNS}/apg_r*/log.jsonl', t0), key=os.path.getmtime)
|
||||
frac_in, lab = 0, f'第 {k + 1} 轮：挖错出题'
|
||||
if cur and len(cur) > k:
|
||||
name = os.path.basename(os.path.dirname(cur[-1])); r = last_rec(name)
|
||||
if r and r.get('total'): frac_in = 0.3 + 0.6 * r['it'] / r['total']; lab = f'第 {k + 1} 轮：APG {r["it"]:,} / {r["total"]:,} 步'
|
||||
return {'frac': min(1, (k + frac_in) / R), 'label': f'{lab}（共最多 {R} 轮，提升不足会提前停）', 'run': name}
|
||||
return {'frac': min(1, k / R), 'label': f'{lab}（共最多 {R} 轮，提升不足会提前停）'}
|
||||
if st == 'm5':
|
||||
tag = info.get('tag', 'base')
|
||||
n = len(newer(f'{RUNS}/m5_{tag}_policy.json', t0)) + len(newer(f'{RUNS}/m5_{tag}_safety.json', t0))
|
||||
return {'frac': n / 2, 'label': ['纯策略评估中', '加安全层评估中', '汇总'][min(n, 2)]}
|
||||
return {'frac': None, 'label': '进行中'}
|
||||
|
||||
|
||||
def pipeline():
|
||||
d = jload(os.path.join(ROOT, RUNS, 'pipeline.json'))
|
||||
if not d: return None
|
||||
cur = d.get('current')
|
||||
if cur and cur in d['stages']:
|
||||
try: d['progress'] = stage_progress(cur, d['stages'][cur])
|
||||
except Exception as e: d['progress'] = {'frac': None, 'label': f'进度读取失败：{e}'}
|
||||
return d
|
||||
|
||||
|
||||
def status():
|
||||
P = lambda *a: os.path.join(ROOT, *a)
|
||||
mine = []
|
||||
for d in sorted(glob.glob(P(RUNS, 'mine', 'r*')), key=lambda x: int(re.sub(r'\D', '', os.path.basename(x)) or 0)):
|
||||
rep, gate = jload(os.path.join(d, 'report.json')), jload(os.path.join(d, 'gate.json'))
|
||||
if rep or gate:
|
||||
mine.append({'round': os.path.basename(d), 'report': rep and {k: rep.get(k) for k in ('n', 'E1', 'E2', 'E3', 'E4', 'E5', 'ok', 'pool')},
|
||||
'gate': gate and {k: gate.get(k) for k in ('gain_pp', 'A_drop_pp', 'fixed_kept', 'R7_left_share_shift_pp', 'R7_travel_median_ratio', 'accept')}})
|
||||
stats = jload(P(DATA, 'stats.json'))
|
||||
if stats: stats = {k: stats.get(k) for k in ('tasks', 'keep', 'gapDrop', 'minGapFilterMm', 'keepRate', 'samplesEvery3', 'samplesPerTask', 'projectedSamplesFor50k', 'consensusDrop', 'vertexMinGapMm', 'execS', 'travelDeg')}
|
||||
shards = glob.glob(P(DATA, 'shard_*.jsonl'))
|
||||
gen_lines = 0
|
||||
for s in shards:
|
||||
try:
|
||||
with open(s, 'rb') as fh: gen_lines += sum(1 for _ in fh)
|
||||
except Exception: pass
|
||||
pb = jload(P(RUNS, 'policy_bench.json'))
|
||||
if pb: pb = {k: {kk: v.get(kk) for kk in ('n', 'ok', 'reached', 'collisionFrames', 'brakeRate', 'recoveryRate', 'livelock', 'inferMsP99', 'safetyMsP99')}
|
||||
for k, v in pb.items() if isinstance(v, dict) and 'n' in v}
|
||||
return {'now': time.time(), 'gpu': gpu(), 'pipeline': pipeline(),
|
||||
'data': {'consistency': (jload(P(DATA, 'consistency.json')) or {}).get('consistentByGateRule'),
|
||||
'gen_tasks_written': gen_lines, 'shards': len(shards), 'stats': stats,
|
||||
'index': jload(P(DATA, 'samples', 'index.json')), 'r2': jload(P(DATA, 'r2.json'))},
|
||||
'm5': [x for x in (jload(f) for f in sorted(glob.glob(P(RUNS, 'm5_gate_*.json')))) if x],
|
||||
'selected': open(P(RUNS, 'selected_ckpt')).read().strip() if os.path.exists(P(RUNS, 'selected_ckpt')) else None,
|
||||
'mine': mine, 'bench': pb, 'acceptance': jload(P(RUNS, 'acceptance.json'))}
|
||||
|
||||
|
||||
STAGES = ['setup', 'kin', 'gate1', 'gen', 'prep', 'train', 'm5', 'apg', 'mine', 'export', 'bench', 'accept']
|
||||
ENV_KEYS = {'N_TASKS': r'\d+', 'NPROC': r'\d+', 'TARGET_SAMPLES': r'\d+', 'CFG': r'ml/configs/(base|wta)\.json',
|
||||
'CONSENSUS': r'[01]?', 'ROUNDS': r'\d+', 'APG_STEPS': r'\d+', 'MINE_N': r'\d+'}
|
||||
CONSOLE = os.path.join(ROOT, RUNS, 'console.log')
|
||||
|
||||
|
||||
def find_bash():
|
||||
"""Windows 优先 Git Bash（System32\\bash.exe 是 WSL，不能用）；其他系统用 PATH 里的 bash。"""
|
||||
if os.name == 'nt':
|
||||
for p in [os.environ.get('BASH', ''), r'C:\Program Files\Git\bin\bash.exe', r'C:\Program Files (x86)\Git\bin\bash.exe',
|
||||
os.path.expandvars(r'%LOCALAPPDATA%\Programs\Git\bin\bash.exe')]:
|
||||
if p and os.path.exists(p): return p
|
||||
g = shutil.which('git')
|
||||
if g:
|
||||
p = os.path.join(os.path.dirname(os.path.dirname(g)), 'bin', 'bash.exe')
|
||||
if os.path.exists(p): return p
|
||||
return None
|
||||
return shutil.which('bash')
|
||||
|
||||
|
||||
class Job:
|
||||
def __init__(self):
|
||||
self.p = None; self.info = {}; self.lock = threading.Lock()
|
||||
|
||||
def state(self):
|
||||
with self.lock:
|
||||
run = self.p is not None and self.p.poll() is None
|
||||
if self.p is not None and not run and self.info.get('rc') is None:
|
||||
self.info['rc'] = self.p.returncode; self.info['ended'] = time.time()
|
||||
return {'running': run, **self.info, 'bash': find_bash() is not None}
|
||||
|
||||
def start(self, frm, to, env):
|
||||
with self.lock:
|
||||
if self.p is not None and self.p.poll() is None: return {'error': '已经在运行'}
|
||||
pj = jload(os.path.join(ROOT, RUNS, 'pipeline.json')) or {}
|
||||
cur = pj.get('current')
|
||||
if cur and time.time() - (pj['stages'].get(cur, {}).get('start') or 0) < 7 * 86400 and not self.p:
|
||||
# 终端里另起的流水线还在跑（或上次被强杀没写结束）：看有没有最近更新的日志
|
||||
fresh = [r for r in list_runs() if time.time() - r['mtime'] < 120]
|
||||
if fresh: return {'error': f'检测到终端里正在运行阶段 {cur}（日志 {fresh[0]["name"]} 2 分钟内有更新），请先在终端结束它'}
|
||||
if frm not in STAGES or to not in STAGES or STAGES.index(frm) > STAGES.index(to): return {'error': '起止阶段无效'}
|
||||
bash = find_bash()
|
||||
if not bash: return {'error': '找不到 bash：Windows 请安装 Git for Windows（自带 Git Bash）'}
|
||||
e = dict(os.environ, FROM=frm, TO=to, MONITOR='0', PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
|
||||
if ' ' not in sys.executable: e['PYTHON'] = sys.executable.replace('\\', '/') # 与控制台同一个 Python
|
||||
for k, v in (env or {}).items():
|
||||
if k in ENV_KEYS and v not in (None, '') and re.fullmatch(ENV_KEYS[k], str(v)): e[k] = str(v)
|
||||
os.makedirs(os.path.dirname(CONSOLE), exist_ok=True)
|
||||
with open(CONSOLE, 'w', encoding='utf-8') as f: f.write(f'$ FROM={frm} TO={to} bash ml/pipeline.sh all [{time.strftime("%Y-%m-%d %H:%M:%S")}]\n')
|
||||
log = open(CONSOLE, 'a', encoding='utf-8') # 追加模式：网页写入的"已停止"提示不会被子进程覆盖
|
||||
kw = {'creationflags': subprocess.CREATE_NEW_PROCESS_GROUP} if os.name == 'nt' else {'start_new_session': True}
|
||||
self.p = subprocess.Popen([bash, 'ml/pipeline.sh', 'all'], cwd=ROOT, env=e, stdout=log, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, **kw)
|
||||
self.info = {'pid': self.p.pid, 'from': frm, 'to': to, 'started': time.time(), 'rc': None, 'ended': None,
|
||||
'env': {k: e[k] for k in ENV_KEYS if k in e}}
|
||||
return {'ok': True, 'pid': self.p.pid}
|
||||
|
||||
def stop(self):
|
||||
with self.lock:
|
||||
if self.p is None or self.p.poll() is not None: return {'error': '没有在运行'}
|
||||
PJ = os.path.join(ROOT, RUNS, 'pipeline.json')
|
||||
cur = (jload(PJ) or {}).get('current') # 先记下正在跑的阶段：被杀的子进程可能以返回码 0 写结束
|
||||
if os.name == 'nt':
|
||||
subprocess.run(['taskkill', '/T', '/F', '/PID', str(self.p.pid)], capture_output=True)
|
||||
else:
|
||||
try: os.killpg(self.p.pid, signal.SIGTERM)
|
||||
except ProcessLookupError: pass
|
||||
try: self.p.wait(10)
|
||||
except subprocess.TimeoutExpired: os.killpg(self.p.pid, signal.SIGKILL)
|
||||
self.info['stopped'] = True
|
||||
with open(CONSOLE, 'a', encoding='utf-8') as f: f.write(f'\n[已在网页上停止 {time.strftime("%H:%M:%S")}]\n')
|
||||
time.sleep(0.5); pj = jload(PJ)
|
||||
if pj and cur and cur in pj['stages']: # 该阶段记为"已停止"（返回码 -15）
|
||||
pj['stages'][cur].update(end=time.time(), rc=-15); pj['current'] = None
|
||||
with open(PJ, 'w', encoding='utf-8') as f: json.dump(pj, f, ensure_ascii=False, indent=1)
|
||||
return {'ok': True}
|
||||
|
||||
|
||||
JOB = Job()
|
||||
|
||||
|
||||
def console(offset):
|
||||
if not os.path.exists(CONSOLE): return {'text': '', 'offset': 0, 'reset': True}
|
||||
size = os.path.getsize(CONSOLE); reset = offset > size or offset == 0
|
||||
if reset: offset = max(0, size - 65536) # 首次只给最后 64 KB
|
||||
with open(CONSOLE, 'rb') as f: f.seek(offset); b = f.read(min(size - offset, 1 << 20))
|
||||
return {'text': b.decode('utf-8', 'replace').replace('\r', ''), 'offset': offset + len(b), 'reset': reset}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a): pass
|
||||
|
||||
def send(self, code, body, ctype):
|
||||
b = body if isinstance(body, bytes) else body.encode('utf-8')
|
||||
self.send_response(code); self.send_header('Content-Type', ctype); self.send_header('Content-Length', str(len(b)))
|
||||
self.send_header('Cache-Control', 'no-store'); self.end_headers(); self.wfile.write(b)
|
||||
|
||||
def do_GET(self):
|
||||
u = urlparse(self.path); q = parse_qs(u.query)
|
||||
try:
|
||||
if u.path in ('/', '/index.html'):
|
||||
with open(HTML, 'rb') as f: return self.send(200, f.read(), 'text/html; charset=utf-8')
|
||||
if u.path == '/api/runs': return self.send(200, json.dumps(list_runs()), 'application/json')
|
||||
if u.path == '/api/log':
|
||||
return self.send(200, json.dumps(read_log(q.get('run', [''])[0], int(q.get('offset', ['0'])[0]))), 'application/json')
|
||||
if u.path == '/api/status': return self.send(200, json.dumps(status()), 'application/json')
|
||||
if u.path == '/api/job': return self.send(200, json.dumps(JOB.state()), 'application/json')
|
||||
if u.path == '/api/console': return self.send(200, json.dumps(console(int(q.get('offset', ['0'])[0]))), 'application/json')
|
||||
self.send(404, 'not found', 'text/plain')
|
||||
except Exception as e:
|
||||
self.send(500, json.dumps({'error': str(e)}), 'application/json')
|
||||
|
||||
|
||||
def do_POST(self):
|
||||
u = urlparse(self.path)
|
||||
if not (self.server.remote_control or self.client_address[0] in ('127.0.0.1', '::1')):
|
||||
return self.send(403, json.dumps({'error': '开始/停止只接受本机操作'}), 'application/json')
|
||||
try:
|
||||
n = int(self.headers.get('Content-Length') or 0); body = json.loads(self.rfile.read(n) or b'{}')
|
||||
if u.path == '/api/start': r = JOB.start(body.get('from', 'kin'), body.get('to', 'accept'), body.get('env') or {})
|
||||
elif u.path == '/api/stop': r = JOB.stop()
|
||||
else: return self.send(404, 'not found', 'text/plain')
|
||||
self.send(200 if 'ok' in r else 409, json.dumps(r, ensure_ascii=False), 'application/json')
|
||||
except Exception as e:
|
||||
self.send(500, json.dumps({'error': str(e)}), 'application/json')
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--port', type=int, default=8100); ap.add_argument('--host', default='127.0.0.1')
|
||||
ap.add_argument('--open', action='store_true', help='启动后自动打开浏览器')
|
||||
ap.add_argument('--remote-control', action='store_true', help='允许其他机器点开始/停止（默认只能看）')
|
||||
a = ap.parse_args()
|
||||
srv = ThreadingHTTPServer((a.host, a.port), H); srv.remote_control = a.remote_control
|
||||
url = f'http://localhost:{a.port}/'
|
||||
print(f'训练控制台：{url} （关闭本窗口会停止控制台；已开始的训练会一并结束，先在网页上确认跑完）', flush=True)
|
||||
if a.open: threading.Timer(0.8, lambda: webbrowser.open(url)).start()
|
||||
try: srv.serve_forever()
|
||||
except KeyboardInterrupt: pass
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+40
-7
@@ -2,6 +2,7 @@
|
||||
# 一条龙：bash ml/pipeline.sh <阶段> ；阶段按顺序：
|
||||
# setup 安装 Python/Node 依赖（torch 需先按 CUDA 版本装好）
|
||||
# kin 导出运动学/碰撞模型 + JS↔PyTorch 对拍（必须 PASS）
|
||||
# r4 浏览器推理延迟摸底（随机初始化模型导出 ONNX；训练前做，不过就改模型规模，不必等训完）
|
||||
# gate1 老师一致性门禁（决定生成时是否加 --consensus 4）
|
||||
# gen 生成老师轨迹（NPROC 进程；CONSENSUS=1 时加共识过滤；样本不足 TARGET_SAMPLES 自动补生成）
|
||||
# prep 打包训练样本 + R2 标签一致性检查
|
||||
@@ -12,12 +13,13 @@
|
||||
# export 导出 ONNX（fp32）到 webapp/models/
|
||||
# bench 部署形态评测（Node，策略 + 安全层，含 R8 对抗集）
|
||||
# accept 验收汇总（方案 §9.2 各项 + R2/R7/R8/R10）→ ml/runs/acceptance.json
|
||||
# all 依次执行 kin → accept
|
||||
# all 一键依次执行 kin → accept（FROM=gen TO=train 可只跑一段），并在后台自动打开训练监控（r4 需在浏览器里看，不含在 all 里；R2、M5 不通过会停下）
|
||||
# monitor 训练控制台网页 http://localhost:8100：点「开始训练」一键跑全流程、看进度、可停止（也可双击 ml/console.bat / ml/console.sh）
|
||||
# 可调环境变量：PYTHON（默认 python；Linux 无 python 时设 PYTHON=python3）、NPROC、N_TASKS、TARGET_SAMPLES、
|
||||
# CFG（ml/configs/base.json | wta.json）、CKPT（覆盖后续阶段的起点模型）、CONSENSUS
|
||||
# 2026-10-18：按另一台机器的审查改——头分目录不互相覆盖、统一 $PYTHON（Windows Git Bash 可跑）、
|
||||
# gen 自动补量、m5 双口径、export 默认 fp32、新增 accept。
|
||||
set -e
|
||||
set -e -o pipefail # pipefail：`node ... | tee` 里 node 失败也算失败（否则门禁中断会留下残缺的 consistency.json 继续往下跑）
|
||||
cd "$(dirname "$0")/.."
|
||||
PY=${PYTHON:-python}
|
||||
command -v "$PY" >/dev/null 2>&1 && "$PY" -c "import sys" 2>/dev/null || { echo "找不到可用的 Python（$PY），请设 PYTHON=..."; exit 1; }
|
||||
@@ -31,10 +33,21 @@ SEL=ml/runs/selected_ckpt
|
||||
ckpt() { if [ -n "${CKPT:-}" ]; then echo "$CKPT"; elif [ -f $SEL ]; then cat $SEL; else echo ml/runs/$TAG/best.pt; fi; }
|
||||
stage=${1:-help}
|
||||
run() { echo "== $*"; "$@"; }
|
||||
# 2026-10-18：每个阶段开始/结束写 ml/runs/pipeline.json（监控网页据此显示全流程进度与各步用时）
|
||||
stage_info() { case $1 in
|
||||
gen) echo "target_tasks=$(( (N_TASKS + NPROC - 1) / NPROC * NPROC ))" ;;
|
||||
train) echo "tag=$TAG bench_steps=2000" ;;
|
||||
m5) echo "tag=$TAG" ;;
|
||||
mine) echo "rounds=${ROUNDS:-6} apg_steps=${APG_STEPS:-1000}" ;;
|
||||
esac; }
|
||||
case " setup kin gate1 gen prep train m5 apg mine export bench accept r4 " in *" $stage "*)
|
||||
$PY -m ml.pstate begin $stage $(stage_info $stage) || true
|
||||
trap 'rc=$?; $PY -m ml.pstate end '"$stage"' $rc >/dev/null 2>&1 || true' EXIT ;;
|
||||
esac
|
||||
case $stage in
|
||||
setup) run $PY -m pip install -r ml/requirements.txt; (cd ml && npm install --no-audit --no-fund) ;;
|
||||
kin) run node ml/prep/export_kin.cjs; run node ml/eval/dump_parity.cjs 2000; run $PY -m ml.eval.check_parity ;;
|
||||
gate1) mkdir -p ml/data; run node ml/gen/consistency.cjs 300 4242 | tee ml/data/consistency.json ;;
|
||||
gate1) mkdir -p ml/data; run node ml/gen/consistency.cjs 300 4242 | tee ml/data/consistency.json.tmp; mv ml/data/consistency.json.tmp ml/data/consistency.json ;; # 中断时不留残缺结果
|
||||
gen) per=$(( (N_TASKS + NPROC - 1) / NPROC ))
|
||||
if [ -z "${CONSENSUS:-}" ] && [ -f ml/data/consistency.json ]; then
|
||||
CONSENSUS=$($PY -c "import json;print(1 if json.load(open('ml/data/consistency.json'))['consistentByGateRule']<0.9 else 0)")
|
||||
@@ -48,11 +61,15 @@ case $stage in
|
||||
need=$($PY -c "import json,math;s=json.load(open('ml/data/stats.json'));d=$TARGET_SAMPLES-s['samplesEvery3'];print(0 if d<=0 else math.ceil(d/(s['samplesEvery3']/s['tasks'])*1.05))")
|
||||
[ "$need" = "0" ] && break
|
||||
echo "样本不足目标 $TARGET_SAMPLES，补生成 $need 条任务"
|
||||
run bash ml/gen/run_parallel.sh $(( (need + NPROC - 1) / NPROC )) $NPROC $s0 "$extra"
|
||||
per2=$(( (need + NPROC - 1) / NPROC ))
|
||||
$PY -m ml.pstate set gen target_tasks=$(( $($PY -c "import json;print(json.load(open('ml/runs/pipeline.json'))['stages']['gen']['info']['target_tasks'])") + per2 * NPROC )) || true
|
||||
run bash ml/gen/run_parallel.sh $per2 $NPROC $s0 "$extra"
|
||||
s0=$((s0 + NPROC))
|
||||
done ;;
|
||||
prep) run bash ml/prep/run_prep.sh $NPROC
|
||||
run $PY -m ml.eval.check_r2 --samples ml/data/samples --out ml/data/r2.json ;;
|
||||
run $PY -m ml.eval.check_r2 --samples ml/data/samples --out ml/data/r2.json
|
||||
# R2 明确不通过（pass=false）就停在这里，不让一键流程白训 3 小时；近邻对不足（pass=null）不拦
|
||||
$PY -c "import json,sys;sys.exit(4 if json.load(open('ml/data/r2.json')).get('pass') is False else 0)" || { echo "R2 不通过：近邻样本标签不自洽，先把 ml/data/r2.json 发我；确认要继续就从「预训练」开始（FROM=train）"; exit 4; } ;;
|
||||
train) run $PY -m ml.train --samples ml/data/samples --out ml/runs/bench_$TAG --steps 2000 --no-eval --config $CFG
|
||||
run $PY -m ml.train --samples ml/data/samples --out ml/runs/$TAG --config $CFG ;;
|
||||
m5) [ -f ml/runs/nc_gate.jsonl ] || run $PY -m ml.mining.mine_round --make-gate 1000 --seed 4321 --mix 0,0,1,0 --name nc_gate.jsonl --out ml/runs
|
||||
@@ -80,11 +97,27 @@ PY
|
||||
;;
|
||||
apg) ck=$(ckpt); run $PY -m ml.apg.apg_finetune --init $ck --samples ml/data/samples --out ml/runs/apg; cp ml/runs/apg/best.pt ml/runs/current.pt ;;
|
||||
mine) run bash ml/mining/loop.sh ${ROUNDS:-6} ${APG_STEPS:-1000} ${MINE_N:-50000} ;;
|
||||
r4) run $PY -m ml.export_onnx --random $CFG --out webapp/models/traj_policy.onnx
|
||||
echo "下一步：python ml/serve.py → 浏览器打开 http://localhost:8000/?policy=1 → 拖动黄球 1 分钟 → 控制台 __POLICY.latency()" ;;
|
||||
monitor) run $PY -m ml.monitor ${MONITOR_ARGS:-} ;;
|
||||
export) ck=ml/runs/current.pt; [ -f $ck ] || ck=$(ckpt)
|
||||
run $PY -m ml.export_onnx --ckpt $ck --out webapp/models/traj_policy.onnx ;;
|
||||
bench) run node tests/policy_bench.cjs webapp/models/traj_policy.onnx nc,route,r8 --nc1000 --out ml/runs/policy_bench.json ;;
|
||||
accept) ck=ml/runs/current.pt; [ -f $ck ] || ck=$(ckpt)
|
||||
run $PY -m ml.eval.acceptance --ckpt $ck --samples ml/data/samples --out ml/runs/acceptance.json ;;
|
||||
all) for s in kin gate1 gen prep train m5 apg mine export bench accept; do bash ml/pipeline.sh $s; done ;;
|
||||
*) sed -n '2,20p' "$0" ;;
|
||||
all) # FROM / TO 指定起止阶段（默认 kin → accept），网页控制台的「开始训练」也走这里
|
||||
FROM=${FROM:-kin}; TO=${TO:-accept}; PLAN=""; on=0
|
||||
for s in setup kin gate1 gen prep train m5 apg mine export bench accept; do
|
||||
[ "$s" = "$FROM" ] && on=1; [ $on = 1 ] && PLAN="$PLAN $s"; [ "$s" = "$TO" ] && break
|
||||
done
|
||||
PLAN=${PLAN# }; [ -n "$PLAN" ] || { echo "FROM=$FROM / TO=$TO 不是有效阶段"; exit 1; }
|
||||
echo "计划：$PLAN"
|
||||
$PY -m ml.pstate reset ${PLAN// /,}
|
||||
# 监控网页随 all 自动在后台启动（MONITOR=0 关闭）；端口已被占用说明已经开着，不重复启动
|
||||
if [ "${MONITOR:-1}" = "1" ] && $PY -c "import socket,sys;s=socket.socket();sys.exit(0 if s.connect_ex(('127.0.0.1',8100)) else 1)"; then
|
||||
mkdir -p ml/runs; $PY -m ml.monitor ${MONITOR_ARGS:-} > ml/runs/monitor.log 2>&1 &
|
||||
echo "训练监控已在后台启动：http://localhost:8100/ （全部跑完后仍可查看，关闭终端即结束）"
|
||||
fi
|
||||
for s in $PLAN; do bash ml/pipeline.sh $s; done ;;
|
||||
*) sed -n '2,22p' "$0" ;;
|
||||
esac
|
||||
|
||||
@@ -25,10 +25,13 @@ const gauss = () => Math.sqrt(-2 * Math.log(rnd() + 1e-12)) * Math.cos(2 * Math.
|
||||
const hash = s => { let h = 2166136261; for (const c of s) h = Math.imul(h ^ c.charCodeAt(0), 16777619); return h >>> 0; };
|
||||
const X = [], Y = [], V = [], SG = [], TID = [], tasks = [];
|
||||
const lines = fs.readFileSync(IN, 'utf8').split('\n').filter(Boolean);
|
||||
let nAug = 0, nAugRej = 0;
|
||||
let nAug = 0, nAugRej = 0, nGapDrop = 0;
|
||||
/* 2026-10-18：认证最小间隙 < 3 mm 的老师轨迹不进训练集；噪声增广接受地板同步为 3 mm（原 1 mm）。 */
|
||||
const MIN_GAP_MM = +(process.env.MIN_GAP_MM || 3);
|
||||
for (const line of lines) {
|
||||
const r = JSON.parse(line);
|
||||
if (!r.keep) continue;
|
||||
if (r.cert && r.cert.minGap != null && r.cert.minGap < MIN_GAP_MM) { nGapDrop++; continue; } // 与 stats.cjs 同一过滤
|
||||
const tid = hash(r.id) & 0x7fffffff;
|
||||
const split = (hash(r.id + '#split') % 100) < 5 ? 'val' : 'train';
|
||||
const P = r.path;
|
||||
@@ -43,7 +46,7 @@ for (const line of lines) {
|
||||
if (rnd() < 0.3) {
|
||||
const qn = q.map((v, j) => { let x = v + gauss(); const t2 = L[O[j]]; if (!t2.cont) x = Math.max(t2.lo, Math.min(t2.hi, x)); return x; });
|
||||
nAug++;
|
||||
if (S.collideFree(qn) && S.clearance(qn, 5) > 1) { off = qn.map((v, j) => v - q[j]); q = qn; qPrev = qPrev.map((v, j) => v + off[j]); aug = true; }
|
||||
if (S.collideFree(qn) && S.clearance(qn, 5) > MIN_GAP_MM) { off = qn.map((v, j) => v - q[j]); q = qn; qPrev = qPrev.map((v, j) => v + off[j]); aug = true; }
|
||||
else nAugRej++;
|
||||
}
|
||||
const pos = k => { const i = Math.min(N, t + k); const w = aug ? Math.max(0, 1 - k / 8) : 0; return traj[i].map((v, j) => v + off[j] * w); };
|
||||
@@ -72,4 +75,4 @@ writeNpy(path.join(OUTDIR, base + '.V.npy'), Float32Array.from(V), [n]);
|
||||
writeNpy(path.join(OUTDIR, base + '.SG.npy'), cat(SG, 13), [n, 13]);
|
||||
writeNpy(path.join(OUTDIR, base + '.TID.npy'), TID, [n], 'int32');
|
||||
fs.writeFileSync(path.join(OUTDIR, base + '.tasks.jsonl'), tasks.map(t => JSON.stringify(t)).join('\n') + '\n');
|
||||
console.log(JSON.stringify({ shard: base, tasks: tasks.length, samples: n, aug: nAug, augRejected: nAugRej, featVer: POL.FEAT_VER }));
|
||||
console.log(JSON.stringify({ shard: base, tasks: tasks.length, samples: n, aug: nAug, augRejected: nAugRej, gapDrop: nGapDrop, featVer: POL.FEAT_VER }));
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""流水线状态文件 ml/runs/pipeline.json（供训练监控网页显示"现在在哪一步、每步用了多久"）。
|
||||
pipeline.sh 每个阶段开始/结束时自动调用，不需要手动运行：
|
||||
python -m ml.pstate reset <阶段1,阶段2,...> # `all` 开始时：清空并写入计划
|
||||
python -m ml.pstate begin <阶段> [键=值 ...] # 阶段开始（值可为数字），如 target_tasks=50000
|
||||
python -m ml.pstate set <阶段> 键=值 ... # 阶段中途更新（如补量后的目标条数）
|
||||
python -m ml.pstate end <阶段> <返回码>
|
||||
2026-10-18 新增（用户要求：从训练开始到结束都能在监控面板看到进度）。"""
|
||||
import json, os, sys, time
|
||||
|
||||
P = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ml', 'runs', 'pipeline.json')
|
||||
|
||||
|
||||
def load():
|
||||
try:
|
||||
with open(P, encoding='utf-8') as f: return json.load(f)
|
||||
except Exception:
|
||||
return {'started': time.time(), 'plan': None, 'stages': {}, 'current': None}
|
||||
|
||||
|
||||
def save(d):
|
||||
os.makedirs(os.path.dirname(P), exist_ok=True)
|
||||
tmp = P + '.tmp'
|
||||
with open(tmp, 'w', encoding='utf-8') as f: json.dump(d, f, ensure_ascii=False, indent=1)
|
||||
os.replace(tmp, P)
|
||||
|
||||
|
||||
def kv(args):
|
||||
out = {}
|
||||
for a in args:
|
||||
k, _, v = a.partition('=')
|
||||
try: v = float(v) if '.' in v else int(v)
|
||||
except ValueError: pass
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
cmd, *rest = sys.argv[1:] or ['?']
|
||||
d = load(); now = time.time()
|
||||
if cmd == 'reset':
|
||||
plan = rest[0].split(',') if rest else []
|
||||
# 从中间某步续跑时保留计划之外（之前已跑完）各步的记录，计划内的清空重来
|
||||
d = {'started': now, 'plan': plan, 'stages': {k: v for k, v in d.get('stages', {}).items() if k not in plan}, 'current': None}
|
||||
elif cmd == 'begin':
|
||||
st = rest[0]
|
||||
d['stages'][st] = {'start': now, 'end': None, 'rc': None, 'info': kv(rest[1:])}
|
||||
d['current'] = st
|
||||
elif cmd == 'set':
|
||||
s = d['stages'].setdefault(rest[0], {'start': now, 'end': None, 'rc': None, 'info': {}})
|
||||
s['info'].update(kv(rest[1:]))
|
||||
elif cmd == 'end':
|
||||
st, rc = rest[0], int(rest[1]) if len(rest) > 1 else 0
|
||||
s = d['stages'].setdefault(st, {'start': now, 'info': {}})
|
||||
s['end'], s['rc'] = now, rc
|
||||
if d.get('current') == st: d['current'] = None
|
||||
else:
|
||||
print(__doc__); return
|
||||
save(d)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+2
-2
@@ -97,14 +97,14 @@ def main():
|
||||
opt.step(); seen += X.shape[0]
|
||||
if step % tc['log_every'] == 0:
|
||||
el = time.time() - t0
|
||||
rec = {'step': step, 'loss': loss.item(), 'mf': mf_raw.item(), 'fk_cm': l_fk.item(), 'smooth': l_sm.item(),
|
||||
rec = {'step': step, 'total': total, 't': time.time(), 'loss': loss.item(), 'mf': mf_raw.item(), 'fk_cm': l_fk.item(), 'smooth': l_sm.item(),
|
||||
'value': l_val.item(), 'sub': l_sub.item(), 'gnorm': gn.item(), 'lr': lr_at(step), 'samples_per_s': seen / max(el, 1e-6),
|
||||
'eta_h': (total - step - 1) * tc['batch'] / max(seen / max(el, 1e-6), 1e-6) / 3600}
|
||||
print(json.dumps(rec), flush=True); log.write(json.dumps(rec) + '\n'); log.flush()
|
||||
if not a.no_eval and (step + 1) % tc['eval_every'] == 0:
|
||||
model.eval()
|
||||
res, _ = evaluate(model, kin, val_tasks, dev)
|
||||
res['step'] = step + 1
|
||||
res['step'] = step + 1; res['t'] = time.time()
|
||||
print('EVAL', json.dumps(res), flush=True); log.write(json.dumps({'eval': res}) + '\n'); log.flush()
|
||||
save('last.pt', step + 1)
|
||||
if res['success'] > best + 1e-4:
|
||||
|
||||
Reference in New Issue
Block a user