jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-10 16:04:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(*): Add idempotent tmux-based training scripts for the Linux box

Browse Source
This commit is contained in:
Josh Creek
2026-07-19 10:27:21 +01:00
parent 1b3c2e063d
commit f4722dbd8d
2 changed files with 63 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TRAINING_LINUX.md
+20 -19
View File
@@ -62,25 +62,31 @@ mini — a 20M-step run in ~4 h. Doubling fps halves that.
## Run training
`run_training.sh <experiment> [train.py args...]` wraps the whole cycle:
`git pull` → train → export the policy JSON → commit and push checkpoints,
logs, and the exported bot. Run it inside `tmux`/`screen` so an SSH
disconnect doesn't kill training. Ctrl-C is safe: the trainer writes
`final.zip` on the way out, and the script still exports, commits, and
pushes what it has.
Fresh run:
One command does everything (needs `tmux`: `sudo apt install tmux`):
```bash
cd ~/ai-training/CosmicClash/training
./run_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24
./start_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24
```
`start_training.sh` launches a detached tmux session with two windows —
**training survives SSH disconnects** — and prints the dashboard URL:
- `train` runs `run_training.sh`: `git pull` → train → export the policy
JSON → commit and push checkpoints, logs, and the exported bot;
- `dashboard` serves TensorBoard on `0.0.0.0:6006` for the whole network
(reused if one is already running).
Re-running `start_training.sh` while a session exists just attaches you to
it (detach again with `Ctrl-B` then `D`) — it will never start a second
trainer. To stop training, attach and Ctrl-C: the trainer writes `final.zip`
on the way out and the script still exports, commits, and pushes what it has.
Resuming a previous policy (continues its timestep counter; `--timesteps` is
*additional* steps). Lessons from run01/run02 hard-coded into flags:
```bash
./run_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24 \
./start_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24 \
--resume checkpoints/run02/final.zip --ent-coef 0.001 --reset-std 0.3
```
@@ -108,16 +114,11 @@ guarantee that training is never lost with a machine.
## Dashboard over the network
On the Linux box, bind TensorBoard to all interfaces instead of localhost:
```bash
cd ~/ai-training/CosmicClash/training
.venv/bin/tensorboard --logdir logs --host 0.0.0.0 --port 6006
```
Then from the Mac (or anything on the LAN): `http://<linux-box-hostname>:6006`.
`start_training.sh` already serves TensorBoard on all interfaces — browse to
the URL it prints (`http://10.0.0.20:6006`) from the Mac or anything on
the LAN.
- If `ufw` is active on the box: `sudo ufw allow 6006/tcp`.
- If you'd rather not open a port, tunnel instead:
`ssh -L 6006:localhost:6006 <linux-box>` from the Mac, then browse
`ssh -L 6006:localhost:6006 10.0.0.20` from the Mac, then browse
`http://localhost:6006`.