mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
Josh Creek
130 lines
5.3 KiB
Markdown
130 lines
5.3 KiB
Markdown
# Training on the Linux / RTX 3090 box
|
||
|
||
Remote-training workflow: run long training sessions on the Linux machine,
|
||
watch the dashboard from any machine on the network, and ship the trained
|
||
model back to the Mac mini automatically when the run finishes. General
|
||
training concepts and the export/evaluate workflow live in
|
||
[TRAINING.md](TRAINING.md) — this doc is only what differs on the Linux box.
|
||
|
||
## One-time setup
|
||
|
||
```bash
|
||
# GitHub auth (one-time): git-over-HTTPS no longer accepts account passwords,
|
||
# so clone over SSH. Generate a key, then add the printed public key at
|
||
# github.com/settings/keys → "New SSH key".
|
||
ssh-keygen -t ed25519 # accept the defaults
|
||
cat ~/.ssh/id_ed25519.pub
|
||
|
||
cd ~/ai-training
|
||
git clone git@github.com:jcreek/CosmicClash.git
|
||
# (submodules are editor tooling only — training doesn't need them)
|
||
|
||
# Godot 4.7.1 Linux binary
|
||
mkdir -p ~/ai-training/godot && cd ~/ai-training/godot
|
||
wget https://github.com/godotengine/godot/releases/download/4.7.1-stable/Godot_v4.7.1-stable_linux.x86_64.zip
|
||
unzip Godot_v4.7.1-stable_linux.x86_64.zip
|
||
echo 'export GODOT_BIN=~/ai-training/godot/Godot_v4.7.1-stable_linux.x86_64' >> ~/.bashrc && source ~/.bashrc
|
||
|
||
# Python env
|
||
cd ~/ai-training/CosmicClash/training
|
||
python3 -m venv .venv
|
||
.venv/bin/pip install -r requirements.txt
|
||
|
||
# CUDA sanity check — should print True
|
||
.venv/bin/python -c "import torch; print(torch.cuda.is_available())"
|
||
|
||
# Headless smoke test — the game must boot without rendering
|
||
$GODOT_BIN --headless --path ../Game res://scenes/free_play.tscn --quit-after 300
|
||
```
|
||
|
||
If the CUDA check prints `False`, reinstall torch from the CUDA index
|
||
(`pip install torch --index-url https://download.pytorch.org/whl/cu121`).
|
||
|
||
## Maximising throughput
|
||
|
||
Env stepping is CPU-bound (each `--n-parallel` instance is one headless Godot
|
||
process simulating 2 agents); the 3090 only accelerates the PPO updates. So
|
||
the two levers are instance count and in-engine speedup:
|
||
|
||
- **`--n-parallel`**: start at `nproc` minus 2 (leave headroom for the
|
||
trainer process itself). The Mac mini sustains 6; a many-core box should
|
||
take considerably more. Each instance opens its own TCP port upward from
|
||
`--port` (default 11008).
|
||
- **`--speedup`**: in-engine physics time-scale. 16 is proven; try 24–32 and
|
||
keep raising while `time/fps` in the console/TensorBoard still scales up.
|
||
Back off if fps stops improving (CPU saturated) or physics glitches appear
|
||
(ball tunnelling, ships escaping the arena — watch for respawn warnings in
|
||
the Godot output).
|
||
|
||
Tune by watching `time/fps`: run a 2-minute smoke run per setting and keep
|
||
the best. Reference: 6 instances × speedup 16 ≈ 1,385 steps/s on an M4 Mac
|
||
mini — a 20M-step run in ~4 h. Doubling fps halves that.
|
||
|
||
## Start a run
|
||
|
||
Fresh run:
|
||
|
||
```bash
|
||
cd ~/ai-training/CosmicClash/training
|
||
.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
|
||
--n-parallel 14 --speedup 24
|
||
```
|
||
|
||
Resuming a previous policy (continues its timestep counter; `--timesteps` is
|
||
*additional* steps). Lessons from run01/run02 hard-coded into flags:
|
||
|
||
```bash
|
||
.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
|
||
--n-parallel 14 --speedup 24 \
|
||
--resume checkpoints/run02/final.zip --ent-coef 0.001 --reset-std 0.3
|
||
```
|
||
|
||
- `--ent-coef` — entropy bonus. `0.0001` collapsed the policy std to 0.075 by
|
||
20M steps (no exploration left); `0.005` blew it up to 3.0 (random play).
|
||
`0.001` is the current middle. Healthy `train/std` drifts between ~0.2 and
|
||
~1.0 — check it 30–45 min in before committing to a long run.
|
||
- `--reset-std` — on resume, restores exploration a collapsed checkpoint lost.
|
||
|
||
Run inside `tmux`/`screen` so an SSH disconnect doesn't kill training.
|
||
Ctrl-C is safe: `final.zip` is written on the way out.
|
||
|
||
## Auto-copy the result to the Mac mini when training finishes
|
||
|
||
One-time: enable **System Settings → General → Sharing → Remote Login** on
|
||
the Mac mini, and `ssh-copy-id jcreek@Joshs-Mac-mini.local` from the Linux
|
||
box so rsync runs unattended.
|
||
|
||
Chain export + copy onto the training command (`;` not `&&`, so the copy
|
||
still happens after a Ctrl-C — `final.zip` exists either way):
|
||
|
||
```bash
|
||
EXP=run03
|
||
.venv/bin/python train.py --experiment $EXP --timesteps 20000000 --n-parallel 14 --speedup 24 ; \
|
||
.venv/bin/python export_policy.py checkpoints/$EXP/final.zip ../Game/bots/$EXP.json && \
|
||
rsync -av checkpoints/$EXP/final.zip \
|
||
jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/training/checkpoints/$EXP/ && \
|
||
rsync -av ../Game/bots/$EXP.json \
|
||
jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/Game/bots/
|
||
```
|
||
|
||
That lands both the raw checkpoint (for future `--resume` / evaluation on the
|
||
Mac) and the exported JSON policy (immediately playable — point Match or
|
||
Spectate mode at `res://bots/<exp>.json`). Add a third rsync of `logs/` if
|
||
you also want the TensorBoard history archived on the Mac.
|
||
|
||
## Dashboard over the network
|
||
|
||
On the Linux box, bind TensorBoard to all interfaces instead of localhost:
|
||
|
||
```bash
|
||
cd ~/ai-training/CosmicClash/training
|
||
.venv/bin/tensorboard --logdir logs --host 0.0.0.0 --port 6006
|
||
```
|
||
|
||
Then from the Mac (or anything on the LAN): `http://<linux-box-hostname>:6006`.
|
||
|
||
- If `ufw` is active on the box: `sudo ufw allow 6006/tcp`.
|
||
- If you'd rather not open a port, tunnel instead:
|
||
`ssh -L 6006:localhost:6006 <linux-box>` from the Mac, then browse
|
||
`http://localhost:6006`.
|