Training on the Linux / RTX 3090 box

Remote-training workflow: run long training sessions on the Linux machine, watch the dashboard from any machine on the network, and ship the trained model back to the Mac mini automatically when the run finishes. General training concepts and the export/evaluate workflow live in TRAINING.md — this doc is only what differs on the Linux box.

One-time setup

# GitHub auth (one-time): git-over-HTTPS no longer accepts account passwords,
# so clone over SSH. Generate a key, then add the printed public key at
# github.com/settings/keys → "New SSH key".
ssh-keygen -t ed25519        # accept the defaults
cat ~/.ssh/id_ed25519.pub

cd ~/ai-training
git clone git@github.com:jcreek/CosmicClash.git
# (submodules are editor tooling only — training doesn't need them)

# Godot 4.7.1 Linux binary
mkdir -p ~/ai-training/godot && cd ~/ai-training/godot
wget https://github.com/godotengine/godot/releases/download/4.7.1-stable/Godot_v4.7.1-stable_linux.x86_64.zip
unzip Godot_v4.7.1-stable_linux.x86_64.zip
echo 'export GODOT_BIN=~/ai-training/godot/Godot_v4.7.1-stable_linux.x86_64' >> ~/.bashrc && source ~/.bashrc

# Python env
cd ~/ai-training/CosmicClash/training
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# CUDA sanity check — should print True
.venv/bin/python -c "import torch; print(torch.cuda.is_available())"

# Headless smoke test — the game must boot without rendering
$GODOT_BIN --headless --path ../Game res://scenes/free_play.tscn --quit-after 300

If the CUDA check prints False, reinstall torch from the CUDA index (pip install torch --index-url https://download.pytorch.org/whl/cu121).

Maximising throughput

Env stepping is CPU-bound (each --n-parallel instance is one headless Godot process simulating 2 agents); the 3090 only accelerates the PPO updates. So the two levers are instance count and in-engine speedup:

--n-parallel: start at nproc minus 2 (leave headroom for the trainer process itself). The Mac mini sustains 6; a many-core box should take considerably more. Each instance opens its own TCP port upward from --port (default 11008).
--speedup: in-engine physics time-scale. 16 is proven; try 24–32 and keep raising while time/fps in the console/TensorBoard still scales up. Back off if fps stops improving (CPU saturated) or physics glitches appear (ball tunnelling, ships escaping the arena — watch for respawn warnings in the Godot output).

Tune by watching time/fps: run a 2-minute smoke run per setting and keep the best. Reference: 6 instances × speedup 16 ≈ 1,385 steps/s on an M4 Mac mini — a 20M-step run in ~4 h. Doubling fps halves that.

Start a run

Fresh run:

cd ~/ai-training/CosmicClash/training
.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
    --n-parallel 14 --speedup 24

Resuming a previous policy (continues its timestep counter; --timesteps is additional steps). Lessons from run01/run02 hard-coded into flags:

.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
    --n-parallel 14 --speedup 24 \
    --resume checkpoints/run02/final.zip --ent-coef 0.001 --reset-std 0.3

--ent-coef — entropy bonus. 0.0001 collapsed the policy std to 0.075 by 20M steps (no exploration left); 0.005 blew it up to 3.0 (random play). 0.001 is the current middle. Healthy train/std drifts between ~0.2 and ~1.0 — check it 30–45 min in before committing to a long run.
--reset-std — on resume, restores exploration a collapsed checkpoint lost.

Run inside tmux/screen so an SSH disconnect doesn't kill training. Ctrl-C is safe: final.zip is written on the way out.

Auto-copy the result to the Mac mini when training finishes

One-time: enable System Settings → General → Sharing → Remote Login on the Mac mini, and ssh-copy-id jcreek@Joshs-Mac-mini.local from the Linux box so rsync runs unattended.

Chain export + copy onto the training command (; not &&, so the copy still happens after a Ctrl-C — final.zip exists either way):

EXP=run03
.venv/bin/python train.py --experiment $EXP --timesteps 20000000 --n-parallel 14 --speedup 24 ; \
.venv/bin/python export_policy.py checkpoints/$EXP/final.zip ../Game/bots/$EXP.json && \
rsync -av checkpoints/$EXP/final.zip \
    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/training/checkpoints/$EXP/ && \
rsync -av ../Game/bots/$EXP.json \
    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/Game/bots/

That lands both the raw checkpoint (for future --resume / evaluation on the Mac) and the exported JSON policy (immediately playable — point Match or Spectate mode at res://bots/<exp>.json). Add a third rsync of logs/ if you also want the TensorBoard history archived on the Mac.

Dashboard over the network

On the Linux box, bind TensorBoard to all interfaces instead of localhost:

cd ~/ai-training/CosmicClash/training
.venv/bin/tensorboard --logdir logs --host 0.0.0.0 --port 6006

Then from the Mac (or anything on the LAN): http://<linux-box-hostname>:6006.

If ufw is active on the box: sudo ufw allow 6006/tcp.
If you'd rather not open a port, tunnel instead: ssh -L 6006:localhost:6006 <linux-box> from the Mac, then browse http://localhost:6006.

5.3 KiB Raw Blame History Unescape Escape