# Training on the Linux / RTX 3090 box

Remote-training workflow: run long training sessions on the Linux machine,
watch the dashboard from any machine on the network, and ship the trained
model back to the Mac mini automatically when the run finishes. General
training concepts and the export/evaluate workflow live in
[TRAINING.md](TRAINING.md) — this doc is only what differs on the Linux box.

## One-time setup

```bash
# GitHub auth (one-time): git-over-HTTPS no longer accepts account passwords,
# so clone over SSH. Generate a key, then add the printed public key at
# github.com/settings/keys → "New SSH key".
ssh-keygen -t ed25519        # accept the defaults
cat ~/.ssh/id_ed25519.pub

cd ~/ai-training
git clone git@github.com:jcreek/CosmicClash.git
# (submodules are editor tooling only — training doesn't need them)

# Godot 4.7.1 Linux binary
mkdir -p ~/ai-training/godot && cd ~/ai-training/godot
wget https://github.com/godotengine/godot/releases/download/4.7.1-stable/Godot_v4.7.1-stable_linux.x86_64.zip
unzip Godot_v4.7.1-stable_linux.x86_64.zip
echo 'export GODOT_BIN=~/ai-training/godot/Godot_v4.7.1-stable_linux.x86_64' >> ~/.bashrc && source ~/.bashrc

# Python env
cd ~/ai-training/CosmicClash/training
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# CUDA sanity check — should print True
.venv/bin/python -c "import torch; print(torch.cuda.is_available())"

# Headless smoke test — the game must boot without rendering
$GODOT_BIN --headless --path ../Game res://scenes/free_play.tscn --quit-after 300
```

If the CUDA check prints `False`, reinstall torch from the CUDA index
(`pip install torch --index-url https://download.pytorch.org/whl/cu121`).

## Maximising throughput

Env stepping is CPU-bound (each `--n-parallel` instance is one headless Godot
process simulating 2 agents); the 3090 only accelerates the PPO updates. So
the two levers are instance count and in-engine speedup:

- **`--n-parallel`**: start at `nproc` minus 2 (leave headroom for the
  trainer process itself). The Mac mini sustains 6; a many-core box should
  take considerably more. Each instance opens its own TCP port upward from
  `--port` (default 11008).
- **`--speedup`**: in-engine physics time-scale. 16 is proven; try 24–32 and
  keep raising while `time/fps` in the console/TensorBoard still scales up.
  Back off if fps stops improving (CPU saturated) or physics glitches appear
  (ball tunnelling, ships escaping the arena — watch for respawn warnings in
  the Godot output).

Tune by watching `time/fps`: run a 2-minute smoke run per setting and keep
the best. Reference: 6 instances × speedup 16 ≈ 1,385 steps/s on an M4 Mac
mini — a 20M-step run in ~4 h. Doubling fps halves that.

## Start a run

Fresh run:

```bash
cd ~/ai-training/CosmicClash/training
.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
    --n-parallel 14 --speedup 24
```

Resuming a previous policy (continues its timestep counter; `--timesteps` is
*additional* steps). Lessons from run01/run02 hard-coded into flags:

```bash
.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
    --n-parallel 14 --speedup 24 \
    --resume checkpoints/run02/final.zip --ent-coef 0.001 --reset-std 0.3
```

- `--ent-coef` — entropy bonus. `0.0001` collapsed the policy std to 0.075 by
  20M steps (no exploration left); `0.005` blew it up to 3.0 (random play).
  `0.001` is the current middle. Healthy `train/std` drifts between ~0.2 and
  ~1.0 — check it 30–45 min in before committing to a long run.
- `--reset-std` — on resume, restores exploration a collapsed checkpoint lost.

Run inside `tmux`/`screen` so an SSH disconnect doesn't kill training.
Ctrl-C is safe: `final.zip` is written on the way out.

## Auto-copy the result to the Mac mini when training finishes

One-time: enable **System Settings → General → Sharing → Remote Login** on
the Mac mini, and `ssh-copy-id jcreek@Joshs-Mac-mini.local` from the Linux
box so rsync runs unattended.

Chain export + copy onto the training command (`;` not `&&`, so the copy
still happens after a Ctrl-C — `final.zip` exists either way):

```bash
EXP=run03
.venv/bin/python train.py --experiment $EXP --timesteps 20000000 --n-parallel 14 --speedup 24 ; \
.venv/bin/python export_policy.py checkpoints/$EXP/final.zip ../Game/bots/$EXP.json && \
rsync -av checkpoints/$EXP/final.zip \
    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/training/checkpoints/$EXP/ && \
rsync -av ../Game/bots/$EXP.json \
    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/Game/bots/
```

That lands both the raw checkpoint (for future `--resume` / evaluation on the
Mac) and the exported JSON policy (immediately playable — point Match or
Spectate mode at `res://bots/<exp>.json`). Add a third rsync of `logs/` if
you also want the TensorBoard history archived on the Mac.

## Dashboard over the network

On the Linux box, bind TensorBoard to all interfaces instead of localhost:

```bash
cd ~/ai-training/CosmicClash/training
.venv/bin/tensorboard --logdir logs --host 0.0.0.0 --port 6006
```

Then from the Mac (or anything on the LAN): `http://<linux-box-hostname>:6006`.

- If `ufw` is active on the box: `sudo ufw allow 6006/tcp`.
- If you'd rather not open a port, tunnel instead:
  `ssh -L 6006:localhost:6006 <linux-box>` from the Mac, then browse
  `http://localhost:6006`.