feat(*): Add exported Linux binary training path for faster parallel instances

This commit is contained in:
Josh Creek
2026-07-28 20:56:14 +01:00
parent 4408ec3ecd
commit 01dbfc7ede
10 changed files with 179 additions and 11 deletions
+50
View File
@@ -112,6 +112,56 @@ On the Mac (or anywhere), collecting the results is just `git pull`. A
20M-step run adds roughly 40 MB of checkpoints — acceptable growth for the
guarantee that training is never lost with a machine.
## Exported-binary training (faster parallel startup)
By default (and in every example above) `train.py` runs the project from
source via `--godot_bin` — each of the `--n-parallel` instances re-parses
project settings and re-imports scripts/resources on launch. An **exported**
build skips that: resources are pre-imported and packed once at export time,
so each instance just loads a binary. Worth it once `--n-parallel` is large
enough that per-instance startup overhead adds up (i.e. this box, not the Mac
mini's `--n-parallel 6`).
Opt in once — `run_training.sh` (and so `start_training.sh`/`next_run.sh`/
`curriculum.sh`, which all funnel through it) takes it from there automatically:
```bash
./export_linux.sh # one-time opt-in: builds training/build/CosmicClash.x86_64
./next_run.sh # from here on, every standing/curriculum run uses it
```
You don't need to (and shouldn't) pass `--exported-binary` yourself through
those entry points — `run_training.sh` adds it whenever `training/build/`
exists, after re-exporting against whatever `git pull` just fetched. Calling
`train.py` directly still takes it explicitly, same as any other flag:
```bash
.venv/bin/python train.py --experiment run04 --exported-binary build/CosmicClash.x86_64 \
--timesteps 20000000 --n-parallel 14 --speedup 24
```
To go back to a source run permanently, delete `training/build/` — with it
gone, `run_training.sh` stops re-exporting and stops adding the flag, so
`next_run.sh`/`curriculum.sh` revert to plain source runs with no code changes.
`export_linux.sh` builds from the "Linux Training" preset in
`Game/export_presets.cfg`, which is training-only — its
`custom_features="training"` activates project.godot's
`run/main_scene.training` override, so the resulting binary boots straight
into `training.tscn` on its own. This indirection is required, not
incidental: official Godot export templates have path/scene overrides
compiled out, so passing `--scene` at launch time (the way the source run
does) hard-aborts an exported binary with "compiled without support for path
overrides" — there's no way to redirect an exported build to a different
scene at runtime. Because the main scene is baked in at export time, this
preset can't later double as a normal "ship the game" Linux build (which
would need `main_menu.tscn` and no training feature tag) — a real game export
would need its own separate preset.
`setup_linux.sh` installs the export templates this needs alongside the
Godot binary. If you never opt in (no `training/build/` directory), this
costs nothing — training stays a plain source run.
## Dashboard over the network
`start_training.sh` already serves TensorBoard on all interfaces — browse to