Vizualize training using TensorBoard and Weights and Biases(W&B)¶

Few things to keep in mind when visualizing training on HPC clusters:

Compute nodes have no internet
Login and/or devel nodes have internet
Logs live on a filesystem accessible from the nodes you use to view/sync (typically $SCRATCH, $PROJECT, etc.)

TensorBoard¶

If your training wrote TensorBoard event files under a directory like runs/exp_001/tb, start TensorBoard from a node that can access that path:

tensorboard --logdir runs/exp_001/tb --port 6006

Tips:

If you manage multiple experiments under one root (e.g., runs/), you can point TensorBoard at the root:

tensorboard --logdir runs --port 6006

Keep TensorBoard running after disconnecting by using tmux:

tmux new -s tb
tensorboard --logdir runs/exp_001/tb --port 6006
# detach with: Ctrl-b d

View TensorBoard from your local machine(laptop) via SSH tunneling¶

From your laptop, open a tunnel to the login/devel node where TensorBoard is running:

ssh -N -L 6006:localhost:6006 user@<login-or-devel-host>

Make sure the hostname of the login-node is exactly the one where you started TensorBoard. Also make sure the port on your local machine is not used by other processes

Then open in your browser: http://localhost:6006

Weights & Biases (W&B) (Offline → Sync Later)¶

Setup once (on login/devel with internet)¶

On a node with internet (login/devel), authenticate your W&B client:

wandb login

Do this once per environment (or whenever your auth expires).

Sync offline runs from login/devel (internet)¶

Compute-node jobs should produce offline run directories on shared storage. Later, from a node with internet (login/devel), sync them to W&B cloud.

If you know the run directory:

wandb sync /path/to/WANDB_DIR/wandb/offline-run-*

If you have multiple runs and want to sync everything under the folder:

wandb sync /path/to/WANDB_DIR/wandb

Notes:

Sync is safe to re-run; it will skip already-uploaded data in most cases.
If you don’t see offline-run-* directories, you may be pointing at the wrong location (see quick checks below).

Quick checks:

ls -lah /path/to/WANDB_DIR
find /path/to/WANDB_DIR -maxdepth 3 -type d -name "offline-run-*" | head

W&B online on devel nodes only (optional)¶

If you sometimes run on a devel node that has internet and you want W&B to upload live:

Enable online mode:

export WANDB_MODE=online

For compute nodes without internet, ensure offline mode:

export WANDB_MODE=offline

This lets you keep the same workflow while controlling behavior per node type.