Observability & Logging¶
Status: current
GraphIDS has three observability surfaces:
- Per-run saved configuration, authoritative manifest and lifecycle events.
- MLflow SQLite for post-run params, tags, metrics, and artifacts.
- SLURM stdout/stderr logs for process text.
Run Directory¶
{run_root}/{dataset}/{experiment_name}/{run_name}/
resolved-run.yml
.graphids/
manifest.json
events.jsonl
checkpoints/
artifacts/
gx exp status <run_dir> reads .graphids/manifest.json and
.graphids/events.jsonl.
MLflow¶
By default:
Training and evaluation jobs do not write to the shared MLflow SQLite database
while running. Schema-2 manifests hold final status, metrics and failures; a
separate serialized step derives an MLflow export from that manifest and
resolved-run.yml. Historical mlflow_ingest.json files remain supported:
For controlled debugging, GRAPHIDS_MLFLOW_MODE=online restores the live
Lightning MLFlowLogger. In either path, the default MLflow tracking URI is:
GRAPHIDS_MLFLOW_MODE=disabled (also off or none) keeps results local.
The saved manifest records this choice; batch ingestion skips those runs.
Experiment names are created as:
Run names are RunConfig.name, normally the experiment name.
Tags¶
graphids.exp.tracking.mlflow_tags(run) writes:
graphids.stage
graphids.run_dir
graphids.git_sha
graphids.dataset
graphids.seed
graphids.plan_id
graphids.representation
Params¶
graphids.exp.tracking.mlflow_hparams(run) writes:
graphids.stage
graphids.dataset
graphids.seed
graphids.plan_id
graphids.git_sha
graphids.run_dir
graphids.representation
graphids.representation_cfg/*
graphids.payload/*
graphids.resource/*
Nested payload fields are flattened by gx exp ingest.
Metrics¶
The event model logs separately normalized classification and normal-only payload
losses, plus exact frame metrics such as val/classification_frame_ap.
graphids.exp.run captures scalar trainer.callback_metrics after
fit and test into the authoritative manifest; vector metrics are omitted.
graphids._mlflow.MLflowSystemMetricsCallback records phase throughput under
.graphids/phase_metrics.json. When a live MLflow run is available it also starts
MLflow's native system metrics monitor.
Phase bookkeeping accumulates detached integer event counts and float64 time
sums on each batch device. It transfers totals once at phase end before recording
elapsed wall time. The scored mask determines event counts independently of
label availability; exceptions discard unfinished counters.
Offline diagnostic runs can include
graphids._mlflow.MLflowSystemMetricsSummaryCallback. It maps native MLflow
histories to resource/* CPU, system-memory, and GPU summaries, which are copied
from final callback metrics into the manifest for subsequent MLflow export.
resource/rss_gib_* is a retained metric name for system memory usage, not
process RSS. Missing optional GPU histories leave GPU summary keys absent.
Summary metric families share history collection and reductions. GPU utilization
means pool samples across devices, memory summaries retain their existing units,
and resource/samples is the largest individual history length. Disabling GPU
summaries avoids GPU-history lookups; unavailable histories remain optional.
SLURM Logs¶
Current jobs write under:
/fs/ess/PAS1266/graphids/slurm_logs/
scripts/{experiment_name}.sbatch
{experiment_name}_{job_id}.out
{experiment_name}_{job_id}.err
Examples:
tail -n 80 /fs/ess/PAS1266/graphids/slurm_logs/graph_mamba_events_source_seed42_47912353.err
sacct -j 47912353 --format=JobID,JobName%36,State,ExitCode,Elapsed -P
squeue -j 47912353 -o '%i %j %T %M %R'
Querying MLflow¶
from graphids._mlflow import configure_tracking_uri
from mlflow.tracking import MlflowClient
configure_tracking_uri()
client = MlflowClient()
experiments = client.search_experiments(filter_string="name LIKE 'graphids/%'")
exp_ids = [e.experiment_id for e in experiments]
runs = client.search_runs(
exp_ids,
filter_string="tags.graphids.plan_id = 'graph_mamba_events_source_seed42'",
)
for run in runs:
print(run.info.run_id, run.data.metrics)