Skip to content

Message-preserving graph–Mamba input

The active model path uses can_event_dm with event_graph_mamba. Every observed CAN frame enters the temporal encoder in physical recording order. This follows the requirement to preserve raw observations and let the model learn its own representations. The snapshot implementation is retired to the verified source archive.

The active preset is configs/experiments/graph_mamba_events_source.yml. It selects the per-ID timing-gap model from the October 30 matched ablations: source-validation classification AP 0.9257 at epoch 25, compared with 0.0478 for the one-stream baseline. This is one seed and a source-validation comparison; weak attack-family coverage and independent operating-point evaluation remain open. Snapshot weights and calibration use a separate architecture and cannot load here.

Selected per-ID timing gap

The model and can_event representation default to schema 2. Each frame retains raw elapsed time and bus interarrival time, plus three header features: normalized preceding same-ID gap, normalizer support, and first-seen indicator. The encoder has 51 input fields and the forecast header has 35. Representation 2 prepares same-ID predecessor indices across chunks without changing the lossless on-disk caches. The index costs eight bytes per event; sorting needs additional temporary CPU memory during setup. Preceding payload copies and graph-entry timestamps are no longer carried.

Normalizers fit only verified-normal train_query captures, using at most 65,536 deterministic evenly spaced pairs per capture. They apply log1p(gap / scale) with median positive-gap scale, robust median/IQR centering, and a frozen 99.9th percentile clip at least as large as the scale. Near-zero IQR uses unit spread. Supported numeric IDs have their own tables; other IDs use the pooled same-ID-gap table and a false support indicator. timing_min_support defaults to 32 sampled pairs. Fitted tables are checkpoint buffers, validated on load and never refitted from evaluation traffic. The legacy unused bus-normalizer row remains in the buffer to preserve selected-checkpoint identity.

The active preset uses one stream, unweighted BCE, a joint payload objective, 25 epochs, 16,384-event chunks and a 65,536-event optimizer window. python -m scripts.event_quality_suite prepare OUTPUT --seed 42 freezes one candidate. submit OUTPUT creates three jobs: a CUDA gate, the fit after a successful gate, and a report after the fit terminates. status OUTPUT refreshes accounting and candidate-results.json. Separate directories and --seed select repeat runs. Preparation alone does not submit anything.

Each epoch summary includes recording_macro_ap over mixed-class recordings and recordings_by_ap ordered weakest first. Undefined single-class AP is excluded from that mean; all recording metrics remain available. The CLI shows the three weakest recordings at the aggregate-AP-selected epoch. Checkpoint selection still uses aggregate AP.

The weakest-recording lines compare classification and payload AP at that same epoch, including supported payload rows versus known classification rows. These APs may cover different populations. Classification-only runs show untrained instead of presenting their payload scores as a detector result.

For the separate classifier diagnostic, run python -m scripts.event_classifier_pilot --config WINNER.yml --checkpoint WINNER.ckpt --output NEW_DIRECTORY inside one A100 allocation. It freezes the checkpoint and timing buffers, disables dropout, and trains two identically initialized linear heads for five epochs: ordinary BCE and training-recording-balanced attack positives capped at 10. Negatives and total positive weight retain their original exposure. Both arms share each causal forward pass and the original loader/windows. The final heads and original classifier are evaluated on the same complete validation cohort. pilot.json contains metrics, weights and coverage; classifier-heads.pt contains only the adapted heads and base checkpoint identity. This is a diagnostic adaptation stage, not full-model resume or a change to the selected preset. The repository's MODEL_IMPROVEMENT_PLAN.md records the pilot protocol and measured results.

python -m scripts.event_failure_report --pilot PILOT_ROOT --focus FOCUS.json --output NEW_DIRECTORY diagnoses saved pilot predictions in a CPU allocation. The focus file maps explicit validation recording keys to lists of matched training keys, including empty lists for unmatched captures. The report verifies all source-validation rows, fits vehicle-scoped byte/transition support only on verified-normal training queries, reconstructs checkpoint-owned timing support, and compares heads at empirical normal-validation frame budgets. Unsupported payload rows remain in recall denominators. report.json and report.md record exposure, support, score distributions, disagreement, matched-budget decisions and physical-row examples. These are source-selection diagnostics; they do not refit the encoder, open target files or provide calibrated alerts/hour.

python -m scripts.event_causal_controls --pilot PILOT_ROOT --output NEW_DIRECTORY compares two simple controls with the original classifier and payload scores on every source-validation capture, inside a CPU allocation. ID novelty flags IDs absent from that vehicle's verified-normal training union. The timing control uses the absolute normalized same-ID gap, with separate frozen training tables per vehicle and the existing log/median/IQR/clip rule. It requires 32 sampled training pairs per ID; first occurrences and unsupported IDs abstain. Neither control assumes matching numeric IDs have shared semantics across vehicles. Unprofiled vehicles abstain. These controls are source diagnostics, separate from the model's checkpoint-owned, pooled normalizer.

The report includes the fixed ID-novelty alarm, empirical normal-frame budgets, known-ID/startup/common-gap-support strata, supported AP denominators and recording macro AP. Unsupported positives remain in budgeted-recall denominators. All saved classifier arms are checked for row alignment and unchanged payload scores; original classifier/payload metrics must reproduce exactly. report.json retains frozen profiles and provenance, and controls-*.npz preserves every control score and support/first-seen mask in physical-row order. Selection-data thresholds and vehicle-specific support do not establish calibrated alert rates or held-out-vehicle improvements.

Append --payload-transition to that command to include paired categorical byte-frequency and same-ID byte-transition controls. They count all bytes and preceding byte pairs from verified-normal training queries with verified lengths, separately for each vehicle, ID and byte position. The marginal distribution has one uniform prior observation across 256 byte values; the conditional distribution has 32 prior observations distributed according to that marginal. Sparse transition contexts blend toward the marginal; missing preceding bytes, first occurrences and unseen contexts use it directly. These constants are fixed before evaluation. Query traffic never updates counts.

Both controls score mean negative log likelihood across current valid bytes and share exactly the same support: every valid current byte position must have a normal-training profile. Missing/unverified payloads, untrained positions and unprofiled vehicles abstain for the whole frame. The byte_supported stratum compares both controls and neural heads on identical rows. byte_backoff and byte_fully_conditioned partition supported frames according to whether all current byte positions have observed training transition contexts. Even these observed contexts retain marginal smoothing. Per-frame byte counts, scores and masks are saved with the control predictions; payload-profile-*.npz preserves the sparse integer training tables, with hashes and fit exposure in the report.

The no-elapsed, normalized-bus, payload delta/XOR and graph-age paths and the weighting/interleaving/classification-only generators have been retired. Their completed artifacts and frozen source remain in the October 30 suite. The selected schema-2 checkpoint and schema-1 baseline checkpoints remain loadable; neutral historical feature flags are accepted and removed during configuration validation. Active flags for removed variants fail explicitly and require the original frozen source. Schema 1 requires use_id_gap: false.

Input contract

One event contains the numeric CAN arbitration ID, observed payload bytes, payload length, and timestamp. The cache preserves all 29 ID bits, exact byte values, FP64 physical timestamps, equal-timestamp row ordering, and every physical CSV row, including duplicates and messages subsequently overwritten within the same 10 ms interval. Source rows already subject to reviewed clock repair retain their reviewed ordering; this path does not independently repair or sort clocks. It does not reconstruct unavailable wire fields such as an absent IDE/RTR flag.

The neural input directly encodes the available frame fields: eight byte values scaled by 255, eight byte-presence masks, 29 ID bits, length scaled by eight, elapsed recording time, the preceding bus interarrival time, and the three per-ID timing features described above. Elapsed time and interarrival are computed in FP64 and converted to the model's usual floating point dtype at projection. Absolute clock origin is retained in the cache but is not a learned recording-identity feature. There are no binned counts, payload-change summaries, byte-difference heuristics, or correlation-selected graph edges. Normal timing statistics supply only the explicit per-ID gap normalization. Labels, normality flags, recording names, roles, vehicles, and source interfaces do not enter the encoder. Normality and payload provenance only control eligible training targets.

Chunks are execution units, not summaries. Every constituent event is encoded and scored individually. During training, each frame is a scored target exactly once per epoch. Both Mamba caches, latest per-ID memory, and previous-bus state carry across chronological chunks and detach at each chunk boundary for truncated backpropagation. History resets at recording boundaries. Training and inference use the same functional scan; inference wraps it in no_grad. Unknown labels remain unknown and are excluded from supervised loss.

stream_batch_size selects 1 to 16 independent recordings on one GPU, limited by the distinct reviewed training recordings available. The scheduler batches equal-length chunks and groups short tails by their exact length, replacing completed recordings with fresh streams. It never inserts fake events, replays warmup rows or drops tails. chunk_events * stream_batch_size must fit both max_microbatch_events (default 65536) and effective_batch_events (default 65536). Raise the explicit microbatch cap when testing more streams at a fixed chunk length; it can be configured up to 262144 events for at most 16 streams. Each optimizer window declares its complete eligible classification and normal-payload target counts on the CPU, and each objective uses its own denominator across all chunks. Groups can split between independent streams to fill optimizer windows, without splitting their chronological chunks. The final partial window uses its actual eligible counts. Use accumulate_grad_batches=1; accumulation belongs to the data schedule. The retired warmup_events argument has been removed.

CPU preparation supplies routing indices plus recording, source, row and incoming ID-history provenance. Each cache retains only compact first-appearance metadata; dense routing exists only for scheduled chunks. Host timestamp metadata avoids GPU scalar reads during continuity checks. An independent serial graph/routing/loss oracle lives in test support; the unused scan(reference=...) argument has been removed. Model execution batches projections and heads across recording banks, then attends within each stream using local history indices and bounded matrix-product tiles. Unknown/future ID slots receive zero attention mass, including frames with entirely empty history.

num_workers (default 2) controls CPU preparation; pinned memory defaults to true. Explicitly select zero workers for synchronous CPU diagnostics. prefetch_factor (default 2) and persistent_workers (default true) apply only with workers enabled. Workers never perform CUDA operations. cuda_prefetch (default false) opts training into a main-process loader that stages one future optimizer window on a dedicated CUDA copy stream; use pin_memory: true for asynchronous host transfers. The loader remains a DataLoader for Lightning. CPU execution and the synchronous path remain available, and loader seeding uses a private RNG. Prefetched windows count toward the same GPU memory ceiling as model activations.

The official scan computes the transformed time step once and shares it between cumulative decay and outgoing-state reconstruction, retaining both gradient paths. At epoch start, the model captures expected source-training recording counts; each optimizer step compares physical coverage against those counts.

The model clips gradients to norm 1 before each AdamW update. Checkpoints may be saved only after complete epoch coverage, including source/config identities, optimizer state (Lightning), and Python/NumPy/PyTorch/CUDA RNG states. Resume via config.ckpt_path starts at the next epoch; missing epoch identity and mid-epoch resume are rejected. Dropout and all existing model dimensions and objectives retain their defaults. Production uses the pinned official CUDA operators in FP32, without compilation or automatic precision changes.

Learned temporal and graph context

A shared Mamba stack advances on the chronological bus message sequence. Its output at each event therefore incorporates earlier messages from all observed IDs. A graph readout uses learned query/key/value attention to the latest preceding Mamba state for each encountered CAN ID. The current event's encoded state queries those historical node states; its updated temporal state becomes that ID's memory for subsequent events. Future IDs and future observations are masked out. Attention learns relationships instead of receiving a graph chosen from count/change correlations. This differs from the snapshot model's per-ID clock-tick Mamba streams and its fixed reference-neighbor readout.

These hidden states are learned compression after individual messages enter the network. Raw inputs are not replaced by a hand-selected per-bin payload. Node and graph-work capacities fail explicitly when exceeded; they never evict IDs, truncate messages, or merge events. This first version supports one training or evaluation device. It rejects multi-device configurations rather than silently padding evaluation or splitting a stream's chronological state across ranks.

Prediction and learning targets

The supervised head predicts attacker-frame labels with BCE. A separate normal-only head predicts eight categorical byte distributions from preceding bus/per-ID states and the arriving frame's header/timing, before that frame's payload can enter the predictor. Valid observed bytes from independently verified-normal training recordings supply this objective. It is a simple factorized byte likelihood, not a joint payload distribution or an operational fault-state model. Source attack recordings do not train the normal-only loss.

Payload NLL is reported as an uncalibrated anomaly ranking. It does not reuse snapshot calibration. Both scores remain distinct: an abnormal payload score is not an assertion of attacker origin. No operational fault labels have been added. An observed-frame interface has no prediction unit during complete silence; absence/suppression detection will require a separately evaluated clock or next-arrival survival objective. No such objective is implied by frame scores.

The preset retains the reviewed Impala/Silverado source split, filters out target roles before opening data files, and uses neither target evaluation nor reference summary features. Reference recordings remain reserved and are not used as query training examples. Explicit source fitting is:

gx exp config configs/experiments/graph_mamba_events_source.yml
gx exp submit configs/experiments/graph_mamba_events_source.yml -C ascend

Do not interpret this preset as a controlled architecture ablation: event units, temporal routing, graph construction, supervision granularity, normality targets, and optimizer-step exposure differ from the snapshot experiment. A subsequent learning experiment must report frame attribution and common bus-bin comparison separately, fit calibration on source data only, and retain existing model results as controls.

Verification and artifacts

Run tests in Slurm, including tests/test_event_graph_mamba.py. They cover exact payload and row preservation, timestamp ties, empty payloads, unknown labels, source-only loading, continuous state and exact scored coverage, future-perturbation causality, current-payload exclusion from forecasting, arbitrary chunk boundaries, state identity, capacity failures, checkpoint round trips, a Lightning learning smoke test, and pinned official CUDA versus reference outputs and gradients.

For a complete source-ingestion audit, run in a CPU allocation:

gx can verify-events --config configs/experiments/graph_mamba_events_source.yml \
  --output "$GRAPHIDS_RUN_ROOT/experimentruns/graph-mamba-events-source-v1/verification/source-input-audit.json"

This rereads every selected source query row and checks every cache field against it, persisting per-record progress and hashes. A failed audit raises and marks the report incomplete. Model evaluation appends bounded chunks to one scores.bin per epoch, plus index.json recording physical row offsets, source identities, exact coverage and frame-level metrics. Incomplete validation is rejected. Thresholds and snapshot calibration are not silently transferred to the new model.

Evaluation computes categorical per-byte NLL once for publishing ranks and reuses it in loss logging and anomaly scores. Loss eligibility remains normal-only, verified, and scored; anomaly scores use the existing payload-support mask. Sanity checks and non-publishing ranks retain the loss-only path.

graphids.evaluation.EventScores owns packed score writing, coverage checks and exact epoch AP independently of the model and Lightning. The model supplies the six score columns and logs the returned metrics. Artifact schema 2 stores little-endian FP64 values in row-major order, with shape/dtype and chunk offsets in the index; the six score columns and checkpoint identities are unchanged. iter_score_chunks reads complete schema-1 NPZ epochs and schema-2 packed epochs. Metric scratch space uses local TMPDIR rather than the persistent lake. scripts/pack_event_scores.py EPOCH --remove-shards migrates a complete legacy epoch: it verifies source metadata, exact coverage and the packed-file SHA256 before replacing the index and removing the old shards. Existing metrics are preserved exactly. Direct artifact and metric tests live in tests/test_evaluation.py and tests/test_metrics.py; end-to-end coverage and checkpoint tests remain in tests/test_event_graph_mamba.py. The CI cpu-tests job includes these direct tests; training and CUDA checks run through the Slurm gate.

Implementation verification on 2026-10-16 checked all 21,285,666 query frames across 21 source recordings, with every cached field matching its CSV row. CPU regression job 56288831 passed 49 tests (two GPU-only cases skipped). A100 job 7891355 passed 15 tests, including official/reference kernel parity; final event-specific checks passed across jobs 7891363 and 7891364 (the latter corrected a test that checked device placement after Lightning teardown instead of during training). The complete ingestion audit is CPU job 56288850. Machine-readable verification, logs, and an implementation source archive are at /fs/ess/PAS1266/graphids/dev/rf15/experimentruns/graph-mamba-events-source-v1/verification/. These are correctness and small learning smoke checks, not a new detection result.

Bounded implementation gate

Run python scripts/event_model_gate.py --mode cpu --output <directory> in a CPU Slurm allocation. sbatch scripts/slurm/event_model_gate.sh requests one A100 for at most two hours and runs official/reference correctness, source-only benchmarks, and a tiny complete synthetic fit through gx exp launch and Ray/Lightning. It does not run a complete source learning epoch or target evaluation.

The gate resolves defaults from the selected experiment YAML and measures only requested configurations. The default ten-recording cohort includes both reviewed vehicles, normal traffic and attacks, with 65,536 scored frames per recording (655,360 total) and continuous preceding history. Three independently reset repetitions use identical weights, optimizer, recording histories and RNG. At the production 16,384-event optimizer window this means 40 updates.

python scripts/event_model_gate.py --mode gpu --output <external-directory>
# Explicit larger-chunk comparison at a common optimizer window:
python scripts/event_model_gate.py --mode benchmark --chunks 16384 32768 65536 \
  --streams 1 --max-microbatch-events 65536 --effective-events 65536 \
  --events-per-recording 65536 --output <external-directory>

Supported overrides include --workers, --cuda-prefetch / --no-cuda-prefetch, --chunks, --streams, --effective-events, --max-microbatch-events, --events-per-recording, --repeats and --no-profiles. Sweeps must explicitly raise their microbatch and optimizer-window limits where needed; invalid requests are errors. OOM configurations are recorded without shrinking them. Correctness failures abort the gate. Historical pipeline, --pipeline, control and --steps interfaces have been removed; their original source is preserved externally.

Throughput timing synchronizes at measurement boundaries. Separate profiles record CUDA-event phases, CPU/CUDA operators, transfers and launch gaps for each successful requested configuration. The 32 GiB reserved-memory ceiling includes prefetched tensors. Graph attention uses 256 MiB temporary tiles and Mamba incoming-state correction retains its 64 MiB bound and official CUDA operators. The GPU mode also runs a bounded public Ray/Lightning fit using production settings and synthetic source data.

The measured production defaults are 16,384-event chunks, one stream, a 16,384-event optimizer window, two persistent workers, prefetch factor two, pinned memory, CUDA prefetch off, graph capacity 2,097,152 and FP32. The earlier full-pipeline measurement was approximately 344k events/s versus 243k for the serial baseline at equal optimizer exposure, with 3.87 GiB reserved memory. A larger-chunk comparison at a common 65,536-event optimizer window found 32K chunks 4.9% slower and 64K chunks 16.7% slower than 16K.

Detailed historical results remain in docs/empirical-notes/, including 2026-10-16-event-batched-pipeline.md and 2026-10-16-event-larger-chunks.md. docs/empirical-notes/2026-10-16-event-runtime-consolidation.md records corrected independent verification: the earlier scan(reference=True) switch did not select a separate graph implementation. Historical timings remain measurements of their recorded configurations; those particular parity comparisons were not independent.