Neuro-Symbolic Fraud Detection: Catching Concept Drift Before F1 Drops (Label-Free)

Can the foundations act as a canary? Can neuro-symbolic idea drift monitoring work at inference time, with out labels?

Three Methods Fraud Can Change

Idea drift fraud detection is tougher than it sounds as a result of solely one of many three frequent drift sorts truly modifications what the mannequin’s realized associations imply. The experiment simulates three kinds of drift on the Kaggle Credit score Card Fraud dataset (284,807 transactions, 0.17% fraud price) throughout 8 progressive home windows every [9].

Covariate drift. The enter function distributions shift. V14, V4, and V12 transfer by as much as +3.0σ progressively. Fraud patterns keep the identical. The world simply appears a little bit completely different.

Prior drift. The fraud price will increase from 0.17% towards 2.0%. Options are unchanged. Fraud turns into extra frequent.

Idea drift. The signal of V14 is progressively flipped for fraud instances throughout 8 home windows. By the top, the transactions the mannequin realized to flag as fraud now appear like official ones. The rule IF V14 < −1.5σ → Fraud is now pointing within the fallacious course.

That third one is the one that ought to fear you in manufacturing. With covariate and prior drift, there are exterior indicators. Enter distributions shift, or fraud charges visibly change. You possibly can monitor these independently. Idea drift leaves no such footprint. The one factor that modifications is what the mannequin’s realized associations imply. You’ll not know till F1 begins falling.

Except one thing sees it first.

The Downside With the First Three Metrics

The mannequin from How a Neural Network Learned Its Own Fraud Rules: A Neuro-Symbolic AI Experiment produced three label-free monitoring indicators as a by-product of the symbolic layer. The concept: if the foundations are studying fraud patterns, modifications in how these guidelines hearth ought to reveal when fraud patterns are shifting.

I anticipated the primary one to be the early warning. It was not.

The issue is particular to how this mannequin trains. All 5 seeds converged between epochs 3 and 10 (Val PR-AUC: 0.7717, 0.6915, 0.6799, 0.7899, 0.7951), when temperature τ continues to be between 3.5 and 4.0. At that temperature, rule activations are tender. Each enter produces a near-identical activation rating no matter its precise options. In plain phrases: the foundations have been firing nearly the identical approach on each transaction, clear or drifted. A similarity metric on near-constant vectors returns 1.000 nearly on a regular basis. The primary sign solely fired in 2 of 5 seeds for idea drift, and in each instances it was the identical window as F1 or later.

Why excessive temperature makes monitoring tougher

The LearnableDiscretizer makes use of a sigmoid gated by temperature τ: σ((x − θ) / τ). At τ = 5.0 (epoch 0), that sigmoid is sort of flat — each function worth produces an activation near 0.5 no matter the place it sits relative to the realized threshold. As τ anneals towards 0.1, the sigmoid sharpens right into a near-binary step. Early stopping fires at τ ≈ 3.5–4.0 — earlier than the foundations have totally crystallised. The consequence: activation vectors are near-constant throughout all inputs, so any similarity metric between them stays close to 1.000 even when fraud patterns are genuinely shifting.

The second sign had the other drawback. Absolutely the change in any function’s contribution is tiny (values within the 0.001–0.005 vary) as a result of the rule weights themselves are small at an early-stopped checkpoint. In plain phrases: the sign was actual however invisible on the scale we have been measuring it. A set absolute threshold of 0.02 by no means fires.

Here’s what these three authentic indicators are:

• RWSS (Rule Weight Stability Rating): cosine similarity between the baseline imply rule activation vector and the present one. In easy phrases: are the foundations nonetheless firing the identical approach they did on clear knowledge?
• FIDI (Characteristic Significance Drift Index): how a lot every function’s contribution to rule activations has modified from the baseline. In easy phrases: has any particular function grow to be kind of necessary to the foundations?
• RFR (Rule Firing Price): what fraction of transactions hearth every rule.

That analysis led to the fitting query. As a substitute of asking “has FIDI modified by greater than X?”, the fitting query is “has FIDI modified by greater than X normal deviations from its personal historical past?”

That query has a special reply. And the reply is V14.

The Metrics: Constructing a Label-Free Drift Detection System

Three new metrics joined the unique three.

RWSS Velocity measures the per-window price of change: RWSS[w] − RWSS[w−1]. A sudden drop of greater than 0.03 per window fires an alert even earlier than absolutely the worth crosses the brink. If RWSS is falling at −0.072 in a single step, that could be a sign no matter the place it began.

FIDI Z-Rating is the one that really labored. Relatively than a model new sign, it’s a easy extension of FIDI utilizing Z-score normalisation towards the function’s personal window historical past. As a substitute of asking whether or not absolutely the change crosses a hard and fast threshold, it asks whether or not the change is anomalous relative to what that function has been doing. Not like conventional drift detection strategies that depend on enter distributions or output labels, this strategy operates purely on the symbolic layer, which suggests it really works at inference time, with no floor fact required. It builds on differentiable rule-learning work together with ∂ILP [3], FINRule [4], RIFF [5], and Neuro-Symbolic Rule Lists [6], extending these representations with Z-score normalisation relatively than fastened thresholds. V14’s contribution to rule activations throughout the clear baseline home windows is small and flat. Close to zero, steady, predictable. When idea drift begins at window 3, it shifts. Not by a lot in absolute phrases. However by 9.53 normal deviations relative to the historical past it constructed throughout steady home windows. That is a gigantic relative anomaly, and no threshold calibration is required to catch it.

PSI on Rule Activations was designed to catch distributional shift within the symbolic layer earlier than the MLP’s compensation masks it on the output stage. It didn’t work right here. The tender activations from early-stopped coaching (τ ≈ 3.5–4.0 on the saved checkpoint) cluster close to 0.5, producing near-uniform distributions that PSI can not distinguish. PSI_rules = 0.0049 all through the whole experiment. PSI_rules by no means fired. It’s within the codebase for when fashions with totally crystallised guidelines (τ < 0.5) can be found. On this experiment it contributed nothing.

The supposed detection order, from earliest to newest:

RWSS Velocity → FIDI Z-Rating → PSI(guidelines) → RWSS absolute → F1 (label-based)

Here’s what truly occurred.

Outcomes: What Every Metric Did

Idea Drift

FIDI Z-Rating fires in 5 of 5 seeds, at all times at window 3. F1 fires at W3 in three seeds and W4 in two. The imply FIDIZ detection lag is +0.40 home windows, that means it leads F1 on common. In seeds 7 and 2024 it fires one full window earlier than F1 drops. Within the remaining three seeds it fires concurrently. It by no means fires after F1 for idea drift. Not as soon as.

Throughout all drift sorts, FIDI Z-Rating is the one metric that detected idea drift in each seed and by no means lagged behind F1. For label-free drift detection fraud techniques, that’s the headline consequence.

RWSS fires in 2 of 5 seeds and in each instances concurrently with or after F1. Velocity matches RWSS precisely, similar window, each time. PSI on rule activations by no means fires in any respect.

Idea Drift vs Covariate Drift: Why Symbolic Monitoring Has Blind Spots

Covariate drift is the place the symbolic layer goes utterly silent.

Each symbolic metric: 0 of 5 seeds. Not one sign. Not one window. F1 ultimately fires in 4 of 5 seeds at W6 or W7, slowly and late, and the symbolic layer had nothing to do with it. This isn’t a niche that higher tuning will shut. It’s a basic property of what the symbolic layer measures.

The reason being mechanical. When V14, V4, and V12 shift by +3.0σ, the shift is uniform throughout all samples. The learnable discretizer computes thresholds relative to the info. Every pattern nonetheless lands in roughly the identical threshold bin relative to its neighbours. Guidelines hearth on roughly the identical proportion of transactions. Nothing within the activation sample modifications. Cosine similarity of imply activations stays at 1.0.

In easy phrases: if each transaction shifts by the identical quantity, the foundations nonetheless see the identical relative image. Transaction A was above the brink earlier than. It’s nonetheless above the brink after. The fraud-vs-legitimate ordering is preserved. RWSS measures that ordering, not absolutely the values. Consider it as a tide that lifts all boats equally. The boats keep in the identical order. RWSS solely measures the order.

If covariate drift is a priority in your deployment, you want a separate input-space monitor: PSI on uncooked options, a KS check on V14, or an information high quality verify. The symbolic layer can not make it easier to there. Symbolic layer drift monitoring has one blind spot, and covariate shift is it.

Prior Drift

FIDIZ fires in 5 of 5 seeds, at all times at W3. However prior drift causes F1 to drop at W0 (seed 123) or W2 (seed 2024) within the two seeds the place F1 fires in any respect. FIDIZ detection lag for prior drift: −2.00 home windows. It fires two home windows after F1.

This isn’t a calibration drawback. FIDIZ wants a minimal of three clear home windows to construct a historical past earlier than its Z-score is significant. Prior drift that causes a direct fraud price soar is already seen in F1 earlier than FIDIZ may even begin computing. A rolling fraud price counter will at all times be quicker right here.

The Alert Demo: Window 3

Right here is the second the entire system was constructed for.

DriftAlertSystem is constructed as soon as from the validation set instantly after coaching. It shops the baseline. Then .verify() known as on every new window. No labels. No retraining. That is inference-only drift detection: the system reads the symbolic layer and nothing else.

Seed 42, idea drift, 8 home windows:

Window 0: severity=none      RWSS=0.999  fired=False
Window 1: severity=none      RWSS=0.999  fired=False
Window 2: severity=none      RWSS=0.999  fired=False
Window 3: severity=warning   RWSS=1.000  fired=True   ← FIDI Z fires right here
Window 4: severity=crucial  RWSS=0.928  fired=True   ← RWSS absolute confirms
Window 5: severity=warning   RWSS=0.928  fired=True
Window 6: severity=warning   RWSS=0.928  fired=True
Window 7: severity=warning   RWSS=0.928  fired=True

At window 3, RWSS is precisely 1.000. The activation sample is completely an identical to baseline. Output possibilities haven’t modified. Nothing in the usual monitoring stack has moved.

And the alert fires at WARNING severity.

The reason being V14. Its Z-score is −9.53. Which means V14’s contribution to rule activations has shifted to almost 10 normal deviations beneath the baseline it established throughout clear home windows. The mannequin’s output doesn’t know but. The MLP is compensating. However the rule path can not compensate. It was skilled to specific a hard and fast symbolic relationship. It’s screaming.

One window later, the MLP stops holding. RWSS drops to 0.928. Velocity falls 0.072 in a single step. Severity escalates to CRITICAL.

═══════════════════════════════════════════════════════
  DRIFT ALERT  |  severity: CRITICAL
  Earliest sign: VELOCITY
═══════════════════════════════════════════════════════
  ── Early-Warning Layer ─────────────────────────────
  RWSS Velocity : -0.0720  [threshold -0.03]  ⚠ FIRED
  FIDI Z-Rating  : ⚠ FIRED
       V14  Z = -9.53
  PSI (guidelines)   : 0.0049  [moderate≥0.10]  steady
  ── Confirmed Layer ─────────────────────────────────
  RWSS absolute : 0.9276  [threshold 0.97]  ⚠ FIRED
  Guidelines gone silent: 0  OK
  Imply RFR change  : -0.001
  Beneficial motion:
    → Retrain instantly. Don't deploy.
═══════════════════════════════════════════════════════

The report names VELOCITY because the earliest layer. That could be a precedence order within the inside logic. In precise window timing, FIDI Z-Rating fired one window earlier at W3. The W3 WARNING is the sooner human-facing alert. The one that provides you time to behave earlier than the CRITICAL fires.

Why FIDI Z-Rating Sees It Earlier than F1 Does

The mannequin has two paths working in parallel from the identical enter.

The MLP path carries 88.6% of the ultimate output (imply α = 0.886 throughout seeds; α is the realized mix weight; 0.886 means the neural community does 88.6% of the prediction work and the symbolic guidelines do the remaining 11.4%). When idea drift progressively reverses V14’s relationship to fraud labels, the MLP, skilled on 284,000 transactions, partially absorbs that change. Its inside representations shift. Output possibilities keep roughly steady for a minimum of one window. That is the MLP compensating.

The rule path carries 11.4%. It was skilled to specific the MLP’s data in symbolic kind: V14 beneath a threshold means fraud [2]. That relationship is fastened and express. When V14 flips signal for fraud instances, the rule’s V14 contribution doesn’t modify. It merely stops working. The bit activations for V14 change course. The rule begins firing on the fallacious transactions.

The neural community adapts. The symbolic layer doesn’t. And that’s precisely why the symbolic layer detects the drift first.

That asymmetry is what FIDI Z-Rating exploits.

Absolutely the change in V14’s contribution is tiny (values within the 0.001 to 0.005 vary) as a result of rule weights are small at an early-stopped checkpoint. A set absolute threshold by no means catches it.

However V14’s historical past by the clear home windows is simply as flat. When idea drift strikes it at window 3, the Z-score is −9.53. Identical sample: near-zero absolute change, excessive relative shift.

The symbolic layer is much less compensating than the MLP, so it reveals the drift first. FIDI Z-Rating makes the sign seen by evaluating every function to not a hard and fast threshold, however to its personal historical past.

However this solely holds for one of many three drift sorts. The opposite two are a special story totally.

What This System Can’t Do

A system that claims early warning invitations overstatement. Here’s what the info truly says. That is label-free anomaly detection fraud monitoring, which suggests the constraints are structural, not tunable.

Covariate drift is a whole blind spot. 0 of 5 seeds. The mechanism is defined within the Outcomes part above. Use PSI on uncooked options or a KS check on V14 as a substitute.

FIDIZ fires late on prior drift by design. When the fraud price jumps, F1 reacts at W0 or W2. FIDIZ structurally can not hearth earlier than W3. It wants historical past that doesn’t but exist. A rolling fraud price monitor responds quicker.

PSI on rule activations produced nothing. PSI_rules = 0.0049 all through each window of each seed. Comfortable activations from early-stopped coaching cluster close to 0.5, and PSI on near-uniform distributions is insensitive regardless of what’s truly occurring. This metric is within the codebase and may go with totally annealed fashions (τ < 0.5). On this experiment it was silent.

5 seeds is proof, not proof. FIDIZ fires at W3 for idea drift throughout all 5 seeds. That’s constant and inspiring. It isn’t the identical as dependable in manufacturing throughout datasets, fraud sorts, and drift severities you haven’t examined. 5 seeds is a place to begin, not a conclusion. Extra seeds, extra drift configurations, and real-world validation are wanted earlier than robust deployment claims.

Outcomes Abstract

The sample is clearest when said plainly first. Consider this as an early warning idea drift system with three distinct modes relying on what’s altering. Covariate drift: the symbolic layer noticed nothing, F1 caught it slowly. Prior drift: the symbolic layer fired after F1, not earlier than. Idea drift: FIDI Z-Rating fired in each single seed, at all times at or earlier than F1, averaging +0.40 home windows of lead time.

Lag = home windows earlier than F1 alert. Constructive = FIDIZ fires first. Detrimental = F1 fires first.

Constructing It

The system is designed for use in manufacturing, not simply in a pocket book.

# As soon as, instantly after coaching
X_val_t      = torch.FloatTensor(X_val)
alert_system = DriftAlertSystem.from_trained_model(mannequin, X_val_t, feature_names)
alert_system.save("outcomes/drift_alert_baseline_seed42.pkl")

# Each scoring run — weekly, each day, per-batch
alert_system = DriftAlertSystem.load("outcomes/drift_alert_baseline_seed42.pkl")
alert = alert_system.verify(mannequin, X_this_week)

if alert.fired:
    print(alert.report())

No labels. No retraining. No infrastructure past saving a pickle file subsequent to the mannequin checkpoint. The .verify() name computes RWSS velocity, FIDI Z-Rating, PSI on activations, and RWSS absolute in that order, utilizing PyTorch [7] and scikit-learn [8]. Severity escalates from none to warning to crucial primarily based on what number of hearth and the way far RWSS has dropped.

The three early-warning computations are every just a few traces.

RWSS Velocity: price of change per window.

def compute_rwss_velocity(rwss_history: Record[float]) -> float:
    if len(rwss_history) < 2:
        return 0.0
    return float(rwss_history[-1] - rwss_history[-2])

# Alert fires when drop > 0.03 per window
vel_fired = rwss_velocity < -0.03

FIDI Z-Rating: normalise function contribution anomaly towards historical past.

def compute_fidi_zscore(fidi_history, current_fidi, min_history=3):
    if len(fidi_history) < min_history:
        return {ok: 0.0 for ok in current_fidi}
    z_scores = {}
    for feat_idx, current_val in current_fidi.objects():
        history_vals = [h.get(feat_idx, 0.0) for h in fidi_history]
        mean_h = np.imply(history_vals)
        std_h  = np.std(history_vals)
        z_scores[feat_idx] = (current_val - mean_h) / std_h if std_h > 1e-8 else 0.0
    return z_scores

# Alert fires when any function Z > 2.5
fidi_z_fired = any(abs(z) > 2.5 for z in z_scores.values())

PSI on Rule Activations: distributional shift within the symbolic layer (included for completeness).

def compute_psi_rules(baseline_acts, current_acts, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    psi_per_rule = []
    for r in vary(baseline_acts.form[1]):
        b = np.histogram(baseline_acts[:, r], bins=bins)[0] + 1e-6
        c = np.histogram(current_acts[:,  r], bins=bins)[0] + 1e-6
        b /= b.sum(); c /= c.sum()
        psi_per_rule.append(float(np.sum((c - b) * np.log(c / b))))
    return np.imply(psi_per_rule)

V14: Three Articles, One Characteristic

That is the half I didn’t plan. However V14 idea drift behaviour seems to be the thread that ties all three articles collectively.

Guiding Neural Networks with Domain Rules: I wrote guidelines about giant transaction quantities and anomalous PCA norms. Cheap intuitions. Nothing to do with V14.

How a Neural Network Learned Its Own Fraud Rules: The mannequin discovered V14 anyway. Given 30 anonymised options and no steering, the gradient landed on the one function with the very best absolute correlation to fraud. Twice, throughout two unbiased seeds.

This Article, I intentionally made V14 break. I flipped its signal for fraud instances, progressively, throughout 8 home windows. And FIDI Z-Rating registered the collapse at −9.53 normal deviations whereas RWSS was nonetheless 1.000 and F1 had not moved.

The identical function, three completely different roles: ignored, found, then monitored as the very first thing to fail. That coherence was not engineered. It’s what reproducible multi-seed analysis on a constant dataset retains producing.

What to Do With This

Use FIDI Z-Rating for idea drift detection with out labels. It fires in 5 of 5 seeds, requires solely 3 home windows of historical past, by no means fires after F1, and wishes no labels. Maintain the Z-score threshold at 2.5 and minimal historical past at 3 home windows.

Add a separate input-space monitor for covariate drift. PSI on uncooked options or a KS check on crucial options like V14. The symbolic layer is blind to distributional shifts that protect relative activation order.

Use a rolling fraud price counter for prior drift. FIDIZ structurally can not hearth earlier than W3. A label-based price counter fires at W0.

Construct the alert baseline instantly after coaching. Not after drift is suspected. Do it after coaching. When you wait, you might have already misplaced your clear reference level. Put it aside alongside the checkpoint file.

One window of early warning is actual. Whether or not it’s one week or in the future is determined by your scoring cadence. For many manufacturing fraud groups, the distinction between a scheduled retrain and an emergency one is measured in precisely these models.

Three Issues That Will Catch You Utilizing This Idea Drift Early Warning System

The three-window blind interval. FIDIZ has no historical past to work with for the primary 3 home windows after deployment. You’re monitoring with RWSS and RFR solely throughout that point. Plan for it explicitly.

Comfortable activations will silence PSI_rules. In case your greatest checkpoint arrives when τ ≥ 1.0 (which occurs each time early stopping fires earlier than coaching is full), rule activations cluster close to 0.5 and PSI_rules returns noise. Test τ at your saved checkpoint. On this experiment τ was nonetheless 3.5–4.0 at convergence. That’s the reason PSI_rules was silent all through.

Retrain means re-audit. This method is a fraud mannequin retraining set off, not a retrain alternative. After retraining, the foundations change. V14 could now not dominate, or new options could have entered. The compliance sign-off from the earlier mannequin doesn’t carry ahead. Construct the audit into the retrain course of, not as a step after, however because the step that closes the loop.

Closing

Three articles. One function saved showing.

Guiding Neural Networks with Domain Rules: I ignored it. How a Neural Network Learned Its Own Fraud Rules: The gradient discovered it. Article 3: When it broke, the symbolic layer observed earlier than the output layer did.

The experiment has a selected, trustworthy scope: FIDI Z-Rating detects idea drift in 5 of 5 seeds, generally one window earlier than F1, by no means after it, totally with out labels. For covariate drift it’s blind. For prior drift it’s late. These aren’t caveats added on the finish to melt the declare. They’re findings that let you know precisely the place to make use of this and the place to not.

A neuro-symbolic mannequin offers you two channels. The MLP is best at prediction. The symbolic layer is best at understanding when prediction is about to go fallacious. They don’t seem to be redundant. They’re watching completely different points of the identical drawback.

The MLP compensates. The symbolic layer can not. That’s its weak point. On this experiment, it turned out to even be its earliest warning.

Sequence