How a Neural Network Learned Its Own Fraud Rules: A Neuro-Symbolic AI Experiment

programs inject guidelines written by people. However what if a neural community may uncover these guidelines itself?

On this experiment, I prolong a hybrid neural community with a differentiable rule-learning module that routinely extracts IF-THEN fraud guidelines throughout coaching. On the Kaggle Credit score Card Fraud dataset (0.17% fraud charge), the mannequin discovered interpretable guidelines reminiscent of:

IF V14 < −1.5σ AND V4 > +0.5σ → Fraud

the place σ denotes the function normal deviation after normalization.

The rule learner achieved ROC-AUC 0.933 ± 0.029, whereas sustaining 99.3% constancy to the neural community’s predictions.

Most apparently, the mannequin independently rediscovered V14 — a function lengthy recognized by analysts to correlate strongly with fraud — with out being instructed to search for it.

This text presents a reproducible neuro-symbolic AI experiment exhibiting how a neural community can uncover interpretable fraud guidelines immediately from information.

Full code: github.com/Emmimal/neuro-symbolic-ai-fraud-pytorch

What the Mannequin Found

Earlier than the structure, the loss operate, or any coaching particulars — here’s what got here out the opposite finish.

After as much as 80 epochs of coaching (with early stopping, most seeds converged between epochs 56–78), the rule learner produced these within the two seeds the place guidelines emerged clearly:

Seed 42 — cleanest rule (5 circumstances, conf=0.95)

Discovered Fraud Rule — Seed 42 · Guidelines have been by no means hand-coded

IF   V14 < −1.5σ
AND V4  > +0.5σ
AND V12 < −0.9σ
AND V11 > +0.5σ
AND V10 < −0.8σ

THEN FRAUD

Seed 7 — complementary rule (8 circumstances, conf=0.74)

Discovered Fraud Rule — Seed 7 · Guidelines have been by no means hand-coded

IF   V14 < −1.6σ
AND V12 < −1.3σ
AND V4  > +0.3σ
AND V11 > +0.5σ
AND V10 < −1.0σ
AND V3  < −0.8σ
AND V17 < −1.5σ
AND V16 < −1.0σ

THEN FRAUD

In each circumstances, low values of V14 sit on the coronary heart of the logic — a hanging convergence given zero prior steering.

The mannequin was by no means instructed which function mattered.

But it independently rediscovered the identical function human analysts have recognized for years.

A neural community discovering its personal fraud guidelines is precisely the promise of neuro-symbolic AI: combining statistical studying with human-readable logic. The remainder of this text explains how — and why the gradient stored discovering V14 even when instructed nothing about it.

From Injected Guidelines to Discovered Guidelines — Why It Issues

Each fraud mannequin has a call boundary. Fraud groups, nevertheless, function utilizing guidelines. The hole between them, between what the mannequin discovered and what analysts can learn, audit, and defend to a regulator — is the place compliance groups reside and die.

In my previous article in this series, I encoded two analyst guidelines immediately into the loss operate: if the transaction quantity is unusually excessive and if the PCA signature is anomalous, deal with the pattern as suspicious. That method labored. The hybrid mannequin matched the pure neural web’s detection efficiency whereas remaining interpretable.

However there was an apparent limitation I left unaddressed. I wrote these guidelines. I selected these two options as a result of they made intuitive sense to me. Hand-coded guidelines encode what you already know, they’re answer when fraud patterns are steady and area data is deep. They’re a poor answer when fraud patterns are shifting, when a very powerful options are anonymized (as they’re on this dataset), or while you need the mannequin to floor indicators you haven’t thought to search for.

The pure subsequent query: what options would the gradient select, if given the liberty to decide on?

This sample extends past fraud. Medical prognosis programs want guidelines that medical doctors can confirm earlier than appearing. Cybersecurity fashions want guidelines that engineers can audit. Anti-money laundering programs function beneath regulatory frameworks requiring explainable choices. In any area combining uncommon occasions, area experience, and compliance necessities, the flexibility to extract auditable IF-THEN guidelines from a skilled neural community is immediately invaluable.

Architecturally, the change is surprisingly easy. You aren’t changing the MLP, you’re including a second path that learns to precise the MLP’s choices as human-readable symbolic guidelines. The MLP trains usually. The rule module learns to agree with it, in symbolic type. That’s the topic of this text: differentiable rule induction in ~250 traces of PyTorch, with no prior data of which options matter.

“You aren’t changing the neural community. You might be instructing it to clarify itself.”

The Structure: Three Learnable Items

The structure retains a typical neural community intact, however provides a second path that learns symbolic guidelines explaining the community’s choices. The 2 paths run in parallel from the identical enter and their outputs are mixed by a learnable weight α:

The MLP path is equivalent to the earlier article: three totally related layers with batch normalization. The rule path is new. Alpha is a learnable scalar that the mannequin makes use of to weight the 2 paths, it begins at 0.5 and is skilled by gradient descent like another parameter. After coaching, α converged to roughly 0.88 on common throughout seeds (vary: 0.80–0.94). The mannequin discovered to weight the neural path at roughly 88% and the rule path at 12% on common. The foundations should not changing the MLP, they’re a structured symbolic abstract of what the MLP discovered.

1. Learnable Discretizer

Guidelines want binary inputs — is V14 beneath a threshold? sure or no. Neural networks want steady, differentiable operations. The mushy sigmoid threshold bridges each.

For every function f and every learnable threshold t:

$$b_{f,t} = sigma!left(frac{x_f – theta_{f,t}}{tau}proper)$$

The place:

• $x_f$​ is the worth of function *f* for this transaction
• $theta_{f,t}$t​ is a learnable threshold, initialized randomly, skilled by backpropagation
• $tau$ is temperature — excessive early in coaching (exploratory), low later (crisp)
• $b_{f,t}$ is the mushy binary output: “is function *f* above threshold *t*?”

The mannequin learns three thresholds per function, giving it three “cuts” per dimension. Every threshold is impartial — the mannequin can unfold them throughout the function’s vary or focus them round essentially the most discriminative cutpoint.

At τ=5.0 (epoch 0): the sigmoid is nearly flat. Each function worth produces a gradient. The mannequin explores freely. At τ=0.1 (epoch 79): the sigmoid is almost a step operate. Thresholds have dedicated. The boundaries are readable as human circumstances.

class LearnableDiscretizer(nn.Module):
    def __init__(self, n_features, n_thresholds=3):
        tremendous().__init__()
        # One learnable threshold per (function × bin)
        self.thresholds = nn.Parameter(
            torch.randn(n_features, n_thresholds) * 0.5
        )
        self.n_thresholds = n_thresholds

    def ahead(self, x, temperature=1.0):
        # x: [B, F] → output: [B, F * n_thresholds] mushy binary options
        x_exp = x.unsqueeze(-1)               # [B, F, 1]
        t_exp = self.thresholds.unsqueeze(0)  # [1, F, T]
        soft_bits = torch.sigmoid(
            (x_exp - t_exp) / temperature
        )
        return soft_bits.view(x.dimension(0), -1)  # [B, F*T]

2. Rule Learner Layer

Every rule is a weighted mixture of binarized options, handed via a sigmoid:$$textual content{rule}_r(x) = sigma!left(frac{sum_i w_{r,i} cdot b_i}{tau}proper)$$

The signal of every weight has a direct interpretation after tanh squashing:

• $w > +0.5$ → function have to be HIGH for this rule to fireside
• $w < -0.5$ → function have to be LOW for this rule to fireside
• $|w| < 0.5$ → function is irrelevant to this rule

Rule extraction follows immediately: threshold absolutely the weight values after coaching to determine which options every rule makes use of. That is how IF-THEN statements emerge from steady parameters — by studying the burden matrix.

class RuleLearner(nn.Module):
    def __init__(self, n_bits, n_rules=4):
        tremendous().__init__()
        # w_{r,i}: which binarized options matter for every rule
        self.rule_weights = nn.Parameter(
            torch.randn(n_rules, n_bits) * 0.1
        )
        # confidence: relative significance of every rule
        self.rule_confidence = nn.Parameter(torch.ones(n_rules))

    def ahead(self, bits, temperature=1.0):
        w = torch.tanh(self.rule_weights)        # bounded in (-1, 1)
        logits = bits @ w.T                       # [B, R]
        rule_acts = torch.sigmoid(logits / temperature)  # [B, R]
        conf = torch.softmax(self.rule_confidence, dim=0)
        fraud_prob = (rule_acts * conf.unsqueeze(0)).sum(dim=1, keepdim=True)
        return fraud_prob, rule_acts

3. Temperature Annealing

The temperature follows an exponential decay schedule:$$tau(t) = tau_{textual content{begin}} cdot left(frac{tau_{textual content{finish}}}{tau_{textual content{begin}}}proper)^{t/T}$$

With τ_start=5.0, τ_end=0.1, T=80 epochs:

def get_temperature(epoch, total_epochs, tau_start=5.0, tau_end=0.1):
    progress = epoch / max(total_epochs - 1, 1)
    return tau_start * (tau_end / tau_start) ** progress

With out annealing, the mannequin stays mushy and guidelines by no means crystallize into something a fraud analyst can learn or a compliance crew can log off on. Annealing is what converts a steady optimization right into a symbolic output.

Earlier than the loss operate — a fast observe on the place this concept comes from, and what makes this implementation completely different from prior work.

Standing on the Shoulders of ∂ILP, NeuRules, and FINRule

It’s value situating this work within the present literature not as a full survey, however to make clear what concepts are borrowed and what’s new.

Differentiable Inductive Logic Programming launched the core concept that inductive logic programming historically a combinatorial search downside — will be reformulated as a differentiable program skilled with gradient descent. The important thing perception used right here is using mushy logical operators that permit gradients to circulate via rule-like constructions. Nevertheless, ∂ILP requires predefined rule templates and background data declarations, which makes it more durable to combine into normal deep studying pipelines.

Current work making use of differentiable guidelines to fraud detection reminiscent of FINRule — exhibits that rule-learning approaches can carry out nicely even on extremely imbalanced monetary datasets. These research exhibit that discovered guidelines can match hand-crafted detection logic whereas adapting extra simply to new fraud patterns.

Different programs reminiscent of RIFF and Neuro-Symbolic Rule Lists introduce decision-tree-style differentiable guidelines and emphasize sparsity to keep up interpretability. The L1 regularization used on this implementation follows the identical precept: encouraging guidelines to depend on just a few circumstances reasonably than all accessible options.

The implementation on this article combines these concepts differentiable discretization plus conjunction studying — however reduces them to roughly 250 traces of dependency-free PyTorch. No template language. No background data declarations. The aim is a minimal rule-learning module that may be dropped into a typical coaching loop.

Three-Half Loss: Detection + Consistency + Sparsity

The total coaching goal:

$$mathcal{L}_{textual content{complete}} = mathcal{L}_{textual content{BCE}} + lambda_c cdot mathcal{L}_{textual content{consistency}} + lambda_s cdot mathcal{L}_{textual content{sparsity}} + lambda_{textual content{conf}} cdot mathcal{L}_{textual content{confidence}}$$

L_BCE — Weighted Binary Cross-Entropy

An identical to the previous article. pos_weight = rely(y=0) / rely(y=1) ≈ 578. One labeled fraud pattern generates 578× the gradient of a non-fraud pattern. This time period is unchanged the rule path provides no complexity to the core detection goal.

L_consistency — The New Time period

Guidelines ought to agree with the MLP the place the MLP is assured. Operationally: MSE between rule_prob and mlp_prob, masked to predictions the place the MLP is both clearly fraud (>0.7) or clearly non-fraud (<0.3):

confident_mask = (mlp_prob > 0.7) | (mlp_prob < 0.3)
if confident_mask.sum() > 0:
    consist_loss = F.mse_loss(
        rule_prob.squeeze()[confident_mask],
        mlp_prob.squeeze()[confident_mask].detach()  # ← vital
    )

The .detach() is vital: we’re instructing the principles to observe the MLP, not the opposite means round. The MLP stays the first learner. The unsure area (0.3–0.7) is intentionally excluded that’s the place guidelines would possibly catch one thing the MLP misses.

L_sparsity — Maintain Guidelines Easy

L1 penalty on the uncooked (pre-tanh) rule weights: imply(|W_rules|). With out this, guidelines take up all 30 options and turn out to be unreadable. With λ_s=0.25, the optimizer pushes irrelevant options towards zero whereas leaving genuinely helpful options — V14, V4, V12 — at |w| ≈ 0.5–0.8 after tanh squashing.

L_confidence — Kill Noise Guidelines

A small L1 penalty on the arrogance logits (λ_conf=0.01) drives low-confidence guidelines towards zero weight within the output mixture, successfully eliminating them. With out this, a number of technically energetic however meaningless guidelines seem with confidence 0.02–0.04 that obscure the true sign.

Last hyperparameters: λ_c=0.3, λ_s=0.25, n_rules=4, λ_conf=0.01.

With the equipment in place here’s what it produced.

Outcomes: Does Rule Studying Work — and What Did It Discover?

Experimental Setup

• Dataset: Kaggle Credit score Card Fraud, 284,807 transactions, 0.173% fraud charge
• Break up: 70/15/15 stratified by class label, 5 random seeds [42, 0, 7, 123, 2024]
• Threshold: F1-maximizing on validation set, utilized symmetrically to check set
• Similar analysis protocol as Article 1

Detection Efficiency

Notice: Isolation Forest numbers from Article 1 for reference. All different fashions evaluated with equivalent splits, thresholds, and seeds.

The rule learner sits barely beneath the pure neural baseline on all three detection metrics, roughly 1.5 F1 factors on common. The tradeoff is explainability. The per-seed breakdown exhibits the complete image:

In seeds 42 and 0, the rule learner exceeds the pure neural baseline on F1. In seed 2024, it exceeds on ROC-AUC. The efficiency variance throughout seeds is the trustworthy image of what gradient-based rule induction produces on a 0.17% imbalanced dataset.

Rule High quality — The New Contribution

Three metrics, Every solutions a special query a compliance officer would ask.

Rule Constancy — can I belief this rule set to signify the mannequin’s precise choices?

def rule_fidelity(mlp_probs, rule_probs, threshold=0.5):
    mlp_preds  = (mlp_probs  > threshold).astype(int)
    rule_preds = (rule_probs > threshold).astype(int)
    return (mlp_preds == rule_preds).imply()

Rule Protection — what fraction of precise fraud does at the very least one rule catch?

def rule_coverage(rule_acts, y_true, threshold=0.5):
    any_rule_fired = (rule_acts > threshold).any(axis=1)
    return any_rule_fired[y_true == 1].imply()

Rule Simplicity — what number of distinctive function circumstances per rule, after deduplication?

def rule_simplicity(rule_weights_numpy, weight_threshold=0.50):
    # Divide by n_thresholds (=3) to get distinctive options,
    # the significant readability metric. Goal: < 8.
    energetic = (np.abs(rule_weights_numpy) > weight_threshold).sum(axis=1)
    unique_features = np.ceil(energetic / 3.0)
    unique_features = unique_features[unique_features > 0]
    return float(unique_features.imply()) if len(unique_features) > 0 else 0.0

This highlights an actual stress in differentiable rule studying: robust sparsity regularization produces clear guidelines after they seem, however could cause the symbolic path to go darkish in some initializations. Reporting imply ± std throughout seeds reasonably than cherry-picking one of the best seed is important exactly due to this variance.

Constancy at 0.993 implies that in seeds the place guidelines are energetic, they agree with the MLP on 99.3% of binary choices — the consistency loss working precisely as designed.

The Extracted Guidelines — What the Gradient Discovered

Each guidelines are proven in full on the prime of this text. The brief model: seed 42 produced a decent 5-condition rule (conf=0.95), seed 7 a broader 8-condition rule (conf=0.74). In each, V14 < −1.5σ (or −1.6σ) seems because the main situation.

The cross-seed function evaluation confirms the sample throughout all 5 seeds:

Even with solely two seeds producing seen guidelines, V14 ranked first or second in each — a statistically hanging convergence given zero prior function steering. The mannequin didn’t must be instructed what to search for.

“The mannequin acquired 30 anonymized options and a gradient sign. It discovered V14 anyway.”

What the Mannequin Discovered — and Why It Makes Sense

V14 is one in every of 28 PCA elements extracted from anonymized bank card transaction information. Precisely what it represents shouldn’t be public data — that’s the level of the anonymization. What a number of impartial analyses have established is that V14 has the best absolute correlation with the fraud label of any function within the dataset.

Why did the rule learner discover it? The mechanism is the consistency loss. By coaching guidelines to agree with the MLP’s assured predictions, the rule learner is studying the MLP’s inside representations and translating them into symbolic type. The MLP had already discovered from the labels that V14 was necessary. The consistency loss transferred that sign into the rule weight matrix. Temperature annealing then hardened that weight right into a crisp threshold situation.

That is the basic distinction between Rule Injection (Article 1) and Rule Studying (this text). Rule injection encodes what you already know. Rule studying discovers what you don’t. On this experiment, the invention was V14 — a sign the gradient discovered independently, with out being instructed to search for it.

Throughout 5 seeds, readable guidelines emerged in two — persistently highlighting V14. That could be a highly effective demonstration that gradient descent can rediscover domain-critical indicators with out being instructed to search for them.

A compliance crew can now learn Rule 1, confirm that V14 < −1.5σ makes area sense, and log off on it — with out opening a single weight matrix. That’s what neuro-symbolic rule studying is for.

4 Issues to Watch Earlier than Deploying This

• Annealing velocity is your most delicate hyperparameter Too quick: guidelines crystallize earlier than the MLP has discovered something — you get crisp nonsense. Too sluggish: τ by no means falls low sufficient and guidelines keep mushy. Deal with τ_end as the primary parameter to tune on a brand new dataset.
• n_rules units your interpretability price range Above 8–10 guidelines, you’ve gotten a lookup desk, not an auditable rule set. Under 4, you might miss tail fraud patterns. The candy spot for compliance use is 4–8 guidelines.
• The consistency threshold assumes a calibrated MLP In case your base MLP is poorly calibrated — widespread on severely imbalanced information — the masks fires too not often. Run a calibration plot on validation outputs. Think about Platt scaling if calibration is poor.
• Discovered guidelines want auditing after each retrain Not like frozen hand-coded guidelines, discovered guidelines replace each time the mannequin retrains. The compliance crew can not log off as soon as and stroll away — the sign-off should occur each retrain cycle.

Rule Injection vs. Rule Studying — When to Use Which

The rule learner provides roughly 200 traces of code and a hyperparameter sweep. It’s not free. On very small datasets, the consistency loss might not accumulate sufficient sign to study significant guidelines — validate constancy earlier than treating extracted guidelines as authoritative. The method is a software, not an answer.

One trustworthy commentary from the five-seed experiment: in 3 of 5 seeds, robust sparsity stress drove all rule weights beneath the extraction threshold. The mannequin converged to the appropriate detection reply however expressed it purely via the MLP path. This variance is actual. Single-seed outcomes would give a misleadingly clear image — which is why multi-seed analysis is non-negotiable for any paper or article making claims about discovered rule habits.

The following query on this sequence is whether or not these extracted guidelines can flag idea drift — detecting when fraud patterns have shifted sufficient that the principles want updating earlier than mannequin efficiency degrades. When V14’s significance drops within the rule weights whereas detection metrics maintain regular, the fraud distribution could also be altering. That early warning sign is the topic of the subsequent article.

Disclosure

This text is predicated on impartial experiments utilizing publicly accessible information (Kaggle Credit score Card Fraud dataset, CC-0 Public Domain) and open-source instruments (PyTorch, scikit-learn). No proprietary datasets, firm assets, or confidential data have been used. The outcomes and code are totally reproducible as described, and the GitHub repository comprises the entire implementation. The views and conclusions expressed listed here are my very own and don’t signify any employer or group.

References

[1] Evans, R., & Grefenstette, E. (2018). Studying Explanatory Guidelines from Noisy Knowledge. JAIR, 61, 1–64. https://arxiv.org/abs/1711.04574

[2] Wolfson, B., & Acar, E. (2024). Differentiable Inductive Logic Programming for Fraud Detection. arXiv preprint arXiv:2410.21928. https://arxiv.org/abs/2410.21928

[3] Martins, J. L., Bravo, J., Gomes, A. S., Soares, C., & Bizarro, P. (2024). RIFF: Inducing Guidelines for Fraud Detection from Determination Bushes. RuleML+RR 2024. arXiv:2408.12989. https://arxiv.org/abs/2408.12989

[4] Xu, S., Walter, N. P., & Vreeken, J. (2024). Neuro-Symbolic Rule Lists. arXiv preprint arXiv:2411.06428. https://arxiv.org/abs/2411.06428

[5] Kusters, R., Kim, Y., Collery, M., de Sainte Marie, C., & Gupta, S. (2022). Differentiable Rule Induction with Discovered Relational Options. arXiv preprint arXiv:2201.06515. https://arxiv.org/abs/2201.06515

[6] Dal Pozzolo, A. et al. (2015). Calibrating Chance with Undersampling for Unbalanced Classification. IEEE SSCI. Dataset: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud (CC-0)

[7] Alexander, E. P. (2026). Hybrid Neuro-Symbolic Fraud Detection. In direction of Knowledge Science. https://towardsdatascience.com/hybrid-neuro-symbolic-fraud-detection-guiding-neural-networks-with-domain-rules/

[8] Liu, F. T., Ting, Okay. M., & Zhou, Z.-H. (2008). Isolation Forest. In 2008 Eighth IEEE Worldwide Convention on Knowledge Mining (ICDM), pp. 413–422. IEEE. https://doi.org/10.1109/ICDM.2008.17

[9] Paszke, A. et al. (2019). PyTorch. NeurIPS 32. https://pytorch.org

[10] Pedregosa, F. et al. (2011). Scikit-learn: Machine Studying in Python. JMLR, 12, 2825–2830. https://scikit-learn.org

Code: github.com/Emmimal/neuro-symbolic-ai-fraud-pytorch

Earlier article: Hybrid Neuro-Symbolic Fraud Detection: Guiding Neural Networks with Domain Rules