Most manufacturing ML fashions don’t decay easily — they fail in sudden, unpredictable shocks. After we match an exponential “forgetting curve” to 555,000 production-like fraud transactions, it returned R² = −0.31, that means it carried out worse than predicting the imply.
Earlier than you set (or belief) any retraining schedule, run this 3-line diagnostic in your present weekly metrics:
report = tracker.report()
print(report.forgetting_regime) # "{smooth}" or "episodic"
print(report.fit_r_squared) # < 0.4 → abandon schedule assumptions- R² ≥ 0.4 → Clean regime → scheduled retraining works
- R² < 0.4 → Episodic regime → use shock detection, not calendars
In case your R² is under 0.4, your mannequin isn’t “decaying” — and every part derived from a half-life estimate is probably going deceptive.
It Began With One Week
It was Week 7.
Recall dropped from 0.9375 to 0.7500 in seven days flat. No alert fired. The mixture month-to-month metric moved a couple of factors — effectively inside tolerance. The dashboard confirmed inexperienced.
That single week erased three weeks of mannequin enchancment. Dozens of fraud instances that the mannequin used to catch walked straight by means of undetected. And the usual exponential decay mannequin — the mathematical spine of each retraining schedule I had ever constructed — didn’t simply fail to foretell it.
It predicted the alternative.
R² = −0.31. Worse than a flat line.
That quantity broke one thing in how I take into consideration MLOps. Not dramatically. Quietly. The sort of break that makes you return and re-examine an assumption you could have been carrying for years with out ever questioning it.
This text is about that assumption, why it’s flawed for a whole class of manufacturing ML programs, and what to do as a substitute — backed by actual numbers on a public dataset you possibly can reproduce in a day.
Full code: https://github.com/Emmimal/mlops-forgetting-curve/
The Assumption No person Questions
All the retraining-schedule business is constructed on a single concept borrowed from a Nineteenth-century German psychologist.
In 1885, Hermann Ebbinghaus performed a sequence of self-experiments on human reminiscence — memorising lists of nonsense syllables, measuring his personal retention at mounted intervals, and plotting the outcomes over time [1]. What he documented was a clear exponential relationship:
R(t) = R₀ · e^(−λt)Reminiscence fades easily. Predictably. At a price proportional to how a lot reminiscence stays. The curve turned one of the vital replicated findings in cognitive psychology and stays a foundational reference in reminiscence analysis to today.
A century later, the machine studying neighborhood borrowed it wholesale. The logic felt sound: a manufacturing mannequin is uncovered to new patterns over time that it was not skilled on, so its efficiency degrades progressively and repeatedly. Set a retraining cadence primarily based on the decay price. Estimate a half-life. Schedule accordingly.
Each main MLOps platform, each “retrain each 30 days” rule of thumb, each automated decay calculator, is downstream of this assumption.
The issue is that no person verified it towards manufacturing knowledge.
So I did.
The Experiment
I used the Kaggle Credit score Card Fraud Detection dataset created by Kartik Shenoy [2] — an artificial dataset of 1.85 million transactions generated utilizing the Sparkov Knowledge Technology software [3], overlaying the interval January 2019 to December 2020. The check break up comprises 555,719 transactions spanning June to December 2020, with 2,145 confirmed fraud instances (0.39% prevalence).
The simulation was designed to reflect a practical manufacturing deployment:
- Mannequin: LightGBM [4] skilled as soon as on historic knowledge, by no means retrained in the course of the check interval
- Main metric: Recall — in fraud detection, a missed fraud prices orders of magnitude greater than a false alarm, making recall the operationally appropriate goal [5]
- Analysis: Weekly rolling home windows on the hold-out check set, every window containing between 15,000 and 32,000 transactions
- High quality filters: Home windows with fewer than 30 confirmed fraud instances had been excluded — under that threshold, weekly recall estimates are statistically unreliable as a result of excessive class imbalance
The baseline was established utilizing the imply of the top-3 recall values throughout the primary six qualifying weeks — a way designed to disregard early warm-up noise whereas monitoring near-peak efficiency.
26 Weeks of Manufacturing Efficiency
Here’s what the complete simulation produced throughout 26 qualifying weekly home windows:
Week 1 [2020-06-21] n=19,982 fraud= 68 R=0.7647
Week 2 [2020-06-28] n=20,025 fraud= 100 R=0.8300
Week 3 [2020-07-05] n=20,182 fraud= 83 R=0.7831
Week 4 [2020-07-12] n=19,777 fraud= 52 R=0.8462
Week 5 [2020-07-19] n=19,898 fraud= 99 R=0.8586
Week 6 [2020-07-26] n=19,733 fraud= 64 R=0.9375 ← peak
Week 7 [2020-08-02] n=20,023 fraud= 152 R=0.7500 ← worst shock (−0.1875)
Week 8 [2020-08-09] n=19,637 fraud= 82 R=0.7439
Week 9 [2020-08-16] n=19,722 fraud= 59 R=0.7966
Week 10 [2020-08-23] n=19,605 fraud= 102 R=0.8922
Week 11 [2020-08-30] n=18,081 fraud= 84 R=0.8690
Week 12 [2020-09-06] n=16,180 fraud= 67 R=0.7910
Week 13 [2020-09-13] n=16,087 fraud= 63 R=0.8413
Week 14 [2020-09-20] n=15,893 fraud= 90 R=0.7444
Week 15 [2020-09-27] n=16,009 fraud= 81 R=0.8272
Week 16 [2020-10-04] n=15,922 fraud= 121 R=0.8264
Week 17 [2020-10-11] n=15,953 fraud= 111 R=0.8559
Week 18 [2020-10-18] n=15,883 fraud= 53 R=0.9245 ← restoration
Week 19 [2020-10-25] n=15,988 fraud= 73 R=0.8630
Week 20 [2020-11-01] n=15,921 fraud= 70 R=0.7429 ← second shock
Week 21 [2020-11-08] n=16,098 fraud= 59 R=0.9322 ← restoration
Week 22 [2020-11-15] n=15,835 fraud= 63 R=0.9206
Week 23 [2020-11-22] n=15,610 fraud= 91 R=0.9121
Week 24 [2020-11-29] n=30,246 fraud= 57 R=0.8596 ← quantity doubles
Week 25 [2020-12-06] n=31,946 fraud= 114 R=0.7895
Week 26 [2020-12-13] n=31,789 fraud= 67 R=0.8507Two home windows had been excluded: the week of December 20 (solely 20 fraud instances) and December 27 (zero fraud instances recorded — an information artefact in step with the vacation interval).
What −0.31 Really Means
R² — the coefficient of willpower — measures how a lot variance within the noticed knowledge is defined by the fitted mannequin [6].
- R² = 1.0: Good match. The mannequin explains all noticed variance.
- R² = 0.0: The mannequin does no higher than predicting the imply of the information for each level.
- R² < 0.0: The mannequin is actively dangerous — it introduces extra prediction error than a flat imply line would.
When the exponential decay mannequin returned R² = −0.3091 on this dataset, it was not becoming poorly. It was becoming backwards. The mannequin predicts a mild slope declining from a secure peak. The info exhibits repeated sudden drops and recoveries with no constant directional pattern.
This isn’t a decay curve. It’s a seismograph.
Two Regimes of Mannequin Forgetting
After observing this sample, I formalised a classification framework primarily based on the R² of the exponential match. Two regimes emerge cleanly:
The sleek regime is the world Ebbinghaus described. Characteristic distributions shift progressively — demographic modifications, gradual financial cycles, seasonal behaviour patterns that evolve over months. The exponential mannequin suits the noticed knowledge effectively. The half-life estimate is actionable. A scheduled retraining cadence is the proper operational response.
The episodic regime is what fraud detection, content material suggestion, provide chain forecasting, and any area with sudden exterior discontinuities really appears like in manufacturing. Efficiency doesn’t decay — it switches. A brand new fraud sample emerges in a single day. A platform coverage change flips consumer behaviour. A competitor exits the market and their prospects arrive with totally different traits. A regulatory change alters the transaction combine.
These aren’t factors on a decay curve. They’re discontinuities. And the R² diagnostic identifies which regime you might be in earlier than you decide to an operations technique constructed on the flawed assumption.
This sample extends past fraud detection — related episodic conduct seems in suggestion programs, demand forecasting, and consumer conduct modeling.
Why Fraud Detection Is At all times Episodic
The Week 7 collapse was not random noise. Allow us to take a look at what the information really exhibits.
Week 6 (July 26): 64 fraud instances. Recall = 0.9375. The mannequin is close to peak efficiency.
Week 7 (August 2): 152 fraud instances — 2.4 instances extra fraud than the earlier week — and recall collapses to 0.7500. The mannequin missed 38 frauds it could have detected seven days earlier.
A 137% improve in fraud quantity in a single week doesn’t symbolize a gradual distribution shift. It represents a regime change — a brand new fraud ring, a newly exploited vulnerability, or an organised marketing campaign that the mannequin had by no means encountered in its coaching knowledge. The mannequin’s realized patterns turned instantly inadequate, not progressively inadequate.
Then think about Week 24 (November 29). Transaction quantity almost doubles — from roughly 16,000 transactions per week to 30,246 — because the Thanksgiving and Black Friday interval begins. Concurrently, the fraud rely drops to 57, giving a fraud price of 0.19%, the bottom in all the check interval. The mannequin encounters a volume-to-fraud ratio it has by no means seen. Recall holds at 0.860, however solely as a result of absolutely the fraud rely is low. The precision concurrently collapses, flooding any downstream assessment queue with false positives.
Neither of those occasions is a degree on a decay curve. Neither could be predicted by a retraining schedule. Each could be caught instantly by a single-week shock detector.
The Diagnostic Framework
The classification is a three-step course of that may be utilized to any present efficiency log.
Step 1: Match the forgetting curve and compute R²
from model_forgetting_curve import ModelForgettingTracker
tracker = ModelForgettingTracker(
metric_name="Recall",
baseline_method="top3", # imply of top-3 early weeks
baseline_window=6, # weeks used to ascertain baseline
retrain_threshold=0.07 # alert threshold: 7% drop from baseline
)
# log your present weekly metrics
for week_recall in your_weekly_metrics:
tracker.log(week_recall)
report = tracker.report()
print(f"Regime : {report.forgetting_regime}")
print(f"R² : {report.fit_r_squared:.3f}")Step 2: Department on regime
if report.fit_r_squared >= 0.4:
# SMOOTH — exponential mannequin is legitimate
print(f"Schedule retrain in {report.predicted_days_to_threshold:.0f} days")
print(f"Half-life: {report.half_life_days:.1f} days")
else:
# EPISODIC — exponential mannequin is invalid, use shock detection
print("Deploy shock detection. Abandon calendar schedule.")Step 3: If episodic, exchange the schedule with these three mechanisms
import pandas as pd
import numpy as np
recall_series = pd.Sequence(your_weekly_metrics)
fraud_counts = pd.Sequence(your_weekly_fraud_counts)
# Mechanism 1 — single-week shock detector
rolling_mean = recall_series.rolling(window=4).imply()
shock_flags = recall_series < (rolling_mean * 0.92)
# Mechanism 2 — volume-weighted recall (extra dependable than uncooked recall)
weighted_recall = np.common(recall_series, weights=fraud_counts)
# Mechanism 3 — two-consecutive-week set off (reduces false retrain alerts)
breach = recall_series < (recall_series.imply() * (1 - 0.07))
retrain_trigger = breach & breach.shift(1).fillna(False)
print(f"Shock weeks detected : {shock_flags.sum()}")
print(f"Quantity-weighted recall : {weighted_recall:.4f}")
print(f"Retrain set off activated : {retrain_trigger.any()}")The brink of 0.92 within the shock detector (alert when recall drops greater than 8% under the 4-week rolling imply) and the retrain threshold of 0.07 relative to the long-run baseline are beginning factors, not mounted guidelines. Calibrate each towards your area’s price asymmetry — the ratio of missed-fraud price to false-alarm price — and your labelling latency.
The Full Diagnostic Report
============================================================
FORGETTING CURVE REPORT
============================================================
Baseline Recall : 0.8807 [top3]
Present Recall : 0.8507
Retention ratio : 96.6%
Decay price lambda : 0.000329
Half-life : 2107.7 days ← statistically meaningless
Forgetting velocity : STABLE
Forgetting regime : EPISODIC
Curve match R-squared : -0.3091 ← the operative quantity
Snapshots logged : 26
Retrain really useful NOW : False
Days till retrain alert : 45.7 ← unreliable in episodic regime
Advisable retrain date : 2026-05-20 ← disregard in episodic regime
Worst single-week drop : Week 7 (−0.1875)
============================================================The stress seen on this report is intentional and necessary. The system concurrently stories Forgetting velocity: STABLE (decay price λ = 0.000329 implies a half-life of two,107 days) and Forgetting regime: EPISODIC (R² = −0.31). Each are appropriate.
The common efficiency throughout 26 weeks is secure — present recall of 0.8507 towards a baseline of 0.8807 provides a retention ratio of 96.6%, comfortably above the 93% retrain threshold. An mixture month-to-month dashboard would present a mannequin working effectively inside acceptable bounds.
The week-to-week behaviour is violently unstable. The worst single shock dropped recall by 18.75 factors in seven days. Three separate weeks dropped under 75% recall. These occasions are completely invisible in mixture metrics and fully unpredictable by a decay mannequin with a 2,107-day half-life.
That is the core failure mode of calendar-based retraining: the granularity at which monitoring occurs determines what’s seen. Episodic shocks solely seem at weekly or sub-weekly decision with per-window fraud case counts as high quality weights.
On the Selection of Baseline Methodology
One calibration choice materially impacts the diagnostic: how the baseline recall is computed from the primary N qualifying weeks.
Three strategies can be found, every with totally different sensitivity traits:
"imply" — arithmetic imply of the primary N weeks. Applicable when early-week efficiency is constant. Delicate to warm-up noise when the mannequin has not but encountered the complete check distribution, which is widespread in fraud detection the place label arrival is delayed.
"max" — peak efficiency within the first N weeks. Probably the most conservative possibility: any subsequent drop under the historic peak is instantly seen. Threat: a single anomalously good week completely inflates the baseline, producing false retrain alerts for weeks which might be performing usually.
"top3" — imply of the highest three values within the first N weeks. The tactic used on this simulation. It filters warm-up noise whereas preserving proximity to true peak efficiency. Advisable for imbalanced classification issues with delayed labelling.
The selection of baseline_window — what number of weeks are included in baseline estimation — issues equally. Six weeks is the minimal for statistical stability at typical fraud prevalence charges. Fewer than six weeks dangers a baseline dominated by early distributional artefacts.
What This Means for MLOps Apply
The sensible implications break cleanly alongside regime traces.
Within the {smooth} regime, calendar-based retraining is legitimate — however the cadence ought to be derived from the empirical decay price, not from conference. A mannequin with a half-life of 180 days ought to be retrained each 120 to 150 days. A mannequin with a half-life of 30 days wants weekly retraining. The precise schedule ought to be calibrated to the purpose the place retention falls to the edge — not picked as a result of month-to-month feels affordable.
Within the episodic regime, calendar-based retraining is operationally wasteful. A mannequin that experiences sudden shocks however recovers will set off scheduled retrains throughout secure restoration intervals, losing compute and labelling finances, whereas the precise shock occasions — the weeks that matter — happen between scheduled dates and go unaddressed till the following calendar set off.
The substitute isn’t extra frequent scheduled retraining. It’s event-driven retraining triggered by the shock detection mechanisms described above: a sudden drop under the rolling imply, sustained throughout two consecutive home windows, with affirmation that the drop isn’t an information artefact (quantity test, fraud price flooring, labelling delay indicator).
That is the excellence that the R² diagnostic makes actionable: it tells you which ones toolbox to open.
Limitations
This evaluation has a number of boundaries that ought to be said explicitly.
The dataset is artificial. The Kaggle fraud dataset used right here was generated utilizing the Sparkov simulation software [3] and doesn’t symbolize actual cardholder or service provider knowledge. The fraud patterns replicate the simulation’s generative mannequin, not precise fraud ring behaviour. The episodic shocks noticed could differ in character from these encountered in dwell manufacturing programs, the place shocks typically contain novel assault vectors that haven’t any prior illustration in coaching knowledge.
A single area. The 2-regime classification framework is proposed primarily based on evaluation of fraud detection knowledge and casual commentary of different domains. A scientific research throughout a number of manufacturing ML programs, together with healthcare danger fashions, content material suggestion engines, and demand forecasting programs, could be required to validate the R² cutoff of 0.4 as a sturdy regime boundary.
Label availability assumption. The simulation assumes weekly recall is computable from ground-truth labels out there throughout the identical week. In lots of fraud programs, confirmed fraud labels arrive with delays of days to weeks as investigations full. The shock detection mechanisms described right here require adaptation for delayed labelling environments — particularly, the rolling window ought to be constructed from label-available transactions solely, and the shock threshold ought to be widened to account for label-arrival variance.
The retrain threshold of seven%. It is a place to begin, not a common suggestion. The operationally appropriate threshold is a operate of the price ratio between missed fraud and false alarms in a selected deployment, which varies considerably throughout service provider classes, transaction values, and assessment staff capability.
Reproducing This Evaluation
The whole implementation — ModelForgettingTracker, the fraud simulation, all 4 diagnostic charts, and the dwell monitoring dashboard — is accessible at https://github.com/Emmimal/mlops-forgetting-curve/.
Necessities:
pip set up pandas numpy scikit-learn matplotlib scipy lightgbmDataset: Obtain fraudTrain.csv and fraudTest.csv from the Kaggle dataset [2] and place them in the identical listing as fraud_forgetting_demo.py.
Run:
python fraud_forgetting_demo.pyTo use the tracker to an present efficiency log with out the Kaggle dependency:
from model_forgetting_curve import load_from_dataframe
tracker = load_from_dataframe(
df,
metric_col="weekly_recall",
metric_name="Recall",
baseline_method="top3",
retrain_threshold=0.07,
)
report = tracker.report()
print(f"Regime: {report.forgetting_regime}")
print(f"R²: {report.fit_r_squared:.3f}")
figs = tracker.plot(save_dir="./charts", dark_mode=True)Conclusion
The Ebbinghaus forgetting curve is a foundational lead to cognitive psychology. As an assumption about manufacturing ML system behaviour, it’s unverified for a whole class of domains the place efficiency is pushed by exterior occasions fairly than gradual distributional drift.
The R² diagnostic offered here’s a one-time, zero-infrastructure test that classifies a system’s forgetting regime from present weekly efficiency logs. If R² ≥ 0.4, the exponential mannequin is legitimate and a retraining schedule is the proper software. If R² < 0.4, the mannequin is within the episodic regime, the half-life is meaningless, and the retraining schedule ought to be changed with event-driven shock detection.
On 555,000 real-synthetic transactions spanning six months of simulated manufacturing, the fraud detection mannequin returned R² = −0.31. The exponential decay mannequin carried out worse than predicting the imply. The worst shock dropped recall by 18.75 factors in seven days with no aggregate-level sign.
The conclusion is exact: scheduled retraining is a symptom of not realizing which regime you might be in. Run the diagnostic first. Then determine whether or not a schedule is smart in any respect.
Disclosure
The creator has no monetary relationship with Kaggle, the dataset creator, or any of the software program libraries referenced on this article. All instruments used — LightGBM, scikit-learn, SciPy, pandas, NumPy, and Matplotlib — are open-source tasks distributed underneath their respective licences. The dataset used for this evaluation is a publicly out there artificial dataset distributed underneath the database contents licence (DbCL) on Kaggle [2]. No actual cardholder, service provider, or monetary establishment knowledge was used. The ModelForgettingTracker implementation described and linked on this article is unique work by the creator, launched underneath the MIT licence.
References
[1] Ebbinghaus, H. (1885). Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie. Duncker & Humblot, Leipzig. English translation: Ebbinghaus, H. (1913). Reminiscence: A Contribution to Experimental Psychology (H. A. Ruger & C. E. Bussenius, Trans.). Academics Faculty, Columbia College. https://doi.org/10.1037/10011-000
[2] Shenoy, Ok. (2020). Credit score Card Transactions Fraud Detection Dataset [Data set]. Kaggle. Retrieved from https://www.kaggle.com/datasets/kartik2112/fraud-detection
Distributed underneath the Database Contents Licence (DbCL) v1.0.
[3] Mullen, B. (2019). Sparkov Knowledge Technology [Software]. GitHub. https://github.com/namebrandon/Sparkov_Data_Generation
[4] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T. Y. (2017). LightGBM: A extremely environment friendly gradient boosting choice tree. Advances in Neural Info Processing Programs, 30, 3146–3154. https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html
[5] Dal Pozzolo, A., Caelen, O., Johnson, R. A., & Bontempi, G. (2015). Calibrating chance with undersampling for unbalanced classification. 2015 IEEE Symposium Sequence on Computational Intelligence, 159–166. https://doi.org/10.1109/SSCI.2015.33
[6] Nagelkerke, N. J. D. (1991). A word on a normal definition of the coefficient of willpower. Biometrika, 78(3), 691–692. https://doi.org/10.1093/biomet/78.3.691
[7] Tsymbal, A. (2004). The issue of idea drift: Definitions and associated work (Technical Report TCD-CS-2004-15). Division of Pc Science, Trinity Faculty Dublin. https://www.scss.tcd.ie/publications/tech-reports/reports.04/TCD-CS-2004-15.pdf
[8] Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on idea drift adaptation. ACM Computing Surveys, 46(4), 44:1–44:37. https://doi.org/10.1145/2523813
[9] Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2019). Studying underneath idea drift: A assessment. IEEE Transactions on Information and Knowledge Engineering, 31(12), 2346–2363. https://doi.org/10.1109/TKDE.2018.2876857
[10] Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Shiny, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, Ok. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., … SciPy 1.0 Contributors. (2020). SciPy 1.0: Elementary algorithms for scientific computing in Python. Nature Strategies, 17, 261–272. https://doi.org/10.1038/s41592-020-0772-5
[11] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesneau, É. (2011). Scikit-learn: Machine studying in Python. Journal of Machine Studying Analysis, 12, 2825–2830. https://jmlr.org/papers/v12/pedregosa11a.html
[12] Hunter, J. D. (2007). Matplotlib: A 2D graphics atmosphere. Computing in Science & Engineering, 9(3), 90–95. https://doi.org/10.1109/MCSE.2007.55
