The data substrate of predictive maintenance: a review of benchmark provenance, label derivation and evidence coverage
DOI:
https://doi.org/10.63646/BBDP3896Keywords:
predictive maintenance; benchmark datasets; data provenance; label derivation; run-to-failure data; maintenance work orders; evidence coverageAbstract
Predictive maintenance research is evaluated almost entirely on a small stock of public benchmark datasets, yet the properties of that stock are rarely examined as objects of study. This review characterizes sixteen widely used predictive-maintenance benchmarks along five provenance attributes: the origin of the record, the derivation of the label, the density of genuine end-of-life events, the span of operating conditions, and the coverage of channels that describe the maintenance decision rather than the machine. Coding these attributes reveals a corpus that is structurally narrow. Every benchmark carries at least one condition-monitoring channel, but only a quarter carry any channel describing the maintenance action, the parts consumed, the energy used or the disposition of the replaced component. Seven of the sixteen carry labels that were never observed on a real in-service failure, being derived instead from a simulation model or from a fault deliberately machined into a test specimen. Not one benchmark in the corpus records energy consumption, and only one records what happened to the component after removal. We then introduce a claim-evidence admissibility mapping that states, for each class of claim made in the predictive-maintenance literature, which data properties are necessary to support it, and we show that claims about energy saving, carbon reduction, lifecycle extension and human-machine collaboration have no admissible substrate in the current stock. This suggests a complementary explanation for the persistent sustainability deficit reported in framework-level gap analyses: the field may be unable to demonstrate environmental benefit not because its architectures exclude it, but because nothing it measures would reveal it. We set out the minimum contents of a sustainability-capable maintenance dataset and a corresponding research agenda.References
[1] Jardine, A.K.S., Lin, D., Banjevic, D., A review on machinery diagnostics and prognostics implementing condition-based maintenance, Mechanical Systems and Signal Processing 20 (2006) 1483–1510. https://doi.org/10.1016/j.ymssp.2005.09.012
[2] Ahmad, R., Kamaruddin, S., An overview of time-based and condition-based maintenance in industrial application, Computers & Industrial Engineering 63 (2012) 135–149. https://doi.org/10.1016/j.cie.2012.02.002
[3] Lee, J., Wu, F., Zhao, W., Ghaffari, M., Liao, L., Siegel, D., Prognostics and health management design for rotary machinery systems — reviews, methodology and applications, Mechanical Systems and Signal Processing 42 (2014) 314–334. https://doi.org/10.1016/j.ymssp.2013.06.004
[4] Sikorska, J.Z., Hodkiewicz, M., Ma, L., Prognostic modelling options for remaining useful life estimation by industry, Mechanical Systems and Signal Processing 25 (2011) 1803–1836. https://doi.org/10.1016/j.ymssp.2010.11.018
[5] Lei, Y., Li, N., Guo, L., Li, N., Yan, T., Lin, J., Machinery health prognostics: a systematic review from data acquisition to RUL prediction, Mechanical Systems and Signal Processing 104 (2018) 799–834. https://doi.org/10.1016/j.ymssp.2017.11.016
[6] Zhang, W., Yang, D., Wang, H., Data-driven methods for predictive maintenance of industrial equipment: a survey, IEEE Systems Journal 13 (2019) 2213–2227. https://doi.org/10.1109/JSYST.2019.2905565
[7] Carvalho, T.P., Soares, F.A.A.M.N., Vita, R., Francisco, R.P., Basto, J.P., Alcalá, S.G.S., A systematic literature review of machine learning methods applied to predictive maintenance, Computers & Industrial Engineering 137 (2019) 106024. https://doi.org/10.1016/j.cie.2019.106024
[8] Zonta, T., da Costa, C.A., da Rosa Righi, R., de Lima, M.J., da Trindade, E.S., Li, G.P., Predictive maintenance in the Industry 4.0: a systematic literature review, Computers & Industrial Engineering 150 (2020) 106889. https://doi.org/10.1016/j.cie.2020.106889
[9] Zio, E., Prognostics and Health Management (PHM): where are we and where do we (need to) go in theory and practice, Reliability Engineering & System Safety 218 (2022) 108119. https://doi.org/10.1016/j.ress.2021.108119
[10] Breque, M., De Nul, L., Petridis, A., European Commission, Directorate-General for Research and Innovation, Industry 5.0: Towards a Sustainable, Human-Centric and Resilient European Industry, Publications Office of the European Union, Luxembourg, 2021. https://doi.org/10.2777/308407
[11] Nahavandi, S., Industry 5.0 — a human-centric solution, Sustainability 11 (2019) 4371. https://doi.org/10.3390/su11164371
[12] Xu, X., Lu, Y., Vogel-Heuser, B., Wang, L., Industry 4.0 and Industry 5.0 — inception, conception and perception, Journal of Manufacturing Systems 61 (2021) 530–535. https://doi.org/10.1016/j.jmsy.2021.10.006
[13] Leng, J., et al., Industry 5.0: prospect and retrospect, Journal of Manufacturing Systems 65 (2022) 279–295. https://doi.org/10.1016/j.jmsy.2022.09.017
[14] Murtaza, A.A., Saher, A., Zafar, M.H., Moosavi, S.K.R., Aftab, M.F., Sanfilippo, F., Paradigm shift for predictive maintenance and condition monitoring from Industry 4.0 to Industry 5.0: A systematic review, challenges and case study, Results in Engineering 24 (2024) 102935. https://doi.org/10.1016/j.rineng.2024.102935
[15] Grieves, M., Vickers, J., Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems, in: Transdisciplinary Perspectives on Complex Systems, Springer, Cham, 2017, pp. 85–113. https://doi.org/10.1007/978-3-319-38756-7_4
[16] Tao, F., Zhang, H., Liu, A., Nee, A.Y.C., Digital twin in industry: state-of-the-art, IEEE Transactions on Industrial Informatics 15 (2019) 2405–2415. https://doi.org/10.1109/TII.2018.2873186
[17] Dalzochio, J., et al., Machine learning and reasoning for predictive maintenance in Industry 4.0: current status and challenges, Computers in Industry 123 (2020) 103298. https://doi.org/10.1016/j.compind.2020.103298
[18] Ran, Y., Zhou, X., Lin, P., Wen, Y., Deng, R., A survey of predictive maintenance: systems, purposes and approaches, arXiv:1912.07383, 2019. https://doi.org/10.48550/arXiv.1912.07383
[19] Moubray, J., Reliability-Centred Maintenance, 2nd ed., Butterworth-Heinemann, Oxford, 1997.
[20] Saxena, A., Goebel, K., Simon, D., Eklund, N., Damage propagation modeling for aircraft engine run-to-failure simulation, in: Proceedings of the International Conference on Prognostics and Health Management, Denver, USA, 2008, pp. 1–9. https://doi.org/10.1109/PHM.2008.4711414
[21] Nectoux, P., et al., PRONOSTIA: an experimental platform for bearings accelerated degradation tests, in: Proceedings of the IEEE International Conference on Prognostics and Health Management, Denver, USA, 2012, pp. 1–8.
[22] Lessmeier, C., Kimotho, J.K., Zimmer, D., Sextro, W., Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: a benchmark data set for data-driven classification, in: Proceedings of the European Conference of the Prognostics and Health Management Society, Bilbao, Spain, 2016. https://doi.org/10.36001/phme.2016.v3i1.1577
[23] Smith, W.A., Randall, R.B., Rolling element bearing diagnostics using the Case Western Reserve University data: a benchmark study, Mechanical Systems and Signal Processing 64–65 (2015) 100–131. https://doi.org/10.1016/j.ymssp.2015.04.021
[24] Page, M.J., et al., The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, BMJ 372 (2021) n71. https://doi.org/10.1136/bmj.n71
[25] Kelly, M., Longjohn, R., Nottingham, K., The UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences.
[26] Saxena, A., Goebel, K., Turbofan Engine Degradation Simulation Data Set, NASA Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA, 2008.
[27] Arias Chao, M., Kulkarni, C., Goebel, K., Fink, O., Aircraft engine run-to-failure dataset under real flight conditions for prognostics and diagnostics, Data 6 (2021) 5. https://doi.org/10.3390/data6010005
[28] Sateesh Babu, G., Zhao, P., Li, X.-L., Deep convolutional neural network based regression approach for estimation of remaining useful life, in: Database Systems for Advanced Applications, Springer, Cham, 2016, pp. 214–228. https://doi.org/10.1007/978-3-319-32025-0_14
[29] Zheng, S., Ristovski, K., Farahat, A., Gupta, C., Long short-term memory network for remaining useful life estimation, in: Proceedings of the IEEE International Conference on Prognostics and Health Management, Dallas, USA, 2017, pp. 88–95. https://doi.org/10.1109/ICPHM.2017.7998311
[30] Li, X., Ding, Q., Sun, J.-Q., Remaining useful life estimation in prognostics using deep convolution neural networks, Reliability Engineering & System Safety 172 (2018) 1–11. https://doi.org/10.1016/j.ress.2017.11.021
[31] Wang, Y., Zhao, Y., Addepalli, S., Remaining useful life prediction using deep learning approaches: a review, Procedia Manufacturing 49 (2020) 81–88. https://doi.org/10.1016/j.promfg.2020.06.015
[32] Fink, O., Wang, Q., Svensén, M., Dersin, P., Lee, W.-J., Ducoffe, M., Potential, challenges and future directions for deep learning in prognostics and health management applications, Engineering Applications of Artificial Intelligence 92 (2020) 103678. https://doi.org/10.1016/j.engappai.2020.103678
[33] Matzka, S., Explainable artificial intelligence for predictive maintenance applications, in: Proceedings of the Third International Conference on Artificial Intelligence for Industries, Irvine, USA, 2020, pp. 69–74. https://doi.org/10.1109/AI4I49448.2020.00023
[34] Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P., Domain randomization for transferring deep neural networks from simulation to the real world, in: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Vancouver, Canada, 2017, pp. 23–30. https://doi.org/10.1109/IROS.2017.8202133
[35] Sünderhauf, N., et al., The limits and potentials of deep learning for robotics, International Journal of Robotics Research 37 (2018) 405–420. https://doi.org/10.1177/0278364918770733
[36] Randall, R.B., Antoni, J., Rolling element bearing diagnostics — a tutorial, Mechanical Systems and Signal Processing 25 (2011) 485–520. https://doi.org/10.1016/j.ymssp.2010.07.017
[37] Wang, B., Lei, Y., Li, N., Li, N., A hybrid prognostics approach for estimating remaining useful life of rolling element bearings, IEEE Transactions on Reliability 69 (2020) 401–412. https://doi.org/10.1109/TR.2018.2882682
[38] Qiu, H., Lee, J., Lin, J., Yu, G., Wavelet filter-based weak signature detection method and its application on rolling element bearing prognostics, Journal of Sound and Vibration 289 (2006) 1066–1090. [no DOI assigned — cite by standard number or repository URL + access date]. https://doi.org/10.1016/j.jsv.2005.03.007
[39] Zhao, Z., et al., Deep learning algorithms for rotating machinery intelligent diagnosis: an open source benchmark study, ISA Transactions 107 (2020) 224–255. https://doi.org/10.1016/j.isatra.2020.08.010
[40] Saha, B., Goebel, K., Battery Data Set, NASA Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA, 2007. [no DOI assigned — cite by standard number or repository URL + access date]
[41] Birkl, C.R., Oxford Battery Degradation Dataset 1, University of Oxford Research Archive, Oxford, 2017. [no DOI assigned by the publisher — cite by proceedings/repository URL + access date]
[42] Purohit, H., et al., MIMII dataset: sound dataset for malfunctioning industrial machine investigation and inspection, in: Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop, New York, USA, 2019, pp. 209–213. https://doi.org/10.33682/m76f-d618
[43] Koizumi, Y., Saito, S., Uematsu, H., Harada, N., Imoto, K., ToyADMOS: a dataset of miniature-machine operating sounds for anomalous sound detection, in: Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, USA, 2019, pp. 313–317. https://doi.org/10.1109/WASPAA.2019.8937164
[44] McCann, M., Johnston, A., SECOM Data Set, UCI Machine Learning Repository, University of California, Irvine, 2008. [no DOI assigned — cite by standard number or repository URL + access date]
[45] Veloso, B., Ribeiro, R.P., Gama, J., Pereira, P.M., The MetroPT dataset for predictive maintenance, Scientific Data 9 (2022) 764. https://doi.org/10.1038/s41597-022-01877-3
[46] Stetco, A., et al., Machine learning methods for wind turbine condition monitoring: a review, Renewable Energy 133 (2019) 620–635. [NOTE: this is a survey, not a data descriptor. If a specific SCADA release is used, add its own citation.] https://doi.org/10.1016/j.renene.2018.10.047
[47] Backblaze, Hard Drive Data and Stats, Backblaze Drive Stats open dataset, San Mateo, CA, published quarterly. [no DOI assigned — cite by standard number or repository URL + access date]
[48] Kaplan, E.L., Meier, P., Nonparametric estimation from incomplete observations, Journal of the American Statistical Association 53 (1958) 457–481. https://doi.org/10.1080/01621459.1958.10501452
[49] Cox, D.R., Regression models and life-tables, Journal of the Royal Statistical Society: Series B 34 (1972) 187–202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
[50] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., SMOTE: synthetic minority over-sampling technique, Journal of Artificial Intelligence Research 16 (2002) 321–357. https://doi.org/10.1613/jair.953
[51] Japkowicz, N., Stephen, S., The class imbalance problem: a systematic study, Intelligent Data Analysis 6 (2002) 429–449. https://doi.org/10.3233/IDA-2002-6504
[52] Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., Schwabacher, M., Metrics for evaluating performance of prognostic techniques, in: Proceedings of the International Conference on Prognostics and Health Management, Denver, USA, 2008, pp. 1–17. https://doi.org/10.1109/PHM.2008.4711436
[53] Saxena, A., Celaya, J., Saha, B., Saha, S., Goebel, K., Metrics for offline evaluation of prognostic performance, International Journal of Prognostics and Health Management 1 (2010) 4–23. https://doi.org/10.36001/ijphm.2010.v1i1.1336
[54] International Organization for Standardization, Energy management systems — Requirements with guidance for use, ISO 50001, ISO, Geneva. [no DOI assigned — cite by standard number or repository URL + access date]
[55] Strubell, E., Ganesh, A., McCallum, A., Energy and policy considerations for deep learning in NLP, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019, pp. 3645–3650. https://doi.org/10.18653/v1/P19-1355
[56] Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O., Green AI, Communications of the ACM 63 (2020) 54–63. https://doi.org/10.1145/3381831
[57] Patterson, D., et al., Carbon emissions and large neural network training, arXiv:2104.10350, 2021. https://doi.org/10.48550/arXiv.2104.10350
[58] Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., Pineau, J., Towards the systematic reporting of the energy and carbon footprints of machine learning, Journal of Machine Learning Research 21 (2020) 1–43. https://doi.org/10.48550/arXiv.2002.05651
[59] International Organization for Standardization, Environmental management — Life cycle assessment — Principles and framework, ISO 14040, ISO, Geneva.
[60] International Organization for Standardization, Greenhouse gases — Carbon footprint of products — Requirements and guidelines for quantification, ISO 14067, ISO, Geneva.
[61] World Resources Institute and World Business Council for Sustainable Development, The Greenhouse Gas Protocol: A Corporate Accounting and Reporting Standard, revised ed., WRI/WBCSD, Washington, DC.
[62] Kirchherr, J., Reike, D., Hekkert, M., Conceptualizing the circular economy: an analysis of 114 definitions, Resources, Conservation and Recycling 127 (2017) 221–232. https://doi.org/10.1016/j.resconrec.2017.09.005
[63] Stahel, W.R., The circular economy, Nature 531 (2016) 435–438. https://doi.org/10.1038/531435a
[64] Ellen MacArthur Foundation, Towards the Circular Economy: Economic and Business Rationale for an Accelerated Transition, Ellen MacArthur Foundation, Cowes, 2013.
[65] Parasuraman, R., Riley, V., Humans and automation: use, misuse, disuse, abuse, Human Factors 39 (1997) 230–253. https://doi.org/10.1518/001872097778543886
[66] Lee, J.D., See, K.A., Trust in automation: designing for appropriate reliance, Human Factors 46 (2004) 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
[67] Sexton, T., Hodkiewicz, M., Brundage, M.P., Smoker, T., Benchmarking for keyword extraction methodologies in maintenance work orders, in: Proceedings of the Annual Conference of the PHM Society, Philadelphia, USA, 2018. https://doi.org/10.36001/phmconf.2018.v10i1.541
[68] Brundage, M.P., Sexton, T., Hodkiewicz, M., Dima, A., Lukens, S., Technical language processing: unlocking maintenance knowledge, Manufacturing Letters 27 (2021) 42–46. https://doi.org/10.1016/j.mfglet.2020.11.001
[69] Vaswani, A., et al., Attention is all you need, in: Advances in Neural Information Processing Systems 30, 2017, pp. 5998–6008. https://doi.org/10.48550/arXiv.1706.03762
[70] Lewis, P., et al., Retrieval-augmented generation for knowledge-intensive NLP tasks, in: Advances in Neural Information Processing Systems 33, 2020, pp. 9459–9474. https://doi.org/10.48550/arXiv.2005.11401
[71] Gebru, T., et al., Datasheets for datasets, Communications of the ACM 64 (2021) 86–92. https://doi.org/10.1145/3458723
[72] Mitchell, M., et al., Model cards for model reporting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, USA, 2019, pp. 220–229. https://doi.org/10.1145/3287560.3287596
[73] Wilkinson, M.D., et al., The FAIR guiding principles for scientific data management and stewardship, Scientific Data 3 (2016) 160018. https://doi.org/10.1038/sdata.2016.18
[74] McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Agüera y Arcas, B., Communication-efficient learning of deep networks from decentralized data, in: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017, pp. 1273–1282. https://doi.org/10.48550/arXiv.1602.05629
[75] Recht, B., Roelofs, R., Schmidt, L., Shankar, V., Do ImageNet classifiers generalize to ImageNet?, in: Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, 2019, pp. 5389–5400. https://doi.org/10.48550/arXiv.1902.10811
[76] Raji, I.D., Bender, E.M., Paullada, A., Denton, E., Hanna, A., AI and the everything in the whole wide world benchmark, in: Proceedings of the NeurIPS Datasets and Benchmarks Track, 2021. https://doi.org/10.48550/arXiv.2111.15366
[77] Paullada, A., Raji, I.D., Bender, E.M., Denton, E., Hanna, A., Data and its (dis)contents: a survey of dataset development and use in machine learning research, Patterns 2 (2021) 100336. https://doi.org/10.1016/j.patter.2021.100336
[78] Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., Aroyo, L.M., Everyone wants to do the model work, not the data work: data cascades in high-stakes AI, in: Proceedings of the CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 2021. https://doi.org/10.1145/3411764.3445518
[79] Northcutt, C.G., Athalye, A., Mueller, J., Pervasive label errors in test sets destabilize machine learning benchmarks, in: Proceedings of the NeurIPS Datasets and Benchmarks Track, 2021. https://doi.org/10.48550/arXiv.2103.14749
[80] Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D. (Eds.), Dataset Shift in Machine Learning, MIT Press, Cambridge, 2009. https://doi.org/10.7551/mitpress/9780262170055.001.0001
[81] Rudin, C., Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1 (2019) 206–215. https://doi.org/10.1038/s42256-019-0048-x
[82] Lundberg, S.M., Lee, S.-I., A unified approach to interpreting model predictions, in: Advances in Neural Information Processing Systems 30, 2017, pp. 4765–4774. https://doi.org/10.48550/arXiv.1705.07874
[83] Ribeiro, M.T., Singh, S., Guestrin, C., Why should I trust you? Explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, USA, 2016, pp. 1135–1144. https://doi.org/10.1145/2939672.2939778
[84] International Organization for Standardization, Condition monitoring and diagnostics of machine systems — Data processing, communication and presentation, ISO 13374, ISO, Geneva.
[85] International Organization for Standardization, Condition monitoring and diagnostics of machines — General guidelines, ISO 17359, ISO, Geneva.
[86] International Organization for Standardization, Asset management — Overview, principles and terminology, ISO 55000, ISO, Geneva. [no DOI assigned — cite by standard number or repository URL + access date]
[87] International Electrotechnical Commission, Dependability management — Part 3-11: Application guide — Reliability centred maintenance, IEC 60300-3-11, IEC, Geneva.