Asynchronous Multimodal Sensor Data Fusion for Label-Scarce Health Indicator Mining in Composite Structure Monitoring
DOI:
https://doi.org/10.63646/FEPP4520Keywords:
Asynchronous sensor fusion; health indicator mining; semi-supervised learning; acoustic emission; guided waves; composite structures; structural health monitoringAbstract
Structural health monitoring (SHM) of composite aerospace panels requires comprehensive health indicators (HIs) that capture multiscale damage evolution under complex fatigue loading. However, constructing reliable HIs is hampered by the asynchronous acquisition rates of passive and active sensing modalities, the scarcity of ground-truth labels, and the heterogeneous information content of complementary sensor streams. This study proposes a multi-level asynchronous multimodal sensor data fusion framework that integrates passive acoustic emission (AE) and active guided wave (GW) sensing to mine high-quality HIs for T-stiffener composite panels without labeled failure data. Three intra-modality AE pipelines and one GW pipeline are developed under an inductive semi-supervised learning paradigm, using physics-informed proxy labels and criteria-driven regularization to enforce monotonicity, prognosability, and trendability. A causal zero-order-hold synchronization scheme aligns the dense AE timeline with sparse GW acquisitions. Late-fusion regression models—including Gaussian process regression, gradient boosting, support vector machines, multilayer perceptrons, and long short-term memory networks—are benchmarked for inter-modality HI fusion. Leave-one-out cross-validation on 12 AE units and 5 GW units demonstrates that the fused HIs consistently exceed a Fitness score of 90%, with the LSTM-based fusion achieving up to 97% ± 2% cohort-level performance. The proposed framework offers a scalable and deployable blueprint for next-generation prognostics in safety-critical composite structures.
How to Cite
References
Alleyne, D., & Cawley, P. (1992). A two-dimensional Fourier transform method for the measurement of propagating multimode signals. Journal of the Acoustical Society of America, 89(3), 1159–1168. https://doi.org/10.1121/1.400530
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798–1828. https://doi.org/10.1109/TPAMI.2013.50
Boller, C., Chang, F. K., & Fujino, Y. (Eds.). (2009). Encyclopedia of structural health monitoring. John Wiley & Sons. https://doi.org/10.1002/9780470061626
Chapelle, O., Scholkopf, B., & Zien, A. (Eds.). (2009). Semi-supervised learning. MIT Press. https://doi.org/10.7551/mitpress/9780262033589.001.0001
Christensen, R. M. (2013). The theory of materials failure. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199662111.001.0001
Coble, J. B., & Hines, J. W. (2011). Identifying optimal prognostic parameters from data: A genetic algorithms approach. Annual Conference of the Prognostics and Health Management Society, 2(1). https://doi.org/10.36001/phmconf.2011.v2i1.2009
Dasarathy, B. V. (1994). Decision fusion. IEEE Computer Society Press.
Dietterich, T. G. (2000). Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems (pp. 1–15). Springer. https://doi.org/10.1007/3-540-45014-9_1
Farrar, C. R., & Worden, K. (2007). An introduction to structural health monitoring. Philosophical Transactions of the Royal Society A, 365(1851), 303–315. https://doi.org/10.1098/rsta.2006.1928
Farrar, C. R., & Worden, K. (2012). Structural health monitoring: A machine learning perspective. John Wiley & Sons. https://doi.org/10.1002/9781118443118
Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811. https://doi.org/10.48550/arXiv.1807.02811
Fromme, P., Wilcox, P. D., Lowe, M. J. S., & Cawley, P. (2006). On the development and testing of a guided ultrasonic wave array for structural integrity monitoring. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 53(4), 777–785. https://doi.org/10.1109/TUFFC.2006.1621502
Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the 33rd International Conference on Machine Learning, 48, 1050–1059.
Giurgiutiu, V. (2008). Structural health monitoring with piezoelectric wafer active sensors. Academic Press. https://doi.org/10.1016/B978-012088760-6.50007-8
Goebel, K., Daigle, M., Saxena, A., Roychoudhury, I., Sankararaman, S., & Celaya, J. (2008). Prognostics: The science of making predictions. CreateSpace Independent Publishing.
Graves, A. (2012). Supervised sequence labelling with recurrent neural networks. Springer. https://doi.org/10.1007/978-3-642-24797-2
Grieves, M., & Vickers, J. (2017). Digital twin: Mitigating unpredictable, undesirable emergent behavior in complex systems. In Transdisciplinary Perspectives on Complex Systems (pp. 85–113). Springer. https://doi.org/10.1007/978-3-319-38756-7_4
Grosse, C. U., & Ohtsu, M. (Eds.). (2008). Acoustic emission testing. Springer. https://doi.org/10.1007/978-3-540-69972-9
Haghani, R., Al-Emrani, M., & Heshmati, M. (2012). Fatigue-prone details in steel bridges. Buildings, 2(4), 456–476. https://doi.org/10.3390/buildings2040456
Hall, D. L., & Llinas, J. (1997). An introduction to multisensor data fusion. Proceedings of the IEEE, 85(1), 6–23. https://doi.org/10.1109/5.554205
Hamstad, M. A. (1994). Thirty years of advances and some remaining challenges in the application of acoustic emission to composite materials. In P. J. Shull (Ed.), Acoustic emission: Current practice and future directions. ASTM International. https://doi.org/10.1520/STP13003S
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. International Conference on Prognostics and Health Management, 1–6. https://doi.org/10.1109/PHM.2008.4711422
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hullermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3), 457–506. https://doi.org/10.1007/s10994-021-05946-3
Jardine, A. K. S., Lin, D., & Banjevic, D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing, 20(7), 1483–1510. https://doi.org/10.1016/j.ymssp.2005.09.012
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422–440. https://doi.org/10.1038/s42254-021-00314-5
Kim, S., & Bhattacharyya, R. (2012). Decision fusion in structural health monitoring. Expert Systems with Applications, 39(3), 2744–2751. https://doi.org/10.1016/j.eswa.2011.08.161
Lamb, H. (1917). On waves in an elastic plate. Proceedings of the Royal Society of London A, 93(648), 114–128. https://doi.org/10.1098/rspa.1917.0008
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
Lee, J., Wu, F., Zhao, W., Ghaffari, M., Liao, L., & Siegel, D. (2014). Prognostics and health management design for rotary machinery systems—Reviews, methodology and applications. Mechanical Systems and Signal Processing, 42(1–2), 314–334. https://doi.org/10.1016/j.ymssp.2013.06.003
Li, Y., Kurfess, T. R., & Liang, S. Y. (2012). Stochastic prognostics for rolling element bearings. Mechanical Systems and Signal Processing, 14(5), 747–762. https://doi.org/10.1006/mssp.2000.1301
Lissenden, C. J. (2021). Nonlinear ultrasonic guided waves—principles for nondestructive evaluation. Journal of Applied Physics, 129(2), 021101. https://doi.org/10.1063/5.0038340
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Unsupervised fault detection in F0: Data from a subsea gas compressor. Data in Brief, 23, 103804. https://doi.org/10.1016/j.dib.2019.103804
Liu, C., Gryllias, K., & Kanfoud, J. (2020). A new deep learning network for condition monitoring and fault diagnosis. Applied Sciences, 10(4), 1372. https://doi.org/10.3390/app10041372
Liu, H., Liu, C., & Huang, Y. (2019a). Adaptive feature extraction using sparse coding for machinery fault diagnosis. Mechanical Systems and Signal Processing, 107, 1–14. https://doi.org/10.1016/j.ymssp.2018.02.040
Liu, J., Wang, W., Ma, F., Yang, Y. B., & Yang, C. (2019b). A data-model-fusion prognostic framework for dynamic system state forecasting. Engineering Applications of Artificial Intelligence, 26(7), 1680–1689. https://doi.org/10.1016/j.engappai.2013.05.001
Migot, A., Bhuiyan, M. Y., & Giurgiutiu, V. (2022). Multi-sensor guided wave fusion for structural health monitoring of composites. Sensors, 22(10), 3836. https://doi.org/10.3390/s22103836
Mosallam, A., Medjaher, K., & Zerhouni, N. (2016). Data-driven prognostic method based on Bayesian approaches for direct remaining useful life prediction. Journal of Intelligent Manufacturing, 27(5), 1037–1048. https://doi.org/10.1007/s10845-014-0933-4
Mujica, L. E., Vehi, J., Ruiz, M., Verleysen, M., Staszewski, W., & Worden, K. (2010). Multivariate statistics process control for dimensionality reduction in structural assessment. Mechanical Systems and Signal Processing, 22(1), 155–171. https://doi.org/10.1016/j.ymssp.2007.07.004
Nascimento, R. G., & Viana, F. A. C. (2019). Fleet prognosis with physics-informed recurrent neural networks. Structural Health Monitoring, 20(2), 561–576. https://doi.org/10.1177/1475921720954819
Nitta, K., Nakayama, T., & Yamashita, M. (2022). Semi-supervised learning for health condition evaluation from industrial IoT data. Advanced Engineering Informatics, 52, 101570. https://doi.org/10.1016/j.aei.2022.101570
Nuhaa, N., Hasan, M. K., Islam, S., & Marzouk, T. (2022). A comprehensive review on health indicator construction methods for data-driven machinery prognostics. IEEE Access, 10, 87756–87795. https://doi.org/10.1109/ACCESS.2022.3200022
Ostachowicz, W., Kudela, P., Krawczuk, M., & Zak, A. (2012). Guided waves in structures for SHM: The time-domain spectral element method. John Wiley & Sons. https://doi.org/10.1002/9781119965855
Pecht, M. G. (2008). Prognostics and health management of electronics. Wiley. https://doi.org/10.1002/9780470385845
Polikar, R. (2006). Ensemble based systems in decision making. IEEE Circuits and Systems Magazine, 6(3), 21–45. https://doi.org/10.1109/MCAS.2006.1688199
Qing, X., Kumar, A., Zhang, C., Gonzalez, I. F., Guo, G., & Chang, F. K. (2007). A hybrid piezoelectric/fiber optic diagnostic system for structural health monitoring. Smart Materials and Structures, 14(3), S98–S103. https://doi.org/10.1088/0964-1726/14/3/012
Quaresimin, M., Susmel, L., & Talreja, R. (2010). Fatigue behaviour and life assessment of composite laminates under multiaxial loadings. International Journal of Fatigue, 32(1), 2–16. https://doi.org/10.1016/j.ijfatigue.2009.02.012
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707. https://doi.org/10.1016/j.jcp.2018.10.045
Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. MIT Press. https://doi.org/10.7551/mitpress/3206.001.0001
Rizzo, P., Scalea, F. L. di., & Uckan, M. (2007). Wavelet-based unsupervised and semi-supervised anomaly detection in structural health monitoring. Smart Structures and Systems, 3(3), 297–314. https://doi.org/10.12989/sss.2007.3.3.297
Rose, J. L. (2014). Ultrasonic guided waves in solid media. Cambridge University Press. https://doi.org/10.1017/CBO9781107273610
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository, NASA Ames Research Center.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine run-to-failure simulation. International Conference on Prognostics and Health Management, 1–9. https://doi.org/10.1109/PHM.2008.4711414
Seera, M., & Lim, C. P. (2014). A hybrid intelligent system for medical data classification. Expert Systems with Applications, 41(5), 2237–2251. https://doi.org/10.1016/j.eswa.2013.09.022
Si, X. S., Wang, W., Hu, C. H., & Zhou, D. H. (2011). Remaining useful life estimation—A review on the statistical data driven approaches. European Journal of Operational Research, 213(1), 1–14. https://doi.org/10.1016/j.ejor.2010.11.018
Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959.
Soutis, C. (2005). Fibre reinforced composites in aircraft construction. Progress in Aerospace Sciences, 41(2), 143–151. https://doi.org/10.1016/j.paerosci.2005.02.004
Staszewski, W. J. (2002). Intelligent signal processing for damage detection in composite materials. Composites Science and Technology, 62(7–8), 941–950. https://doi.org/10.1016/S0266-3538(02)00008-8
Staszewski, W., Boller, C., & Tomlinson, G. (Eds.). (2004). Health monitoring of aerospace structures: Smart sensor technologies and signal processing. Wiley. https://doi.org/10.1002/0470092866
Su, Z., Ye, L., & Lu, Y. (2006). Guided Lamb waves for identification of damage in composite structures: A review. Journal of Sound and Vibration, 295(3–5), 753–780. https://doi.org/10.1016/j.jsv.2006.01.020
Talreja, R., & Singh, C. V. (2012). Damage and failure of composite materials. Cambridge University Press. https://doi.org/10.1017/CBO9781139016551
Tao, F., Zhang, H., Liu, A., & Nee, A. Y. C. (2018). Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 15(4), 2405–2415. https://doi.org/10.1109/TII.2018.2873186
Torres, M. E., Colominas, M. A., Schlotthauer, G., & Flandrin, P. (2011). A complete ensemble empirical mode decomposition with adaptive noise. IEEE International Conference on Acoustics, Speech and Signal Processing, 4144–4147. https://doi.org/10.1109/ICASSP.2011.5947265
Vachtsevanos, G., Lewis, F. L., Roemer, M., Hess, A., & Wu, B. (2006). Intelligent fault diagnosis and prognosis for engineering systems. Wiley. https://doi.org/10.1002/9780470117842
Van Engelen, J. E., & Hoos, H. H. (2020). A survey on semi-supervised learning. Machine Learning, 109(2), 373–440. https://doi.org/10.1007/s10994-019-05855-6
Virkler, D. A., Hillberry, B. M., & Goel, P. K. (1979). The statistical nature of fatigue crack propagation. Journal of Engineering Materials and Technology, 101(2), 148–153. https://doi.org/10.1115/1.3443666
Worden, K., & Farrar, C. R. (2007). The fundamental axioms of structural health monitoring. Proceedings of the Royal Society A, 463(2082), 1639–1664. https://doi.org/10.1098/rspa.2007.1834
Wu, Z., & Huang, N. E. (2009). Ensemble empirical mode decomposition: A noise-assisted data analysis method. Advances in Adaptive Data Analysis, 1(1), 1–41. https://doi.org/10.1142/S1793536909000047
Yan, J., Koc, M., & Lee, J. (2004). A prognostic algorithm for machine performance assessment and its application. Production Planning & Control, 15(8), 796–801. https://doi.org/10.1080/09537280412331283241
Zhang, B., Sconyers, C., Byington, C., Patrick, R., Orchard, M. E., & Vachtsevanos, G. (2011). A probabilistic fault detection approach: Application to bearing fault detection. IEEE Transactions on Industrial Electronics, 58(5), 2011–2018. https://doi.org/10.1109/TIE.2010.2058072
Zhang, Q., Hua, C., & Xu, G. (2019). A mixture Weibull proportional hazard model for mechanical system failure prediction utilising lifetime and monitoring data. Mechanical Systems and Signal Processing, 43(1–2), 103–112. https://doi.org/10.1016/j.ymssp.2013.10.013
Zhang, W., Li, C., Peng, G., Chen, Y., & Zhang, Z. (2018). A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load. Mechanical Systems and Signal Processing, 100, 439–453. https://doi.org/10.1016/j.ymssp.2017.06.022
Zhang, X., Wan, S., He, Y., Wang, X., & Dou, L. (2021a). Teager energy spectral kurtosis of wavelet packet transform and its application in locating the sound source of fault bearing of belt conveyor. Measurement, 173, 108565. https://doi.org/10.1016/j.measurement.2020.108565
Zhang, Z., Song, Y., & Li, S. (2020). Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering & System Safety, 203, 107093. https://doi.org/10.1016/j.ress.2020.107093
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019). Deep learning and its applications to machine health monitoring. Mechanical Systems and Signal Processing, 115, 213–237. https://doi.org/10.1016/j.ymssp.2018.05.050
Zhao, R., Yan, R., Wang, J., & Mao, K. (2017). Learning to monitor machine health with convolutional bi-directional LSTM networks. Sensors, 17(2), 273. https://doi.org/10.3390/s17020273
Zhao, Z., Zhang, Q., Yu, X., Sun, C., Wang, S., Yan, R., & Chen, X. (2020). Applications of unsupervised deep transfer learning to intelligent fault diagnosis: A survey and comparative study. IEEE Transactions on Instrumentation and Measurement, 70, 3525921. https://doi.org/10.1109/TIM.2021.3053515
Zhu, J., Nostrand, T., Spiegel, C., & Morton, B. (2014). Survey of condition indicators for condition monitoring systems. Annual Conference of the Prognostics and Health Management Society, 5(1). https://doi.org/10.36001/phmconf.2014.v5i1.2464
Zhu, X. J. (2005). Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison.
Zhu, Y., Liu, C., & Jiang, W. (2018). Joint operational and financial risk management of power utilities using Markov decision processes. Applied Energy, 226, 527–535. https://doi.org/10.1016/j.apenergy.2018.06.019
