Deep Learning Automated System for Thermal Defectometry of Multilayer Materials

Currently, along with growth in industrial production, the requirements for product quality testing are also increasing. In the tasks of defectoscopy and defectometry of multilayer materials, the use of thermal nondestructive testing method is promising. At the same time, interpretation of thermal testing data is complicated by a number of factors, which makes the use of traditional methods of data processing ineffective. Therefore, an urgent task is to search for new methods of thermal testing that will automate the diagnostic process and increase information content of obtained results. The purpose of article is to use the advances in deep learning for processing results of active thermal testing of products made of multilayer materials and development of an automated system for thermal defectoscopy and defectometry of such products. The proposed system consists of a heating source, an infrared camera for recording sequences of thermograms and a digital information processing unit. Three neural network modules are used for automated data processing, each of which performs one of the tasks: defects detection and classification, determination of the defect depth and thickness. The software algorithms and user interface for interacting with system are programmed in the NI LabVIEW development environment. Experimental studies on samples made of multilayer fiberglass have shown a significant advantage of the developed system over using traditional methods for analyzing thermal testing data. The defect classification (determining the type) error on the test dataset was 15.7 %. Developed system ensured determination of defect depth with a relative error of 3.2 %, as well as the defect thickness with a relative error of 3.5 %.

Ключевые слова: тепловой контроль, композиционные материалы, глубокое обучение. DOI: 10.21122/2220-9506-2021-12-2-98-107 Introduction Nowadays, products made of multilayer and composite materials are widely used in various industries. In particular, composite materials are increasingly used in aircraft industry, from which the most responsible elements of aircraft construction are made. At the same time, there is a tendency to increase the requirements for product quality testing. Timely detection of hidden defects makes it possible to prevent significant material and sometimes human losses. Due to a number of advantages, methods of active thermal non-destructive testing (TNDT) are used in composite materials testing tasks. Therefore, it is important to create automated systems for determining characteristics of defects based on the results of active TNDT, which will have increased informativeness, reliability and accuracy of defectometry in conditions of significant levels of noise and complex internal structure of the object of testing (OT).
The results of multilayer materials testing are influenced by a large number of random factors due to changes in the properties of composites, which occur due to complexity of their manufacturing processes, a large number of types of possible defects that cannot be formalized, imperfect inspection methods and defectoscopic equipment. Features of properties and physical characteristics of multilayer materials complicate the use of many existing methods of TNDT, which use mainly deterministic models and their corresponding data processing methods. Such methods do not provide the necessary noise immunity, measurement accuracy and reliability of testing [1].
A rather limited number of scientific papers are devoted to the analysis of thermal fields for the purpose of automated simultaneous detection, classification of defects and determination of their parameters. Initial researches were aimed at performing defectometry by solving the inverse tasks of TNDT. With the development of modern technologies of digital data processing, development trends have shifted to the application of latest statistical methods and intelligent systems based on deep learning.
Today, classical methods of digital signal processing, such as Fourier transform or wavelet analysis, are used to analyze the data of active TNDT [2,3]. In particular, the algorithm of dynamic thermal tomography is implemented with the use of these methods [4]. Another approach is based on a comprehensive statistical analysis of the entire recorded sequence of thermograms, which uses the principal components analysis method [5]. Each of these methods has its advantages and disadvantages, but they are all used to solve a narrow range of tasks and are not universal and adaptive [6].
In [7,8] the method of recognition of threedimensional defects is described. It uses the method of degree of similarity estimating for surface thermal field of OT with the existing 3D surface models, which were obtained by numerical modeling of three-dimensional thermal conductivity task. This approach in practice demonstrates low adaptability, as it requires construction of mathematical models of OT for each new testing task.
The work [9] is devoted to the study of deep learning application for composites testing. Study shows results of processing experimental data on carbon fiber testing using two neural networks, which provide both qualitative detection of hidden defects and defectometry elements. The first neural network is designed to detect defective areas, and the second is used to classify defects by depth. The high efficiency of the neural network in both types of problems is proved.
The authors of [10] conducted a study of the effectiveness of method for determining defects depth in multilayer materials using deep learning. It is presented and implemented a new algorithm based on the use of a multilayer neural network to determine the depth of defects in real time. Study uses computer simulations to create an artificial data set. An experimental validation of neural networks efficiency was performed, which showed an up to 10 % error in determining defects depth at the level of 0.5 mm.
Analysis of existing publications shows that the use of modern intelligent systems allows to solve the problems of thermal defectometry with increased efficiency. Existing studies prove the prospects of using deep learning for defect classification and defectometry. The error in measuring defects depth by traditional methods reaches 7-10 %, while neural networks can reduce it to 2.5-3 %. At the same time, existing works do not provide a quantitative assessment of the effectiveness of determining defects thickness using deep learning. The authors mainly focus on solving one specific testing task, while the modern approach requires a comprehensive automated analysis of OT in order to describe it as fully as possible. Currently, there are no systems that in practice implement a simultaneous automated solution to the problems of detecting defects of multilayer materials by the active thermal method, their classification and defectometry.
Thus, there is a need to develop new methods and automated testing systems for products made of multilayer materials. A large number of interconnected informative parameters, impossibility of linear separation of defects classes on diagnostic grounds, need for automation and increasing testing informativeness require to use the latest data processing systems, in particular, based on deep learning algorithms.

Physical principles of active thermal nondestructive testing
Dynamic thermal field is described by the function T (x, y, τ). During the active thermal non-destructive testing, the character of change in instantaneous temperature values over time at surface points of OT is considered. To obtain these dependences, the OT is heated by a heat source for a certain time. The process of heating and further cooling of OT is registered using a thermal imager. Resulting sequence of thermograms reflects the change in temperature field on the surface of the OT over time [11].
Considering the temperature dynamics at individual points (pixels) of thermograms, which correspond to the coordinates of OT surface, it is possible build a temperature profile -a chart of temperature change over time for this point ( Figure 1). As a rule, in defect-free areas, the nature of temperature change is constant and is considered known. In this case, we can enter some reference temperature T nd (x nd , y nd , τ), which is considered defect-free. In the defect zone, the regular nature of the thermal field is disturbed and local temperature differences T d (x, y, τ), occurs, which lead to a change in the temperature profile. Thus, it is possible to calculate the value of temperature difference between defective and defect-free areas: The time τopt, at which the value of ΔT (x, y, τ) in this area of OT becomes the maximum, is called the optimal time of testing: As the size of the defect increases, its heating rate decreases, which leads to a change in the shape of the temperature profile. In particular, for deeper defects the value of ΔТ max decreases and the time of optimal observation τ opt increases. Quality of obtained thermograms significantly depends on the characteristics of heat source and instrument for recording the thermal field. Ensuring uniform heating in practice is a difficult task, as the nature of heating is influenced by imperfections of the heat source and numerous external factors, such as influence of external emitters, air movement etc. Due to the anisotropy of characteristics, composite materials have different values of thermal conductivity along the coordinate axes, which leads to shape distortion of defects thermal imprints [12]. Therefore, task of testing process automating and finding new or improving existing testing methods that will provide high informativeness, reliability and accuracy in such conditions is relevant.

Automated system of thermal defectometry structure
Trends in the development of TNDT place the following requirements on testing systems: a high level of automation; high informativeness, speed and productivity of testing; versatility and high adaptability; high reliability of testing and accuracy of defectometry. To meet these requirements, it is necessary to use modern hardware and software. At the same time, the general scheme of active thermal testing remains unchanged. The object of testing is exposed by a heat source. Inside a solid, thermal energy is distributed in all directions due to the diffusion process. In the presence of hidden defects, the heat fluxes inside OT are redistributed, which leads to the appearance of specific temperature anomalies on its front and rear surfaces.  2021, vol. 12, no. 2, pp. 98-107 A.S. Momot et al. Приборы и методы измерений 2021 The temperature field of OT is observed and registered using an infrared camera. Temperature signals, presented in the form of thermograms, are transmitted to an automated data processing system on a PC to detect defects and determine their parameters [13]. The choice of testing scheme, characteristics of heat source and thermal imaging equipment significantly affect the diagnostic result. The efficiency of traditional methods of thermogram sequence processing directly depends on these factors. This reduces the versatility of testing systems that use standard data processing algorithms. In particular, changing the OT, heat source or thermal imager in many cases leads to the need for a complete recalculation of system parameters. The use of modern methods of TNDT data processing on the basis of deep learning allows you to add information about new OT or take into account changes in testing conditions without losing previous data. Because all information about network experience is contained in weights, retraining the system in the event of inspection of new objects or the appearance of new types of defects will not necessitate changes in further data processing algorithms.
Based on the analysis of existing schemes of active thermal testing, it is possible to synthesize the scheme of realization of automated TNDT data processing system using an improved method of determining the defects characteristics. This method involves automated data analysis in three neural network modules. The modular structure facilitates construction and modification of the system and increases overall efficiency of its work by optimizing the settings of modules to solve specific problems [14].
General block diagram of the automated system for determining defects characteristics is shown in Figure 2. The system is universal and can be used for various testing schemes and regardless of the characteristics of heat source, infrared camera or OT parameters. The core of each neural network module uses a deep feedforward network. Software algorithms of the system are implemented in NI LabVIEW environment.

Experimental studies of the proposed system
In order to conduct experimental studies of the efficiency of automated thermal defectometry system, two training and one test sample of multilayer fiberglass were developed. This material is used as a structural for manufacture of critical parts with high strength. Developed samples are square plates of five layers fiberglass. Total thickness of each sample is 5 mm, the thickness of one layer is 1 mm. The side of the plate is 100 mm.
The scheme of the test sample is shown in Figure 3. It contains hidden artificial defects of three types: air cavities (white in Figure 3), paper foreign inclusions (red) and aluminum third inclusions (blue). Defects have a square shape, the size of side is from 10 mm to 4 mm. Hidden artificial defects are located at depths of 1 to 3 mm and have different values of thickness: 1 mm, 2 mm or 3 mm.
The scheme of bilateral active TNDT was used during the experiment. The power of infrared heat source was 1 kW. To minimize the impact of thermal radiation from the heat source on results, a steel protective plate was used, which contains a hole and a mount for OT. The plate with OT was located at a distance of 100 mm from the heater. The distance   The infrared camera and heat source are controlled by operator in manual mode. Ambient temperature during the experimental studies was 20 °C. At the beginning of experiment, the first thermogram was registered OT at the initial time. After turning on the infrared heater and putting it to work, recording of thermograms begins. Time interval between adjacent thermograms is 6 s. Heating and recording of the experimental sequence of thermograms was carried out for 120 s. After the thermograms recording procedure is completed, the heater is switched off. Experiment resulted in a sequence of 20 thermograms. Obtained results reflect the process of OT thermal field changing at the stage of heating.
Recorded thermogram sequences were exported to a PC. The initial processing of thermograms was carried out using proprietary Testo IRSoft software. The resolution of the each obtained thermogram is 320 × 240 pixels. Thermograms are stored as arrays of pixel temperatures. Based on the obtained results, a set of initial data for further processing is formed. The thermogram of test sample at the optimal time of testing is shown in Figure 4. On the optimal thermogram it is possible to distinguish visually 8 thermal prints of artificial defects. Due to significant boundary effects, information on bottom row of defects is lost. In general, the OT thermogram is characterized by uneven heating, which complicates its automated processing by standard methods. Next, only the region of interest (which is directly OT) is considered. Figure 5a shows samples of temperature profiles of the defect-free and defective areas for different types of defects, lying at a depth of 3 mm. An example of differential temperature signals from artificial defects of the test sample, which are located at different depths, is shown in Figure 5b. To form a set of training vectors for neural network modules, two training samples were developed and manufactured. The material, structure and geometric dimensions of the training samples correspond to similar parameters of test sample. The procedure of training samples testing took place according to the method and conditions described for the test sample. Training samples contain artificial internal defects in the form of air cavities, foreign aluminum and paper inclusions with different geometric dimensions, thickness and depth values. In total, 6 artificial models with different parameters were created for each type of defects.
As a result, a set of temperature profiles vectors with a total number of 6545 samples was formed. This set includes 3605 examples of temperature profiles from defect-free areas, 1414 profiles of defects in the form of air cavities, 1019 profiles of defects in the form of paper inclusions and Devices andMethods of Measurements 2021, vol. 12, no. 2, pp. 98-107 A.S. Momot et al. Приборы и методы измерений 2021. -Т. 12, № 2. -С. 98-107 A.S. Momot et al. 104 507 profiles of defects in the form of aluminum inclusions. Training dataset is characterized by a certain unevenness, which arose due to the limited number of training samples. The set of training vectors was divided into training / validation / test subsets in the proportion of 70 % / 15 % / 15 % respectively. In order to process the experimental sequence of thermograms of the test sample, neural networks of appropriate modules for detection and classification, determination of depth and thickness of defects were created and trained. To solve these tasks, it is advisable to choose the architecture of neural networks for the detection and classification of defects, which is shown in Figure 6. Architecture of networks for determining defects depth and thickness is similar. The input layer contains 20 neurons, which corresponds to the number of thermograms in sequence. The source layer contains 4 (according to the number of classes) or 1 neuron.
The Levenberg-Markard algorithm was used as an optimizer. Loss function -MSE, metric -MAE. According to the training results, MAE of defects depth determination on the validation set was 0.028 mm, MAE of thickness determination was 0.019 mm.
The defects map, obtained by the results of work of trained neural network module for defects detection and classification, is shown in Figure 7. All 12 artificial defects were detected and unmistakably classified on the map. Defects color on the map corresponds to their depth.
The shape and size of the defects are close to true ones. In the image we can see some dots of incorrectly classified temperature profiles, which can be filtered by a median filter. In addition, Figure 7 also shows binary defect maps obtained using classical methods: optimal thermogram, Fourier and wavelet analysis methods, principal components analysis method (PCA), and dynamic thermal tomography (DTT). Visually it is possible to notice the increased efficiency of offered system on the basis of deep learning in comparison with classical methods.   Devices andMethods of Measurements 2021, vol. 12, no. 2, pp. 98-107 A.S. Momot et al. Приборы и методы измерений 2021. -Т. 12, № 2. -С. 98-107 A.S. Momot et al. Figure 8 shows the thermal tomogram obtained by the DTT method and the image of test sample internal structure, which was built on the results of proposed system. At the defects boundaries there are negative edge effects in the form of anomalous emissions. This effect can be eliminated using median filtering. In general, image of the internal structure is reliable. Quantitative evaluation of certain parameters and comparison of results with traditional methods is given below. The neural network module for defects detection and classification allows to determine the size of defects by their thermal imprints with the highest accuracy among considered methods. Temperature profiles were automatically classified with an error of 15.7 %. The value of Tanimoto criterion [15] at 88.7 % confirms the high reliability of constructed defects map.
In considered conditions of testing the use of deep learning is the only method that gives chance to define defects depth effectively. Corresponding neural network module allows to determine the depth

Discussion
According to the results of quantitative evaluation of effectiveness of defects detecting in test sample by different methods (Table), it is established that the best results are demonstrated by developed automated system based on deep learning. In particular, the use of neural networks is only method by which it was possible to detect all 12 artificial defects and automatically classify them. Depth, mm Devices andMethods of Measurements 2021, vol. 12, no. 2, pp. 98-107 A.S. Momot et al. Приборы и методы измерений 2021. -Т. 12, № 2. -С. 98-107 A.S. Momot et al. of artificial defects of in test sample with a relative error within ± 3.2 %. DTT method in a similar task showed negative results, which makes it impossible to assess the accuracy of determining defects depth. Relative error in determining defects thickness by neural network module is ± 3.5 %. Constructed thermal tomogram of the internal structure of sample is generally reliable, but at the boundaries of some defects there are negative boundary effects. Therefore, the thickness measurement must be performed at the central points of thermal imprints of the defects.
Consequently, the use of proposed automated system based on deep learning demonstrates the advantages of neural network modules over traditional methods in all testing tasks. Due to the high noise immunity and generalizing properties of neural networks, the presence of non-uniform heating has a weak effect on the efficiency of defects detection in multilayer materials and the accuracy of measuring their parameters.

Conclusion
In the paper offered to use the deep learning approach for automation of thermal defectometry of products from multilayer materials. The system for implementation of this method consists of three modules based on neural networks. Modules are designed to solve tasks of defects classification by type, determining their depth and thickness. Experimentally established that developed automated system allowed to detect and classify all artificial defects embedded in the test sample, and to estimate their depth with an error within ± 3.2 % and thickness with an error up to ± 3.5 %. Defect maps constructed as a result of processing experimental data using the proposed system have a high reliability according to Tanimoto criterion (88.1 %). In addition, the results of comparative analysis show that the developed system has an advantage over traditional methods in qualitative and quantitative indicators.
The main direction for further research is to optimize the architecture of neural networks of relevant system modules by using the latest advances in deep learning. In particular, it is proposed to introduce normalization and dropout layers into the network architecture, to change the training optimization algorithm and activation function of fully connected layers. An important task is also the formation of a wide training samples dataset with different defects and materials configurations. This will expand the scope of developed automated system without the need to retrain neural networks for each individual task or type of multilayer material.