Hand Motion Analysis for Recognition of Qualified and Unqualified Welders using 9-DOF IMU Sensors and Support Vector Machine (SVM) Approach

This research aimed to find out how to identify qualified and unqualified welders of shielded metal arc welding (SMAW) in the shipyard industry. A cost-effective system that can identify the welder skills in real time is needed to reduce the cost of inspection and to maintain weldment quality. In this study, 9-degree of freedom (DOF) sensors of the inertial measurement unit (IMU) were applied to measure and to record the typical hand motions of welders. These sensors consisted of an accelerometer, a gyroscope, and a magnetometer installed in a microcontroller board, known as a wearable device. The wearable device was fitted on a welder's hand to monitor and to record wrist-hand motions of both qualified and unqualified welders. The data on inertial measurements of the welder's hand motions were sent through a Bluetooth connection and then saved in a memory card of a smartphone. Some properties, such as the root mean square (RMS), correlation index, spectral peaks, and spectral power, were extracted from the time-series data to characterize hand motions. The support vector machine (SVM) method, a part of the artificial intelligence (AI) technique, was applied to classify and to recognize the typical hand motions of the two types of welders using a supervised learning approach. The validation results showed that the proposed system was able to identify qualified and unqualified welders.  

9-DOF IMU sensors; Support vector machine; Qualified and unqualified welders

       Shielded metal arc welding (SMAW) is one type of the welding process that is commonly used in steel construction, including that in the shipbuilding industry. Such a welding process is executed mostly using the E6013 electrode, which is easily found on the market (Purnama et al., 2020). This process is still performed manually, and the weldment quality depends largely on welder skills when maintaining welding parameters, such as travel speed, the distance between electrode tips and the base metal, electric current, and voltage, with reference to the qualified welding procedure specification (WPS). The effect of those parameters on temperature distribution was simulated numerically using the finite are affected accordingly. Presently, such effects of hand motion in manual welding have been reduced by using automatic welding equipment and applying robotic technology as described by Sunar et al. (2019). However, the application of robotics technology in the shipbuilding industry is still limited due to many variations of the construction joint.

The verification of qualified (Q) or unqualified (UQ) welders of SMAW during the welding process is one of the problems in the shipbuilding industry. Even Q welders who stop doing welding for a certain period may experience a decline in their competency. Additionally, there are still many practices to replace Q welders with UQ ones for the purpose of cost reduction. Another case is the decrease in weldment quality when a welder applies improper hand motions due to fatigue, less focus, or different WPS applications. To overcome these problems, welding training or familiarization according to the WPS is frequently required from all welders before they are assigned to a new construction.

It has been observed and identified that the typical characteristics and steadiness of a welder's hand motions during the welding process significantly contribute to the weldment quality (Pribadi and Shinoda, 2020). If these hand motion characteristics can be monitored and recorded, the weldment quality can then be evaluated accordingly by analyzing the created graph. It is made possible using wearable sensors that can measure the linear acceleration, degree acceleration, and magnetic force of wrist-hand motions. In this case, the pattern of magnetic force signals can reflect the stability of the electric voltage and current that are applied during the welding process.

Many researchers have tried the application of motion sensors to monitor and to record various types of motion. An accelerometer sensor was used to measure the moment of the object position, including the classification of activities (Godfrey et al., 2008; Bayat et al., 2014). Recently, it was shown that the welder's hand motion pattern can be monitored and recorded by a wearable device with a 9-degree of freedom (DOF) inertial measurement unit (IMU). A multilayer perceptron approach was developed to classify welder activities in various butt-welding positions (flat-horizontal-vertical) commonly done in the shipbuilding industry (Pribadi and Shinoda, 2020). The application of a motion analysis system (MAS) is becoming increasingly popular in assisting with real-time monitoring in civil construction projects. A MAS makes a significant contribution to the manufacturing process related to increasing productivity and enhancing ergonomic aspects (Bortolini et al., 2018). Recently, sensor systems with a wearable device have been developed and researched for use by construction workers (Valero et al., 2017).

Many classifying methods that are parts of artificial intelligence (AI) technology can make the system work. The support vector machine (SVM) model is a frequently used approach to classifying activities (Gyllensten, 2010; Attal et al., 2015). The others are the multilayer perceptron model (Gyllensten, 2010; Bayat et al., 2014) and the deep convolutional neural networks model (Jiang and Yin, 2015). This utilization of AI techniques has also been combined with cloud infrastructure development (Keshavarzian et al., 2019).

1.1.  Human Motion

Human motion or movement is described in general as related to mobility, function, and occupation, which is an end product of an intentional or unintentional consequence of joint motion and muscle activity (Everett and Kell, 2010). On one hand, anatomical and physiological functions are the focus of studies in kinesiology. On the other hand, mechanical knowledge and methods focus on biomechanical studies (Godfrey et al., 2008). Kinematics and kinetics are part of biomechanical motion. Kinematic motion focuses on the motion characteristics that do not take into account the force factor that causes the motion. Information regarding the position, velocity, acceleration, and both linear and angular motions can be used in kinematic analysis (Godfrey et al., 2008). Several researchers have developed the study of motions to monitor the characteristics of physiological and physical effects of the human body. Some examples are monitoring physiological conditions and activities in the construction work environment (Lee et al., 2017), measuring physical intensity (Kong et al., 2018), and measuring body fatigue of workers (Bowen et al., 2019). The use of kinematic sensors to monitor the motion characteristics is the focus of this research.

1.2.  Welder Motions in Manual Welding

SMAW is one of the popular types of welding technology in which the process is fully controlled by the welder's hand. Welders perform welding functions and must be capable of controlling their hand movements (Erasmus+, 2014). Personal skill is essential for obtaining welding results, following applicable standards (Moore and Booth, 2015). Welders must be able to maintain arc distance and electrodes while manipulating the electrodes continuously during welding. They must undergo welding training to develop sufficient skills in electrode manipulation in various positions of butt welding, such as 1G-flat, 2G-horizontal, and 3G-vertical (Erasmus+, 2014).

Various hand motions, such as weaving, pushing, and dragging produce the different forms of welds. A specific weaving technique should be applied to ensure sound beads of welding. Pulling techniques are utilized for maximum penetration and for generating solid-looking welds. A combination of pushing and weaving techniques is applicable in 3G- vertical welding positions to prevent the molten metal from slipping. The semi-circle weave method can be applied for keeping the overheating in the weld pool. The welding process with weaving techniques can be repeated through pool welding to get more heat in the process. A combination of the values of linear acceleration, angular acceleration, and magnetic force can be used to monitor typical signal-forming characteristics of welder hand motion.

1.3.  Recognition of Qualified (Q) and Unqualified (UQ) Welders

Recently, developments in human activity monitoring have been increasing in various sectors. Researchers compete in undertaking significant developments to meet the industrial demand for real-time monitoring. Various types of equipment, such as cameras, PIR, and wearable motion sensors (GPS, accelerometers, gyroscopes, and magnetometers) are often applied in monitoring human activity (Dong et al., 2019; Qin et al., 2020). However, only common human activities (sleeping, standing, walking, standing, and climbing) have been carried out by the researchers. No such system has been implemented in industrial welding activities.

The present study aims to determine whether a wearable device equipped with an accelerometer, a gyroscope, and a magnetometer can be used to monitor and record wrist-hand motions. Each sensor has 3 DOFs in the direction of X, Y, and Z, providing a total of 9- DOF motion monitoring. After the measurement data are recorded, the use of the SVM method with a supervised learning approach is explored to classify and to recognize the wrist-hand motions of Q and UQ welders. The MATLAB AI software (MATLAB, 2020) is used to process the data, to perform a data-training procedure, and to test the accuracy of the recognition.

This study examined the potential of utilizing real-time monitoring by means of a wearable device to differentiate between Q and UQ welders. Various data patterns can be extracted using wearable devices. A database consisting of hand motion patterns by Q and UQ welders was developed as a reference for further identification of welders' qualifications. The proposed method can be considered a novel and cost-effective approach to overcoming some practical steel construction problems, including those in shipbuilding. Welding supervisors could use this method to reduce contact with welders, without compromising the quality of the results in uncertain conditions, such as during the COVID-19 pandemic. This system could also be used to identify Q welders who might apply improper hand motions because of fatigue or decreased focus, as well as to identify a replacement of a Q welder with a UQ one. The supervisors could monitor the performance of welders using a web-based monitoring system and respond accordingly.

    The authors thank all members of PT Dok dan Perkapalan (DPS) Surabaya for the opportunity to observe the welding practices in the shipbuilding process.

Alpaydin, E., 2014. Introduction to Machine Learning Ethem Alpaydin. Introduction to Machine Learning, Third Edition. MIT Press. Massachusetts. pp. 32–36

Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi, F., Oukhellou, L., Amirat, Y., 2015. Physical Human Activity Recognition using Wearable Sensors. Sensors, Volume 15(12), pp. 31314–31338

Bayat, A., Pomplun, M., Tran, D.A., 2014. A Study on Human Activity Recognition using Accelerometer Data from Smartphones. Procedia Computer Science. Volume 34, pp. 450–457

Bortolini, M., Faccio, M., Gamberi, M., Pilati, F., 2018. Motion Analysis System (MAS) for Production and Ergonomics Assessment in the Manufacturing Processes. Computers & Industrial Engineering, Volume 139, https://doi.org/10.1016/j.cie.2018.10.046

Bowen, J., Hinze, A., Griffiths, C., 2019. Investigating Real-Time Monitoring of Fatigue Indicators of New Zealand Forestry Workers. Accident Analysis and Prevention, Volume 126, pp. 122–141

Dong, J., Xia, Z., Yan, W., Zhao, Q., 2019. Dynamic Gesture Recognition by Directional Pulse Coupled Neural Networks for Human-Robot Interaction in Real Time. Journal of Visual Communication and Image Representation, Volume 63, https://doi.org/10.1016/j.jvcir.2019.102583

Erasmus+, 2014. Mechanized, Orbital and Robot Welding-Mechanized Welding, pp. 1–94

Everett, T., Kell, C., 2010. Human Movement: An Introductory Text. Churchill Livingstone. Churchill Livingstone/Elsevier, New York, pp. 105–109

Godfrey, A., Conway, R., Meagher, D., ÓLaighin, G., 2008. Direct Measurement of Human Movement by Accelerometry. Medical Engineering and Physics, Volume 30(10), pp. 1364–1386

Gyllensten, I.C., 2010. Physical Activity Recognition in Daily Life using a Triaxial Accelerometer. PhD thesis, KTH Royal Institute of Technology.

Hsu, C.W., Lin, C.J., 2002. A Comparison of Methods for Multiclass Support Vector Machines. IEEE Transactions on Neural Networks, Volume 13(2), pp. 415–425

Jiang, W., Yin, Z., 2015. Human Activity Recognition using Wearable Sensors by Deep Convolutional Neural Networks. In: Proceedings of the 2015 ACM Multimedia Conference, pp. 1307–1310

Keshavarzian, A., Sharifian, S., Seyedin, S., 2019. Modified Deep Residual Network Architecture Deployed on Serverless Framework of IoT Platform based on Human Activity Recognition Application. Future Generation Computer Systems, Volume 101, pp. 14–28

Kong, L., Li, H., Yu, Y., Luo, H., Skitmore, M., Antwi-Afari, M.F., 2018. Quantifying the Physical Intensity of Construction Workers, a Mechanical Energy Approach. Advanced Engineering Informatics, Volume 38, pp. 404–419

Lee, W., Lin, K.Y., Seto, E., Migliaccio, G.C., 2017. Wearable Sensors for Monitoring On-Duty and Off-Duty Worker Physiological Status and Activities in Construction. Automation in Construction, Volume 83, pp. 341–353

MATLAB, 2020. Natick, Massachusetts: The MathWorks Inc. Mathworks: USA

Moore, P., Booth, G., 2015. 3-Welding problems and defects. P. Moore & G. Booth, ed., The Welding Engineer's Guide to Fracture and Fatigue. Oxford: Woodhead Publishing, pp. 23–36

Pribadi, T.W., Shinoda, T., 2020. Hand Motion Recognition of Shipyard Welder using 9-DOF Inertial Measurement Unit and Multi Layer Perceptron Approach. In: IOP Conference Series: Earth and Environmental Science, Volume 557, pp. 1–15

Purnama, D., Winarto, W., Sofyan, N., Prihastomo, A., Ito, K., 2020. Microstructure and Mechanical Properties of Ah-36 Steel Weldment Welded using Magnesium Modified E6013 Electrode. International Journal of Technology, Volume 11(1), pp. 48–59

Qian, H., Mao, Y., Xiang, W., Wang, Z., 2010. Recognition of Human Activities using SVM Multi-Class Classifier. Pattern Recognition Letters, Volume 31(2), pp. 100–111

Qin, Z., Zhang, Y., Meng, S., Qin, Z., Choo, K.-K.R., 2020. Imaging and Fusing Time Series for Wearable Sensor-Based Human Activity Recognition. Information Fusion, Volume 53, pp. 80–87

Sunar, A., Reggi, B., Kurniawan, P., Haikal., 2019. Evaluation of the 2-Axis Movement of a 5-Axis Gantry Robot for Welding Applications. International Journal of Technology, Volume 10(5), pp. 1024–1032

Valero, E., Sivanathan, A., Bosché, F., Abdel-Wahab, M., 2017. Analysis of Construction Trade Worker Body Motions using a Wearable and Wireless Motion Sensor Network. Automation in Construction, Volume 83, pp. 48–55