Towards Reliable Osteoarthritis Classification: Fine-Tuned Convolutional Neural Networks, Vision Transformers, and Ensemble Learning Approaches

Osteoarthritis (OA) is a widespread degenerative condition affecting millions of people worldwide. Early detection and precise classification are crucial for effective disease management. This study investigated the use of deep learning techniques to classify the severity of knee OA from X-ray images, specifically targeting three categories: Normal (KL Grade 0), Moderate (KL Grade 3), and Severe (KL Grade 4). We utilized a dataset from the Osteoarthritis Initiative (OAI), containing 3,221 X-ray images of the knee, and fine-tuned eight pretrained CNNs (DenseNet201, EfficientNetB7, InceptionV3, InceptionResNetV2, ResNet50V2, ResNet152V2, Vision Transformer B32, and Xception). A custom CNN and ensemble deep learning models (hard and weighted voting) were also proposed with a total of 11 models. The models were assessed using a dataset split of 70% for training, 15% for validation, and 15% for testing, ensuring comprehensive evaluation across all development stages. DenseNet201 achieved the highest classification accuracy of 97.11% among the individual models, while Vision Transformer B32 showed the lowest accuracy of 59.38%. Ensemble methods using hard and weighted voting, incorporating the top five models, achieved a consistent accuracy of 97.11%. These results demonstrate the potential of deep learning, particularly ensemble strategies, in accurately classifying knee OA severity. This method can help build smarter tools that assist doctors in making better decisions, aiding in the early detection and management of OA, offering a robust tool for improving patient outcomes.

Classification; Deep Learning; Ensemble Methods; Knee X-ray; Osteoarthritis

Ahmed, R., & Imran, A. S. (2024). Knee osteoarthritis analysis using deep learning and XAI on X-rays. IEEE Access, 12, 68870–68879. https://doi.org/10.1109/ACCESS.2024.3400987

Alshamrani, H. A., Rashid, M., Alshamrani, S. S., & Alshehri, A. H. D. (2023). Osteo-Net: An automated system for predicting knee osteoarthritis from X-ray images using transfer-learning-based neural networks approach. Healthcare, 11(9). https://doi.org/10.3390/healthcare11091206

Belagatti, P. (2024). Understanding the softmax activation function: A comprehensive guide [SingleStore Blog]. https://www.singlestore.com/blog/a-guide-to-softmax-activation-function/

Bhandari, A. (2020). AUC-ROC curve in machine learning clearly explained [Online article].

Borawar, L., & Kaur, R. (2023). ResNet: Solving vanishing gradient in deep networks. In Lecture Notes in Networks and Systems (Vol. 600, pp. 235–247). https://doi.org/10.1007/978-981-19-8825-7_21

Chauhan, R., Ghanshala, K. K., & Joshi, R. C. (2018). Convolutional neural network (CNN) for image detection and recognition. Proceedings of the International Conference on Secure Cyber Computing and Communications (ICSCCC), 278–282. https://doi.org/10.1109/ICSCCC.2018.8703316

Chen, P. (2018). Knee osteoarthritis severity grading dataset. Mendeley Data.

Demir, A., & Yilmaz, F. (2020). Inception-ResNet-v2 with LeakyReLU and average pooling for more reliable and accurate classification of chest X-ray images. TIPTEKNO 2020 Medical Technologies Congress. https://doi.org/10.1109/TIPTEKNO50054.2020.9299232

DenseNets, F., Background, I., & What, M. (2021). Understanding and visualizing DenseNets [Technical article].

Duklan, N., Kumar, S., Maheshwari, H., Singh, R., Sharma, S. D., & Swami, S. (2024). CNN architectures for image classification: A comparative study using ResNet50V2, ResNet152V2, InceptionV3, Xception, and MobileNetV2. SSRG International Journal of Electronics and Communication Engineering, 11(9), 11–21. https://doi.org/10.14445/23488549/IJECE-V11I9P102

Fajarani, R., Rahman, S. F., Pangesty, A. I., Katili, P. A., Park, D. H., & Basari. (2024). Physical and chemical characterization of collagen/alginate/poly(vinyl alcohol) scaffold with the addition of multi-walled carbon nanotube, reduced graphene oxide, titanium dioxide, and zinc oxide materials. International Journal of Technology, 15(2), 332–341. https://doi.org/10.14716/ijtech.v15i2.6693

Foti, G., & Longo, C. (2024). Deep learning and AI in reducing magnetic resonance imaging scanning time: Advantages and pitfalls in clinical practice. Polish Journal of Radiology, 89, 443–451. https://doi.org/10.5114/pjr/192822

Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022). Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115, 105151. https://doi.org/10.1016/j.engappai.2022.105151

Ganesh Kumar, M., & Goswami, A. D. (2023). Automatic classification of the severity of knee osteoarthritis using enhanced image sharpening and CNN. Applied Sciences, 13(3). https://doi.org/10.3390/app13031658

Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., & Tao, D. (2023). A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 87–110. https://doi.org/10.1109/TPAMI.2022.3152247

Hossin, M., & bin Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International Journal of Data Mining and Knowledge Management Process, 5, 1–11. https://doi.org/10.5121/ijdkp.2015.5201

Kaur, P., Kohli, G. S., Bedi, J., & Wasly, S. (2024). A novel deep learning approach for automated grading of knee osteoarthritis severity. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-024-20322-8

Keith Sinusas, M. (2012). Osteoarthritis: Diagnosis and treatment. American Family Physician, 85(1), 49–56.

Kingma, D. P., & Ba, J. L. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations, 1–15.

Kondal, S., Kulkarni, V., Gaikwad, A., Kharat, A., & Pant, A. (2022). Automatic grading of knee osteoarthritis on the Kellgren–Lawrence scale from radiographs using convolutional neural networks. In Lecture Notes in Networks and Systems (Vol. 249, pp. 163–173). https://doi.org/10.1007/978-3-030-85365-5_16

Koonce, B. (2021). ResNet-50. In Convolutional neural networks with Swift for TensorFlow (pp. 63–72). https://doi.org/10.1007/978-1-4842-6168-2_6

Mlsna, P. A., & Rodriguez, J. J. (2009). Gradient and Laplacian edge detection. In The essential guide to image processing (pp. 495–524). https://doi.org/10.1016/B978-0-12-374457-9.00019-6

National Institutes of Health. (2012). Osteoarthritis Initiative (OAI) dataset. https://nda.nih.gov/oai

Olsson, S., Akbarian, E., Lind, A., Razavian, A. S., & Gordon, M. (2021). Automating classification of osteoarthritis according to Kellgren–Lawrence in the knee using deep learning in an unfiltered adult population. BMC Musculoskeletal Disorders, 22(1), 1–8. https://doi.org/10.1186/s12891-021-04722-7

Ou, J., Zhang, J., Alswadeh, M., Zhu, Z., Tang, J., Sang, H., & Lu, K. (2025). Advancing osteoarthritis research: The role of AI in clinical, imaging, and omics fields. Bone Research, 13(1). https://doi.org/10.1038/s41413-025-00423-2

Patro, S. G. K., & Sahu, K. K. (2015). Normalization: A preprocessing stage. IARJSET, 20–22. https://doi.org/10.17148/iarjset.2015.2305

Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021). Vision transformers for dense prediction. Proceedings of the IEEE International Conference on Computer Vision, 12159–12168. https://doi.org/10.1109/ICCV48922.2021.01196

Rani, S., Memoria, M., Almogren, A., Bharany, S., Joshi, K., Altameem, A., Rehman, A. U., & Hamam, H. (2024). Deep learning to combat knee osteoarthritis and severity assessment by using CNN-based classification. BMC Musculoskeletal Disorders, 25(1). https://doi.org/10.1186/s12891-024-07942-9

Sarvamangala, D. R., & Kulkarni, R. V. (2021). Grading of knee osteoarthritis using convolutional neural networks. Neural Processing Letters, 53(4), 2985–3009. https://doi.org/10.1007/s11063-021-10529-3

Sharma, R., & Kamra, A. (2023). A review on CLAHE-based enhancement techniques. Proceedings of the International Conference on Contemporary Computing and Informatics (IC3I), 321–325. https://doi.org/10.1109/IC3I59117.2023.10397722

Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, 10691–10700.

Tariq, T., Suhail, Z., & Nawaz, Z. (2025). A review for automated classification of knee osteoarthritis using KL grading scheme for X-rays. Biomedical Engineering Letters, 15(1). https://doi.org/10.1007/s13534-024-00437-5

Touahema, S., Zaimi, I., Zrira, N., Ngote, M. N., Doulhousne, H., & Aouial, M. (2024). Med-Knee: A new deep learning-based software for automated prediction of radiographic knee osteoarthritis. Diagnostics, 14(10). https://doi.org/10.3390/diagnostics14100993

Vorontsov, I. E., Kulakovskiy, I. V., & Makeev, V. J. (2013). Jaccard index-based similarity measure to compare transcription factor binding site models. Algorithms for Molecular Biology, 8(1). https://doi.org/10.1186/1748-7188-8-23

Wing, N., Van Zyl, N., Wing, M., Corrigan, R., Loch, A., & Wall, C. (2021). Reliability of three radiographic classification systems for knee osteoarthritis among observers of different experience levels. Skeletal Radiology, 50(2), 399–405. https://doi.org/10.1007/s00256-020-03551-4

Xie, X., Zhang, K., Li, Y., Li, Y., Li, X., Lin, Y., Huang, L., & Tian, G. (2025). Global, regional, and national burden of osteoarthritis from 1990 to 2021 and projections to 2035: A cross-sectional study for the Global Burden of Disease Study 2021. PLOS One, 20(5). https://doi.org/10.1371/journal.pone.0324296

Xin Teoh, Y., Othmani, A., Li Goh, S., Usman, J., & Lai, K. W. (2024). Deciphering knee osteoarthritis diagnostic features with explainable artificial intelligence: A systematic review. IEEE Access, 12, 109080–109108. https://doi.org/10.1109/ACCESS.2024.3439096

Yong, C. W., Teo, K., Murphy, B. P., Hum, Y. C., Tee, Y. K., Xia, K., & Lai, K. W. (2022). Knee osteoarthritis severity classification with ordinal regression module. Multimedia Tools and Applications, 81(29), 41497–41509. https://doi.org/10.1007/s11042-021-10557-0

Yoon, J. S., Yon, C. J., Lee, D., Lee, J. J., Kang, C. H., Kang, S. B., Lee, N. K., & Chang, C. B. (2023). Assessment of a novel deep learning-based software developed for automatic feature extraction and grading of radiographic knee osteoarthritis. BMC Musculoskeletal Disorders, 24(1), 1–10. https://doi.org/10.1186/s12891-023-06951-4

Zhang, D., Dong, Y., Xu, Y., Qian, J., Ye, M., Yuan, Q., & Luo, J. (2024). Enhancing knee osteoarthritis diagnosis with DMS: A novel dense multi-scale convolutional neural network approach. Journal of Orthopaedic Surgery and Research, 19(1), 1–9. https://doi.org/10.1186/s13018-024-05352-0

Zhao, H., Ou, L., Zhang, Z., Zhang, L., Liu, K., & Kuang, J. (2025). The value of deep learning-based X-ray techniques in detecting and classifying KL grades of knee osteoarthritis: A systematic review and meta-analysis. European Radiology, 35(1), 327–340. https://doi.org/10.1007/s00330-024-10928-9

Zhu, Y., & Newsam, S. (2018). DenseNet for dense flow. Proceedings of the International Conference on Image Processing (ICIP), 790–794. https://doi.org/10.1109/ICIP.2017.8296389