Supply Chain Management of Drug Products in Blockchain Using Reinforcement Learning

Supply chain management (SCM) is a complex system that consists of two parts: a management system for the medication supply chain and a recommendation system. The first part of the system is the management system powered by reinforcement learning. The reinforcement Learning model is trained to recommend the most suitable medications for a diverse set of customers. The model will be fed customer ratings and reviews via the proposed framework client application, where it will learn over time to make the most informed drug recommendations possible. The proposed system will provide patients with recommendations for successful drug therapies.

Blockchain; Drug; Product; Reinforcement learning; Supply chain management

The effective management and security of pharmaceutical product distribution chains necessitate the adoption of distributed ledger technology. An increasing number of pharmaceutical firms are integrating blockchain into their supply chains due to its numerous advantageous applications (Hannah et al., 2022). Blockchain offers a decentralized electronic ledger accessible and verified by all network nodes, making it potentially beneficial across various industries (Singh et al., 2021).

Blockchain technology presents a decentralized electronic ledger that is both accessible and verified by all network nodes, offering potential advantages across a range of industries (Saurabh and Dey, 2021). To enable efficient and effective operations for manufacturers, suppliers, customers, and distributors within a supply chain, the establishment of a secure and dependable technical platform is imperative. According to Shahbazi and Byun (2021a), a supply chain involves the collaboration of suppliers and manufacturers from order inception to conclusion. This encompasses activities ranging from raw material sourcing to product recycling, allowing them to maximize their investments of time and capital (Yuvaraj et al., 2020).

      However, attaining a heightened level of network resilience necessitates modifications to conventional supply chain networks. Rapid adaptability and response become crucial during natural disasters like earthquakes, floods, or recent events such as virus outbreaks. Supply chain (SC) networks have evolved to establish a robust framework for conducting business in unforeseeably challenging circumstances, such as when suppliers are unable to meet customer demands. These Resilient Supply Chain (RSC) networks were instituted to provide a stable foundation for business operations in unpredictable and challenging environments (Shukla et al., 2021).
     It is imperative to note that key supplier selection and segmentation factors, such as robust enhancers like backup providers and risk reducers, primarily come into play post-catastrophic events. This awareness is crucial because these criteria may face implementation challenges across a supply chain (SC) due to inadequate communication and coordination among constituent organization. The adoption of these measures during crises becomes an absolute necessity (Ahamed and Karthikeyan, 2020; Brilly-Sangeetha et al., 2020; Liu et al., 2020). Key supplier selection and segmentation factors, like backup providers and risk reduction, mainly apply after catastrophic events. This awareness is crucial as these criteria, vital in emergencies, may encounter implementation challenges due to insufficient communication and coordination within a supply chain (SC). Implementing these measures during crises is essential (Ahamed and Karthikeyan, 2020; Brilly-Sangeetha et al., 2020; Liu et al., 2020).
    This highlights blockchain technology's potential for enhancing transparency and traceability in pharmaceutical supply chains. The work underscores blockchain's role as an immutable ledger in preventing counterfeit drugs and ensuring product authenticity. Additionally, the study focuses on applying machine learning algorithms for personalized medication recommendations. They stress the importance of patient data and reviews in enhancing drug recommendation accuracy, aligning with our system's medication recommendation component (Patel et al. 2022). The research explores the use of reinforcement learning in optimizing supply chain operations, demonstrating the potential for RL agents to make informed decisions in complex supply chain environments, mirroring our proposed RL approach. In the realm of patient-centric healthcare (Passerat et al., 2020). The concept of tailoring medical treatments to individual patient needs aligns with our medication recommendation system’s objective of providing personalized drug therapy suggestions based on user-specific data. Additionally, their research addresses issues concerning transaction throughput and latency in blockchain networks, which are relevant to our system’s scalability considerations (Sahoo et al., 2019).
      The development of the platform is necessary to accomplish this objective. The emerging technology known as blockchain has the potential to facilitate a resilient technological platform that has the potential to revolutionize SC, particularly in circumstances in which time is of the essence.

2.    Literature Review

Wong et al. (2021) evaluate the technological feasibility of blockchain research. Blockchain functions as a data transfer and communication network within a community. However, the immutability, substantial accumulation, and heterogeneity of supply chain transaction data on peer-to-peer blockchains may pose efficiency and memory challenges for reliant supply chain management systems. To counter this, an advanced cloud-based blockchain architecture was established to maintain high standards in supply chain management, with careful attention to scalability, accessibility, security, and virtualization as the platform expands.

        Abbas et al. (2020) introduced the DSCMR, an innovative system merging blockchain technology and machine learning for medication supply chain management. Our solution features a machine learning-based medication recommendation system alongside a blockchain-driven pharmaceutical supply chain governance system. This technology facilitates real-time monitoring of medication distribution by pharmaceutical companies. The machine learning component, utilizing N-gram models, provides guidance on treatments likely to yield desired results, using specific datasets for model training. Challenges include network scope limitations and industry hesitance to adopt real-time technology.

        Cendrawati et al. (2023) provide a thorough review of machine-learning (ML) applications in supply chains, addressing implications, limitations, and managerial recommendations. They emphasize the need for additional research on AI's trajectory and real-time pricing using RL techniques to enhance supply chain management.

        Fong et al. (2023) utilized a variety of machine learning strategies to optimize inventory management. They introduced a scalable deep neural network architecture to rectify supply-and-demand disparities. This framework processes multiple input variables, predicting supply and demand patterns based on transaction data. Model accuracy was assessed for a customizable demand forecasting model. Integrative layers facilitated the mapping of high-dimensional features to a lower-dimensional subdomain, fostering data harmonization from various sources.

       Milani et al. (2020) focused on supply chain management concerning chronic diseases. They investigated the use of forecasting models for non-communicable diseases (NCDs) and effective data acquisition and assessment methods. The study also examined the application of predictive models in infectious disease supply chain management. This novel approach utilizes numerical forecasting models and machine learning to analyze vertical and horizontal healthcare supply chain interactions, incorporating diverse data collection techniques and analytical methodologies for anticipating adverse medical outcomes.

       Shah et al. (2021) delineate essential components for efficient supply chain operations, particularly in pharmaceutical manufacturing. Recent advancements in supply chain optimization include the adoption of machine learning-driven software applications, which reduce lead times and enhance forecasting for more time and cost-effective methods. The research focuses on paracetamol, a significant pharmaceutical ingredient produced in large quantities in India.

    The first part of the proposed system is a management system for the medication supply chain, and the second part of the system is a recommendation system.

3.1. Management System for Medication Supply Chain

The first component of the proposed system concentrates on the management of the medication supply chain using blockchain technology. This entails utilizing blockchain as a decentralized and transparent ledger to record and monitor various drug supply chain transactions and events. Key features and benefits of the management system may include:

     Transparency: Blockchain ensures real-time visibility in pharmaceutical supply chains, reducing counterfeit risks and boosting stakeholder trust.

    Traceability: The system records transactions on the blockchain, creating an unchangeable record of the drug's journey, improving accountability, and facilitating issue resolution.

   Security: Blockchain's decentralization ensures data security by preventing tampering and fraud, thereby strengthening the integrity of the medication supply chain.

    Streamlined Processes: By harnessing blockchain automation, the management system optimizes inventory, orders and logistics, thereby reducing errors and enhancing operational efficiency.

3.2. Recommendation System

       The second component employs reinforcement learning to analyze customer data for personalized medication recommendations:

     Personalization: The reinforcement learning model utilizes customer feedback and historical data to personalize drug recommendations, considering efficacy, side effects, and preferences.

    Continuous Learning: The recommendation system continually updates its knowledge, enhancing accuracy through new data in the reinforcement learning model. It adapts to changing trends and preferences.

      Enhanced Patient Outcomes: The system aims to enhance patient outcomes and satisfaction by providing tailored drug recommendations. Patients receive suggestions for pharmacological therapies with higher success probabilities, minimizing adverse effects.

      Decision Support: The recommendation system serves as a valuable decision-support tool for healthcare professionals, assisting them in selecting appropriate medications. It offers insights and recommendations derived from aggregated data and best practices, aiding informed decision-making.

      By integrating these two components, the proposed system provides end-to-end management of the medication supply chain and offers personalized and effective drug recommendations through reinforcement learning. This comprehensive strategy has the potential to optimize supply chain operations, enhance patient care, and boost the overall performance of the pharmaceutical industry.

       The blockchain's primary and vital role is to maintain decentralized data records by consolidating numerous transactions within a single block. This foundational function lends its name to distributed ledger technology. To ensure maximum security, each transaction undergoes hashing and encryption before storage. The proposed solution is a user-centric architecture that offers distributed ledger and smart contract features on a pay-per-use model, enhancing accessibility and flexibility. The front-end web application handles a wide variety of tasks, including but not limited to medicine orders, the supply of raw materials, data updates, order updates, record updates, and medication deliveries. It is also capable of handling many of these tasks simultaneously. The prevention of the sale of legal medications on the black market is the primary objective of this approach; hence, this goal serves as the central focal point of this strategy.

Blockchains, with their inherent cryptographic security measures and data integrity maintenance procedures, ensure the absolute safety of our proposed solution. They enable real-time tracking of prescribed medication, providing users with location and status information. Additionally, our method allows interconnected peers to execute create, read, update, and delete operations on shared data. To segment the entire network into distinct private networks, we have implemented channel principles. The channel's objective is to enable users to establish secure and anonymous private networks. In our system, customers are required to scan drug package barcodes for authenticity verification before making purchases at the pharmacy, requiring barcode-equipped packaging. Patients access detailed medication information, encompassing production date, manufacturer, cost, expiry date, and more. The genesis of blockchain channels traces back to this observed phenomenon's origins. Our capability allows suppliers to communicate solely with factories regarding raw materials. Users maintain privacy through concealed channels for confidential discussions, preserving conversation confidentiality.

       Our system incorporates a machine learning-driven recommendation engine, assisting pharmaceutical company clients in optimal medication selection. Deep learning algorithms, trained on customer feedback from medicine-related websites, and sentiment analysis contribute to personalized medication recommendations. While existing drug SCM systems exist, our proposed system uniquely provides patients with recommendations for effective therapies. Users receive private credentials and enrollment certificates from the user administrator for successful network enrollment and authentication. These trustworthy blockchain networks are made accessible to their users. In addition, the consensus mechanism oversees attaching the user who has registered to the private network. This is the place where transactions can be carried out and orders can be controlled, and it is the responsibility of the consensus mechanism to do so. This enables each peer node to participate in the blockchain functionality.

This paper introduces a comprehensive pharmaceutical supply chain management (SCM) system seamlessly integrated with advanced learning algorithms and blockchain technology. The SCM component optimizes inventory, order fulfillment, and distribution across supply chain stages. The Learning Algorithm, trained on extensive patient reviews, enhances drug recommendations for personalized, data-driven suggestions. The Blockchain component ensures data security, transparency, traceability, user authentication, and data ownership, forming a robust and interconnected ecosystem that transforms pharmaceutical distribution, improves patient care, and safeguards supply chain integrity.

3.3. Supply Chain Environment

       Markov Decision Process (MDP) provides both the theoretical foundation and a structured framework for learning with well-defined objectives, achieved through interactions with a digital environment. Within a specified number of time steps, a reinforcement learning (RL) agent engages in conversations with its surrounding world (see Figure1).

Figure 1 Markov Decision Process

     At each discrete time step t T within a sequence, an interaction occurs between the RL agent and the environment. After successfully executing an action at time step t, the agent receives a numerical reward denoted as Rt R and obtains a representation of the current state of the environment, marked as St S. Subsequently, the agent A_tA progresses from state S_t to state S_t+1. A trajectory is made up of the following sequence of states, actions, and rewards (eq.1):

where            

       S_n - terminal state.

The agent engages in recurrent interactions with the stochastic environment at each time step, referred to as episodes. While St and Rt adhere to the policy , the agent closely monitors them. In such scenarios, the agent's primary objective is to devise a strategy that maximizes the potential cumulative advantage. The dynamics of an RL agent operating within the MDP framework (eq. 2), and the probability of the agent reaching state S* and receiving reward R* by taking action A* in state St, are denoted by the symbol p(.).

In SCE, it is the agent’s job to determine, at each time step t, the quantity of products that need to be ordered for each stage m. This calculation falls under the purview of the SCE. The value for each reorder quantity can be located in the ‘At’ action, represented as an integer. This integer represents the quantity of each reorder at different locations within the supply chain.

      The primary role of the RL agent is to coordinate production to maximize income over the planning horizon. This is achieved by constructing a vector representing the state St, which integrates inventory levels across stages and prior activities. SCE pertains to a single-product supply chain with multiple tiers. The model operates based on certain assumptions, including no temporal decay in product sales and refill quantities presented in whole units.

      The set M = {0, 1, …, mend} represents various economic actors in the supply chain. Stores fulfil consumer requirements at this stage. The stage mend, symbolizing the raw material source, is referred to as stage repair. The product lifecycle involves intermediaries, including merchants and wholesalers, from Stage-1 to Stage-1.

      In each subsequent stage, one unit breaks down into its fundamental components, forming the starting point for the next stage. Manufacturing and shipping of new parts occur with consistent lead times to prevent inventory depletion. All stages, except the last, have production and storage limits. The final stage assumes an uninterrupted supply of raw materials. During the simulation, the following events unfold at each time step t  T:

• All the processes, except for the region responsible for storing raw materials, are responsible for submitting orders for replenishment.
• The process of determining whether reorders can be fulfilled is going to involve looking at both the manufacturing capabilities and the stock levels.
• With the rise in demand, the retail establishment is making better use of the inventory that it already possesses to satisfy the orders placed by customers.
• Any demand or restocking requests that cannot be fulfilled are canceled; there is no backlog of unfulfilled requests.
• There is a cost involved in maintaining each individual item of surplus stock, and this cost must be paid.

The following equations can be used to describe the dynamic behavior of the SCE in the case where  are both identical to one another (eq.3):

In each phase m and period t, I represent the initial stock quantity. The received but undelivered items are denoted as V, indicating their presence in the pipeline inventory system. Both the requested reorder quantity (Q) and the accepted reorder quantity (Q) are identical. A time interval L must pass between iterations to acquire new supplies. Demand (D) follows a Poisson discontinuous distribution.

       The total sales  in each period are directly linked to the sum of customer demand at stage 0 and approvals at stage 1. U represents net demand, which is obtained by subtracting item acquisition costs, penalties for unmet demand, and inventory maintenance costs from sales revenue. and h denote unit sales price, procurement cost, penalty for unfulfilled demand, and inventory holding charges, respectively. The unit penalty for unmet demand is also represented by h. When production capacity and stock levels are at their maximum without exceeding capacity restrictions (c), Q equals Qˆ.

Resource limitations may restrict the number of allowed reorders. It's assumed an infinite supply of raw materials is available at this stage, with predetermined replenishment sizes. The desire is for accurate and comprehensive information accessibility at every supply chain link. However, achieving this assumption appears unattainable in many practical contexts due to limited digitization. Yet, in numerous other supply chains, this assumption holds true. Many organizations are now opting to outsource supply chain responsibilities in areas where they hold a significant competitive advantage. These strategies, from inception to execution, must first gain widespread acceptance before they can be effetely implemented in the real world.

       SCE can proxy the evaluation of generic algorithms in real-world supply chain management, contingent on some information availability. This hinges on presumed information access. The critic accurately predicts reward function behavior, expressed as Loss(), minimizing parameter updates. This unique loss function distinguishes PPO, ensuring stable learning across diverse benchmarks (eq.4).

        The policy undergoes k rounds of revisions (variable k). The constraint, cl(.), is expressed as , with  as a hyper-parameter for stricter policy change limits. The temporal difference error T measures the gap between actual and critic neural network-estimated rewards. The sum of discounted prediction errors Aˆt sums actual and estimated time steps. Both the temporal difference error and the total of discounted prediction errors  are computed here.

The model segments trajectories uniformly, handles N agents concurrently over T time steps, and fine-tunes the surrogate loss using mini-batch stochastic gradient descent. The goal is to update the policy while preserving comparability, balancing method complexity and ease of implementation.

The UCI Machine Learning Datasets contain patient evaluations and ratings of various drugs, sourced from well-known pharmaceutical databases like Drug.com and Druglib.com. For this study, data exclusively from Drug.com was utilized, considered a consensus among industry experts. Users can rate medications on a 0 to 10 scale, with reviews categorized by the treated medical condition. While the dataset isn't extensive, its well-organized nature makes it suitable for training deep learning models despite its size.

The evaluations are broken down into three distinct parts, which are as follows: positive impacts, and consensus. With the use of a web crawler and the soup package in Python, the dataset was constructed from several sources, which included ratings and reviews submitted by people. Following the completion of the crawling procedure, the dataset contains a total of 215,063 drug reviews, and the number of medicines that were intended to be the focus of the analysis has been settled on 541.

During the data collection phase, a total of 63,376 predefined destinations were under consideration. Our machine-learning models have been rigorously developed and tested using this dataset. The model will receive customer ratings and reviews through the proposed framework client application, continuously learning to provide well-informed drug recommendations. After being trained and integrated into the blockchain framework via a RESTful API, the system will offer condition-specific drug recommendations. This will occur following the framework's initial provision of recommendations for condition-specific medications.

Figure 3 illustrates the SCM system's accuracy, showcasing its ability to provide correct recommendations and effectively manage the supply chain. The increased accuracy demonstrates the system's enhanced precision and reliability, leading to more successful outcomes. Figure 4's F-Measure combines precision and recall, offering a comprehensive performance measure. A higher F-Measure indicates a better balance between precision and recall, highlighting the system's improved capability to provide precise drug recommendations without missing relevant ones. Figure 5, representing precision, quantifies the proportion of accurate drug recommendations among all suggestions, showing the system's efficiency in delivering valuable drug suggestions. Evaluating system scalability, tests with 100, 200, and 300 users showed minimal improvements in response time only with the third user group, as depicted in Figures 2 to 6.

Customers can perform this action by logging into the front end of our program and conducting a drug search. Despite this, customers will receive the most accurate suggestions possible regarding alcohol use and other related situations. While evaluating the blockchain network, it is also possible to provide information regarding transaction latency, throughput, success rate, and performance measures. Anyone who needs a medicine suggestion is welcome to use our platform; however, only authorized users are permitted to execute transactions utilizing our blockchain-based platform.

    Our recommendation system assures customers by exclusively suggesting highly-rated and proven-effective medications based on their medical information. Through our user-friendly client application, patients can seek answers to their health-related queries. Compared to prior scholarly efforts, our system exhibits superior performance. While blockchain technology has garnered attention in pharmaceutical supply chain research, our system stands out as it pioneers medication recommendations. Future research should prioritize enhancing scalability and performance, possibly through improved distributed ledger tech, refined reinforcement learning algorithms, and resource allocation. Minimizing latency and optimizing response times is vital for maintaining a seamless user experience during system expansion.

Abbas, K., Afaq, M., Ahmed-Khan, T., Song, W.C., 2020. A Blockchain and Machine Learning-Based Drug Supply Chain Management and Recommendation System for Smart Pharmaceutical Industry. Electronics, Volume 9(5), p. 852

Ahamed, N.N., Karthikeyan, P., 2020. A Reinforcement Learning Integrated in Heuristic Search Method for Self-Driving Vehicle Using Blockchain in Supply Chain Management. International Journal of Intelligent Networks, Volume 1, pp. 92–101

Brilly-Sangeetha, S., Wilfred-Blessing, N.R., Yuvaraj, N., Adeline-Sneha, J., 2020. Improving The Training Pattern in Back-Propagation Neural Networks Using Holt-Winters’ Seasonal Method and Gradient Boosting Model. In: Applications of Machine Learning, pp. 189–198

Cendrawati, D.G., Hesty, N.W., Pranoto, B., Aminuddin, Kuncoro, A.H., Fudholi, A., 2023. Short-Term Wind Energy Resource Prediction Using Weather Research Forecasting Model for a Location in Indonesia. International Journal of Technology, Volume 14(3), pp. 584–595

Chen, H., Chen, Z., Lin, F., Zhuang, P., 2021. Effective Management for Blockchain-Based Agri-Food Supply Chains Using Deep Reinforcement Learning. IEEE Access, Volume 9, pp. 36008–36018

Fong, S.W.L., Ismail, H., Kian, T.P., 2023. Role of Brand Trust in Private Label Adoption Model-an Affective and Trust-based Innovation Characteristic. International Journal of

         Technology, Volume 14(5), pp. 993–1008

Hannah, S., Deepa, A.J., Chooralil, V.S., Brilly-Sangeetha, S., Yuvaraj, N., Arshath-Raja, R., Alene, A. 2022. Blockchain-Based Deep Learning to Process Iot Data Acquisition in Cognitive Data. BioMed Research International, Volume 2022, p. 5038851

Liu, Y., Yu, F.R., Li, X., Ji, H., Leung, V.C., 2020. Blockchain and Machine Learning For Communications and Networking Systems. IEEE Communications Surveys & Tutorials, Volume 22(2), pp. 1392–1431

Milani, M.M., Begawan, B.S., Darussalam, B., Imran, M.M.A.H., Ramashini, M., Krishani, M., 2020, Forecasting of Supply Chain Management for Non-Communicable Diseases. In: Proceedings of the International Conference on Advances in Computing and Technology (ICACT–2020), Kelaniya, Sri Lanka, pp. 85–87

Natarajan, Y., Srihari, K., Dhiman, G., Chandragandhi, S., Gheisari, M., Liu, Y., Lee, C.C., Singh, K.K., Yadav, K., Alharbi, H.F., 2022. An IoT and Machine Learning?Based Routing Protocol for Reconfigurable Engineering Application. IET Communications, Volume 16(5), pp. 464–475

Patel, V., Shah, M., 2022. Artificial Intelligence and Machine Learning in Drug discovery and Development. Intelligent Medicine, Volume 2(3), pp. 134–140

Passerat, P., Farnan, T., McCoy, M., Harris, J. D., Manion, S. T., Flannery, H. L., Gleim, B.,  2020. Blockchain-Orchestrated Machine Learning for Privacy Preserving Federated Learning in Electronic Health Data. IEEE International Conference on Blockchain, pp. 550–555

Sahoo, M., Singhar, S. S., Sahoo, S. S., 2020. A Blockchain Based Model to Eliminate Drug counterfeiting. Machine Learning and Information Processing, pp. 213–222

Saurabh, S., Dey, K., 2021. Blockchain Technology Adoption, Architecture, And Sustainable Agri-Food Supply Chains. Journal of Cleaner Production, Volume 284, p. 124731

Shah, P., 2021. Post COVID-19 Supply Chain Optimization for the Indian Pharmaceutical Industry using AI Techniques. Intersect: The Stanford Journal of Science, Technology, and Society, Volume 15(1), pp. 1–20

Shahbazi, Z., Byun, Y.C., 2021a. Integration of Blockchain, IoT and Machine Learning for Multistage Quality Control and Enhancing Security in Smart Manufacturing. Sensors, Volume 21(4), p. 1467

Shukla, A., Kalnoor, G., Kumar, A., Yuvaraj, N., Manikandan, R., Ramkumar, M., 2021. Improved Recognition Rate of Different Material Category Using Convolutional Neural Networks. In: Materials Today: Proceedings, Volume 81(2), pp. 947–950

Singh, S.K., Azzaoui, A.E., Kim, T.W., Pan, Y., Park, J.H., 2021. Deepblockscheme: A Deep Learning-Based Blockchain Driven Scheme for Secure Smart City. Human-centric Computing and Information Sciences, Volume 11(12), pp. 1–13

Wong, S., Yeung, J.K.W., Lau, Y.Y., So, J., 2021. Technical Sustainability of Cloud-Based Blockchain Integrated with Machine Learning for Supply Chain Management. Sustainability, Volume 13(15), p. 8270

Yuvaraj, N., Raja, R.A., Kousik, N.V., Johri, P., Diván, M.J., 2020. Analysis On the Prediction of Central Line-Associated Bloodstream Infections (CLABSI) Using Deep Neural Network Classification. In: Computational Intelligence and Its Applications in Healthcare, pp. 229–244