Roforco

roforco

Authors:
(1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu);
(2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu);
(3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Table of Links Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.3 Agent Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References ABSTRACT Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely to decide where to trade, at what price and what quantity, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework FinRL as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, full-stack framework, customization, reproducibility and hands-on tutoring Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging workloads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simulates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor’s risk-aversion. Moreover, serving as practitioners’ stepping stones, typical trading tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc. 1 INTRODUCTION Deep reinforcement learning (DRL), that balances exploration (of uncharted territory) and exploitation (of current knowledge), is a promising approach to automate trading in quantitative finance [50][51][47][54][21][13]. DRL algorithms are powerful in solving dynamic decision-making problems by learning through interactions with an unknown environment and offer two major advantages of portfolio scalability and market model independence [6]. In quantitative finance, algorithmic trading is essentially making dynamic decisions, namely, to decide where to trade, at what price and what quantity, in a highly stochastic and complex financial market. Incorporating many financial factors, as shown in Fig. 1, a DRL trading agent builds a multi-factor model to trade automatically, which is difficult for human traders to accomplish [4, 53]. Therefore, DRL has been envisioned to have a competitive edge in quantitative finance. Many existing works have applied DRL in quantitative financial tasks. Both researchers and industry practitioners are actively designing trading strategies fueled by DRL, since deep neural networks are significantly powerful at estimating the expected return of taking a certain action at a state. Moody and Saffell [33] utilized a policy search for stock trading; Deng et al. [9] showed that DRL can obtain more profits than conventional methods. More applications include stock trading [35, 47, 51, 54], futures contracts [54], alternative data (news sentiments) [22, 35], high frequency trading [15], liquidation strategy analysis [3], and hedging [6]. DRL is also being actively explored in the cryptocurrency market, e.g., automated trading, portfolio allocation, and market making. However, designing a DRL trading strategy is not easy. The programming is error-prone with tedious debugging. The development pipeline includes preprocessing market data, building a training environment, managing trading states, and backtesting trading performance. These steps are standard for implementation but yet time consuming especially for beginners. Therefore, there is an urgent demand for an open-source library to help researchers and quantitative traders to overcome the steep learning curve. In this paper, we present a FinRL framework that automatically streamlines the development of trading strategies, so as to help researchers and quantitative traders to iterate their strategies at a high turnover rate. Users specify the configurations, such as picking data APIs and DRL algorithms, and analyze the performance of trading results. To achieve this, FinRL introduces a three-layer framework. At the bottom is an environment layer that simulates financial markets using actual historical data, such as closing price, shares, trading volume, and technical indicators. In the middle is the agent layer that implements fine-tuned DRL algorithms and common reward functions. The agent interacts with the environment through properly defined reward functions on the state space and action space. The top layer includes applications in automated trading, where we demonstrate several use cases, namely stock trading, portfolio allocation, cryptocurrency trading, etc. We provide baseline trading strategies to alleviate debugging workloads. Under the three-layer framework, FinRL is developed with three primary principles: • Full-stack framework. To provide a full-stack DRL framework with finance-oriented optimizations, including market data APIs, data preprocessing, DRL algorithms, and automated backtesting. Users can transparently make use of such a development pipeline. • Customization. To maintain modularity and extensibility in development by including state-of-the-art DRL algorithms and supporting design of new algorithms. The DRL algorithms can be used to construct trading strategies by simple configurations. • Reproducibility and hands-on tutoring. To provide tutorials such as step-by-step Jupyter notebooks and user guide to help users walk through the pipeline and reproduce the use cases. This leads to a unified framework where developers are able to efficiently explore ideas through high-level configurations and specifications, and to customize their own strategies at request. Our contributions are summarized as follows: • FinRL is the first open-source framework that demonstrates the great potential of applying DRL algorithms in quantitative finance. We build an ecosystem around the FinRL framework, which seeds the rapidly growing AI4Finance community. • The application layer provides interfaces for users to customize FinRL to their own trading tasks. Automated backtesting module and performance metrics are provided to help quantitative traders iterate trading strategies at a high turnover rate. Profitable trading strategies are reproducible and hands-on tutorials are provided in a beginner-friendly fashion. Adjusting the trained models to the rapid changing markets is also possible. • The agent layer provides state-of-the-art DRL algorithms that are adapted to finance with fine-tuned hyperparameters. Users can add new DRL algorithms. • The environment layer includes not only a collection of historical data APIs, but also live trading APIs. They are reconfigured into standard OpenAI gym-style environments [5]. Moreover, it incorporates market frictions and allows users to customize the trading time granularity. The remainder of this paper is organized as follows. Section 2 reviews related works. Section 3 presents the FinRL framework. Section 4 demonstrates benchmark trading tasks using FinRL. We conclude this paper in Section 5. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Authors: Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang ( Corresponding Author ), New York University Shanghai (christina.wang@nyu.edu). Corresponding Author Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.2 Application Layer 3.3 Agent Layer 3.3 Agent Layer 3.4 Environment Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References 5 Ecosystem of FinRL and Conclusions, and References ABSTRACT Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely to decide where to trade, at what price and what quantity, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework FinRL as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, full-stack framework, customization, reproducibility and hands-on tutoring Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging workloads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simulates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor’s risk-aversion. Moreover, serving as practitioners’ stepping stones, typical trading tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc. 1 INTRODUCTION Deep reinforcement learning (DRL), that balances exploration (of uncharted territory) and exploitation (of current knowledge), is a promising approach to automate trading in quantitative finance [50][51][47][54][21][13]. DRL algorithms are powerful in solving dynamic decision-making problems by learning through interactions with an unknown environment and offer two major advantages of portfolio scalability and market model independence [6]. In quantitative finance, algorithmic trading is essentially making dynamic decisions, namely, to decide where to trade, at what price and what quantity, in a highly stochastic and complex financial market. Incorporating many financial factors, as shown in Fig. 1, a DRL trading agent builds a multi-factor model to trade automatically, which is difficult for human traders to accomplish [4, 53]. Therefore, DRL has been envisioned to have a competitive edge in quantitative finance. Many existing works have applied DRL in quantitative financial tasks. Both researchers and industry practitioners are actively designing trading strategies fueled by DRL, since deep neural networks are significantly powerful at estimating the expected return of taking a certain action at a state. Moody and Saffell [33] utilized a policy search for stock trading; Deng et al. [9] showed that DRL can obtain more profits than conventional methods. More applications include stock trading [35, 47, 51, 54], futures contracts [54], alternative data (news sentiments) [22, 35], high frequency trading [15], liquidation strategy analysis [3], and hedging [6]. DRL is also being actively explored in the cryptocurrency market, e.g., automated trading, portfolio allocation, and market making. However, designing a DRL trading strategy is not easy. The programming is error-prone with tedious debugging. The development pipeline includes preprocessing market data, building a training environment, managing trading states, and backtesting trading performance. These steps are standard for implementation but yet time consuming especially for beginners. Therefore, there is an urgent demand for an open-source library to help researchers and quantitative traders to overcome the steep learning curve. In this paper, we present a FinRL framework that automatically streamlines the development of trading strategies, so as to help researchers and quantitative traders to iterate their strategies at a high turnover rate. Users specify the configurations, such as picking data APIs and DRL algorithms, and analyze the performance of trading results. To achieve this, FinRL introduces a three-layer framework. At the bottom is an environment layer that simulates financial markets using actual historical data, such as closing price, shares, trading volume, and technical indicators. In the middle is the agent layer that implements fine-tuned DRL algorithms and common reward functions. The agent interacts with the environment through properly defined reward functions on the state space and action space. The top layer includes applications in automated trading, where we demonstrate several use cases, namely stock trading, portfolio allocation, cryptocurrency trading, etc. We provide baseline trading strategies to alleviate debugging workloads. Under the three-layer framework, FinRL is developed with three primary principles: • Full-stack framework. To provide a full-stack DRL framework with finance-oriented optimizations, including market data APIs, data preprocessing, DRL algorithms, and automated backtesting. Users can transparently make use of such a development pipeline. • Full-stack framework. • Customization. To maintain modularity and extensibility in development by including state-of-the-art DRL algorithms and supporting design of new algorithms. The DRL algorithms can be used to construct trading strategies by simple configurations. • Customization. • Reproducibility and hands-on tutoring. To provide tutorials such as step-by-step Jupyter notebooks and user guide to help users walk through the pipeline and reproduce the use cases. • Reproducibility and hands-on tutoring. This leads to a unified framework where developers are able to efficiently explore ideas through high-level configurations and specifications, and to customize their own strategies at request. Our contributions are summarized as follows: • FinRL is the first open-source framework that demonstrates the great potential of applying DRL algorithms in quantitative finance. We build an ecosystem around the FinRL framework, which seeds the rapidly growing AI4Finance community. • The application layer provides interfaces for users to customize FinRL to their own trading tasks. Automated backtesting module and performance metrics are provided to help quantitative traders iterate trading strategies at a high turnover rate. Profitable trading strategies are reproducible and hands-on tutorials are provided in a beginner-friendly fashion. Adjusting the trained models to the rapid changing markets is also possible. • The agent layer provides state-of-the-art DRL algorithms that are adapted to finance with fine-tuned hyperparameters. Users can add new DRL algorithms. • The environment layer includes not only a collection of historical data APIs, but also live trading APIs. They are reconfigured into standard OpenAI gym-style environments [5]. Moreover, it incorporates market frictions and allows users to customize the trading time granularity. The remainder of this paper is organized as follows. Section 2 reviews related works. Section 3 presents the FinRL framework. Section 4 demonstrates benchmark trading tasks using FinRL. We conclude this paper in Section 5. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

পরিমাণগত অর্থায়নে ট্রেডিং স্বয়ংক্রিয় করার জন্য গভীর শক্তিবৃদ্ধি শেখার কাঠামো

Deep Reinforcement Learning Framework zur Automatisierung des Handels im quantitativen Finanzwesen

Marco de aprendizaje por refuerzo profundo para automatizar el comercio en finanzas cuantitativas

Cadre d'apprentissage par renforcement profond pour automatiser le trading en finance quantitative

क्वांटिटेटिव फाइनेंस में ट्रेडिंग को स्वचालित करने के लिए डीप रीइनफोर्समेंट लर्निंग फ्रेमवर्क

定量金融における取引を自動化する深層強化学習フレームワーク

양적 금융 거래를 자동화하기 위한 심층 강화 학습 프레임워크

Estrutura de aprendizagem por reforço profundo para automatizar negociações em finanças quantitativas

Структура глубокого обучения с подкреплением для автоматизации торговли в области количественных финансов

Kantitatif Finansta Ticareti Otomatikleştirmek için Derin Takviyeli Öğrenme Çerçevesi

Khung học tập củng cố sâu để tự động hóa giao dịch trong tài chính định lượng

深度强化学习框架实现量化金融交易自动化

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References ABSTRACT Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to prompt and more capable in complex settings. RLHF at its core is providing a new toolkit to optimize LLMs other than next-token prediction, enabling the integration of qualitative training goals. The attempted match between user preferences and downstream performance, which happens in a learned reward model, results in an optimization landscape where training and evaluation metrics can appear correlated. The apparent correlation can lead to unexpected behaviors and stories of “too much RLHF.” In RLHF, challenges emerge because the following sub-modules are not consistent with each other: the reward model training, the policy model training, and the policy model evaluation. This mismatch results in models that sometimes avoid user requests through false safety flags, are difficult to steer to an intended characteristic, or always answer in a specific style. As chat model evaluation becomes increasingly nuanced, the reliance on a perceived link between reward model score and downstream performance drives the objective mismatch issue. In this paper, we illustrate the cause of this issue, reviewing relevant literature from model-based reinforcement learning, and discuss relevant solutions to encourage further research. By solving objective mismatch in RLHF, the LLMs of the future will be more precisely aligned to user instructions for both safety and helpfulness 1 Introduction Reinforcement learning from human feedback (RLHF) is a powerful tool for integrating qualitative styles and values into large machine learning models (Bai et al., 2022; Christiano et al., 2017; Ouyang et al., 2022). RLHF was popularized with its use to integrated human values into large language models (LLMs) for aligning chat tools (Schulman, Zoph, Kim, & more, 2022). In doing so, RLHF has become an important technique in the process of making models better at responding to user requests, often referred to as instruction-tuned, steerability, chat-models, etc. RLHF methods typically operate in a two step process following the training of a base language model, first they learn a model of human preferences that acts as a reward function, and second they use this model within a reinforcement learning (RL) optimization loop. In the RLHF process, these two steps are often executed independently, with an accurate reward model being trained on human preference data and then the RL optimizer is used to extract maximum information into the chat model. A common challenge of modern LLMs trained with RLHF is difficulties in extracting intended behaviors from the model. Sometimes, the models decline benign requests for safety reasons and other times they need clever prompt tuning to extract full performance. In this paper, we detail a fundamental challenge in modern RLHF learning schemes: the objective mismatch issue. In RLHF, three important parts of training are numerically decoupled: the design of evaluation metrics, the training of a reward model, and the training of the generating model. This mismatch between the reward model and the RL training is visualized in Fig. 1, yet other links exist between the goals of evaluation and simulating human values. Specifically, there are many avenues to better align reward model training to the literature in preference quantification (Lambert, Gilbert, & Zick, 2023) and fundamental optimization challenges need to be solved in RLHF practices (Casper et al., 2023). ChatGPT, the most popular model trained with RLHF, shows signs this limitation through issues such as verbosity, self-doubt and question refusals, repeated phrases, hedging, and more (Schulman, 2023). These traits of overoptimization are results of the subtle proxy objective problem that objective mismatch provides a frame for studying and solving – the reward model attributes excess value to phrases that do not contribute to user benefit, which the RL optimizer exploits, such as safety flags. On the hand, the current training setups are not fully aligned with evaluation tools because the RLHF’d models still need sophisticated prompting techniques such as “thinking step by step” (J. Wei et al., 2022) or “take a deep breath” (Yang et al., 2023) to reach maximum performance. Solving objective mismatch will remove the need for these advanced techniques and reduce the likelihood of out-of-scope refusals from an LLM. The phrase objective mismatch originates from model-based reinforcement learning (MBRL), where an agent iteratively learns a dynamics model that it later uses to solve a control task (Lambert, Amos, Yadan, & Calandra, 2020; R. Wei, Lambert, McDonald, Garcia, & Calandra, 2023). In this context, the mismatch is between learning an accurate dynamics model rather than one that is optimized for high task reward. In RLHF, the problem is related, but with added complexity, as the reward model is optimized for preference data over a closed distribution, which does not match the end users. Second, the task of open-ended language generation is less specific to a notion of reward than that of RL control policies. For these reasons, as we explore in this paper, the objective mismatch issue is more nuanced and critical to RLHF. In this position paper, we make three contributions: • Clearly explain the origins and potential manifestations of objective mismatch in chat-tuned LLMs, • Connect related work from NLP and RL literature around objective mismatch, • Propose directions of study to solve the mismatch and foster better RLHF practices. This paper is under CC 4.0 license. available on arxiv

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

সারিবদ্ধকরণ সিলিং: মানব প্রতিক্রিয়া থেকে শক্তিবৃদ্ধি শিক্ষায় উদ্দেশ্যগত অমিল

Die Ausrichtungsobergrenze: Objektive Diskrepanz beim verstärkenden Lernen aus menschlichem Feedback

El techo de alineación: desajuste objetivo en el aprendizaje por refuerzo a partir de la retroalimentación humana

Le plafond d’alignement : inadéquation des objectifs dans l’apprentissage par renforcement à partir de la rétroaction humaine

संरेखण छत: मानव प्रतिक्रिया से सुदृढीकरण सीखने में उद्देश्य बेमेल

アライメントの天井: 人間のフィードバックからの強化学習における目標の不一致

정렬 천장: 인간 피드백을 통한 강화 학습의 객관적 불일치

O Teto de Alinhamento: Incompatibilidade de Objetivos na Aprendizagem por Reforço a partir do Feedback Humano

Потолок согласования: объективное несоответствие в обучении с подкреплением на основе обратной связи с человеком

Hizalama Tavanı: İnsan Geri Bildiriminden Takviyeli Öğrenimde Hedef Uyuşmazlığı

Mức trần liên kết: Sự không phù hợp khách quan trong việc học tăng cường từ phản hồi của con người

对齐上限：来自人类反馈的强化学习中的目标不匹配

Contextual bandits are a class of reinforcement learning algorithms used in decision-making models where a learner must choose actions that result in the most reward. They are named after the classical "one-armed bandit" problem of gambling, where a player needs to decide which slot machines to play, how many times to play each machine, and in what order to play them. What sets contextual bandits apart is that the decision-making process is informed by the context. The context, in this case, refers to a set of observable variables that can impact the result of the action. This addition makes the bandit problem closer to real-world applications, such as personalized recommendations, clinical trials, or ad placement, where the decision depends on specific circumstances. The Formulation of the Contextual Bandit Problems The contextual bandit problem can be implemented using various algorithms. Below, I provide a general high-level overview of the algorithmic structure for contextual bandit problems: And following algorithm here is the underlying math of the contextual bandit problem: To sum up all above generally speaking the goal of the algorithm is to maximize the cumulative reward over some time horizon. Note that in each round, the reward for only the chosen action is observed. This is known as partial feedback, which differentiates bandit problems from supervised learning. The most common approach for solving a contextual bandit problem involves balancing exploration and exploitation . Exploitation means using the current knowledge to make the best decision, while exploration involves taking less certain actions to gather more information. The trade-off between exploration and exploitation is manifested in various algorithms designed to solve the Contextual Bandit problem, such as LinUCB, Neural Bandit, or Decision Tree Bandit. All of these algorithms employ their unique strategies to address this trade-off. The code representation of this idea for Bandit is: class Bandit:
    def __init__(self, n_actions, n_features):
        self.n_actions = n_actions
        self.n_features = n_features
        self.theta = np.random.randn(n_actions, n_features)

    def get_reward(self, action, x):
        return x @ self.theta[action] + np.random.normal()

    def get_optimal_reward(self, x):
        return np.max(x @ self.theta.T)

# Usage example

# Define the bandit environment
n_actions = 10
n_features = 5
bandit = Bandit(n_actions, n_features)

# Define the model agent
model_agent = model(n_features, *args)

# Define the context (features)
x = np.random.randn(n_features)

# The agent makes a prediction
pred_rewards = np.array([model_agent.predict(x) for _ in range(n_actions)])

# The agent chooses an action
action = np.argmax(pred_rewards)

# The agent gets a reward
reward = bandit.get_reward(action, x)

# The agent updates its parameters
model_agent.update(x, reward) LinUCB Algorithm The LinUCB algorithm is a contextual bandit algorithm that models the expected reward of an action given a context as a linear function, and it selects actions based on the Upper Confidence Bound (UCB) principle to balance exploration and exploitation. It exploits the best option available according to the current model (which is a linear model), but it also explores options that could potentially provide higher rewards, considering the uncertainty in the model's estimates. Mathematically it can be presented as: This approach ensures that the algorithm explores actions for which it has high uncertainty. After selecting an action and receiving a reward, LinUCB updates the parameters as follows: And here is the code implementation in Python: import numpy as np


class LinUCB:
    def __init__(self, n_actions, n_features, alpha=1.0):
        self.n_actions = n_actions
        self.n_features = n_features
        self.alpha = alpha

        # Initialize parameters
        self.A = np.array(
            [np.identity(n_features) for _ in range(n_actions)]
        )  # action covariance matrix
        self.b = np.array(
            [np.zeros(n_features) for _ in range(n_actions)]
        )  # action reward vector
        self.theta = np.array(
            [np.zeros(n_features) for _ in range(n_actions)]
        )  # action parameter vector

    def predict(self, context):
        context = np.array(context)  # Convert list to ndarray
        context = context.reshape(
            -1, 1
        )  # reshape the context to a single-column matrix
        p = np.zeros(self.n_actions)
        for a in range(self.n_actions):
            theta = np.dot(
                np.linalg.inv(self.A[a]), self.b[a]
            )  # theta_a = A_a^-1 * b_a
            theta = theta.reshape(-1, 1)  # Explicitly reshape theta
            p[a] = np.dot(theta.T, context) + self.alpha * np.sqrt(
                np.dot(context.T, np.dot(np.linalg.inv(self.A[a]), context))
            )  # p_t(a|x_t) = theta_a^T * x_t + alpha * sqrt(x_t^T * A_a^-1 * x_t)
        return p

    def update(self, action, context, reward):
        self.A[action] += np.outer(context, context)  # A_a = A_a + x_t * x_t^T
        self.b[action] += reward * context  # b_a = b_a + r_t * x_tx Decision Tree Algorithm Decision Tree Bandit models the reward function as a decision tree, where each leaf node corresponds to an action and each path from the root to a leaf node represents a decision rule based on the context. It performs exploration and exploitation through a statistical framework, making splits and merges in the decision tree based on statistical significance tests. The objective is to learn the best decision tree that maximizes the expected cumulative reward: Decision tree learning usually involves two steps: : Starting with a single node (root), recursively split each node based on a feature and a threshold that maximizes the expected reward. This process continues until some stopping condition is met, such as reaching a maximum depth or minimum number of samples per leaf. Tree Growing : Starting from the fully grown tree, recursively merge the children of each node if it increases or does not decrease the expected reward significantly. This process continues until no further merging can be performed. Tree Pruning The splitting criterion and merging criterion are usually defined based on statistical tests to ensure the statistical significance of the improvement. from sklearn.tree import DecisionTreeRegressor


class DecisionTreeBandit:
    def __init__(self, n_actions, n_features, max_depth=5):
        self.n_actions = n_actions
        self.n_features = n_features
        self.max_depth = max_depth

        # Initialize the decision tree model for each action
        self.models = [
            DecisionTreeRegressor(max_depth=self.max_depth) for _ in range(n_actions)
        ]
        self.data = [[] for _ in range(n_actions)]

    def predict(self, context):
        return np.array(
            [self._predict_for_action(a, context) for a in range(self.n_actions)]
        )

    def _predict_for_action(self, action, context):
        if not self.data[action]:
            return 0.0
        X, y = zip(*self.data[action])
        self.models[action].fit(np.array(X), np.array(y))
        context_np = np.array(context).reshape(
            1, -1
        )  # convert list to NumPy array and reshape
        return self.models[action].predict(context_np)[0]

    def update(self, action, context, reward):
        self.data[action].append((context, reward)) For DecisionTreeBandit, the method uses the current model parameters to predict the expected reward for each action given a context. The method updates the model parameters based on the observed reward from the selected action. This implementation use for DecisionTreeBandit. predict update scikit-learn Contextual Bandits with Neural Networks Deep learning models can be used to approximate the reward function in high-dimensional or non-linear cases. The policy is typically a neural network that takes the context and available actions as input and outputs the probabilities of taking each action. A popular deep learning approach is to use an actor-critic architecture, where one network (the actor) decides which action to take, and the other network (the critic) evaluates the action taken by the actor. For more complex scenarios where the relationship between the context and the reward is not linear, we can use a neural network to model the reward function. One popular method is to use a policy gradient method, such as REINFORCE or actor-critic. To sum up the following Neural Bandit uses neural networks to model the reward function, taking into account the uncertainty in the parameters of the neural network. It further introduces exploration through policy gradients where it updates the policy in the direction of higher rewards. This form of exploration is more directed, which can be beneficial in large action spaces. Simplest code implementation is the following: import torch
import torch.nn as nn
import torch.optim as optim


class NeuralNetwork(nn.Module):
    def __init__(self, n_features):
        super(NeuralNetwork, self).__init__()
        self.layer = nn.Sequential(
            nn.Linear(n_features, 32), nn.ReLU(), nn.Linear(32, 1)
        )

    def forward(self, x):
        return self.layer(x)


class NeuralBandit:
    def __init__(self, n_actions, n_features, learning_rate=0.01):
        self.n_actions = n_actions
        self.n_features = n_features
        self.learning_rate = learning_rate

        # Initialize the neural network model for each action
        self.models = [NeuralNetwork(n_features) for _ in range(n_actions)]
        self.optimizers = [
            optim.Adam(model.parameters(), lr=self.learning_rate)
            for model in self.models
        ]
        self.criterion = nn.MSELoss()

    def predict(self, context):
        context_tensor = torch.tensor(context, dtype=torch.float32)  # Convert to tensor
        with torch.no_grad():
            return torch.cat(
                [model(context_tensor).reshape(1) for model in self.models]
            )

    def update(self, action, context, reward):
        self.optimizers[action].zero_grad()
        context_tensor = torch.tensor(context, dtype=torch.float32)  # Convert to tensor
        reward_tensor = torch.tensor(reward, dtype=torch.float32)  # Convert to tensor
        pred_reward = self.models[action](context_tensor)
        loss = self.criterion(pred_reward, reward_tensor)
        loss.backward()
        self.optimizers[action].step() For NeuralBandit the same picture with method and method that use the current model parameters to predict the expected reward for each action given a context and  updates the model parameters based on the observed reward from the selected action. This implementation use PyTorch for NeuralBandit. predict update Comparative performance of Models Each model comes with its unique advantages and disadvantages, and the choice of model depends on the specific requirements of the problem at hand. Let's delve into the specifics of some of these models and then compare their performances. LinUCB The Linear Upper Confidence Bound (LinUCB) model uses linear regression to estimate the expected reward for each action given a context. It also keeps track of the uncertainty of these estimates and uses it to encourage exploration. Advantages: It is simple and computationally efficient. It provides theoretical guarantees for its regret bound. Disadvantages: It assumes the reward function is a linear function of the context and action, which may not hold for more complex problems. Decision Tree Bandit The Decision Tree Bandit model represents the reward function as a decision tree. Each leaf node corresponds to an action, and each path from the root to a leaf node represents a decision rule based on the context. Advantages: It provides interpretable decision rules. It can handle complex reward functions. Disadvantages: It may suffer from overfitting, especially for large trees. It requires tuning hyperparameters such as the maximum depth of the tree. Neural Bandit The Neural Bandit model uses a neural network to estimate the expected reward for each action given a context. It uses policy gradient methods to encourage exploration. Advantages: It can handle complex, non-linear reward functions. It can perform directed exploration, which is beneficial in large action spaces. Disadvantages: It requires tuning hyperparameters such as the architecture and learning rate of the neural network. It may be computationally expensive, especially for large networks and large action spaces. Code comparison and general running example: Here is Python code to compare the performances of all these model’s bandits mentioned and defined before with running a simulation for each model. import matplotlib.pyplot as plt

# Define the bandit environment
n_actions = 10
n_features = 5
bandit = Bandit(n_actions, n_features)

# Define the agents
linucb_agent = LinUCB(n_actions, n_features, alpha=0.1)
neural_agent = NeuralBandit(n_actions, n_features, learning_rate=0.01)
tree_agent = DecisionTreeBandit(n_actions, n_features, max_depth=5)

# Run the simulation for each agent
n_steps = 1000
agents = [linucb_agent, tree_agent, neural_agent]
cumulative_rewards = {agent.__class__.__name__: np.zeros(n_steps) for agent in agents}
cumulative_regrets = {agent.__class__.__name__: np.zeros(n_steps) for agent in agents}
for agent in agents:
    print(agent)
    for t in range(n_steps):
        x = np.random.randn(n_features)
        pred_rewards = agent.predict([x])
        action = np.argmax(pred_rewards)
        reward = bandit.get_reward(action, x)
        optimal_reward = bandit.get_optimal_reward(x)
        agent.update(action, x, reward)
        cumulative_rewards[agent.__class__.__name__][t] = (
            reward
            if t == 0
            else cumulative_rewards[agent.__class__.__name__][t - 1] + reward
        )
        cumulative_regrets[agent.__class__.__name__][t] = (
            optimal_reward - reward
            if t == 0
            else cumulative_regrets[agent.__class__.__name__][t - 1]
            + optimal_reward
            - reward
        )

# Plot the results
plt.figure(figsize=(12, 6))

plt.subplot(121)
for agent_name, rewards in cumulative_rewards.items():
    plt.plot(rewards, label=agent_name)
plt.xlabel("Steps")
plt.ylabel("Cumulative Rewards")
plt.legend()

plt.subplot(122)
for agent_name, regrets in cumulative_regrets.items():
    plt.plot(regrets, label=agent_name)
plt.xlabel("Steps")
plt.ylabel("Cumulative Regrets")
plt.legend()

plt.show() Application Scenarios : In this scenario, the context could be user information (like age, gender, past purchases) and item information (like item category, price). The actions are the items to recommend, and the reward is whether the user clicked on or bought the item. A contextual bandit algorithm can be used to learn a personalized recommendation policy that maximizes the click-through rate or revenue. Personalized Recommendation : In clinical trials, the context could be patient information (like age, gender, medical history), the actions are the different treatments, and the reward is the health outcome. A contextual bandit algorithm can learn a treatment policy that maximizes the patient's health outcome, balancing the treatment effect and side effects. Clinical Trials : In online advertising, the context could be user information and ad information, the actions are the ads to display, and the reward is whether the user clicked on the ad. A contextual bandit algorithm can learn a policy that maximizes the click-through rate or revenue, considering the ad relevance and user preferences. Online Advertising In each of these applications, it's crucial to balance exploration and exploitation. Exploitation is choosing the best action based on current knowledge, while exploration is trying out different actions to gain more knowledge. Contextual bandit algorithms provide a framework to formalize and solve this exploration-exploitation trade-off. Conclusion Contextual bandits are a powerful tool for decision-making in environments with limited feedback. The ability to leverage context in decision-making allows for more complex and nuanced decisions than traditional bandit algorithms. While we didn't compare performance explicitly, we noted that the choice of algorithm should be influenced by the problem characteristics. For simpler relationships, LinUCB and Decision Trees could excel, while Neural Networks may outperform in complex scenarios.Each method offered unique strengths: LinUCB for its mathematical simplicity, Decision Trees for their interpretability, and Neural Networks for their ability to handle complex relationships. Choosing the right approach is key to effective problem-solving in this exciting domain. Overall, context-based bandit problems present an engaging area within reinforcement learning, with diverse algorithms available to tackle them and we can expect to see even more innovative uses of these powerful models.

Contextual Multi-Armed Bandit Problems in Reinforcement Learning

শক্তিবৃদ্ধি শিক্ষায় প্রাসঙ্গিক বহু-সশস্ত্র দস্যু সমস্যা

Kontextuelle mehrarmige Banditenprobleme beim Reinforcement Learning

Problemas contextuales de bandidos con múltiples brazos en el aprendizaje por refuerzo

Problèmes contextuels de bandit multi-armé dans l'apprentissage par renforcement

सुदृढीकरण सीखने में प्रासंगिक बहु-सशस्त्र दस्यु समस्याएं

強化学習におけるコンテキストマルチアームバンディット問題

강화 학습의 상황별 다중 무장 도적 문제

Problemas contextuais de bandidos multi-armados no aprendizado por reforço

Takviyeli Öğrenmede Bağlamsal Çok Silahlı Haydut Sorunları

Các vấn đề về kẻ cướp nhiều vũ trang theo ngữ cảnh trong học tập tăng cường

强化学习中的情境多臂老虎机问题

Multi-Armed Bandits (MABs) form a cornerstone in the world of Reinforcement Learning, offering a powerful mechanism to negotiate scenarios that necessitate optimal equilibrium between exploration and exploitation. Exploration refers to trying out novel options, while exploitation signifies sticking to the current best option. In such circumstances, MABs often prove to be a stellar choice due to their intelligent learning mechanisms. First, let's unravel the concept behind MABs. Picture a gambler surrounded by an array of slot machines, each known as a "one-armed bandit." The critical decisions to make here involve determining which machines to play, the order of playing, and the frequency of play. The catch is that each machine offers a different, uncertain reward. The ultimate goal is to amass the highest cumulative reward across a sequence of plays. What is a Multi-Armed Bandit? The name "multi-armed bandit" comes from a thought experiment involving a gambler at a row of slot machines (often known as "one-armed bandits"). The gambler needs to decide which machines to play, how many times to play each machine and in what order, with the objective of maximizing their total reward. In a more formal definition, a MAB problem is a tuple (A, R) The goal in the MAB problem is to find a policy (a mapping from historical data to actions) that maximizes the expected total reward over a given number of rounds. π Q The key dilemma in MAB is the trade-off between "exploration" and "exploitation": involves trying different arms to learn more about the reward they might yield. This might involve pulling sub-optimal arms, but it helps to gather information about the system. Exploration involves pulling the arm that the agent currently believes to have the highest expected reward, based on the information it has gathered so far. Exploitation Balancing exploration and exploitation is a core challenge in reinforcement learning and MAB problems. To address this problem, a host of algorithms have been developed, including the Epsilon-Greedy, the Upper Confidence Bound (UCB) and Thompson Sampling strategies. The detailed review of Contextual Bandits as an extension of MAB will be covered in next articles. The Epsilon(ε)-Greedy The Epsilon(ε)-Greedy strategy adheres to a straightforward principle. Most of the time, quantified as (1 - ε), it opts for the action yielding the highest estimated reward, thereby exploiting the best known option. However, the rest of the time, quantified as ε, it chooses a random action, thus exploring new possibilities. The ingenuity of this strategy lies in its simplicity and effectiveness, despite the caveat that it might select less rewarding actions with the same probability as the most rewarding ones during its exploration phase and the value of ε determines the balance between exploration and exploitation. Mathematically it can be defined as: This can be defined by the following pseudocode: Initialize Q(a) for all a
Repeat:
  Generate a random number p between 0 and 1
  If p < epsilon:
    Select a random action a
  Else:
    Select action a with the highest Q(a)
  Execute action a and observe reward r
  Update Q(a) = Q(a) + alpha * (r - Q(a)) The Pythonic implementation of the Epsilon-Greedy strategy is fairly straightforward: import numpy as np

class EpsilonGreedy:
    def __init__(self, epsilon, counts, values):
        self.epsilon = epsilon
        self.counts = counts
        self.values = values

    def select_arm(self):
        if np.random.random() < self.epsilon:
            return np.random.choice(len(self.values))
        else:
            return np.argmax(self.values)

    def update(self, chosen_arm, reward):
        self.counts[chosen_arm] += 1
        n = self.counts[chosen_arm]
        value = self.values[chosen_arm]
        new_value = ((n-1) / float(n)) * value + (1 / float(n)) * reward
        self.values[chosen_arm] = new_value The ε-greedy algorithm has the benefit of simplicity and the guarantee that it will continue to explore (a bit) even after many steps. However, it does not take into account how much is known about each action when it explores, unlike UCB or Thompson Sampling. UCB Conversely, the UCB strategy factors in both the estimated reward and the associated uncertainty or variance when deciding on an action. It exhibits a preference for actions with high uncertainty, seeking to diminish it and ensuring a more productive exploration phase. The UCB strategy is mathematically defined as follows: Its Python implementation is as follows: class UCB:
    def __init__(self, counts, values):
        self.counts = counts
        self.values = values

    def select_arm(self):
        n_arms = len(self.counts)
        for arm in range(n_arms):
            if self.counts[arm] == 0:
                return arm
        ucb_values = [0.0 for arm in range(n_arms)]
        total_counts = sum(self.counts)
        for arm in range(n_arms):
            bonus = sqrt((2 * log(total_counts)) / float(self.counts[arm]))
            ucb_values[arm] = self.values[arm] + bonus
        return np.argmax(ucb_values)

    def update(self, chosen_arm, reward):
        self.counts[chosen_arm] += 1
        n = self.counts[chosen_arm]
        value = self.values[chosen_arm]
        new_value = ((n-1) / float(n)) * value + (1 / float(n)) * reward
        self.values[chosen_arm] = new_value The balance between exploration and exploitation in UCB comes from its formula: the estimated value of an action plus a term that decreases over time (as more about the action is learned) but increases with the uncertainty about that action. Thus, the algorithm tends to explore arms with high uncertainty and high potential reward. Thompson Sampling Thompson Sampling is a Bayesian-inspired algorithm for the multi-armed bandit problem. It assigns a prior distribution to the reward probabilities of each bandit (or each action), then updates these priors as rewards are observed. The action selection is probabilistic, according to the posterior distribution over the best action. In a Beta-Bernoulli framework, we consider each bandit's reward distribution as a Bernoulli distribution (i.e., binary rewards 0 or 1). We then assign a Beta prior to the probability of getting a reward. The Beta distribution is the conjugate prior of the Bernoulli distribution, which allows for an easy posterior update. Logic: Assign each bandit a Beta distribution with parameters α=1, β=1 (Uniform prior). For each round: Sample a random number from each bandit's current Beta distribution. Select the bandit with the highest sampled number and pull that bandit. Observe the reward from the pulled bandit. If it's a success (1), increase the bandit's α by one; if it's a failure (0), increase the bandit's β by one. Repeat steps 2 and 3. import numpy as np
from scipy.stats import beta

class Bandit:
    def __init__(self, true_probability):
        self.true_probability = true_probability
        self.alpha = 1
        self.beta = 1

    def pull(self):
        return np.random.random() < self.true_probability

    def sample(self):
        return np.random.beta(self.alpha, self.beta)

    def update(self, reward):
        self.alpha += reward
        self.beta += (1 - reward)

def Thompson(bandits, num_trials):
    rewards = np.zeros(num_trials)
    for i in range(num_trials):
        # Thompson sampling
        j = np.argmax([b.sample() for b in bandits])

        # Pull the arm for the bandit with the largest sample
        reward = bandits[j].pull()

        # Update rewards log
        rewards[i] = reward

        # Update the distribution for the bandit whose arm we just pulled
        bandits[j].update(reward)
    return rewards

# Suppose we have 3 bandits with these true probabilities
true_probabilities = [0.2, 0.5, 0.75]
bandits = [Bandit(p) for p in true_probabilities]

# Run experiment
rewards = Thompson(bandits, num_trials=10000)

# Print the total reward
print("Total reward earned:", rewards.sum())
print("Overall win rate:", rewards.sum() / len(rewards)) Thompson Sampling thus "explores" actions that it is uncertain about (those for which the distribution of values is spread out) and "exploits" actions that it believes may have high value (those for which the distribution is skewed towards high values). Over time, as more is learned about each action, the distributions become more peaked and the actions chosen by the algorithm tend to converge on the one with the highest expected value. The exploration/exploitation balance in Thompson Sampling comes naturally from the shape of the distributions. This method is quite effective, but can be more complex to implement than UCB or ε-greedy, particularly for problems with large or continuous action spaces, or complex reward structures. Why are Multi-Armed Bandits the Best RL for Your Task? : MAB algorithms are simpler and more computationally efficient than full-fledged RL algorithms, which require maintaining and updating a potentially large state-action value table or approximator. Simplicity : MAB algorithms provide robust methods for managing the trade-off between trying new actions and sticking with known good actions. Balance of exploration and exploitation : MAB algorithms can adapt in real-time to changes in the reward distributions of the actions. Real-time adaptability : The simplicity and efficiency of MABs allow for easy integration into existing systems, providing immediate benefits with minimal disruption. Easy integration : MABs have been successfully applied in diverse fields, including advertising (choosing which ad to show to maximize click-through rate), healthcare (personalizing treatment strategies), and web page optimization (A/B testing). Broad applicability Applications of MABs Multi-Armed Bandits (MABs) have a wide range of applications across various industries and domains. Here are some examples of how they can be used: : MABs can be used to dynamically adjust the selection of advertisements to display to users based on their interactions. This helps maximize click-through rates or conversions over time. Online Advertising : In medical research, MAB algorithms can be used to dynamically assign patients to different treatments. This ensures that more patients receive the most effective treatments, thus reducing regret, i.e., the loss incurred due to not always choosing the best treatment. Clinical Trials : News websites can use MABs to personalize the articles shown to each user. The MAB algorithm can learn over time which types of articles each user is interested in, and adjust the recommendations accordingly. News Article Recommendation : E-commerce platforms or airlines can use MAB algorithms to optimize their pricing strategies in real time, maximizing revenue based on customer behavior and market dynamics. Dynamic Pricing : In computer networking, MAB algorithms can be used to manage congestion and optimize the routing of packets. Each route can be treated as an arm, and the algorithm can dynamically select routes to minimize packet loss or latency. Network Routing : MABs can also be used to optimize the selection of hyperparameters in machine learning models. Each set of hyperparameters can be treated as an arm, and the algorithm can iteratively refine the selection to find the optimal model configuration. Machine Learning Hyperparameter Tuning In essence, the utility of MABs extends far beyond conventional reinforcement learning tasks. They symbolize an effective framework for enhancing decision-making processes in environments of uncertainty, providing practical solutions to real-world problems across various domains. Therefore, when the task at hand involves balancing exploration and exploitation, MABs often emerge as the go-to option, providing a versatile, robust, and adaptive solution to decision-making conundrums. Conclusion Multi-Armed Bandits, with their ability to balance exploration and exploitation effectively, provide a robust solution to many real-world decision-making problems. Their inherent adaptability and versatility make them a valuable tool, not just within the realm of reinforcement learning, but also across a broad range of applications, from healthcare to online advertising. Whether you're a data scientist, a machine learning enthusiast, or a professional looking to enhance your decision-making processes, understanding and implementing MAB strategies can be an enriching and rewarding experience.

Multi-Armed Bandits: The Best Reinforcement Learning Solution for Your Task

মাল্টি-আর্মড দস্যু: আপনার টাস্কের জন্য সেরা শক্তিবৃদ্ধি শেখার সমাধান

Mehrarmige Banditen: Die beste Lernlösung zur Verstärkung für Ihre Aufgabe

Bandidos con múltiples brazos: la mejor solución de aprendizaje por refuerzo para su tarea

Bandits à plusieurs bras : la meilleure solution d'apprentissage par renforcement pour votre tâche

बहु-सशस्त्र डाकू: आपके कार्य के लिए सर्वोत्तम सुदृढीकरण सीखने का समाधान

Multi-Armed Bandits: タスクに最適な強化学習ソリューション

Multi-Armed Bandits: 작업을 위한 최고의 강화 학습 솔루션

Bandidos multi-armados: a melhor solução de aprendizado por reforço para sua tarefa

Çok Silahlı Haydutlar: Göreviniz için En İyi Takviyeli Öğrenme Çözümü

Kẻ cướp nhiều vũ trang: Giải pháp học tăng cường tốt nhất cho nhiệm vụ của bạn

多臂老虎机：适合您任务的最佳强化学习解决方案

Stories, News & Mentions about ROFORCO

Roforco’s Stories on HackerNoon