This is Part 5 of a 11-part series based on the research paper “Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior”. Use the table of links below to navigate to the next part. Table of Links Part 1: Abstract & Introduction Part 2: Important Concepts Part 3: System Description Part 4: Agents & Simulation Details Part 5: Experiment Design Part 6: Continual Learning Part 7: Experiment Results Part 8: Market and Agent Responsiveness to External Events Part 9: Conclusion & References Part 10: Additional Simulation Results Part 11: Simulation Configuration 4. Experiment Design In order to provide realistic answers to questions about RL behavior, it is crucial to ensure that the simulation is similar to a real market. We list a set of properties inherent in real-world financial markets. We then analyze the results of the simulations to see whether they exhibit these properties. In general, RL agents assume unchanged dynamics of the system. However, in real-world scenarios when interacting with other traders and in a dynamic market, the system dynamics evolve. Assuming that the system is realistic, we investigate whether training RL agents during the simulation improves their behavior and enhances their adaptability to changing market conditions. The following two sub-sections discuss in detail the design of the experiment to answer these two questions: Can RL agents simulate a realistic market?


Should agent training persist during simulation? 4.1 Desired Market Properties A realistic simulation should look and behave like a real-world market. To analyze and quantify a realistic behavior we examine two aspects: statistical characteristics and market responsiveness. Statistical characteristics include distributional properties such as the price/return distribution, as well as time dynamics such as auto-correlations of the return series, volatility clustering, and more. Market responsiveness measures changes in several variables in response to a large sell order hitting the market. 4.1.1 Statistical Characteristics Prior research using empirical financial time series data found some statistical properties that are shared by a wide range of assets and across various time periods. For instance, stock returns have a distribution that has a sharper peak and fatter tail, compared to a normal distribution. These statistical characteristics are known as stylized facts [1]. We examine whether the results of our simulation show these stylized facts previously evidenced in literature [1, 23]. We focus on the following well-known single-asset stylized facts: Heavy tails and kurtosis decay with increasing ∆t,


Absence of auto-correlations,


Slow decay of auto-correlation in absolute returns,


Volatility clustering. In addition to these variables which are the result of the entire market activity, we examine individual market makers’ performance records and the evolution of their inventory and PnL component. MM agents should have a stable inventory evolution and the absolute value of the inventory should be close to 0. MM agents should make most of their profit from the spread rather than the inventory PnL. 4.1.2 Market Responsiveness A simulated market within a closed environment where trading is driven by traders without any outside information should be stable. However, we are interested in how the simulation will behave when we additional external impact is introduced. We want to see if changes in behavior align with documented literature on real-world cases. In a real-world market, a large liquidation typically causes price impact. Almgren and Chriss [24] assert that price impact manifests in two forms: temporary impact and permanent impact. Therefore, following a substantial sell order, the price is observed to undergo a transient decline, followed by a recovery to a level that remains below the pre-sale price. Additionally, a large directional order flow carries information, such as the potential for substantial liquidation. Market makers can capture the information from their order flows and adjust their strategies accordingly. In practical scenarios, when more orders come from one side than the other side, market makers tend to widen the spread and shift the center of price spread towards the direction which has fewer orders [25, 26, 27]. We examine the behavior of the MM agents to assess whether similar patterns are observed. To do this, we design two experiments with different types of informed traders. The first experiment introduces a flash-sale agent who places large sell orders to the simulated market. The second experiment involves dynamically altering the buy/sell preferences of LT agents throughout the simulation, prompting them to trade in the same direction within short time periods, thereby influencing price movements. Authors:
(1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu);
(2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu);
(3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu);
(4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This is Part 5 of a 11-part series based on the research paper “Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior”. Use the table of links below to navigate to the next part. This is Part 5 of a 11-part series based on the research paper “ Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior ”. Use the table of links below to navigate to the next part. Part 5 Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior Table of Links Part 1: Abstract & Introduction Part 1: Abstract & Introduction Part 2: Important Concepts Part 2: Important Concepts Part 3: System Description Part 3: System Description Part 4: Agents & Simulation Details Part 4: Agents & Simulation Details Part 5: Experiment Design Part 5: Experiment Design Part 6: Continual Learning Part 6: Continual Learning Part 7: Experiment Results Part 7: Experiment Results Part 8: Market and Agent Responsiveness to External Events Part 8: Market and Agent Responsiveness to External Events Part 9: Conclusion & References Part 9: Conclusion & References Part 10: Additional Simulation Results Part 10: Additional Simulation Results Part 11: Simulation Configuration Part 11: Simulation Configuration 4. Experiment Design In order to provide realistic answers to questions about RL behavior, it is crucial to ensure that the simulation is similar to a real market. We list a set of properties inherent in real-world financial markets. We then analyze the results of the simulations to see whether they exhibit these properties. In general, RL agents assume unchanged dynamics of the system. However, in real-world scenarios when interacting with other traders and in a dynamic market, the system dynamics evolve. Assuming that the system is realistic, we investigate whether training RL agents during the simulation improves their behavior and enhances their adaptability to changing market conditions. The following two sub-sections discuss in detail the design of the experiment to answer these two questions: Can RL agents simulate a realistic market? Should agent training persist during simulation? Can RL agents simulate a realistic market? Can RL agents simulate a realistic market? Should agent training persist during simulation? Should agent training persist during simulation? 4.1 Desired Market Properties A realistic simulation should look and behave like a real-world market. To analyze and quantify a realistic behavior we examine two aspects: statistical characteristics and market responsiveness. Statistical characteristics include distributional properties such as the price/return distribution, as well as time dynamics such as auto-correlations of the return series, volatility clustering, and more. Market responsiveness measures changes in several variables in response to a large sell order hitting the market. 4.1.1 Statistical Characteristics 4.1.1 Statistical Characteristics Prior research using empirical financial time series data found some statistical properties that are shared by a wide range of assets and across various time periods. For instance, stock returns have a distribution that has a sharper peak and fatter tail, compared to a normal distribution. These statistical characteristics are known as stylized facts [1]. We examine whether the results of our simulation show these styl ized facts previously evidenced in literature [1, 23]. We focus on the following well-known single-asset stylized facts: ized facts Heavy tails and kurtosis decay with increasing ∆t, Absence of auto-correlations, Slow decay of auto-correlation in absolute returns, Volatility clustering. Heavy tails and kurtosis decay with increasing ∆t, Heavy tails and kurtosis decay with increasing ∆t, Absence of auto-correlations, Absence of auto-correlations, Slow decay of auto-correlation in absolute returns, Slow decay of auto-correlation in absolute returns, Volatility clustering. Volatility clustering. In addition to these variables which are the result of the entire market activity, we examine individual market makers’ performance records and the evolution of their inventory and PnL component. MM agents should have a stable inventory evolution and the absolute value of the inventory should be close to 0. MM agents should make most of their profit from the spread rather than the inventory PnL. 4.1.2 Market Responsiveness 4.1.2 Market Responsiveness A simulated market within a closed environment where trading is driven by traders without any outside information should be stable. However, we are interested in how the simulation will behave when we additional external impact is introduced. We want to see if changes in behavior align with documented literature on real-world cases. In a real-world market, a large liquidation typically causes price impact. Almgren and Chriss [24] assert that price impact manifests in two forms: temporary impact and permanent impact. Therefore, following a substantial sell order, the price is observed to undergo a transient decline, followed by a recovery to a level that remains below the pre-sale price. Additionally, a large directional order flow carries information, such as the potential for substantial liquidation. Market makers can capture the information from their order flows and adjust their strategies accordingly. In practical scenarios, when more orders come from one side than the other side, market makers tend to widen the spread and shift the center of price spread towards the direction which has fewer orders [25, 26, 27]. We examine the behavior of the MM agents to assess whether similar patterns are observed. To do this, we design two experiments with different types of informed traders. The first experiment introduces a flash-sale agent who places large sell orders to the simulated market. The second experiment involves dynamically altering the buy/sell preferences of LT agents throughout the simulation, prompting them to trade in the same direction within short time periods, thereby influencing price movements. Authors: (1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu); (2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu); (3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu); (4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). Authors: Authors: (1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu); (2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu); (3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu); (4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Realistic Market Simulations Are Essential for Testing Reinforcement Learning Behavior

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI That Learns and Unlearns: The Exceptionally Smart EXPLORER

Reinforcement Learning Revolutionizes Market Insights with Adaptive Simulations

Reinforcement Learning Simulation Features Realism and Adaptability

RL Agents Adapt to Flash Sale Events & Imbalanced Limit Order Books (LOBs)

Market-Making & Liquidity-Taking Agents Leverages Independent Policies & Reward-driven Strategies

AI That Learns and Unlearns: The Exceptionally Smart EXPLORER

Reinforcement Learning Revolutionizes Market Insights with Adaptive Simulations

Reinforcement Learning Simulation Features Realism and Adaptability

RL Agents Adapt to Flash Sale Events & Imbalanced Limit Order Books (LOBs)

Market-Making & Liquidity-Taking Agents Leverages Independent Policies & Reward-driven Strategies

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps