This is Part 7 of a 11-part series based on the research paper “Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior”. Use the table of links below to navigate to the next part. Table of Links Part 1: Abstract & Introduction Part 2: Important Concepts Part 3: System Description Part 4: Agents & Simulation Details Part 5: Experiment Design Part 6: Continual Learning Part 7: Experiment Results Part 8: Market and Agent Responsiveness to External Events Part 9: Conclusion & References Part 10: Additional Simulation Results Part 11: Simulation Configuration 5 Experiment Results 5.1 Statistical Characteristics of Observed Asset Prices We use Google, Apple, and Amazon’s tick level limit order book data on 2012-06-21 from LOBSTER [28]. We are presenting graphs for Group A only due to page limitations. The continual learning Group A is the most interesting one to analyze. Heavy tails and kurtosis decay. We measure leptokurtic behavior using excess kurtosis which should be significantly larger than 0. Decay means that kurtosis decreases as the sampling frequency of returns decreases (seconds, minutes, hours, etc.). Figure 3 shows the Quantile-Quantile plots of simulations for the continual training RL agents (Group A) and ZI agents respectively, compared with real data. The real data, as well as data obtained using RL and ZI agents, all exhibit strong fat tail return and price distributions. The results from the ZI agents simulation show a milder tail. Results from RL agents closely align with the real data. The average kurtosis from the 10 simulations is presented in Table 1. The table shows the kurtosis decays as data is sampled less frequently. Absence of auto-correlations. [1] describes this property as small insignificant values for auto-correlation of the return time series unless within a very short time interval. Figure 3 shows a strong negative correlation in the first lag and that the auto-correlation function decays to 0 for increasing lags. The tighter boxplots in 3 imply the ZI agents generate a very consistent market, while the real data and RL agents exhibit more variability in this analysis. The author of [1] cites the bounce of market orders between bid and ask prices as the reason why the first lag shows stronger negative auto-correlation. Slow decay of auto-correlation for absolute returns. The auto-correlation function of (1-second) absolute returns decays slowly as a function of the time lags, this implies a long-range time dependency of return magnitude. Figure 4 shows auto-correlation of absolute returns from the simulations using RL agents (Figure 4a) and the ones using ZI agents (Figure 4b). The blue lines in both plots indicate the autocorrelation of the absolute returns of the real data. It can be seen that the auto-correlations from all three methods are decreasing slowly with increasing lags. Figure 4b shows again tighter boxes indicating the simulation using ZI agents is more consistent. Comparing both plots, we can see that RL agents simulate a market in which the return magnitude has a larger autocorrelation than the market simulated by ZI agents. Volatility clustering. Cont et al. [1] use autocorrelation of squared return time series to measure the volatility clustering effect. Figure 5 shows decreasing auto-correlations in both types of simulation. This indicates that they all exhibit volatility clustering effects. Similar to the auto-correlations of absolute returns, the auto-correlation in the ZI-agent simulated market shown in Figure 5b is significantly lower than that of the real data, indicating a less pronounced clustering effect in the ZI-agent simulated market than the RL-agent simulated market. To assess realistic behavior, the MM agents should control their inventory and rely on the Profit and Loss (PnL) from spread (providing liquidity) rather than from inventory (long-term investments). Figure 6 shows the average inventory evolution of all MM agents in the three groups over time. The continual training group has the smallest variation in the inventory, while the untrained group has the largest variation. We further separate the MM’s PnL into profit from spread and profit from holding inventory. The MMs in the continual training group profit on average $1.26 million from spread and lose $0.22 million from inventory. The results for MM agents from other groups are similar. Authors:
(1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu);
(2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu);
(3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu);
(4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This is Part 7 of a 11-part series based on the research paper “Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior”. Use the table of links below to navigate to the next part. This is Part 7 of a 11-part series based on the research paper “ Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior ”. Use the table of links below to navigate to the next part. Part 7 Reinforcement Learning In Agent-based Market Simulation: Unveiling Realistic Stylized Facts And Behavior Table of Links Part 1: Abstract & Introduction Part 1: Abstract & Introduction Part 2: Important Concepts Part 2: Important Concepts Part 3: System Description Part 3: System Description Part 4: Agents & Simulation Details Part 4: Agents & Simulation Details Part 5: Experiment Design Part 5: Experiment Design Part 6: Continual Learning Part 6: Continual Learning Part 7: Experiment Results Part 7: Experiment Results Part 8: Market and Agent Responsiveness to External Events Part 8: Market and Agent Responsiveness to External Events Part 9: Conclusion & References Part 9: Conclusion & References Part 10: Additional Simulation Results Part 10: Additional Simulation Results Part 11: Simulation Configuration Part 11: Simulation Configuration 5 Experiment Results 5.1 Statistical Characteristics of Observed Asset Prices We use Google, Apple, and Amazon’s tick level limit order book data on 2012-06-21 from LOBSTER [28]. We are presenting graphs for Group A only due to page limitations. The continual learning Group A is the most interesting one to analyze. Heavy tails and kurtosis decay. We measure leptokurtic behavior using excess kurtosis which should be significantly larger than 0. Decay means that kurtosis decreases as the sampling frequency of returns decreases (seconds, minutes, hours, etc.). Figure 3 shows the Quantile-Quantile plots of simulations for the continual training RL agents (Group A) and ZI agents respectively, compared with real data. The real data, as well as data obtained using RL and ZI agents, all exhibit strong fat tail return and price distributions. The results from the ZI agents simulation show a milder tail. Results from RL agents closely align with the real data. The average kurtosis from the 10 simulations is presented in Table 1. The table shows the kurtosis decays as data is sampled less frequently. Heavy tails and kurtosis decay. Absence of auto-correlations. [1] describes this property as small insignificant values for auto-correlation of the return time series unless within a very short time interval. Figure 3 shows a strong negative correlation in the first lag and that the auto-correlation function decays to 0 for increasing lags. The tighter boxplots in 3 imply the ZI agents generate a very consistent market, while the real data and RL agents exhibit more variability in this analysis. The author of [1] cites the bounce of market orders between bid and ask prices as the reason why the first lag shows stronger negative auto-correlation. Absence of auto-correlations. Slow decay of auto-correlation for absolute returns. The auto-correlation function of (1-second) absolute returns decays slowly as a function of the time lags, this implies a long-range time dependency of return magnitude. Figure 4 shows auto-correlation of absolute returns from the simulations using RL agents (Figure 4a) and the ones using ZI agents (Figure 4b). The blue lines in both plots indicate the autocorrelation of the absolute returns of the real data. It can be seen that the auto-correlations from all three methods are decreasing slowly with increasing lags. Figure 4b shows again tighter boxes indicating the simulation using ZI agents is more consistent. Comparing both plots, we can see that RL agents simulate a market in which the return magnitude has a larger autocorrelation than the market simulated by ZI agents. Slow decay of auto-correlation for absolute returns. Volatility clustering. Cont et al. [1] use autocorrelation of squared return time series to measure the volatility clustering effect. Figure 5 shows decreasing auto-correlations in both types of simulation. This indicates that they all exhibit volatility clustering effects. Similar to the auto-correlations of absolute returns, the auto-correlation in the ZI-agent simulated market shown in Figure 5b is significantly lower than that of the real data, indicating a less pronounced clustering effect in the ZI-agent simulated market than the RL-agent simulated market. Volatility clustering. To assess realistic behavior, the MM agents should control their inventory and rely on the Profit and Loss (PnL) from spread (providing liquidity) rather than from inventory (long-term investments). Figure 6 shows the average inventory evolution of all MM agents in the three groups over time. The continual training group has the smallest variation in the inventory, while the untrained group has the largest variation. We further separate the MM’s PnL into profit from spread and profit from holding inventory. The MMs in the continual training group profit on average $1.26 million from spread and lose $0.22 million from inventory. The results for MM agents from other groups are similar. Authors: (1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu); (2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu); (3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu); (4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). Authors: Authors: (1) Zhiyuan Yao, Stevens Institute of Technology, Hoboken, New Jersey, USA (zyao9@stevens.edu); (2) Zheng Li, Stevens Institute of Technology, Hoboken, New Jersey, USA (zli149@stevens.edu); (3) Matthew Thomas, Stevens Institute of Technology, Hoboken, New Jersey, USA (mthomas3@stevens.edu); (4) Ionut Florescu, Stevens Institute of Technology, Hoboken, New Jersey, USA (ifloresc@stevens.edu). This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

RL Agents Show Superior Realism and Adaptability Over Zero-Intelligence Agents

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

AI That Learns and Unlearns: The Exceptionally Smart EXPLORER

Reinforcement Learning Revolutionizes Market Insights with Adaptive Simulations

Reinforcement Learning Simulation Features Realism and Adaptability

RL Agents Adapt to Flash Sale Events & Imbalanced Limit Order Books (LOBs)

Market-Making & Liquidity-Taking Agents Leverages Independent Policies & Reward-driven Strategies

Continual Learning RL Agents Set New Standard for Realistic Market Simulations

AI That Learns and Unlearns: The Exceptionally Smart EXPLORER

Reinforcement Learning Revolutionizes Market Insights with Adaptive Simulations

Reinforcement Learning Simulation Features Realism and Adaptability

RL Agents Adapt to Flash Sale Events & Imbalanced Limit Order Books (LOBs)

Market-Making & Liquidity-Taking Agents Leverages Independent Policies & Reward-driven Strategies

Continual Learning RL Agents Set New Standard for Realistic Market Simulations

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps