Smarter demand forecasting models at scale start here

PredictHQ

predicthq

Companies are waking up to the fact that remote work isn’t a perk to reserve for a tithe of your workforce. They must embrace it wherever and whenever possible — for the foreseeable future. One way that this is happening outside of the traditional office space is with virtual live events. Due to the fast-tracked nature of the shift online, many companies are turning to API-based solutions to cut down on development time. Here are some of the best ways to enhance your virtual live events with APIs that can leave your attendees with a positive experience every time. 1. Manage Attendance A successful live event always starts with attendance. By running a live event, you’re choosing to restrict yourself to a specific time slot when everything will happen in real-time. This requires the orchestration of dozens, hundreds, and at times even thousands of individuals. You must make them aware of the event, let them sign up, and then give them thorough instructions for how to attend. Selling tickets for a live event is something that in-house virtual live event software often cannot address in a satisfactory manner — which is why many turn to as an external API solution. Eventbrite Eventbrite gives an organization everything it needs to get people “in the virtual doors.” It facilitates the creation of event pages and the purchase of tickets. It also is very easy to embed on a company website or in an email campaign. The API was already a popular option for countless events before the pandemic started. This head start has allowed Eventbrite to deliver a sophisticated, user-friendly API that makes it remarkably easy to sell tickets for your event. 2. Promote Using Social Media Having a good ticket platform creates the “doors” for your event. But you still need people to find that virtual entrance if they’re going to enter when the time comes. That’s why lining up a good event marketing API is also a good idea. Facebook is a common tool used throughout the promotional process. However, most businesses go about much of their social media marketing activities manually. They create pages, write posts, and push marketing content often in a context that is completely disconnected from an event itself. When it comes to events, though, the pioneer in the Meta-verse space has created their to streamline much of the promotional process. This can integrate with your existing virtual live event software, making it much easier to create event pages on Facebook. Official Events API From there, the API streamlines the large-scale distribution and promotion of your event page. This can make it much easier to market the upcoming occasion to a massive audience without pouring endless resources or strategic effort into the ordeal. 3. Level Up Audio The last thing you want is to attract a huge number of attendees through a killer promotional campaign, only to have technical issues once the live event begins. Even if the actual functionality of your software goes off without a hitch, if the quality of the presentation is sub-par, it presents problems. This can undermine the ability of your company to have virtual events that are genuinely as effective as their in-person counterparts. One way to ensure that your presentation quality remains high is by using the io APIs — including its spatial audio with a new manual placement feature. Dolby. The API doesn’t just deliver the crisp, clear audio that earned Dolby.io its reputation. It also utilizes cutting-edge tech to position speaker and attendee audio throughout a virtual event. Users can manually place sound objects throughout a three-dimensional space to impact how they hear the event. This makes it easier to connect visuals with audio, creating a smoother, more life-like feel to the online interaction. 4. Maximize Event Data Live events are a tricky business. It’s difficult to know who will attend, if people will stay for the entire event, and if external factors will impact initial estimations. That’s where an API tool like can be a lifesaver. PredictHQ PredictHQ specializes in compiling and organizing data that can impact demand forecasting. This can be for an endless variety of different kinds of events — including virtual conferences. If you’ve found that your events tend to fluctuate in attendance, PredictHQ can help you identify the reasons behind the fluctuations. This can enable a company to take steps to maximize the effectiveness of each event and properly forecast the demand of upcoming events, as well. Hosting live virtual events is the way of the future. However, that doesn’t mean simply having an event online will equate to success. Every live event needs a support network that can smooth out any wrinkles and ensure that the experience is accessible, positive, and user-friendly. So, review the list above and consider where your own live online events are lacking. Then take steps to nail down the APIs that can help you take your next event from good to great.

4 APIs That Will Enhance Your Virtual Live Event

Stock Market Prediction - the Holy Grail of Time Series Data

AI: Sleep Computational Neuroscience, Dreams, Loneliness, and Predictive Coding

How to Build an AI Prediction Smart Contract Using GenLayer 

GenLayer ব্যবহার করে কীভাবে একটি AI পূর্বাভাস স্মার্ট চুক্তি তৈরি করবেন 

So erstellen Sie mit GenLayer einen Smart Contract zur KI-Vorhersage

Cómo crear un contrato inteligente de predicción de IA utilizando GenLayer 

Comment créer un contrat intelligent de prédiction d'IA à l'aide de GenLayer 

GenLayer का उपयोग करके AI पूर्वानुमान स्मार्ट अनुबंध कैसे बनाएं

GenLayer を使用して AI 予測スマート コントラクトを構築する方法

GenLayer를 사용하여 AI 예측 스마트 계약을 구축하는 방법 

Como construir um contrato inteligente de previsão de IA usando GenLayer 

Как создать смарт-контракт для прогнозирования ИИ с помощью GenLayer 

GenLayer Kullanarak Yapay Zeka Tahmini Akıllı Sözleşmesi Nasıl Oluşturulur 

Cách xây dựng hợp đồng thông minh dự đoán AI bằng GenLayer 

如何使用 GenLayer 构建 AI 预测智能合约

"AI can enhance data collection, analysis, prediction, and evaluation processes"

"এআই ডেটা সংগ্রহ, বিশ্লেষণ, ভবিষ্যদ্বাণী এবং মূল্যায়ন প্রক্রিয়াগুলিকে উন্নত করতে পারে"

„KI kann die Prozesse der Datenerfassung, -analyse, -vorhersage und -auswertung verbessern“

"La IA puede mejorar los procesos de recopilación, análisis, predicción y evaluación de datos"

"L'IA peut améliorer les processus de collecte, d'analyse, de prédiction et d'évaluation de données"

"एआई डेटा संग्रह, विश्लेषण, भविष्यवाणी और मूल्यांकन प्रक्रियाओं को बढ़ा सकता है"

「AIはデータの収集、分析、予測、評価のプロセスを強化できます」

"AI는 데이터 수집, 분석, 예측, 평가 프로세스를 향상시킬 수 있습니다"

"A IA pode aprimorar os processos de coleta, análise, previsão e avaliação de dados"

«ИИ может улучшить процессы сбора, анализа, прогнозирования и оценки данных»

"Yapay zeka veri toplama, analiz, tahmin ve değerlendirme süreçlerini geliştirebilir"

"AI có thể tăng cường quá trình thu thập, phân tích, dự đoán và đánh giá dữ liệu"

“人工智能可以增强数据收集、分析、预测和评估过程”

Note: Hello, I am V. Before you read on I just want to say that I understand the magnitude of my ambition and am not afraid to reach higher. I may not know everything, but that won’t stop me. All feedback is accepted. The more there is to fix, the better the final product will be. MANA - A Community Directed Decentralized Price Prediction Market Abstract The concept of cryptocurrency has been in existence for around 40 years and has now become a significant asset for investors to consider. Cryptocurrencies enable tradable exchange without the involvement of a third party, leading many individuals to pursue cryptocurrency trading as a career. With tradable cryptocurrencies like Bitcoin, Ethereum, Solana, and more, it's increasingly possible for people with limited experience in cryptocurrency and investing to quickly acquire knowledge and skills in this field. This often leads to a deep interest in the financial markets. MANA is positioned as a platform for profitable trading and a community-driven marketplace. While other organizations have introduced features enabling users to make predictions and interact, MANA aims to welcome users of all experience levels to test their expertise. It's a place where users can vote on the price movements of cryptocurrencies. Project Overview MANA is a decentralized platform leveraging blockchain technology to enable community-driven price predictions for various assets. Utilizing the MANA Token (MNAT), participants can vote on future price movements every two hours, stake their tokens, and earn rewards for accurate predictions. The MANA platform aims to harness collective intelligence for market analysis while providing all participants with a fair, transparent, and engaging experience. Price prediction is a valuable tool for traders and investors. Traditional methods often rely on expert analysis or algorithmic predictions, which can be biased or lack transparency, but the markets are run by people. If an entity sold a large amount Bitcoin the price of bitcoin would go down due to supply and demand. MANA offers a decentralized alternative, where community members contribute their insights to predict price movements, creating a collective intelligence that enhances market understanding. Voting Mechanism & Staking Community members must submit their predictions on where the price of a specific asset will be every two hours. This continuous voting cycle ensures up-to-date and relevant predictions that reflect the latest market sentiments. To participate in voting, members must stake MNAT tokens. Staking serves as a commitment to their prediction and helps prevent spam or malicious activities. The amount staked reflects the voter's confidence in their prediction. By staking tokens, users contribute to a larger pool of staked tokens that are then distributed amongst voters with the most accuracy. To maximize potential gains, members will need to stake more tokens. In other words, the more you put in, the more you'll get out. Data Collection The smart contract records all votes and the corresponding staked amounts, ensuring complete transparency and immutability of the voting process. This data is publicly accessible and verifiable by anyone. While the voting data is public, the identity of the voters remains pseudonymous, linked only to their wallet addresses, to maintain the utmost security and privacy possible. Outcome Verification & Reward Distribution The voting period will take place once every two hours, during that time users can vote on where the price will be by the end of the voting period. Although to prevent potential security flaws voting will pause 10 minutes before the next interval. At the end of each voting period an oracle will retrieve the actual price data from a reliable source. The smart contract will then verify the outcome by comparing the actual price change to the community's predictions. Participants who make the closest predictions about price movements will receive rewards from the pool of staked tokens. The rewards are distributed proportionally based on the amount staked, which encourages accurate predictions and active participation. Those who make the furthest incorrect predictions forfeit their staked tokens, which are then redistributed among the correct predictors. Security & Fairness The smart contract ensures that the voting process is secure and tamper-proof. Utilizing a reliable oracle minimizes the risk of manipulation, maintaining the integrity of the prediction outcomes. Community Engagement My vision for MANA is promoting active community engagement by enabling members to actively participate in market predictions. This initiative not only provides invaluable insights into market trends, but also fosters a gamified experience that is both educational and rewarding.  I want to allow users to get people stake as much money as they want so long as they can afford it, essentially balancing balances user autonomy with their own discretion. Use Cases: Trading Signals: Traders can confidently utilize the collective predictions as signals to inform and enhance their trading strategies.
Market Analysis: Analysts and researchers can rely on the aggregated data for in-depth market sentiment analysis.
Gamified Learning: Trading novices can confidently immerse themselves in learning about market dynamics in an engaging, low-risk environment.
Competitive High Risk Earning: Traders with the most confidence can stake MNAT tokens for potentially higher gains allowing users to mix their earning with their day trading tasks. Technical Details Smart Contract & Oracle Integration The MANA smart contract is built on the secure and stable Ethereum network, ensuring efficient and reliable execution of voting, staking, and reward distribution. A robust and decentralized oracle system fetches accurate price data, ensuring the integrity and reliability of the outcome verification process. Tokenomics Token Name: Mana Token Token Symbol: MNAT Total Supply: 333,000,000 MNAT MNAT tokens are used for staking, voting, and reward distribution within the MANA platform. Please take note of the following planned token allocation (ACTIVELY SUBJECT TO CHANGE): Founder's Allocation: 15% (49.95M MNAT)
Development Fund: 15% (49.95M MNAT)
Marketing and Partnerships: 10% (33.3M MNAT)
Community and Ecosystem Incentives: 20% (66.6M MNAT)
Liquidity Pool: 35% (116.M MNAT)
Reserve Fund: 5% (16.65M MNAT) Roadmap Q3 2024: Concept Development and White Paper Release
Q4 2024: Smart Contract Development and Testing
Q1 2025: Oracle Integration and Platform Beta Launch
Q2 2025: Full Platform Launch and Community Building
Q3 2025: Expansion to Additional Assets and Enhanced Features Conclusion MANA represents an approach to decentralized price prediction, leveraging the power of community intelligence and blockchain technology. By allowing participants to stake MNAT tokens and vote on price movements, MANA creates a transparent, fair, and engaging platform for market predictions. Join us in revolutionizing the way we understand and engage with financial markets. Lets see what the effect of the platform (if grown large enough) has on the greater crypto market. Thanks for reading, V. Link to Git Repo (Don’t bother reading the README): MANA - Global Participation in the face of uncertainty Note: Hello, I am V. Before you read on I just want to say that I understand the magnitude of my ambition and am not afraid to reach higher. I may not know everything, but that won’t stop me. All feedback is accepted. The more there is to fix, the better the final product will be. Note: Hello, I am V. Before you read on I just want to say that I understand the magnitude of my ambition and am not afraid to reach higher. I may not know everything, but that won’t stop me. All feedback is accepted. The more there is to fix, the better the final product will be. MANA - A Community Directed Decentralized Price Prediction Market Abstract The concept of cryptocurrency has been in existence for around 40 years and has now become a significant asset for investors to consider. Cryptocurrencies enable tradable exchange without the involvement of a third party, leading many individuals to pursue cryptocurrency trading as a career. With tradable cryptocurrencies like Bitcoin, Ethereum, Solana, and more, it's increasingly possible for people with limited experience in cryptocurrency and investing to quickly acquire knowledge and skills in this field. This often leads to a deep interest in the financial markets. MANA is positioned as a platform for profitable trading and a community-driven marketplace. While other organizations have introduced features enabling users to make predictions and interact, MANA aims to welcome users of all experience levels to test their expertise. It's a place where users can vote on the price movements of cryptocurrencies. Project Overview MANA is a decentralized platform leveraging blockchain technology to enable community-driven price predictions for various assets. Utilizing the MANA Token (MNAT), participants can vote on future price movements every two hours, stake their tokens, and earn rewards for accurate predictions. The MANA platform aims to harness collective intelligence for market analysis while providing all participants with a fair, transparent, and engaging experience. Price prediction is a valuable tool for traders and investors. Traditional methods often rely on expert analysis or algorithmic predictions, which can be biased or lack transparency, but the markets are run by people. If an entity sold a large amount Bitcoin the price of bitcoin would go down due to supply and demand. MANA offers a decentralized alternative, where community members contribute their insights to predict price movements, creating a collective intelligence that enhances market understanding. Voting Mechanism & Staking Community members must submit their predictions on where the price of a specific asset will be every two hours. This continuous voting cycle ensures up-to-date and relevant predictions that reflect the latest market sentiments. To participate in voting, members must stake MNAT tokens. Staking serves as a commitment to their prediction and helps prevent spam or malicious activities. The amount staked reflects the voter's confidence in their prediction. By staking tokens, users contribute to a larger pool of staked tokens that are then distributed amongst voters with the most accuracy. To maximize potential gains, members will need to stake more tokens. In other words, the more you put in, the more you'll get out. Data Collection The smart contract records all votes and the corresponding staked amounts, ensuring complete transparency and immutability of the voting process. This data is publicly accessible and verifiable by anyone. While the voting data is public, the identity of the voters remains pseudonymous, linked only to their wallet addresses, to maintain the utmost security and privacy possible. Outcome Verification & Reward Distribution The voting period will take place once every two hours, during that time users can vote on where the price will be by the end of the voting period. Although to prevent potential security flaws voting will pause 10 minutes before the next interval. At the end of each voting period an oracle will retrieve the actual price data from a reliable source. The smart contract will then verify the outcome by comparing the actual price change to the community's predictions. Participants who make the closest predictions about price movements will receive rewards from the pool of staked tokens. The rewards are distributed proportionally based on the amount staked, which encourages accurate predictions and active participation. Those who make the furthest incorrect predictions forfeit their staked tokens, which are then redistributed among the correct predictors. Security & Fairness The smart contract ensures that the voting process is secure and tamper-proof. Utilizing a reliable oracle minimizes the risk of manipulation, maintaining the integrity of the prediction outcomes. Community Engagement My vision for MANA is promoting active community engagement by enabling members to actively participate in market predictions. This initiative not only provides invaluable insights into market trends, but also fosters a gamified experience that is both educational and rewarding.  I want to allow users to get people stake as much money as they want so long as they can afford it, essentially balancing balances user autonomy with their own discretion. Use Cases: Trading Signals: Traders can confidently utilize the collective predictions as signals to inform and enhance their trading strategies. Market Analysis: Analysts and researchers can rely on the aggregated data for in-depth market sentiment analysis. Gamified Learning: Trading novices can confidently immerse themselves in learning about market dynamics in an engaging, low-risk environment. Competitive High Risk Earning: Traders with the most confidence can stake MNAT tokens for potentially higher gains allowing users to mix their earning with their day trading tasks. Trading Signals: Traders can confidently utilize the collective predictions as signals to inform and enhance their trading strategies. Market Analysis: Analysts and researchers can rely on the aggregated data for in-depth market sentiment analysis. Gamified Learning: Trading novices can confidently immerse themselves in learning about market dynamics in an engaging, low-risk environment. Competitive High Risk Earning: Traders with the most confidence can stake MNAT tokens for potentially higher gains allowing users to mix their earning with their day trading tasks. Technical Details Technical Details Smart Contract & Oracle Integration The MANA smart contract is built on the secure and stable Ethereum network, ensuring efficient and reliable execution of voting, staking, and reward distribution. A robust and decentralized oracle system fetches accurate price data, ensuring the integrity and reliability of the outcome verification process. Tokenomics Token Name: Mana Token Token Symbol: MNAT Total Supply: 333,000,000 MNAT MNAT tokens are used for staking, voting, and reward distribution within the MANA platform. Please take note of the following planned token allocation (ACTIVELY SUBJECT TO CHANGE): Founder's Allocation: 15% (49.95M MNAT) Development Fund: 15% (49.95M MNAT) Marketing and Partnerships: 10% (33.3M MNAT) Community and Ecosystem Incentives: 20% (66.6M MNAT) Liquidity Pool: 35% (116.M MNAT) Reserve Fund: 5% (16.65M MNAT) Founder's Allocation: 15% (49.95M MNAT) Development Fund: 15% (49.95M MNAT) Marketing and Partnerships: 10% (33.3M MNAT) Community and Ecosystem Incentives: 20% (66.6M MNAT) Liquidity Pool: 35% (116.M MNAT) Reserve Fund: 5% (16.65M MNAT) Roadmap Q3 2024: Concept Development and White Paper Release Q4 2024: Smart Contract Development and Testing Q1 2025: Oracle Integration and Platform Beta Launch Q2 2025: Full Platform Launch and Community Building Q3 2025: Expansion to Additional Assets and Enhanced Features Q3 2024: Concept Development and White Paper Release Q4 2024: Smart Contract Development and Testing Q1 2025: Oracle Integration and Platform Beta Launch Q2 2025: Full Platform Launch and Community Building Q3 2025: Expansion to Additional Assets and Enhanced Features Conclusion MANA represents an approach to decentralized price prediction, leveraging the power of community intelligence and blockchain technology. By allowing participants to stake MNAT tokens and vote on price movements, MANA creates a transparent, fair, and engaging platform for market predictions. Join us in revolutionizing the way we understand and engage with financial markets. Lets see what the effect of the platform (if grown large enough) has on the greater crypto market. Thanks for reading, V. Link to Git Repo (Don’t bother reading the README): MANA - Global Participation in the face of uncertainty (Don’t bother reading the README): MANA - Global Participation in the face of uncertainty

Proposal for a Collective Price Prediction Protocol

একটি সমষ্টিগত মূল্য পূর্বাভাস প্রোটোকল জন্য প্রস্তাব

Vorschlag für ein kollektives Preisvorhersageprotokoll

Propuesta de protocolo colectivo de predicción de precios

Proposition d’un protocole collectif de prévision des prix

सामूहिक मूल्य पूर्वानुमान प्रोटोकॉल का प्रस्ताव

集合的価格予測プロトコルの提案

집단 가격 예측 프로토콜에 대한 제안

Proposta de Protocolo Coletivo de Previsão de Preços

Предложение по протоколу коллективного прогнозирования цен

Toplu Fiyat Tahmin Protokolü Önerisi

Đề xuất về Giao thức dự đoán giá tập thể

集体价格预测协议提案

Authors:
(1) Yuan Wang, University of Rochester (e-mail: yuan.wang1@simon.rochester.edu);
(2) Yangxin Fan, University of Rochester (e-mail: yfan24@ur.rochester.edu). Table of Links Abstract and Introduction Related work Methodology Media reporting analysis WP fatal police shooting dataset insight Fatal police shooting rate and victims race prediction Conclusion and References Abstract We believe that “all men are created equal” [11]. With the rise of the police shootings reported by media, more people in the U.S. think that police use excessive force during law enforcement, especially to a specific group of people. We want to add our two cents point of view by multidimensional analysis to reveal more facts than the monotone mainstream media. The more facets we understand the problem, the better solution that the whole society could approach. 1. Introduction Our report has three parts: First, we analyzed and quantified fatal police shooting news reporting deviation of mainstream media. Second, we used FP-growth to mine frequent patterns, clustered hotspots of fatal police shootings, and brought multi-attributes (social economics, demographics, political tendency, education, gun ownership rate, police training hours, etc.) to reveal connections under the iceberg. Third, we built regression models based on correlation analysis for numeric variables selection to predict police shooting rates at the state level. We also built classification models based on Chi-square testing for categorical variables selection to predict the victims’ race of fatal police shootings. The main datasets we choose for our analysis include: 1. Washington Post Fatal Police Shooting Dataset (WP data) [13] covers fatal police shooting from 01-01-2015 to 12- 02-2020. 2. KilledByPolice (KBP): Fatal police shooting reported in KilledByPolice website [6] from 01-01-2015 to 11-04-2020. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yuan Wang, University of Rochester (e-mail: yuan.wang1@simon.rochester.edu); (2) Yangxin Fan, University of Rochester (e-mail: yfan24@ur.rochester.edu). Authors: Authors: (1) Yuan Wang, University of Rochester (e-mail: yuan.wang1@simon.rochester.edu); (2) Yangxin Fan, University of Rochester (e-mail: yfan24@ur.rochester.edu). Table of Links Abstract and Introduction Abstract and Introduction Related work Related work Methodology Methodology Media reporting analysis Media reporting analysis WP fatal police shooting dataset insight WP fatal police shooting dataset insight Fatal police shooting rate and victims race prediction Fatal police shooting rate and victims race prediction Conclusion and References Conclusion and References Abstract We believe that “all men are created equal” [11]. With the rise of the police shootings reported by media, more people in the U.S. think that police use excessive force during law enforcement, especially to a specific group of people. We want to add our two cents point of view by multidimensional analysis to reveal more facts than the monotone mainstream media. The more facets we understand the problem, the better solution that the whole society could approach. We believe that “all men are created equal” [11]. With the rise of the police shootings reported by media, more people in the U.S. think that police use excessive force during law enforcement, especially to a specific group of people. We want to add our two cents point of view by multidimensional analysis to reveal more facts than the monotone mainstream media. The more facets we understand the problem, the better solution that the whole society could approach. 1. Introduction Our report has three parts: First, we analyzed and quantified fatal police shooting news reporting deviation of mainstream media. Second, we used FP-growth to mine frequent patterns, clustered hotspots of fatal police shootings, and brought multi-attributes (social economics, demographics, political tendency, education, gun ownership rate, police training hours, etc.) to reveal connections under the iceberg. Third, we built regression models based on correlation analysis for numeric variables selection to predict police shooting rates at the state level. We also built classification models based on Chi-square testing for categorical variables selection to predict the victims’ race of fatal police shootings. The main datasets we choose for our analysis include: 1. Washington Post Fatal Police Shooting Dataset (WP data) [13] covers fatal police shooting from 01-01-2015 to 12- 02-2020. 2. KilledByPolice (KBP): Fatal police shooting reported in KilledByPolice website [6] from 01-01-2015 to 11-04-2020. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

US Fatal Police Shooting Analysis and Prediction: A Study

Understanding Factors Affecting Neural Network Performance in Diffusion Prediction

Authors:
(1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com);
(2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com);
(3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Methods Results Discussion Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Authors: Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Discussion Discussion Conclusions and References Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Architecting Neural Networks for Diffusion Prediction: A Study on Encoder-Decoder CNNs

Image source: https://www.flickr.com/photos/nida-nih/51015715423/in/photostream/ The observation that the brain makes [something described as] predictions is a gateway to exploring how placebos work. It also includes what follows after, to have therapeutic effects—in the conditions they do. Placebos involve the expectation to feel better and the experience of it. Placebos are used in studies. They, however, have presented a different challenge in testing the efficacies of psychedelics and antidepressants. There is a feature in The Atlantic, Psychedelics Are Challenging the Scientific Gold Standard, stating that "In clinical trials, participants (and the researchers studying them) generally aren’t supposed to know whether they’re getting the actual drug or a placebo, to avoid allowing people’s expectations about a treatment to shape their response to it. Blinding, as this practice is called, is a key component of a randomized controlled clinical trial, or RCT—medicine’s gold standard for demonstrating that a drug actually works. But virtually no one can take a psychedelic drug and not know it. Such failure, if it exists, might extend beyond psychedelics to several kinds of psychoactive drugs. For instance, a 2022 analysis found that many antidepressant trials fail to effectively blind participants, in part because of side effects." This contributed, in part, to a recent report by Reuters, US FDA panel votes against first MDMA-based PTSD treatment, stating that "Advisers to the U.S. Food and Drug Administration on Tuesday voted against a therapy based on the mind-altering drug MDMA for patients with post-traumatic stress disorder, marking a setback to the nascent field of psychedelic treatments. The panel voted 10-to-1 against the first MDMA-based PTSD treatment, saying the benefits did not outweigh its risks, while nine members said the available data did not show its effectiveness in PTSD patients." There might be a few conceptual approaches around this, starting with some questions. How does the human mind do what is described as prediction or anticipation? How do relays occur in the mind—afterwards—to result in relief? How can this be explained to participants and researchers in a way that neutralizes the expected side effects of psychoactive drugs against dampening the placebo effect or group identification? Brain science has established that electrical signals leap from node to node in myelinated axons, going faster in what is called saltatory conduction. This is different from continuous conduction in unmyelinated axons, where they travel directly along the length of the axon without leaping. It is postulated that in a set of electrical signals [in clusters of neurons], some electrical signals split or go ahead of others to interact with chemical signals before others follow. This means that whatever initial perception an individual has, either of sight, smell, touch, taste, or sound, could be a result of this split. If the input matches with the split, then relays continue—with the incoming signals going in the same direction. However, if the input is different, the incoming electrical signals go in the right direction. This explains what is described as predictive coding, processing, and prediction error. Simply, when two letters in a word are first seen, there could be a perception of what the word is, which may be confirmed if the rest of the letters are seen. In this case, the first and second sets of the electrical signals would go in the same direction. If the rest of the word is seen and it is different from what was perceived, then the incoming electrical signals would go in another direction. This is a mechanistic way to explain how the brain appears to predict using electrical signals. It is proposed that electrical signals are always splitting, making it feel like predictions are common in the mind. Splits are postulated to be a division of distributions—a feature in the human mind. Distributions are defined as the way some sets of signals get the outputs of other sets of signals. Distribution happens in attention [a feature] and in awareness [another feature, but less than attention]. For example, a word may result in the memory of an event, the sight of something may result in an emotion of fear, or the sound of something may result in an appetitive feeling. Also, experiences with side effects—or not, can also be described as distributions since electrical and chemical signals are involved in modulations of internal senses. Some other features of mind, including intentionality, may be responsible for distributions as well. When intent does not [a non-intent feature], arrays do. However, destinations of distributions are not always constant—they may vary, sometimes widely. There is a new paper in Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, Dynamic Functional Hyperconnectivity After Psilocybin Intake Is Primarily Associated With Oceanic Boundlessness, where the authors "investigated the relationship between neurobiological and experiential patterns after psilocybin by focusing on the link between dynamic cerebral connectivity and retrospective questionnaire assessment." How does the human mind work for distributions that correlate with hyperconnectivity? How do psychedelics have an effect? What is the relationship between predictions in placebos and how psychedelics work? How can hallucinations in psychedelics be explained in comparison to those in large language models? Can AI shape future clinical trials for psychedelics? Exploring the postulate of the human mind in the action potentials neurotransmitters theory of consciousness​, ​with examples and detailed animations, could work in explaining to participants and researchers that while there are expectations of side effects, features of the human mind ultimately decide experiences. So, it may not proportionately apply that the absence of side effects indicates that the participant is under placebo, open-label, or otherwise. Image source: https://www.flickr.com/photos/nida-nih/51015715423/in/photostream/ Image source: https://www.flickr.com/photos/nida-nih/51015715423/in/photostream/ https://www.flickr.com/photos/nida-nih/51015715423/in/photostream/ The observation that the brain makes [something described as] predictions is a gateway to exploring how placebos work. It also includes what follows after, to have therapeutic effects—in the conditions they do. Placebos involve the expectation to feel better and the experience of it. Placebos are used in studies. They, however, have presented a different challenge in testing the efficacies of psychedelics and antidepressants. There is a feature in The Atlantic , Psychedelics Are Challenging the Scientific Gold Standard , stating that "In clinical trials, participants (and the researchers studying them) generally aren’t supposed to know whether they’re getting the actual drug or a placebo, to avoid allowing people’s expectations about a treatment to shape their response to it. Blinding, as this practice is called, is a key component of a randomized controlled clinical trial, or RCT—medicine’s gold standard for demonstrating that a drug actually works. But virtually no one can take a psychedelic drug and not know it. Such failure, if it exists, might extend beyond psychedelics to several kinds of psychoactive drugs. For instance, a 2022 analysis found that many antidepressant trials fail to effectively blind participants, in part because of side effects." The Atlantic Psychedelics Are Challenging the Scientific Gold Standard This contributed, in part, to a recent report by Reuters , US FDA panel votes against first MDMA-based PTSD treatment , stating that "Advisers to the U.S. Food and Drug Administration on Tuesday voted against a therapy based on the mind-altering drug MDMA for patients with post-traumatic stress disorder, marking a setback to the nascent field of psychedelic treatments. The panel voted 10-to-1 against the first MDMA-based PTSD treatment, saying the benefits did not outweigh its risks, while nine members said the available data did not show its effectiveness in PTSD patients." Reuters US FDA panel votes against first MDMA-based PTSD treatment There might be a few conceptual approaches around this, starting with some questions. How does the human mind do what is described as prediction or anticipation? How do relays occur in the mind—afterwards—to result in relief? How can this be explained to participants and researchers in a way that neutralizes the expected side effects of psychoactive drugs against dampening the placebo effect or group identification? Brain science has established that electrical signals leap from node to node in myelinated axons, going faster in what is called saltatory conduction. This is different from continuous conduction in unmyelinated axons, where they travel directly along the length of the axon without leaping. It is postulated that in a set of electrical signals [in clusters of neurons], some electrical signals split or go ahead of others to interact with chemical signals before others follow. This means that whatever initial perception an individual has, either of sight, smell, touch, taste, or sound, could be a result of this split. If the input matches with the split, then relays continue—with the incoming signals going in the same direction. However, if the input is different, the incoming electrical signals go in the right direction. This explains what is described as predictive coding, processing, and prediction error. Simply, when two letters in a word are first seen, there could be a perception of what the word is, which may be confirmed if the rest of the letters are seen. In this case, the first and second sets of the electrical signals would go in the same direction. If the rest of the word is seen and it is different from what was perceived, then the incoming electrical signals would go in another direction. This is a mechanistic way to explain how the brain appears to predict using electrical signals. It is proposed that electrical signals are always splitting, making it feel like predictions are common in the mind. Splits are postulated to be a division of distributions—a feature in the human mind. Distributions are defined as the way some sets of signals get the outputs of other sets of signals. Distribution happens in attention [a feature] and in awareness [another feature, but less than attention]. For example, a word may result in the memory of an event, the sight of something may result in an emotion of fear, or the sound of something may result in an appetitive feeling. Also, experiences with side effects—or not, can also be described as distributions since electrical and chemical signals are involved in modulations of internal senses. Some other features of mind, including intentionality, may be responsible for distributions as well. When intent does not [a non-intent feature], arrays do. However, destinations of distributions are not always constant—they may vary, sometimes widely. There is a new paper in Biological Psychiatry: Cognitive Neuroscience and Neuroimaging , Dynamic Functional Hyperconnectivity After Psilocybin Intake Is Primarily Associated With Oceanic Boundlessness , where the authors "investigated the relationship between neurobiological and experiential patterns after psilocybin by focusing on the link between dynamic cerebral connectivity and retrospective questionnaire assessment." Biological Psychiatry: Cognitive Neuroscience and Neuroimaging Dynamic Functional Hyperconnectivity After Psilocybin Intake Is Primarily Associated With Oceanic Boundlessness How does the human mind work for distributions that correlate with hyperconnectivity? How do psychedelics have an effect? What is the relationship between predictions in placebos and how psychedelics work? How can hallucinations in psychedelics be explained in comparison to those in large language models? Can AI shape future clinical trials for psychedelics? Exploring the postulate of the human mind in the action potentials neurotransmitters theory of consciousness ​, ​with examples and detailed animations, could work in explaining to participants and researchers that while there are expectations of side effects, features of the human mind ultimately decide experiences. So, it may not proportionately apply that the absence of side effects indicates that the participant is under placebo, open-label, or otherwise. the action potentials neurotransmitters theory of consciousness

Predictive Coding, AI: Modeling Placebos in RCTs for Psychedelics and Antidepressants

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References 2. Related Works Tree-based algorithms are widely used in machine learning for tabular data. Decision trees recursively split the data into multiple parts based on axis-aligned hyper-planes (Hastie et al., 2009). Random Forests (RF) (Breiman, 2001) and Gradient Boosted Decision Trees (GBDT) (Friedman, 2001) are the most commonly used tree based ensembles. RF fits multiple decision trees on random subsets of the data and averages/polls the predictions to alleviate the overfitting characteristic of decision trees. GBDT, XGBoost (Chen and Guestrin, 2016), and CatBoost (Prokhorenkova et al., 2018) are boosted ensemble models that sequentially build decision trees to correct errors made by previous trees, leading to improved performance on complex datasets with non-linear relations. Recently, there has been a lot of interest in deep learning models for tabular data. Some methods introduce differentiable approximations of decision functions used in decision trees to make them differentiable (Hazimeh et al., 2020; Popov et al., 2019). These methods outperform pure tree based problem for some problem statements, however, they are not consistently better (Gorishniy et al., 2021). Other methods have used attention mechanisms to adapt DL methods to tabular data (Arik et al., 2019; Huang et al., 2020; Gorishniy et al., 2021; Somepalli et al., 2021; Chen et al., 2022). TabNet (Arik et al., 2019) proposes a sparse attention mechanism that is stacked in multiple layers to mimic the recursive splitting of decision trees. Inspired from the success of self-attention transformers (Vaswani et al., 2017) in many domains (Devlin et al., 2019; Dosovitskiy et al., 2021; Gong et al., 2021) methods like TabTransformer (Huang et al., 2020), FT-Transformer (Gorishniy et al., 2021) and SAINT (Somepalli et al., 2021) were proposed. TabTransformer embeds all categorical variables into a unified embedding space, and a sentence of categorical embeddings is passed through self-attention transformer layers. FT-Transformer further extends this by attending to numerical features as well, by using continuous embedding. SAINT builds on FT-Transformer by proposing a new kind of attention which captures interactions between samples of a batch. However, SAINT does not provide any advantage over FT-Transformer for our problem statement, because intersample attention is only effective when the number of dimensions is higher in comparision to the number of samples, thus we do not compare RCT against SAINT (Somepalli et al., 2021). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References 2. Related Works Tree-based algorithms are widely used in machine learning for tabular data. Decision trees recursively split the data into multiple parts based on axis-aligned hyper-planes (Hastie et al., 2009). Random Forests (RF) (Breiman, 2001) and Gradient Boosted Decision Trees (GBDT) (Friedman, 2001) are the most commonly used tree based ensembles. RF fits multiple decision trees on random subsets of the data and averages/polls the predictions to alleviate the overfitting characteristic of decision trees. GBDT, XGBoost (Chen and Guestrin, 2016), and CatBoost (Prokhorenkova et al., 2018) are boosted ensemble models that sequentially build decision trees to correct errors made by previous trees, leading to improved performance on complex datasets with non-linear relations. Recently, there has been a lot of interest in deep learning models for tabular data. Some methods introduce differentiable approximations of decision functions used in decision trees to make them differentiable (Hazimeh et al., 2020; Popov et al., 2019). These methods outperform pure tree based problem for some problem statements, however, they are not consistently better (Gorishniy et al., 2021). Other methods have used attention mechanisms to adapt DL methods to tabular data (Arik et al., 2019; Huang et al., 2020; Gorishniy et al., 2021; Somepalli et al., 2021; Chen et al., 2022). TabNet (Arik et al., 2019) proposes a sparse attention mechanism that is stacked in multiple layers to mimic the recursive splitting of decision trees. Inspired from the success of self-attention transformers (Vaswani et al., 2017) in many domains (Devlin et al., 2019; Dosovitskiy et al., 2021; Gong et al., 2021) methods like TabTransformer (Huang et al., 2020), FT-Transformer (Gorishniy et al., 2021) and SAINT (Somepalli et al., 2021) were proposed. TabTransformer embeds all categorical variables into a unified embedding space, and a sentence of categorical embeddings is passed through self-attention transformer layers. FT-Transformer further extends this by attending to numerical features as well, by using continuous embedding. SAINT builds on FT-Transformer by proposing a new kind of attention which captures interactions between samples of a batch. However, SAINT does not provide any advantage over FT-Transformer for our problem statement, because intersample attention is only effective when the number of dimensions is higher in comparision to the number of samples, thus we do not compare RCT against SAINT (Somepalli et al., 2021). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Related Works

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: সম্পর্কিত কাজ

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Verwandte Arbeiten

Revelando el poder de la autoatención para la predicción de costos de envío: trabajos relacionados

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : travaux connexes

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: संबंधित कार्य

配送コスト予測における自己注意の力を明らかにする: 関連研究

배송비 예측을 위한 자기주의의 힘 공개: 관련 연구

Revelando o poder da autoatenção para previsão de custos de envio: trabalhos relacionados

Раскрытие силы самовнимания для прогнозирования стоимости доставки: сопутствующие работы

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: İlgili Çalışmalar

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Các tác phẩm liên quan

揭示自注意力机制在运输成本预测中的作用：相关研究

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References References Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. CoRR, abs/1910.02600, 2019. URL http://arxiv.org/abs/1910.02600. Sercan O Arik, Engin Gedik, Kenan Guney, and Umut Atilla. Tabnet: Attentive interpretable tabular learning. In Advances in Neural Information Processing Systems, pages 10951–10961, 2019. Christopher M Bishop. Pattern recognition and machine learning. In Springer, chapter 2, pages 36–43. 2006. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3930–3938, 2022. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data, 2020. William Falcon and The PyTorch Lightning team. PyTorch Lightning, 3 2019. URL https: //github.com/Lightning-AI/lightning. Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571–575, 2021. doi: 10.21437/Interspeech.2021-698. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932–18943, 2021. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009. Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR, 2020. Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146–3154, 2017. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Daniele Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. SIGKDD Explor. Newsl., 3(1):27–32, jul 2001. ISSN 1931-0145. doi: 10.1145/507533.507538. URL https://doi.org/10.1145/507533. 507538. Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019. Liudmila Prokhorenkova, Gleb Gusev, Alexey Vorobev, Anna Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31:6638–6648, 2018. Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References References Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. CoRR, abs/1910.02600, 2019. URL http://arxiv.org/abs/1910.02600. Sercan O Arik, Engin Gedik, Kenan Guney, and Umut Atilla. Tabnet: Attentive interpretable tabular learning. In Advances in Neural Information Processing Systems, pages 10951–10961, 2019. Christopher M Bishop. Pattern recognition and machine learning. In Springer, chapter 2, pages 36–43. 2006. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3930–3938, 2022. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data, 2020. William Falcon and The PyTorch Lightning team. PyTorch Lightning, 3 2019. URL https: //github.com/Lightning-AI/lightning. Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571–575, 2021. doi: 10.21437/Interspeech.2021-698. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932–18943, 2021. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009. Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR, 2020. Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146–3154, 2017. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Daniele Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. SIGKDD Explor. Newsl., 3(1):27–32, jul 2001. ISSN 1931-0145. doi: 10.1145/507533.507538. URL https://doi.org/10.1145/507533. 507538. Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019. Liudmila Prokhorenkova, Gleb Gusev, Alexey Vorobev, Anna Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31:6638–6648, 2018. Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: References

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: তথ্যসূত্র

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Referenzen

Revelando el poder de la autoatención para la predicción de costos de envío: referencias

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : références

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: संदर्भ

配送コスト予測における自己注意の力を明らかにする: 参考文献

배송비 예측을 위한 자기주의의 힘 공개: 참고자료

Revelando o poder da autoatenção para previsão de custos de envio: referências

Раскрытие силы самовнимания для прогнозирования стоимости доставки: ссылки

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Referanslar

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Tài liệu tham khảo

揭示自注意力机制在运输成本预测中的作用：参考文献

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References Abstract Amazon ships billions of packages to its customers annually within the United States. Shipping cost of these packages are used on the day of shipping (day 0) to estimate profitability of sales. Downstream systems utilize these days 0 profitability estimates to make financial decisions, such as pricing strategies and delisting loss-making products. However, obtaining accurate shipping cost estimates on day 0 is complex for reasons like delay in carrier invoicing or fixed cost components getting recorded at monthly cadence. Inaccurate shipping cost estimates can lead to bad decision, such as pricing items too low or high, or promoting the wrong product to the customers. Current solutions for estimating shipping costs on day 0 rely on tree-based models that require extensive manual engineering efforts. In this study, we propose a novel architecture called the Rate Card Transformer (RCT) that uses self-attention to encode all package shipping information such as package attributes, carrier information and route plan. Unlike other transformer-based tabular models, RCT has the ability to encode a variable list of one-to-many relations of a shipment, allowing it to capture more information about a shipment. For example, RCT can encode properties of all products in a package. Our results demonstrate that cost predictions made by the RCT have 28.82% less error compared to tree-based GBDT model. Moreover, the RCT outperforms the state-of-the-art transformer-based tabular model, FTTransformer, by 6.08%. We also illustrate that the RCT learns a generalized manifold of the rate card that can improve the performance of tree-based models. 1. Introduction Amazon ships packages in the order of billions annually to its customers in the United States alone. The route planning for these packages is done on the day of shipping, day 0. As part of this plan, the shipping cost for each package is estimated by breaking down the package journey into smaller legs, and calculating the cost of each leg using a rate card. Day cost estimates are used to compute initial profitability estimates for accounting purposes, e.g.the estimate of profit/loss for each item as a result of a specific sale to a customer. These profitability estimates are used by several downstream services for decision making and planning. However, the day 0 estimates may differ from the actual cost due to factors like improper rate-card configuration, incorrect package dimensions, wrong delivery address, etc. Inaccurate cost estimates cause skewed profitability estimates, which in turn leads to suboptimal financial decisions by downstream systems. For example, if the shipping cost of an item is consistently overestimated, then the item could be removed from the catalog. On the other hand, underestimated cost can lead pricing systems to lower the price of the item, leading to losses. Further, inaccurate estimation also leads us to promote wrong products to the customer, causing bad customer experience. To improve these shipping cost estimates, we propose a Transformer based deep learning model that accurately predicts the shipping cost at day 0. In the context of shipping, a package is characterized by its physical dimensions, weight, and contents. It also include details about the carrier responsible for transporting it and the intended route. Additionally, a package is associated with a variable number of attributes that describe the item(s) inside and the various charges related to its shipment. Collectively, we refer to these attributes as the rate card associated with the package. For tabular datasets like package rate cards, tree based models like Gradient Boosted Decision Trees (GBDT), XGBoost (Chen and Guestrin, 2016), etc., are considered as state-of-the-art (SOTA) models. However, their effectiveness heavily relies on high-quality input features (Arik et al., 2019) which can require extensive feature engineering. For our use case, this problem is further accentuated by the fact that the target concept depends on high order combinatorial interactions between rate card attributes. For example, if the rate card is improperly configured for large containers with flammable substances shipped from Washington DC to New York by ABC carrier, then the model has to learn to associate property combination < size = large, item = flammable, source = W ashington, destination = New Y ork, carrier = ABC > with high deviation between estimated and actual costs. When dealing with feature combinations, considering all possible higher-order interactions between package properties may be impractical due to the exponential increase in the number of interactions with each increase in order, leading to the curse of dimensionality (Bishop, 2006). Another shortcoming of tree based models is their inability to handle a variable length list of features. A package may contain multiple items, and its ship cost can be broken down into multiple charges types. Previous experiments demonstrated that adding features engineered from multiple items and charges improved GBDT’s performance. However, due to inability of tree based models to handle variable list of features, complete information from them could not be learned. In this paper, inspired by the recent success of transformers in tabular domain (Huang et al., 2020; Somepalli et al., 2021; Gorishniy et al., 2021), we propose a novel architecture called the Rate Card Transformer (RCT) to predict ship cost on day 0. The proposed model is specifically designed to learn an embedding of rate card associated with a package. The RCT leverages self-attention mechanisms to effectively capture the interdependencies between various components in a rate card by learning interactions between input features. Specifically, our contributions in this work include: • Propose a novel architecture, Rate Card Transformer (RCT), which leverages transformer architecture to learn a manifold of the rate card, to predict shipping cost on Day 0. Further, it is demonstrated that RCT outperforms both GBDTs and the stateof-the-art tabular transformer, FT-Transformer, (Gorishniy et al., 2021) in shipping cost prediction. • Extensive experiments are performed to show that the learned embeddings are a sufficient representation of the rate card manifold, and self-attention layers are effective feature interaction learners. Ablation studies are performed to analyze the impact of number of transformer layers and attention heads on model performance. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References Abstract Amazon ships billions of packages to its customers annually within the United States. Shipping cost of these packages are used on the day of shipping (day 0) to estimate profitability of sales. Downstream systems utilize these days 0 profitability estimates to make financial decisions, such as pricing strategies and delisting loss-making products. However, obtaining accurate shipping cost estimates on day 0 is complex for reasons like delay in carrier invoicing or fixed cost components getting recorded at monthly cadence. Inaccurate shipping cost estimates can lead to bad decision, such as pricing items too low or high, or promoting the wrong product to the customers. Current solutions for estimating shipping costs on day 0 rely on tree-based models that require extensive manual engineering efforts. In this study, we propose a novel architecture called the Rate Card Transformer (RCT) that uses self-attention to encode all package shipping information such as package attributes, carrier information and route plan. Unlike other transformer-based tabular models, RCT has the ability to encode a variable list of one-to-many relations of a shipment, allowing it to capture more information about a shipment. For example, RCT can encode properties of all products in a package. Our results demonstrate that cost predictions made by the RCT have 28.82% less error compared to tree-based GBDT model. Moreover, the RCT outperforms the state-of-the-art transformer-based tabular model, FTTransformer, by 6.08%. We also illustrate that the RCT learns a generalized manifold of the rate card that can improve the performance of tree-based models. 1. Introduction Amazon ships packages in the order of billions annually to its customers in the United States alone. The route planning for these packages is done on the day of shipping, day 0. As part of this plan, the shipping cost for each package is estimated by breaking down the package journey into smaller legs, and calculating the cost of each leg using a rate card. Day cost estimates are used to compute initial profitability estimates for accounting purposes, e.g.the estimate of profit/loss for each item as a result of a specific sale to a customer. These profitability estimates are used by several downstream services for decision making and planning. However, the day 0 estimates may differ from the actual cost due to factors like improper rate-card configuration, incorrect package dimensions, wrong delivery address, etc. Inaccurate cost estimates cause skewed profitability estimates, which in turn leads to suboptimal financial decisions by downstream systems. For example, if the shipping cost of an item is consistently overestimated, then the item could be removed from the catalog. On the other hand, underestimated cost can lead pricing systems to lower the price of the item, leading to losses. Further, inaccurate estimation also leads us to promote wrong products to the customer, causing bad customer experience. To improve these shipping cost estimates, we propose a Transformer based deep learning model that accurately predicts the shipping cost at day 0. In the context of shipping, a package is characterized by its physical dimensions, weight, and contents. It also include details about the carrier responsible for transporting it and the intended route. Additionally, a package is associated with a variable number of attributes that describe the item(s) inside and the various charges related to its shipment. Collectively, we refer to these attributes as the rate card associated with the package. For tabular datasets like package rate cards, tree based models like Gradient Boosted Decision Trees (GBDT), XGBoost (Chen and Guestrin, 2016), etc., are considered as state-of-the-art (SOTA) models. However, their effectiveness heavily relies on high-quality input features (Arik et al., 2019) which can require extensive feature engineering. For our use case, this problem is further accentuated by the fact that the target concept depends on high order combinatorial interactions between rate card attributes. For example, if the rate card is improperly configured for large containers with flammable substances shipped from Washington DC to New York by ABC carrier, then the model has to learn to associate property combination < size = large, item = flammable, source = W ashington, destination = New Y ork, carrier = ABC > with high deviation between estimated and actual costs. When dealing with feature combinations, considering all possible higher-order interactions between package properties may be impractical due to the exponential increase in the number of interactions with each increase in order, leading to the curse of dimensionality (Bishop, 2006). Another shortcoming of tree based models is their inability to handle a variable length list of features. A package may contain multiple items, and its ship cost can be broken down into multiple charges types. Previous experiments demonstrated that adding features engineered from multiple items and charges improved GBDT’s performance. However, due to inability of tree based models to handle variable list of features, complete information from them could not be learned. In this paper, inspired by the recent success of transformers in tabular domain (Huang et al., 2020; Somepalli et al., 2021; Gorishniy et al., 2021), we propose a novel architecture called the Rate Card Transformer (RCT) to predict ship cost on day 0. The proposed model is specifically designed to learn an embedding of rate card associated with a package. The RCT leverages self-attention mechanisms to effectively capture the interdependencies between various components in a rate card by learning interactions between input features. Specifically, our contributions in this work include: • Propose a novel architecture, Rate Card Transformer (RCT), which leverages transformer architecture to learn a manifold of the rate card, to predict shipping cost on Day 0. Further, it is demonstrated that RCT outperforms both GBDTs and the stateof-the-art tabular transformer, FT-Transformer, (Gorishniy et al., 2021) in shipping cost prediction. • Extensive experiments are performed to show that the learned embeddings are a sufficient representation of the rate card manifold, and self-attention layers are effective feature interaction learners. Ablation studies are performed to analyze the impact of number of transformer layers and attention heads on model performance. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Abstract and Introduction

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: বিমূর্ত এবং ভূমিকা

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Zusammenfassung und Einführung

Revelando el poder de la autoatención para la predicción de costos de envío: resumen e introducción

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : résumé et introduction

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: सार और परिचय

配送コスト予測における自己注意の力を明らかにする: 概要と概要

배송비 예측을 위한 자기주의의 힘 공개: 요약 및 소개

Revelando o poder da autoatenção para previsão de custos de envio: resumo e introdução

Раскрытие силы самовнимания для прогнозирования стоимости доставки: Краткое содержание и введение

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Özet ve Giriş

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Tóm tắt và giới thiệu

揭示自注意力机制在运输成本预测中的作用：摘要和简介

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References 5. Conclusion and Future Work In this paper, we presented a novel framework based on the Transformer architecture for predicting shipping costs on day 0. Our proposed framework encodes shipping attributes of a package, i.e., the package rate card, into a uniform embedding space. These embeddings are then fed through a Transformer layer, which models complex higher-order interactions and learns an effective representation of the package rate card for predicting shipping costs. Our experimental results demonstrate that the proposed model, called RCT, outperforms GBDT model by 28.8%. Furthermore, demonstrate the RCT performs better than SOTA model FT-Transformer for our problem statement. We also show that when rate card representation learned by RCT is added to GBDT model, its performance is improved by 12.51%. This underscores the fact that RCT is able to learn sufficient representation representations of rate card information. n this work, the route information used was limited to the start and end nodes alone. Future work could explore the use of Graph Neural Networks to encode information about the complete route. Further, the performance of the RCT might be improved by exploring ways to include the item ID as a feature, such as the use of item embeddings which are available internally. Also, while the RCT was trained to predict only the ship cost, it can be modified to predict all the attributes of the invoice by adding a Transformer decoder layer. This would enable other applications like invoice anomaly detection. Additionally, future research could investigate whether the package representations learnt by the RCT can be used to improve the performance of other related tasks or to quantify the model uncertainty in each prediction via approaches like the one proposed in Amini et al. (2019). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References 5. Conclusion and Future Work In this paper, we presented a novel framework based on the Transformer architecture for predicting shipping costs on day 0. Our proposed framework encodes shipping attributes of a package, i.e., the package rate card, into a uniform embedding space. These embeddings are then fed through a Transformer layer, which models complex higher-order interactions and learns an effective representation of the package rate card for predicting shipping costs. Our experimental results demonstrate that the proposed model, called RCT, outperforms GBDT model by 28.8%. Furthermore, demonstrate the RCT performs better than SOTA model FT-Transformer for our problem statement. We also show that when rate card representation learned by RCT is added to GBDT model, its performance is improved by 12.51%. This underscores the fact that RCT is able to learn sufficient representation representations of rate card information. n this work, the route information used was limited to the start and end nodes alone. Future work could explore the use of Graph Neural Networks to encode information about the complete route. Further, the performance of the RCT might be improved by exploring ways to include the item ID as a feature, such as the use of item embeddings which are available internally. Also, while the RCT was trained to predict only the ship cost, it can be modified to predict all the attributes of the invoice by adding a Transformer decoder layer. This would enable other applications like invoice anomaly detection. Additionally, future research could investigate whether the package representations learnt by the RCT can be used to improve the performance of other related tasks or to quantify the model uncertainty in each prediction via approaches like the one proposed in Amini et al. (2019). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Conclusion and Future Work

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: উপসংহার এবং ভবিষ্যতের কাজ

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Fazit und zukünftige Arbeit

Revelando el poder de la autoatención para la predicción de costos de envío: conclusión y trabajo futuro

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : conclusion et travaux futurs

शिपिंग लागत की भविष्यवाणी के लिए आत्म-ध्यान की शक्ति का अनावरण: निष्कर्ष और भविष्य का कार्य

配送コスト予測における自己注意の力の解明：結論と今後の課題

배송비 예측을 위한 자기주의의 힘 공개: 결론 및 향후 연구

Revelando o poder da autoatenção para previsão de custos de envio: conclusão e trabalho futuro

Раскрытие силы самовнимания для прогнозирования стоимости доставки: выводы и будущая работа

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünün Ortaya Çıkarılması: Sonuç ve Gelecekteki Çalışmalar

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Kết luận và công việc trong tương lai

揭示自注意力机制在运输成本预测中的作用：结论和未来工作

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References 4. Experiments In this section, the performance of the RCT is demonstrated on a dataset of packages shipped in 2022. The mean absolute error (MAE) between the predicted and actual shipping cost is selected as the performance metric, as it is representative of the absolute error in monetary terms. In this paper, the MAE values are normalized by the MAE of day 0 heuristic estimate, which is expressed as MAE percentage (MAE%). This metric emphasizes the improvement achieved against the heuristic baseline. 4.1. Experimental Setup 4.1.1. Architecture and Hyperameters The embedding dimension was set to 128, and 6 transformer encoder layers were used, each with 16 self-attention heads. Adam optimizer (Kingma and Ba, 2014) with a starting learning rate of 0.0001 and a batch size of 2048 was used. To improve convergence, the learning rate was reduced by a factor of 0.7 every time the validation metric plateaued. The model code was implemented using the PyTorch (Prokhorenkova et al., 2018) and PyTorch Lightning (Falcon and The PyTorch Lightning team, 2019) frameworks. 4.1.2. Data Preparation A training dataset of 10M packages was sampled from packages shipped during a 45-day period in 2022. The data was preprocessed by label encoding categorical features and standardizing numerical features. The test dataset contains all packages (without sampling) that were shipped during a separate, non-overlapping week from 2022. 4.1.3. Benchmark Methods We compare the performance of RCT against various models with increasing level of complexity: GBDT, AWS AutoGluon (Erickson et al., 2020), Feedforward neural network, TabTransformer and FT-Transformer. For GBDT model, numerical features were not standardized, and target encoding (Micci-Barreca, 2001) was used to encode categorical features instead of label encoding. AWS AutoGluon was configured to learn an ensemble of LightGBM (Ke et al., 2017) models. A feedforward neural network containing 5 layers was used, the input to which was generated by embedding and concatenating dimension, route and service features. Publicly available implementations [1] of TabTransformer and FT-Transformer were used, and all hyperparameters were made consistent with RCT. Since the baselines do not handle collections of items and charges, we only used dimension, route and service features. 4.2. Baseline Comparisons Table 1a compares RCT against the baseline models discussed in section 4.1.3. The models in the table are organized in increasing order of model complexity. Both tree based models, GBDT and AutoGluon, are performing at a similar level. Deep learning models consistently outperform tree based models, indicating that the proposed architecture is efficient for shipping cost prediction. Transformer based models have lower MAE% scores than feedforward neural network, showing that transformers learn effective interaction. The RCT model outperforms both transformer models - TabTransformer and FT-Transformer (SOTA), suggesting that a custom architecture which encodes the latent structure of the rate card is contributing to the improved performance. Table 2 compares the performance of FT-Transformer and RCT models at different model sizes. The results show that RCT outperforms FT-Transformer across all tested model sizes, showing indicates that encoding rate card structure provides performance benefits across varying model capacities. 4.3. Does RCT learn effective representation of rate cards? Transformers have been shown to have strong representation learning capabilities in a variety of tasks. In this experiment, we investigate the effectiveness of rate card representation learned by RCT. To evaluate this, we compare the performance of our GBT model with and without the learned rate card representation as an input feature. The pooled output of the final Transformer layer is treated as the learned representation of the rate card. Adding this feature improved the performance of the GBDT by 9.79% (refer Table 1b). Further, it was observed that even when all manually engineered features are dropped, the GBDT still performs comparably, with an MAE percentage of 69.21%. This indicates that the learned representations of rate cards are not only effective at capturing better feature information, but are also sufficient representation of the package rate card. However, even with this feature, the GBDT has a 13.5% higher MAE% than the RCT. This is likely because the RCT is trained end-to-end, while the GBDT uses features learned as part of a separate model. 4.4. Does self-attention learn better interactions than feed forward neural networks? In section 4.2, it was observed that feed forward (FF) neural networks were outperformed by transformers, leading to the hypothesis that self-attention is a superior interaction learner. This section aims to explore this hypothesis further by utilizing FF instead of self-attention to encode dimension, route and service features while limiting the width of self-attention to only the item and charge features. The output encodings of both FF and self-attention are concatenated and fed into an FF layer to predict shipping cost. As the self-attention width is decreased, it fails to capture the interactions between all rate card features. The resulting model exhibits a higher MAE% of 64.73% in comparison to the RCT’s 55.72%. These results suggest that FF models are inferior interaction learners in comparison to transformers. 4.5. Analysis of Self-Attention In section 3.2, we discussed the proficiency of transformers in feature aggregation, owing to self-attention. In this section, ablation experiments are conducted to analyze the effect of attention depth and attention head count. Increasing the number of attention heads allows the model to learn more independent feature interactions. For this experiment, the model capacity is fixed at 128 dimensions, so an increase in the number of heads also reduces the complexity of interactions learned per head. Thus, choosing optimal head count is a trade-off between learning independent interactions and the complexity of each learned interaction. The trade-off can be observed in Fig. 2a, where the performance improves from 4 heads to 16 heads because the attention learned by each head is complex enough. However, the performance degrades when attention heads are increased from 16 to 32 because the complexity of heads has reduced substantially, negating the benefit of learning more independent interactions. Next, we illustrate the effect of increasing the attention depth by adding transformer encoder layers. Deeper transformer networks learn more complex higher-order interactions, thereby enhancing the model’s performance, as observed in Fig. 2b. However, increasing the number of layers from 6 to 12 reduces the model’s performance due to overfitting, caused by the rise in learnable parameter count. The evidence for overfitting can be found in Fig. 2b, where the gap between train and val MAE increases by 30% when moving from 6 to 12 layers. Finally, in Fig. 3, we display the heat maps generated using Algorithm 1. These heat maps illustrate the number of times each feature was attended to as part of the top five most attended features. Each column corresponds to a head, and each row corresponds to a feature. The heat map on the left was generated using RCT with nheads = 16, and the one on the right was generated with nheads = 4. Comparing both the heat maps, it can be seen that Fig. 3a has less number of active feature interactions per column, confirming our hypothesis that a larger number of attention heads leads to each head learning independent interactions between features. 4.6. How does the Transformer scale with more data? To minimize the experimentation costs, all experiments in this paper were conducted using a training dataset of size 10 million. However, it is important to use the best performing model, the training dataset size can be increased to achieve optimal performance. To verify the scalability of RCT with data, we trained the model on different training dataset sizes and plotted the results in Fig. 4. The results demonstrate that RCT’s performance continues to improve with larger datasets. Therefore, we can confidently expect that models trained on larger datasets will outperform the model explored in this paper. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. [1] https://github.com/lucidrains/tab-transformer-pytorc Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References 4. Experiments In this section, the performance of the RCT is demonstrated on a dataset of packages shipped in 2022. The mean absolute error (MAE) between the predicted and actual shipping cost is selected as the performance metric, as it is representative of the absolute error in monetary terms. In this paper, the MAE values are normalized by the MAE of day 0 heuristic estimate, which is expressed as MAE percentage (MAE%). This metric emphasizes the improvement achieved against the heuristic baseline. 4.1. Experimental Setup 4.1.1. Architecture and Hyperameters The embedding dimension was set to 128, and 6 transformer encoder layers were used, each with 16 self-attention heads. Adam optimizer (Kingma and Ba, 2014) with a starting learning rate of 0.0001 and a batch size of 2048 was used. To improve convergence, the learning rate was reduced by a factor of 0.7 every time the validation metric plateaued. The model code was implemented using the PyTorch (Prokhorenkova et al., 2018) and PyTorch Lightning (Falcon and The PyTorch Lightning team, 2019) frameworks. 4.1.2. Data Preparation A training dataset of 10M packages was sampled from packages shipped during a 45-day period in 2022. The data was preprocessed by label encoding categorical features and standardizing numerical features. The test dataset contains all packages (without sampling) that were shipped during a separate, non-overlapping week from 2022. 4.1.3. Benchmark Methods We compare the performance of RCT against various models with increasing level of complexity: GBDT, AWS AutoGluon (Erickson et al., 2020), Feedforward neural network, TabTransformer and FT-Transformer. For GBDT model, numerical features were not standardized, and target encoding (Micci-Barreca, 2001) was used to encode categorical features instead of label encoding. AWS AutoGluon was configured to learn an ensemble of LightGBM (Ke et al., 2017) models. A feedforward neural network containing 5 layers was used, the input to which was generated by embedding and concatenating dimension, route and service features. Publicly available implementations [1] of TabTransformer and FT-Transformer were used, and all hyperparameters were made consistent with RCT. Since the baselines do not handle collections of items and charges, we only used dimension, route and service features. 4.2. Baseline Comparisons Table 1a compares RCT against the baseline models discussed in section 4.1.3. The models in the table are organized in increasing order of model complexity. Both tree based models, GBDT and AutoGluon, are performing at a similar level. Deep learning models consistently outperform tree based models, indicating that the proposed architecture is efficient for shipping cost prediction. Transformer based models have lower MAE% scores than feedforward neural network, showing that transformers learn effective interaction. The RCT model outperforms both transformer models - TabTransformer and FT-Transformer (SOTA), suggesting that a custom architecture which encodes the latent structure of the rate card is contributing to the improved performance. Table 2 compares the performance of FT-Transformer and RCT models at different model sizes. The results show that RCT outperforms FT-Transformer across all tested model sizes, showing indicates that encoding rate card structure provides performance benefits across varying model capacities. 4.3. Does RCT learn effective representation of rate cards? Transformers have been shown to have strong representation learning capabilities in a variety of tasks. In this experiment, we investigate the effectiveness of rate card representation learned by RCT. To evaluate this, we compare the performance of our GBT model with and without the learned rate card representation as an input feature. The pooled output of the final Transformer layer is treated as the learned representation of the rate card. Adding this feature improved the performance of the GBDT by 9.79% (refer Table 1b). Further, it was observed that even when all manually engineered features are dropped, the GBDT still performs comparably, with an MAE percentage of 69.21%. This indicates that the learned representations of rate cards are not only effective at capturing better feature information, but are also sufficient representation of the package rate card. However, even with this feature, the GBDT has a 13.5% higher MAE% than the RCT. This is likely because the RCT is trained end-to-end, while the GBDT uses features learned as part of a separate model. 4.4. Does self-attention learn better interactions than feed forward neural networks? In section 4.2, it was observed that feed forward (FF) neural networks were outperformed by transformers, leading to the hypothesis that self-attention is a superior interaction learner. This section aims to explore this hypothesis further by utilizing FF instead of self-attention to encode dimension, route and service features while limiting the width of self-attention to only the item and charge features. The output encodings of both FF and self-attention are concatenated and fed into an FF layer to predict shipping cost. As the self-attention width is decreased, it fails to capture the interactions between all rate card features. The resulting model exhibits a higher MAE% of 64.73% in comparison to the RCT’s 55.72%. These results suggest that FF models are inferior interaction learners in comparison to transformers. 4.5. Analysis of Self-Attention In section 3.2, we discussed the proficiency of transformers in feature aggregation, owing to self-attention. In this section, ablation experiments are conducted to analyze the effect of attention depth and attention head count. Increasing the number of attention heads allows the model to learn more independent feature interactions. For this experiment, the model capacity is fixed at 128 dimensions, so an increase in the number of heads also reduces the complexity of interactions learned per head. Thus, choosing optimal head count is a trade-off between learning independent interactions and the complexity of each learned interaction. The trade-off can be observed in Fig. 2a, where the performance improves from 4 heads to 16 heads because the attention learned by each head is complex enough. However, the performance degrades when attention heads are increased from 16 to 32 because the complexity of heads has reduced substantially, negating the benefit of learning more independent interactions. Next, we illustrate the effect of increasing the attention depth by adding transformer encoder layers. Deeper transformer networks learn more complex higher-order interactions, thereby enhancing the model’s performance, as observed in Fig. 2b. However, increasing the number of layers from 6 to 12 reduces the model’s performance due to overfitting, caused by the rise in learnable parameter count. The evidence for overfitting can be found in Fig. 2b, where the gap between train and val MAE increases by 30% when moving from 6 to 12 layers. Finally, in Fig. 3, we display the heat maps generated using Algorithm 1. These heat maps illustrate the number of times each feature was attended to as part of the top five most attended features. Each column corresponds to a head, and each row corresponds to a feature. The heat map on the left was generated using RCT with nheads = 16, and the one on the right was generated with nheads = 4. Comparing both the heat maps, it can be seen that Fig. 3a has less number of active feature interactions per column, confirming our hypothesis that a larger number of attention heads leads to each head learning independent interactions between features. 4.6. How does the Transformer scale with more data? To minimize the experimentation costs, all experiments in this paper were conducted using a training dataset of size 10 million. However, it is important to use the best performing model, the training dataset size can be increased to achieve optimal performance. To verify the scalability of RCT with data, we trained the model on different training dataset sizes and plotted the results in Fig. 4. The results demonstrate that RCT’s performance continues to improve with larger datasets. Therefore, we can confidently expect that models trained on larger datasets will outperform the model explored in this paper. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [1] https://github.com/lucidrains/tab-transformer-pytorc

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Experiment

শিপিং খরচ পূর্বাভাস জন্য স্ব-মনোযোগের শক্তি উন্মোচন: পরীক্ষা

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Experiment

Revelando el poder de la autoatención para la predicción de costos de envío: experimento

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : expérience

शिपिंग लागत की भविष्यवाणी के लिए आत्म-ध्यान की शक्ति का अनावरण: प्रयोग

配送コスト予測における自己注意の力を明らかにする: 実験

배송비 예측을 위한 자기주의의 힘 공개: 실험

Revelando o poder da autoatenção para previsão de custos de envio: experiência

Раскрытие силы самовнимания для прогнозирования стоимости доставки: эксперимент

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Deney

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Thử nghiệm

揭示自注意力机制在运输成本预测中的作用：实验

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References 3. Methodology 3.1. Problem Statement 3.2. Background The Transformer architecture (Vaswani et al., 2017) is constructed by stacking multiple encoder blocks, where each block takes a sequence of embeddings as input and outputs a sequence of context aware embeddings. The encoder block consists of a multi-head self-attention (MHSA) layer followed by a position-wise feed-forward layer, with residual connections and layer norm before each layer. The MHSA layer comprises multiple self-attention units called heads, which learn interactions between input embeddings. The output sequence is then recursively passed through subsequent encoder layers, allowing each successive layer to learn higher order feature interactions. The transformer’s depth controls the complexity of the learned representation, as deeper layers capture more complex interactions between features. Further, multiple self-attention heads are used in MHSA, enabling each head to attend to different feature sub-spaces and learning interactions between them, cumulatively learning multiple independent sets of feature interactions. 3.3. Rate Card Transformer The rate card of a package consists of multiple features types, namely dimensional, route, service, item, and charge (Fig. 1a), where each feature type comprises multiple numerical and categorical features. The dimensional, route and service features are referred to as fixed length feature types, because each of them have a fixed number of features. Fixed length feature types are embedded to a sequence of tokens using a mixed embedding layer (MEL). For example, dimensional features d ∈ S[md, nd] are embedded to a d-dimensional token sequence of length md + nd. The MEL contains multiple embedding blocks, one for each feature in the feature type being embedded. Embedding lookup tables are used for embedding categorical features, while numerical features are embedded using continuous embedding blocks, as introduced in (Gorishniy et al., 2021). The sequence of feature tokens is passed as input to a stack of L Transformer encoder layers that are able to learn complex, higher order interactions between the features. Finally, the pooled Transformer output is fed to a feedforward layer to predict the shipping cost Cˆ as shown in Fig. 1b. We call the complete architecture the Rate Card Transformer (RCT). Trained to minimize the L1 loss between the predicted and actual shipping cost (Equation 3), RCT learns an effective representation of the dynamic rate card that allows it to accurately predict the shipping cost. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References 3. Methodology 3.1. Problem Statement 3.2. Background The Transformer architecture (Vaswani et al., 2017) is constructed by stacking multiple encoder blocks, where each block takes a sequence of embeddings as input and outputs a sequence of context aware embeddings. The encoder block consists of a multi-head self-attention (MHSA) layer followed by a position-wise feed-forward layer, with residual connections and layer norm before each layer. The MHSA layer comprises multiple self-attention units called heads, which learn interactions between input embeddings. The output sequence is then recursively passed through subsequent encoder layers, allowing each successive layer to learn higher order feature interactions. The transformer’s depth controls the complexity of the learned representation, as deeper layers capture more complex interactions between features. Further, multiple self-attention heads are used in MHSA, enabling each head to attend to different feature sub-spaces and learning interactions between them, cumulatively learning multiple independent sets of feature interactions. 3.3. Rate Card Transformer The rate card of a package consists of multiple features types, namely dimensional, route, service, item, and charge (Fig. 1a), where each feature type comprises multiple numerical and categorical features. The dimensional, route and service features are referred to as fixed length feature types, because each of them have a fixed number of features. Fixed length feature types are embedded to a sequence of tokens using a mixed embedding layer (MEL). For example, dimensional features d ∈ S[md, nd] are embedded to a d-dimensional token sequence of length md + nd. The MEL contains multiple embedding blocks, one for each feature in the feature type being embedded. Embedding lookup tables are used for embedding categorical features, while numerical features are embedded using continuous embedding blocks, as introduced in (Gorishniy et al., 2021). The sequence of feature tokens is passed as input to a stack of L Transformer encoder layers that are able to learn complex, higher order interactions between the features. Finally, the pooled Transformer output is fed to a feedforward layer to predict the shipping cost Cˆ as shown in Fig. 1b. We call the complete architecture the Rate Card Transformer (RCT). Trained to minimize the L1 loss between the predicted and actual shipping cost (Equation 3), RCT learns an effective representation of the dynamic rate card that allows it to accurately predict the shipping cost. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Methodology

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: পদ্ধতি

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Methodik

Revelando el poder de la autoatención para la predicción de costos de envío: metodología

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : méthodologie

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: कार्यप्रणाली

配送コスト予測における自己注意の力を明らかにする: 方法論

배송비 예측을 위한 자기주의의 힘 공개: 방법론

Revelando o poder da autoatenção para previsão de custos de envio: metodologia

Раскрытие силы самовнимания для прогнозирования стоимости доставки: методология

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Metodoloji

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Phương pháp luận

揭示自注意力机制在运输成本预测中的作用：方法论

Artificial Intelligence has been advancing at an unprecedented rate, causing fear about whether AI will dominate various aspects of our lives in the near future and “take our jobs.” In this article, by leveraging Moore's Law and other technological growth principles, I’m going to make an educated estimation about the state of AI in the coming years. Let’s explore these predictions and explain why AI may not completely take over within the next five years, and what the timeline might look like for AI to reach a truly advanced state. The Current State of AI AI technology has made significant strides in recent years. From natural language processing to computer vision, AI applications are becoming increasingly sophisticated, whether that’s the recent GPT-4o model, Google’s Gemini, or any one of the models in the race to become the most capable AI model. There are lots of people using these models in positive ways to unlock new methodologies in research and education, but there are also lots of people using these technologies for evil such as deepfakes. However, despite these advancements, there are still considerable challenges and limitations that need to be addressed. Current AI systems need massive amounts of data, computational power, and energy to function effectively. Moore's Law and Its Implications for AI Moore's Law, the observation that the number of transistors on a microchip doubles approximately every two years, has been a reliable predictor of technological progress for decades. More generally, Moore’s Law points to technology getting twice as accessible, cheaper, and efficient every two years. According to Moore's Law, we can expect computational power to continue increasing exponentially. However, there are physical and economic limits to this growth that may impact the pace of AI development. Estimating AI's Growth Using Known Laws By extrapolating from Moore's Law and considering other technological growth trends, we can make predictions about AI's capabilities in the coming years. Computational Power If we consider top-of-the-notch GPUs today, let’s say NVIDIA A100, it provides approximately 312 teraflops (TFLOPS) of performance. Moore's Law suggests a doubling of transistors every two years, which implies a potential 8-fold increase in computational power over five years (2^2.5 ≈ 5.66). Future GPU Performance = 312 TFLOPS × 5.66 ≈ 1766 TFLOPS However, the physical limitations of silicon and the economic costs of producing ever-smaller transistors could slow this growth. The increase might realistically be closer to 4 times due to these constraints: Realistic Future GPU Performance=312 TFLOPS×4≈1248 TFLOPS Data Availability and Quality According to projections by IDC (International Data Corporation), the size of the global datasphere will increase from 64.2 zettabytes in 2020 to 175 zettabytes by 2025. It is essential for the training of increasingly complex AI models that data is growing exponentially. But there are still a lot of obstacles to overcome, including concerns about data privacy, data quality, and effectively labeling and processing this data.The availability of high-quality, labeled data is critical, even with an increase in data output. Preprocessing and data labeling are obstacles that will not scale linearly with data increase, according to current trends. Energy Consumption AI training models like GPT-3 require substantial energy. GPT-3 used approximately 1,287 MWh to train. GPT-4, developed approximately 3 years after its predecessor required over 50 gigawatt-hours, so around 50 times the amount it took to train GPT-3. Assuming AI models become more complex and require more data and computational power, energy consumption could grow significantly. If we estimate a similar 50x increase or a 5000% increase in complexity to around 2500 gigawatt-hours for the next model that could be released in the next 5 years, this would require large improvements in energy production and efficiency. Given current energy efficiency improvements of around 10% per year (source: Lawrence Livermore National Laboratory), this only translates to approximately a 60% increase in energy efficiencies over a 5-year period if the energy production technologies don’t improve significantly. The Need for Increased Complexity in AI Models Current AI models struggle with many major issues today. Lack of Up-to-date Information: Current models are trained on static datasets and do not have real-time access to the internet for the most up-to-date information. Hallucinations: AI models sometimes generate information that is plausible-sounding but incorrect or nonsensical. Understanding Context and Nuance: Handling nuanced human language, context, and intent remains a challenge for AI. To address these issues, future AI models will need to incorporate: Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Increased Model Complexity Solving these problems will significantly increase the complexity of AI models. For example: Real-Time Data Integration: Requires constant data updates and real-time processing capabilities.
Improved Model Architectures: Likely to involve larger and more complex models with more parameters.
Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Let's assume each of these improvements independently doubles the complexity of the AI model. If we start with the current complexity (C), integrating these advancements might lead to: Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Data Quality and Ethics AI systems depend heavily on high-quality, unbiased data for training and decision-making. While the quantity of data is increasing rapidly, the quality and ethical considerations surrounding this data are equally crucial. Data Generation and Processing With the increase from 64.2 to 175 zettabytes, data generation will likely outpace the current capabilities of processing and labeling. Ensuring data quality and addressing biases require significant resources. Let's consider the human labor and computational resources needed for data labeling. Assuming 1 hour of human labor is needed to label 1 GB of data, labeling 175 zettabytes would require: 175 ZB×1 hour/GB×10^9  GB/ZB =175×10^9  hours This is not feasible with current resources, indicating a substantial gap in data quality and ethical processing. Conclusion: When Will AI Be in a Truly Advanced State In the Next, Say 5 Years? The idea that AI will "take over" in the next five years seems improbable, even if it will surely continue to advance and become more deeply integrated into many industries. There are several obstacles to overcome, including issues with energy usage, data quality, and processing power limitations. AI will not be able to evolve as fast without significant advancements in energy efficiency, data ethics, and technology.It seems more reasonable to anticipate that AI will become truly advanced within the next ten to fifteen years, based on existing patterns and reasonable development estimates. This prolonged period enables the development and successful implementation of essential ethical, technological, and resource-based breakthroughs. Sources and References: Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics, 38(8), 114-117.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins.
IDC. (2020). Global DataSphere Forecast 2021-2025.
Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing. Artificial Intelligence has been advancing at an unprecedented rate, causing fear about whether AI will dominate various aspects of our lives in the near future and “take our jobs.” In this article, by leveraging Moore's Law and other technological growth principles, I’m going to make an educated estimation about the state of AI in the coming years. Let’s explore these predictions and explain why AI may not completely take over within the next five years, and what the timeline might look like for AI to reach a truly advanced state. The Current State of AI AI technology has made significant strides in recent years. From natural language processing to computer vision, AI applications are becoming increasingly sophisticated, whether that’s the recent GPT-4o model, Google’s Gemini, or any one of the models in the race to become the most capable AI model. There are lots of people using these models in positive ways to unlock new methodologies in research and education, but there are also lots of people using these technologies for evil such as deepfakes. However, despite these advancements, there are still considerable challenges and limitations that need to be addressed. Current AI systems need massive amounts of data, computational power, and energy to function effectively. Moore's Law and Its Implications for AI Moore's Law , the observation that the number of transistors on a microchip doubles approximately every two years, has been a reliable predictor of technological progress for decades. More generally, Moore’s Law points to technology getting twice as accessible, cheaper, and efficient every two years. According to Moore's Law, we can expect computational power to continue increasing exponentially. Moore's Law However, there are physical and economic limits to this growth that may impact the pace of AI development. Estimating AI's Growth Using Known Laws By extrapolating from Moore's Law and considering other technological growth trends, we can make predictions about AI's capabilities in the coming years. Computational Power If we consider top-of-the-notch GPUs today, let’s say NVIDIA A100, it provides approximately 312 teraflops (TFLOPS) of performance. Moore's Law suggests a doubling of transistors every two years, which implies a potential 8-fold increase in computational power over five years (2^2.5 ≈ 5.66). Future GPU Performance = 312 TFLOPS × 5.66 ≈ 1766 TFLOPS However, the physical limitations of silicon and the economic costs of producing ever-smaller transistors could slow this growth. The increase might realistically be closer to 4 times due to these constraints: Realistic Future GPU Performance=312 TFLOPS×4≈1248 TFLOPS Data Availability and Quality According to projections by IDC (International Data Corporation), the size of the global datasphere will increase from 64.2 zettabytes in 2020 to 175 zettabytes by 2025. It is essential for the training of increasingly complex AI models that data is growing exponentially. But there are still a lot of obstacles to overcome, including concerns about data privacy, data quality, and effectively labeling and processing this data. The availability of high-quality, labeled data is critical, even with an increase in data output. Preprocessing and data labeling are obstacles that will not scale linearly with data increase, according to current trends. Energy Consumption AI training models like GPT-3 require substantial energy. GPT-3 used approximately 1,287 MWh to train. GPT-4, developed approximately 3 years after its predecessor required over 50 gigawatt-hours , so around 50 times the amount it took to train GPT-3. Assuming AI models become more complex and require more data and computational power, energy consumption could grow significantly. 50 gigawatt-hours If we estimate a similar 50x increase or a 5000% increase in complexity to around 2500 gigawatt-hours for the next model that could be released in the next 5 years, this would require large improvements in energy production and efficiency. 2500 gigawatt-hours Given current energy efficiency improvements of around 10% per year (source: Lawrence Livermore National Laboratory), this only translates to approximately a 60% increase in energy efficiencies over a 5-year period if the energy production technologies don’t improve significantly. The Need for Increased Complexity in AI Models Current AI models struggle with many major issues today. Lack of Up-to-date Information: Current models are trained on static datasets and do not have real-time access to the internet for the most up-to-date information. Lack of Up-to-date Information: Hallucinations: AI models sometimes generate information that is plausible-sounding but incorrect or nonsensical. Hallucinations: Understanding Context and Nuance: Handling nuanced human language, context, and intent remains a challenge for AI. Understanding Context and Nuance: To address these issues, future AI models will need to incorporate: Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Real-Time Data Integration: Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Improved Model Architectures: Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Contextual Understanding: Increased Model Complexity Increased Model Complexity Solving these problems will significantly increase the complexity of AI models. For example: Real-Time Data Integration: Requires constant data updates and real-time processing capabilities. Improved Model Architectures: Likely to involve larger and more complex models with more parameters. Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Real-Time Data Integration: Requires constant data updates and real-time processing capabilities. Real-Time Data Integration: Improved Model Architectures: Likely to involve larger and more complex models with more parameters. Improved Model Architectures: Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Contextual Understanding: Let's assume each of these improvements independently doubles the complexity of the AI model. If we start with the current complexity (C), integrating these advancements might lead to: Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Data Quality and Ethics AI systems depend heavily on high-quality, unbiased data for training and decision-making. While the quantity of data is increasing rapidly, the quality and ethical considerations surrounding this data are equally crucial. Data Generation and Processing With the increase from 64.2 to 175 zettabytes, data generation will likely outpace the current capabilities of processing and labeling. Ensuring data quality and addressing biases require significant resources. Let's consider the human labor and computational resources needed for data labeling. Assuming 1 hour of human labor is needed to label 1 GB of data, labeling 175 zettabytes would require: 175 ZB×1 hour/GB×10^9  GB/ZB =175×10^9  hours 175 ZB×1 hour/GB×10^9  GB/ZB = 175×10^9  hours 175×10^9  hours This is not feasible with current resources, indicating a substantial gap in data quality and ethical processing. Conclusion: When Will AI Be in a Truly Advanced State In the Next, Say 5 Years? The idea that AI will "take over" in the next five years seems improbable, even if it will surely continue to advance and become more deeply integrated into many industries. There are several obstacles to overcome, including issues with energy usage, data quality, and processing power limitations. AI will not be able to evolve as fast without significant advancements in energy efficiency, data ethics, and technology. It seems more reasonable to anticipate that AI will become truly advanced within the next ten to fifteen years, based on existing patterns and reasonable development estimates. This prolonged period enables the development and successful implementation of essential ethical, technological, and resource-based breakthroughs. Sources and References : Sources and References Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics, 38(8), 114-117. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins. IDC. (2020). Global DataSphere Forecast 2021-2025. Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing. Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics , 38(8), 114-117. Electronics Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning . MIT Press. Deep Learning Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins . HarperCollins IDC. (2020). Global DataSphere Forecast 2021-2025. Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing.

Predicting the Future of AI With Moore's Law: When Will It Really Take Over?

An In-Depth Study on Predicting Human Falls Using AI

Stories, News & Mentions about PREDICTHQ

Predicthq’s Stories on HackerNoon