Authors:
(1) Vijay Ekambaram, IBM Research;
(2) Arindam Jati, IBM Research;
(3) Nam H. Nguyen, IBM Research;
(4) Pankaj Dayama, IBM Research;
(5) Chandra Reddy, IBM Research;
(6) Wesley M. Gifford, IBM Research;
(7) Jayant Kalagnanam, IBM Research. Editor's note: this is part 3 of 5 of a study detailing the development of a tiny, fast AI model that delivers excellent accuracy. Read the rest below. Table of Links Abstract and 1. Introduction


2 TTM Components


3 TTM Workflows and 3.1 Pre-training Workflow

3.2 Fine-tuning Workflow



4 Experiments and Results

4.1 Experimental Setting
4.2 TTM’s Zero/Few-shot Performance
4.3 Computational Benefits of TTM and 4.4 TTM’s effectiveness in cross-channel and exogenous modeling
4.5 Ablation Studies and 4.6 Comparison with recent concurrent works



5 Conclusions and Future Work, and References

Appendix A 3 TTM Workflows TTM works in 2 stages: pre-train and fine-tune (Figure 1(a)). 3.1 Pre-training Workflow Multi-Resolution Pre-training via TTM Backbone The majority of the pre-training happens in the TTM backbone. The primary challenge with the proposed pre-training technique is that the pre-training data is diverse and has multiple resolutions. There are two main options for pre-training: conducting separate pre-training for each resolution type or pre-training using all resolution data collectively. While it’s common to train a model per resolution type to overcome challenges in learning diverse seasonal patterns, this leads to diminished training data for each resolution due to limited data availability. Consequently, this motivated the exploration of pre-training a single model using datasets from all resolutions. To achieve this, we propose the following 3 enhancements. Data Augmentation via Downsampling: A significant challenge in TS pre-training datasets is the scarcity of public datasets at specific resolutions. To overcome this, we employ a downsampling technique for high-resolution datasets, generating multiple datasets at lower resolutions. For example, from a one-second resolution dataset, we derive datasets at minute and hour resolutions. Note that, the original high-resolution dataset remains within the pool of pre-training datasets. This methodology significantly augments the number of datasets for each resolution which greatly improves the model performance (Section 4.5). Resolution Prefix Tuning: This technique explicitly learns and incorporates a new patch embedding as a prefix into the input data based on the input resolution type (see Figure 1(b)). Similar to the concept of prefix tuning [Li and Liang, 2021], this approach provides an explicit signal to the model about the resolution type for resolution-conditioned modeling. First, we map every resolution to a unique integer, which is then passed through an embedding layer to project it to the hidden dimension, hf. Subsequently, we expand the embedding across all channels to have a representation of shape c×1×hf. This module is optional for the TTM backbone, particularly beneficial when the context length (sl) is short. In these scenarios, automatically detecting the resolution becomes a challenge for the model. Hence, by explicitly fusing the resolution information as a prefix, we can enhance the model’s ability to learn effectively across resolutions. 3.2 Fine-tuning Workflow In the fine-tuning workflow, we deal with data from the target domain that has no overlap with the pre-training datasets. We have three options here: (a) In Zero-shot forecasting, we directly use the pre-trained model to evaluate on the test part of the target data, (b) In Few-shot forecasting, we utilize only a tiny portion (5-10%) of the train part of the target data to quickly update the pre-trained weights of the decoder and head, and subsequently, evaluate it on the test part, (c) In Full-shot forecasting, we fine-tune the pre-trained weights of the decoder and head on the entire train part of the target data, and then, evaluate on the test part. The backbone is completely frozen during fine-tuning, and still operates in a channel-independent univariate fashion. However, the TTM decoder can be fine-tuned via channel-mixing (for multivariate) or a channel-independent (for univariate) way based on the nature of the target data. If pure multivariate modeling is needed, then the channel-mixer block in all the TSMixer components (see Figure 1(b)) in the decoder gets enabled to explicitly capture the channel correlation between the channels. The forecast head and reverse normalization perform similar operations as in the pretraining stage. The fine-tuning also optimizes the forecasting objective with MSE loss. This thoughtful multi-level design choice ensures our backbone excels in channel-independent pre-training, enabling effective temporal correlation modeling across diverse datasets. Simultaneously, the decoder handles target-data-specific tasks like channel-correlation modeling and fine-tuning. In addition, if the target data has exogenous variables, then an exogenous mixer block is applied to the actual forecasts as explained next. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Vijay Ekambaram, IBM Research; (2) Arindam Jati, IBM Research; (3) Nam H. Nguyen, IBM Research; (4) Pankaj Dayama, IBM Research; (5) Chandra Reddy, IBM Research; (6) Wesley M. Gifford, IBM Research; (7) Jayant Kalagnanam, IBM Research. Authors: Authors: (1) Vijay Ekambaram, IBM Research; (2) Arindam Jati, IBM Research; (3) Nam H. Nguyen, IBM Research; (4) Pankaj Dayama, IBM Research; (5) Chandra Reddy, IBM Research; (6) Wesley M. Gifford, IBM Research; (7) Jayant Kalagnanam, IBM Research. Editor's note: this is part 3 of 5 of a study detailing the development of a tiny, fast AI model that delivers excellent accuracy. Read the rest below. Editor's note: this is part 3 of 5 of a study detailing the development of a tiny, fast AI model that delivers excellent accuracy. Read the rest below. Editor's note: this is part 3 of 5 of a study detailing the development of a tiny, fast AI model that delivers excellent accuracy. Read the rest below. Editor's note: this is part 3 of 5 of a study detailing the development of a tiny, fast AI model that delivers excellent accuracy. Read the rest below. Table of Links Abstract and 1. Introduction 2 TTM Components 3 TTM Workflows and 3.1 Pre-training Workflow

3.2 Fine-tuning Workflow 4 Experiments and Results

4.1 Experimental Setting
4.2 TTM’s Zero/Few-shot Performance
4.3 Computational Benefits of TTM and 4.4 TTM’s effectiveness in cross-channel and exogenous modeling
4.5 Ablation Studies and 4.6 Comparison with recent concurrent works 5 Conclusions and Future Work, and References

Appendix A Abstract and 1. Introduction Abstract and 1. Introduction Abstract and 1. Introduction 2 TTM Components 2 TTM Components 2 TTM Components 3 TTM Workflows and 3.1 Pre-training Workflow 3.2 Fine-tuning Workflow 3 TTM Workflows and 3.1 Pre-training Workflow 3 TTM Workflows and 3.1 Pre-training Workflow 3.2 Fine-tuning Workflow 3.2 Fine-tuning Workflow 4 Experiments and Results 4.1 Experimental Setting
4.2 TTM’s Zero/Few-shot Performance
4.3 Computational Benefits of TTM and 4.4 TTM’s effectiveness in cross-channel and exogenous modeling
4.5 Ablation Studies and 4.6 Comparison with recent concurrent works 4 Experiments and Results 4 Experiments and Results 4.1 Experimental Setting 4.2 TTM’s Zero/Few-shot Performance 4.3 Computational Benefits of TTM and 4.4 TTM’s effectiveness in cross-channel and exogenous modeling 4.5 Ablation Studies and 4.6 Comparison with recent concurrent works 4.1 Experimental Setting 4.2 TTM’s Zero/Few-shot Performance 4.3 Computational Benefits of TTM and 4.4 TTM’s effectiveness in cross-channel and exogenous modeling 4.5 Ablation Studies and 4.6 Comparison with recent concurrent works 5 Conclusions and Future Work, and References Appendix A 5 Conclusions and Future Work, and References 5 Conclusions and Future Work, and References Appendix A Appendix A 3 TTM Workflows TTM works in 2 stages: pre-train and fine-tune (Figure 1(a)). 3.1 Pre-training Workflow Multi-Resolution Pre-training via TTM Backbone The majority of the pre-training happens in the TTM backbone. The primary challenge with the proposed pre-training technique is that the pre-training data is diverse and has multiple resolutions. There are two main options for pre-training: conducting separate pre-training for each resolution type or pre-training using all resolution data collectively. While it’s common to train a model per resolution type to overcome challenges in learning diverse seasonal patterns, this leads to diminished training data for each resolution due to limited data availability. Consequently, this motivated the exploration of pre-training a single model using datasets from all resolutions. To achieve this, we propose the following 3 enhancements. Data Augmentation via Downsampling: A significant challenge in TS pre-training datasets is the scarcity of public datasets at specific resolutions. To overcome this, we employ a downsampling technique for high-resolution datasets, generating multiple datasets at lower resolutions. For example, from a one-second resolution dataset, we derive datasets at minute and hour resolutions. Note that, the original high-resolution dataset remains within the pool of pre-training datasets. This methodology significantly augments the number of datasets for each resolution which greatly improves the model performance (Section 4.5). Data Augmentation via Downsampling: Resolution Prefix Tuning: This technique explicitly learns and incorporates a new patch embedding as a prefix into the input data based on the input resolution type (see Figure 1(b)). Similar to the concept of prefix tuning [Li and Liang, 2021], this approach provides an explicit signal to the model about the resolution type for resolution-conditioned modeling. First, we map every resolution to a unique integer, which is then passed through an embedding layer to project it to the hidden dimension, hf. Subsequently, we expand the embedding across all channels to have a representation of shape c×1×hf. This module is optional for the TTM backbone, particularly beneficial when the context length (sl) is short. In these scenarios, automatically detecting the resolution becomes a challenge for the model. Hence, by explicitly fusing the resolution information as a prefix, we can enhance the model’s ability to learn effectively across resolutions. Resolution Prefix Tuning: 3.2 Fine-tuning Workflow In the fine-tuning workflow, we deal with data from the target domain that has no overlap with the pre-training datasets. We have three options here: (a) In Zero-shot forecasting, we directly use the pre-trained model to evaluate on the test part of the target data, (b) In Few-shot forecasting, we utilize only a tiny portion (5-10%) of the train part of the target data to quickly update the pre-trained weights of the decoder and head, and subsequently, evaluate it on the test part, (c) In Full-shot forecasting, we fine-tune the pre-trained weights of the decoder and head on the entire train part of the target data, and then, evaluate on the test part. The backbone is completely frozen during fine-tuning, and still operates in a channel-independent univariate fashion. However, the TTM decoder can be fine-tuned via channel-mixing (for multivariate) or a channel-independent (for univariate) way based on the nature of the target data. If pure multivariate modeling is needed, then the channel-mixer block in all the TSMixer components (see Figure 1(b)) in the decoder gets enabled to explicitly capture the channel correlation between the channels. The forecast head and reverse normalization perform similar operations as in the pretraining stage. The fine-tuning also optimizes the forecasting objective with MSE loss. This thoughtful multi-level design choice ensures our backbone excels in channel-independent pre-training, enabling effective temporal correlation modeling across diverse datasets. Simultaneously, the decoder handles target-data-specific tasks like channel-correlation modeling and fine-tuning. In addition, if the target data has exogenous variables, then an exogenous mixer block is applied to the actual forecasts as explained next. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Abstract

Too Long; Didn't Read

Make resilience your competitive advantage

This AI Model Learns to Forecast With Almost No Training—Here’s How

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

ZeroShape: Data Curation Details - Synthetic Training Dataset Generation and More

To AI or Not to AI: How Startups Can Wisely Add AI to Their Products

Executable Language and the Disappearance of Accountability

A Quick Guide to Quantization for LLMs

Autonomous Agents at Scale: Lessons from Redis + Symfony Scheduler

ZeroShape: Data Curation Details - Synthetic Training Dataset Generation and More

To AI or Not to AI: How Startups Can Wisely Add AI to Their Products

Executable Language and the Disappearance of Accountability

A Quick Guide to Quantization for LLMs

Autonomous Agents at Scale: Lessons from Redis + Symfony Scheduler

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps