Concatenated Masked Autoencoders as Spatial-Temporal Learner: Related Work

Table of Links

• Abstract & Introduction
• Related Work
• Method
• Experiments
• Conclusion & References

II. RELATED WORK

A. Masked Visual Modeling

Masked visual modeling method [14] learn representations from images disrupted by masking. Essentially, they function as denoising autoencoders, disrupting input signals and reconstructing the original undamaged signals to learn effective representations. This paradigm has spawned a range of derivatives, such as reconstructing masking pixels [15], [16] or restoring lost color channels [17].

The success of masked language modeling [18] in NLP’s self-supervised pre-training, along with the popularity of ViT [5], has sparked extensive research into the use of transformer-based architectures for masked visual modeling within the field of computer vision [19], [1], [20], [21], [22], [23]. BEiT [19], PeCo [20], and iBOT [23] naturally inherit the idea of BERT [18] and propose learning representations from images by predicting discrete tokens.

Some other research [24], [1], [22] focuses on using pixels as prediction targets, which is simpler. MAE [1] and SimMIM [22], by learning to reconstruct missing patches from randomly masked input image patches, achieve good visual representation. Moreover, MAE’s encoder only handles visible patches under a high mask ratio, significantly speeding up training and achieving better transfer performance. MAE has also been straightforwardly extended to the video domain [2], [3], reconstructing masked cubes. However, cube masking is sub-optimal for learning correspondences. SiamMAE [4] proposed an asymmetric masking strategy that retains past frames while reconstructing future frames containing a high proportion of masks, encouraging the model to model motion and match inter-frame correspondences. However, this strategy struggle to model long-term continuous motion information. Therefore, we propose a concatenated information channel masking reconstruction strategy that extends motion modeling to a theoretically infinite frame interval.

B. Contrastive Based Self-supervision

Effectively utilizing the temporal dimension is crucial in self-supervised video representation learning. Currently, there are various pretext tasks for pre-training, including predicting the future [25], [26], [27], segmenting pseudo ground truth [28], reconstructing future frames [10], [11], [29], tracking [30], [31], reference coloring [32], and temporal ordering [33], [34], [35], [36]. More advanced contrastive learning methods [37], [38] have been developed, which learn representations by modeling image similarities and dissimilarities [39], [40] or solely similarities [41], [42], [43]. However, these methods rely on large batches [40], multicrops [44], negative key queues [39], or custom strategies to prevent representation collapse [42]. Their performance greatly depends on the choice of image augmentation [40]. In contrast, our method is based on a simple masking and reconstruction pipeline [1].