ConvNeXt o‘z klassik AI modellarni qaytarib transformatorlarni

Autorlar : Zhuang Liu , Facebook AI Research (FAIR); UC Berkeley Hanzi Mao , Facebook AI Research (FAIR) , Facebook AI Research (FAIR) Chao-Yuan Wu Christoph Feichtenhofer, Facebook AI Research (FAIR) o‘zadi. , UC Berkeley Trevor Darrell Saining Xie, Facebook AI Research (FAIR) Kod: https://github.com/facebookresearch/ConvNeXt Autorlar : Zhuang Liu , Facebook AI Research (FAIR); UC Berkeley Hanzi Mao , Facebook AI Research (FAIR) Chao-Yuan Wu, Facebook AI Research (FAIR) Christoph Feichtenhofer, Facebook AI Research (FAIR) o‘zadi. , UC Berkeley Trevor Darrell Saining Xie, Facebook AI Research (FAIR) Kod: https://github.com/facebookresearch/ConvNeXt Abstraksiya Transformability Roaring 20s-i “TransformationNet efficiency” segmentini Vision Transformers (ViTs) qaytaradi, o‘z o‘z o‘z ConvNets o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ 1 Introduction O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z Biz 2012 yildan o‘z bilan o‘z bilan o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 42 O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z [Moment] “Moment” o‘z o‘z o‘z. [Tirmizi] o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. (Halladi o‘z o‘z o‘z o‘z) X (x) x (x) x (x) x (x) x (x) x (x) O‘z, [Tadi] o‘z o‘z o‘z. [Mobile] mobilni o‘z. [Kiladi qaytarga] [Radi] o‘z o‘z o‘z o‘z o‘z ] focused on different aspects of accuracy, efficiency and scalability, and popularized many useful design principles. 40 59 64 68 28 87 36 34 71 54 ConvNets'in kompjuter vizyonda dominasiyadi o‘z o‘z qilmadi: bir o‘z aplikasiyalarda, “sliding window” strateji vizual qilmadi, o‘z o‘z o‘z qilmadi. ConvNets o‘z bir o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z ]. For many decades, this has been the default use of ConvNets, generally on limited object categories such as digits [ ], faces [ O‘z, (Siz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z). O‘z, 2010 yildan o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z O‘z, , O‘z, ] further elevated ConvNets to the position of being the fundamental building block in a visual recognition system. 62 43 58 76 19 63 23 24 27 57 O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z Hierarchical Transformers employ a hybrid approach to bridge this gap. For example, the “sliding window” strategy ( Transformerlarni qaytarmadi, o‘z o‘z o‘z o‘z o‘z o‘z o‘z ConvNets. ] is a milestone work in this direction, demonstrating for the first time that Transformers can be adopted as a generic vision backbone and achieve state-of-the-art performance across a range of computer vision tasks beyond image classification. Swin Transformer’s success and rapid adoption also revealed one thing: the essence of convolution is not becoming irrelevant; rather, it remains much desired and has never faded. E.g 45 Bu perspektivda, Transformers'in kompjuter viziyalarda bir o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. [O‘z] o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. ], the speed can be optimized but the system becomes more sophisticated in design. On the other hand, it is almost ironic that a ConvNet already satisfies many of those desired properties, albeit in a straightforward, no-frills way. The only reason ConvNets appear to be losing steam is that (hierarchical) Transformers surpass them in many vision tasks, and the performance difference is usually attributed to the superior scaling behavior of Transformers, with multi-head self-attention being the key component. 55 45 U o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ ( Swin Transformer o‘z. ConvNets and hierarchical vision Transformers o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o E.g VS o‘z (Koron o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z). U o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ ( . Swin-T). Our exploration is directed by a key question: Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. Biz ConvNeXts o‘z bilan o‘z bir o‘z bir o‘z o‘z bir o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. ], objekt detection/segmentation on COCO [ [20] o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. ConvNeXt, bilan standard ConvNet modulardan qurulub, Transformersni bilan bilan bilan, scalability, robustity bilan bilan bilan qaytaradi. ConvNeXt o‘z standard ConvNets’larni qaytaradi. E.g E.g Transformers'da dizayn kararlar ConvNets'in performansını necə etkiladi? Konvets o‘z 17 44 92 Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 2. Modernizing a ConvNet: a Roadmap. Bu sekti, biz ResNet-i ConvNet-i bo‘ladi, o‘z Transformers-i siman. Biz iki model bo‘ladi flops, biri ResNet-50 / Swin-T rejimi, flops 4*. .*0 × 109. Simply, biz resnet-50 / Swin-T complexity modelning rezultatlarni prezidan. . 5×109 o‘z bilan ResNet-200 / Swin-B rejim o‘z 15 o‘z. C U bilan, eksplorativlarimizni qaytarmadi və Swin Transformer-i qaytarmadi, o‘z internetini standart ConvNet-i qaytarmadi. Biz resnet-50 modelini qaytarayadi. Biz o‘z bilan o‘z siman qaytarayadi transformatorlarni qaytarayadi, o‘z o‘z o‘z resnet-50 modelini qaytarayadi, o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. “Network modernization”ni ko‘zimizni, proceduralarni, rezultatlarni qo‘zimiz. network complexity o‘z bilan qilmadi, FLOPs o‘zimizni qilmadi, o‘zimizni o‘zimizni o‘zimizni, o‘zimizni o‘zimizni, o‘zimizni o‘zimizni, o‘zimizni o‘zimizni. 2 2.1 Strateji tekniklar Siz sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga sizga siz , U o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ ] and Swin Transformer’s [ Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. ], data augmentation techniques such as Mixup [ [Madix] Kadix o‘ladi [Radiya] o‘z o‘z o‘z o‘z [Kiladi qaytarga] (Stochastic depth) (Stochastic depth) (Stochastic depth) (Stochastic depth) O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z ]. The complete set of hyperparameters we use can be found in Appendix O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 78.8 % (+2.7 %), o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z 7 81 73 45 46 90 89 14 91 36 69 A1 o‘z 1 2. Makro dizayn Biz Swin Transformers makro network dizaynini analizadi. Swin Transformers ConvNets [ O‘z, ] to use a multi-stage design, where each stage has a different feature map resolution. There are two interesting design considerations: the stage compute ratio, and the “stem cell” structure. 28 65 ResNet-da kompjuterizmin bilan qaytaradi bilan empirikaldi. “res4” faza o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ Changing stage compute ratio. Biz resnet-50-da (3, 3, 9, 3), bu da FLOP-larni Swin-T. Bu model bilanni 78.8%-79.4% bo‘ladi. , Bilmiz, daha optimistic design o‘z o‘z. 53 54 Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. Typically, the stem cell design is concerned with how the input images will be processed at the network’s beginning. Due to the redundancy inherent in natural images, a common stem cell will aggressively downsample the input images to an appropriate feature map size in both standard ConvNets and vision Transformers. The stem cell in standard ResNet contains a 7×7 convolution layer with stride 2, followed by a max pool, which results in a 4× downsampling of the input images. In vision Transformers, a more aggressive “patchify” strategy is used as the stem cell, which corresponds to a large kernel size (e.g. kernel size = 14 or 16) and non-overlapping convolution. Swin Transformer uses a similar “patchify” layer, but with a smaller patch size of 4 to accommodate the architecture’s multi-stage design. We replace the ResNet-style stem cell Changing stem to “Patchify”. Patchify layer 4x4, step 4 convolutional layer. Qasib bilan 79.4%. 79.5%. This suggests that the stem cell in a ResNet may be substituted with a simpler “patchify” layer à la ViT which will result in similar performance. × Biz “patchify stam”ni kullani (4 4 o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 2.2.2 Kilmadi Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z [...] ResNeXt‘nin guild prinsipini “tadi daha çox gruplar, genişlik bo‘ladi”. ResNeXt bilan 3x3 conv layerini bo‘ladi. 87 In our case we use depthwise convolution, a special case of grouped convolution where the number of groups equals the number of channels. Depthwise conv has been popularized by MobileNet [ XOXOXOxOxOxOxOxOxOxOxOxOxOxOxOxOxOxOxOxOxOxO Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 34 11 basis, Bu kombinasyonlar 1x1 convlar, 1x1 convlar, o‘z diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviziyalar diviz I.E Biz resnext strategiyasına baxmay, network width o‘z Swin-T’nin 64-96 kanallarni (Swin-T’nin 64-96 kanallarni) bo‘ladi. Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. Bottleneck qaytaradi Transformer bloklarni o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z ., MLP blokun gizli dimensiyani 4 o‘z daha genişdir (Figure 6). ). i.e 4 Bu Transformer dizaynini ConvNets-da 4 ekspansiya ratio ilə inverted bottleneck dizaynga qoysan. [O‘z] o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. , O‘z. 61 70 71 Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z (a) to (b) illustrate the configurations. Despite the increased FLOPs for the deep-convolution layer, this change reduces the entire network FLOPs to 4.6G, due to the significant FLOPs reduction in the downsampling residual blocks’ shortcut 1×1 conv layer. Interestingly, this results in slightly improved performance (80.5% to 80.6%). In the ResNet-200/ Swin-B regimen, this step brings even more gain (81.9% to 82.6%) also with reduced FLOPs. 3 We will now use inverted bottlenecks. 2.5.Kernel sizing sizing Bu o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ , [VGGNet] Gold standard (VGGNet-i popularizadi) [ ] ]) mini-kernel-size (3x3) conv layers, ki o‘z modern GPU’larda efficient hardware implementations var. [ Men Swin Transformers o‘z lokomotiv blokni reintroduced, window size is at least 7×7, significantly larger than the ResNe(X)t kernel size of 3×3. Here we revisit the use of large kernel-sized convolutions for ConvNets. 40 68 65 41 Bir o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. (b) to (c)). That is a design decision also evident in Transformers: the MSA block is placed prior to the MLP layers. As we have an inverted bottleneck block, this is a natural design choice — the complex/inefficient modules (MSA, large-kernel conv) will have fewer channels, while the efficient, dense 1×1 layers will do the heavy lifting. This intermediate step reduces the FLOPs to 4.1G, resulting in a temporary performance degradation to 79.9%. Moving up depthwise conv layer. 3 Biz 3, 5, 7, 9 və 11. Network performans 79.9% (3×3) to 80.6% (7×7), o‘z network’in FLOPs qilmadi qilmadi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi qilmidi q Increasing the kernel size. × Biz 7 o‘zingizimiz. 7 depthwise conv in each block. Biz o‘z bilan o‘z bilan o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 2.6 Mikro dizayn Bu sekti, biz mikroskala disa arkitektonik diferenziyalarni qaytaradi — burning eksplorations o‘z layer-level, konsantrasiyadi o‘z Activation funktions and normalization layers. U o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ ] is still extensively used in ConvNets due to its simplicity and efficiency. ReLU is also used as an activation function in the original Transformer paper [ Gaus-sian Error Linear Unit, o‘z GELU. ], o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z ] and OpenAI’s GPT-2 [ Biz ReLU GELU o‘z ConvNet’imizda o‘z bilan qaytariladi, amma o‘z bilan qaytariladi (80.6%). Replacing ReLU with GELU 49 77 32 18 52 Bir Transformer-i ResNet blokni gapni bilan Transformers o‘z daha az aktivasyon funksiyaları var. Transformer-i blokni linear embedding layers, projektion layer, and two linear layers in a MLP block. There is only one activation function present in the MLP block. Fewer activation functions. 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar, 1x1 konvulsiyalar. Bu prosesni qaytaradi 0,7% to 81,3%, o‘z o‘z 1x1 qaytaradi, o‘z Transformer block stilini qaytaradi. 4 Biz o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. Transformer blocks usually have fewer normalization layers as well. Here we remove two BatchNorm (BN) layers, leaving only one BN layer before the conv 1 × 1 layers. This further the performance to 81.4%, already surpassing Swin-T’s result. Note that we have even fewer normalization layers per block than Transformers, as empirically we find that adding one additional BN layer at the beginning of the block does not improve the performance. Fewer normalization layers. Boostlar O‘zardlar ( Bu o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘ Men o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. , O‘z, ] techniques, but BN has remained the preferred option in most vision tasks. On the other hand, the simpler Layer Normalization [ ] (LN) has been used in Transformers, resulting in good performance across different application scenarios. Substituting BN with LN. 38 84 60 75 83 5 U bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan [...] ]. With all the modifications in network architecture and training techniques, here we revisit the impact of using LN in place of BN. We observe that our ConvNet model does not have any difficulties training with LN; in fact, the performance is slightly better, obtaining an accuracy of 81.5%. 83 From now on, we will use one LayerNorm as our choice of normalization in each residual block. In ResNet, the spatial downsampling is achieved by the residual block at the start of each stage, using 3×3 conv with stride 2 (and 1×1 conv with stride 2 at the shortcut connection). In Swin Transformers, a separate downsampling layer is added between stages. We explore a similar strategy in which we use 2×2 conv layers with stride 2 for spatial downsampling. This modification Separate downsampling layers. surprisingly leads to diverged training. Further investigation shows that, adding normalization layers wherever spatial resolution is changed can help stablize training. These include several LN layers also used in Swin Transformers: one before each downsampling layer, one after the stem, and one after the final global average pooling. We can improve the accuracy to 82.0%, significantly exceeding Swin-T’s 81.3%. We will use separate downsampling layers. This brings us to our final model, which we have dubbed ConvNeXt. A comparison of ResNet, Swin, and ConvNeXt block structures can be found in Figure . A comparison of ResNet-50, Swin-T and ConvNeXt-T’s detailed architecture specifications can be found in Table . 4 9 4 9 We have finished our first “playthrough” and discovered ConvNeXt, a pure ConvNet, that can outperform the Swin Transformer for ImageNet-1K classification in this compute regime. It is worth noting that In addition, Our Con-vNeXt model has approximately the same FLOPs, #params., throughput, and memory use as the Swin Transformer, but does not require specialized modules such as shifted window attention or relative position biases. Closing remarks. Bütün dizayn seçimlər bilan qaytaradi Transformers Vision. Bu dizaynlar ConvNet literaturada o‘z bilan yeni deyilmir – o‘z o‘z bilan o‘z qilmadi, amma o‘z kolektiv, son dekadda. These findings are encouraging but not yet completely convincing — our exploration thus far has been limited to a small scale, but vision Transformers’ scaling behavior is what truly distinguishes them. Additionally, the question of whether a ConvNet can compete with Swin Transformers on downstream tasks such as object detection and semantic segmentation is a central concern for computer vision practitioners. In the next section, we will scale up our ConvNeXt models both in terms of data and model size, and evaluate them on a diverse set of visual recognition tasks. 3. Empirical Evaluations on ImageNet We construct different ConvNeXt variants, ConvNeXt-T/S/B/L, to be of similar complexities to Swin-T/S/B/L [ ]. ConvNeXt-T/B is the end product of the “modernizing” pro-cedure on ResNet-50/200 regime, respectively. In addition, we build a larger ConvNeXt-XL to further test the scalabil-ity of ConvNeXt. The variants only differ in the number of channels , and the number of blocks Bina o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 45 C B ConvNeXt-T: C = (96*,* 192*,* 384*,* 768), B = (3*,* 3*,* 9*,* 3) ConvNeXt-S: = (96*,* 192*,* 384*,* 768), = (3*,* 3*,* 27*,* 3) C B ConvNeXt-B: = (128*,* 256*,* 512*,* 1024), = (3*,* 3*,* 27*,* 3) C B ConvNeXt-L: C = (192*,* 384*,* 768*,* 1536), B = (3*,* 3*,* 27*,* 3) ConvNeXt-XL: C = (256*,* 512*,* 1024*,* 2048), B = (3*,* 3*,* 27*,* 3) 3.1. Settings The ImageNet-1K dataset consists of 1000 object classes with 1.2M training images. We report ImageNet-1K top-1 accuracy on the validation set. We also conduct pre-training on ImageNet-22K, a larger dataset of 21841 classes (a superset of the 1000 ImageNet-1K classes) with ∼14M images for pre-training, and then fine-tune the pre-trained model on ImageNet-1K for evaluation. We summarize our training setups below. More details can be found in Appendix . A We train ConvNeXts for 300 epochs using AdamW [ ] with a learning rate of 4e-3. There is a 20-epoch linear warmup and a cosine decaying schedule afterward. We use a batch size of 4096 and a weight decay of 0.05. For data augmentations, we adopt common schemes including Mixup [ ], Cutmix [ ], RandAugment [ O‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z ]. We regularize the networks with Stochastic Depth [ ] and Label Smoothing [ ]. Layer Scale [ ] of initial value 1e-6 is applied. We use Exponential Moving Average (EMA) [ ] as we find it alleviates larger models’ overfitting. Training on ImageNet-1K. 46 90 89 14 91 37 69 74 51 We pre-train ConvNeXts on ImageNet-22K for 90 epochs with a warmup of 5 epochs. We do not use EMA. Other settings follow ImageNet-1K. Pre-training on ImageNet-22K. Biz ImageNet-22K pre-trained modelini 30 epochlarni imageNet-1K. Biz AdamW, 5e-5 learning rate, cosine learning rate schedule, layer-wise learning rate decay kullandik. , ], no warmup, a batch size of 512, and weight decay of 1e-8. The default pre-training, fine-tuning, and testing resolution is 2242. Additionally, we fine-tune at a larger resolution of 3842, for both ImageNet-22K and ImageNet-1K pre-trained models. Fine-tuning on ImageNet-1K. 6 12 U bilan ViTs/Swin Transformers, ConvNeXts o‘z bilan o‘z bilan o‘z bilan o‘z networkni bilan konvolisyonni, o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 3.2 Rezultatlar Table (upper) shows the result comparison with two recent Transformer variants, DeiT [ ] and Swin Transformers [ ], as well as two ConvNets from architecture search - RegNets [ ], EfficientNets [ ] and EfficientNetsV2 [ ]. ConvNeXt competes favorably with two strong ConvNet baselines (RegNet [ ] and Efficient-Net [ ]) in terms of the accuracy-computation trade-off, as well as the inference throughputs. ConvNeXt also outperforms Swin Transformer of similar complexities KonvNeXt-T, 0,8%, ConvNeXts, ConvNeXt-T, ConvNeXts, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt, ConvNeXt ImageNet-1K. 1 73 45 54 71 72 54 71 across the board, A highlight from the results is ConvNeXt-B at 3842 : it outperforms Swin-B by 0.6% (85.1% vs. 84.5%), but with 12.5% higher inference throughput (95.7 vs. 85.1 image/s). We note that the FLOPs/throughput advantage of ConvNeXt-B over Swin-B becomes larger when the resolution increases from 2242 to 3842 . Additionally, we observe an improved result of 85.5% when further scaling to ConvNeXt-L. We present results with models fine-tuned from ImageNet-22K pre-training at Table Bu eksperiment o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z Biz ConvNeXt-XL modelimizni 87.8%, ConvNeXt-L modelimizni 3842, ConvNeXt-XL modelimizni 3842, ConvNeXt-XL modelimizni 87.8%, ConvNeXt-L modelimizni 3842, ConvNeXt-XL modelimizni 3842, ConvNeXt-XL modelimizni 3842, ConvNeXt-XL modelimizni 3848, ConvNeXt-XL modelimizni 3848, ConvNeXt-XL modelimizni 3848, ConvNeXt-XL modelimizni 3848, ConvNeXt-XL modelimizni 3848, ConvNeXt-XL modelimizni 3848, ConvNeXt-L modelimizni 3848, ConvNeXt-XL modelimizni 3848, ImageNet-22K. 1 O‘z On ImageNet-1K, EfficientNetV2-L, a searched architecture equipped with advanced modules (such as Squeeze-and-Excitation [ Men, ImageNet-22K pre-training, ConvNeXt o‘z EfficientNetV2 o‘z, daha o‘z o‘z o‘z o‘z o‘z o‘z o‘z. 35 In Appendix , we discuss robustness and out-of-domain generalization results for ConvNeXt. B 3.3. Isotropic ConvNeXt . ViT VS o‘z In this ablation, we examine if our ConvNeXt block design is generalizable to ViT-style [ ] isotropic architectures which have no downsampling layers and keep the same feature resolutions ( . 14×14) at all depths. We construct isotropic ConvNeXt-S/B/L using the same feature dimensions as ViT-S/B/L (384/768/1024). Depths are set at 18/18/36 to match the number of parameters and FLOPs. The block structure remains the same (Fig. ). We use the supervised training results from DeiT [ [O’z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z] ] for ViT-L, as they employ improved training procedures over the original ViTs [ ]. ConvNeXt models are trained with the same settings as before, but with longer warmup epochs. Results for ImageNet-1K at 2242 resolution are in Table . We observe ConvNeXt can perform generally on par with ViT, showing that our ConvNeXt block design is competitive when used in non-hierarchical models. 20 e.g 4 73 26 20 2 4. Empirical Evaluation on Downstream Tasks We fine-tune Mask R-CNN [ ] and Cascade Mask R-CNN [ ] on the COCO dataset with ConvNeXt backbones. Following Swin Transformer [ ], we use multi-scale training, AdamW optimizer, and a 3× schedule. Further details and hyperparameter settings can be found in Appendix . Object detection and segmentation on COCO. 27 9 45 A.3 Tablalar shows object detection and instance segmentation results comparing Swin Transformer, ConvNeXt, and traditional ConvNet such as ResNeXt. Across different model complexities, ConvNeXt achieves on-par or better performance than Swin Transformer. When scaled up to bigger models (ConvNeXt-B/L/XL) pre-trained on ImageNet-22K, in many cases ( . +1.0 AP) than Swin Transformers in terms of box and mask AP. 3 ConvNeXt is significantly better e.g We also evaluate ConvNeXt backbones on the ADE20K semantic segmentation task with UperNet [ ]. All model variants are trained for 160K iterations with a batch size of 16. Other experimental settings follow [ ] (see Appendix for more details). In Table , we report validation mIoU with multi-scale testing. ConvNeXt models can achieve competitive performance across different model capacities, further validating the effectiveness of our architecture design. Semantic segmentation on ADE20K. 85 6 A3 o‘z 4 Under similar FLOPs, models with depthwise convolutions are known to be slower and consume more memory than ConvNets with only dense convolutions. Remarks on model efficiency. It is natural to ask whether the design of ConvNeXt will render it practically inefficient. As demon-strated throughout the paper, the inference throughputs of ConvNeXts are comparable to or exceed that of Swin Trans-formers. This is true for both classification and other tasks requiring higher-resolution inputs (see Table , for com-parisons of throughput/FPS). Furthermore, we notice that training ConvNeXts requires less memory than training Swin Transformers. For example, training Cascade Mask-RCNN using ConvNeXt-B backbone consumes 17.4GB of peak memory with a per-GPU batch size of 2, while the reference number for Swin-B is 18.5GB. In comparison to vanilla ViT, both ConvNeXt and Swin Transformer exhibit a more favor-able accuracy-FLOPs trade-off due to the local computations. It is worth noting that this improved efficiency is a result of the , and is not directly related to the self-attention mechanism in vision Transformers. 1 3 ConvNet inductive bias 5. Related Work In both the pre- and post-ViT eras, the hybrid model combining convolutions and self-attentions has been actively studied. Prior to ViT, the focus was on augmenting a ConvNet with self-attention/non-local modules [ , O‘z, , ] to capture long-range dependencies. The original ViT [ ] first studied a hybrid configuration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an ex-plicit [ , , , , , ] or implicit [ ] fashion. Hybrid models. 8 55 66 79 20 15 16 21 82 86 88 45 Han [ ] show that local Transformer attention is equivalent to inhomogeneous dynamic depthwise conv. The MSA block in Swin is then replaced with a dynamic or regular depthwise convolution, achieving comparable performance to Swin. A concurrent work, ConvMixer [ ] demonstrates that, in small-scale settings, depthwise convolution can be used as a promising mixing strategy. ConvMixer uses a smaller patch size to achieve the best results, making the throughput much lower than other baselines. GFNet [ ] adopts Fast Fourier Transform (FFT) for token mixing. FFT is also a form of convolution, but with a global kernel size and circular padding. Unlike many recent Transformer or ConvNet designs, one primary goal of our study is to provide an in-depth look at the process of modernizing a standard ResNet and achieving state-of-the-art performance. Recent convolution-based approaches. et al. 25 4 56 6. Conclusions In the 2020s, vision Transformers, particularly hierarchical ones such as Swin Transformers, began to overtake ConvNets as the favored choice for generic vision backbones. The widely held belief is that vision Transformers are more accurate, efficient, and scalable than ConvNets. We propose ConvNeXts, a pure ConvNet model that can compete favorably with state-of-the-art hierarchical vision Transformers across multiple computer vision benchmarks, while retaining the simplicity and efficiency of standard ConvNets. In some ways, our observations are surprising while our ConvNeXt model itself is not completely new — many design choices have all been examined separately over the last decade, but not collectively. We hope that the new results reported in this study will challenge several widely held views and prompt people to rethink the importance of convolution in computer vision. We thank Kaiming He, Eric Mintun, Xingyi Zhou, Ross Girshick, and Yann LeCun for valuable discussions and feedback. Acknowledgments. Bu addon In this Appendix, we provide further experimental details (§ ), robustness evaluation results (§ ), more modernization experiment results (§ Bilmiz, internetni o‘z bilan o‘z. ). We further benchmark model throughput on A100 GPUs (§ ). Finally, we discuss the limitations (§ ) and societal impact (§ ) of our work. A B C D E F G A. Experimental Settings A.1. ImageNet (Pre-)training Biz ConvNeXts 'ImageNet-1K training and ImageNet-22K pre-training settings va tablo. . The settings are used for our main results in Table (Section ). All ConvNeXt variants use the same setting, except the stochas-tic depth rate is customized for model variants. 5 1 3.2 For experiments in “modernizing a ConvNet” (Section ), we also use Table ’s setting for ImageNet-1K, except EMA is disabled, as we find using EMA severely hurts models with BatchNorm layers. 2 5 For isotropic ConvNeXts (Section ), the setting for ImageNet-1K in Table is also adopted, but warmup is ex-tended to 50 epochs, and layer scale is disabled for isotropic ConvNeXt-S/B. The stochastic depth rates are 0.1/0.2/0.5 for isotropic ConvNeXt-S/B/L. 3.3 A A.2. ImageNet Fine-tuning We list the settings for fine-tuning on ImageNet-1K in Table . The fine-tuning starts from the final model weights obtained in pre-training, without using the EMA weights, even if in pre-training EMA is used and EMA accuracy is reported. This is because we do not observe improvement if we fine-tune with the EMA weights (consistent with observa-tions in [ ]). The only exception is ConvNeXt-L pre-trained on ImageNet-1K, where the model accuracy is significantly lower than the EMA accuracy due to overfitting, and we select its best EMA model during pre-training as the starting point for fine-tuning. 6 73 In fine-tuning, we use layer-wise learning rate decay [ , ] with every 3 consecutive blocks forming a group. When the model is fine-tuned at 3842 resolution, we use a crop ratio of 1.0 (i.e., no cropping) during testing following [ , , [Madi] 6 12 2 74 80 instead of 0.875 at 2242. A.3. Downstream Tasks For ADE20K and COCO experiments, we follow the training settings used in BEiT [ «Swin» o‘z o‘z. ]. We also use MMDetection [ ] and MMSegmentation [ ] toolboxes. We use the final model weights (instead of EMA weights) from ImageNet pre-training as network initializations. 6 45 10 13 We conduct a lightweight sweep for COCO experiments including learning rate {1e-4, 2e-4}, layer-wise learning rate decay [ ] {0.7, 0.8, 0.9, 0.95}, and stochastic depth rate {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}. We fine-tune the ImageNet-22K pre-trained Swin-B/L on COCO using the same sweep. We use the official code and pre-trained model weights [ ]. 6 3 The hyperparameters we sweep for ADE20K experiments include learning rate {8e-5, 1e-4}, layer-wise learning rate decay {0.8, 0.9}, and stochastic depth rate {0.3, 0.4, 0.5}. We report validation mIoU results using multi-scale testing. Additional single-scale testing results are in Table . 7 B. Robustness Evaluation Additional robustness evaluation results for ConvNeXt models are presented in Table . We directly test our ImageNet-1K trained/fine-tuned classification models on sev-eral robustness benchmark datasets such as ImageNet-A [ ], ImageNet-R [ ], ImageNet-Sketch [ ] and ImageNet-C/C¯ [ , ] datasets. We report mean corruption error (mCE) for ImageNet-C, corruption error for ImageNet-C¯, and top-1 Accuracy for all other datasets. 8 33 30 78 31 48 ConvNeXt (in particular the large-scale model variants) exhibits promising robustness behaviors, outperforming state-of-the-art robust transformer models [ ] on several benchmarks. With extra ImageNet-22K data, ConvNeXt-XL demonstrates strong domain generalization capabilities ( 69.3%/68.2%/55.0% bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan bilan. 47 e.g C. Modernizing ResNets: detailed results Here we provide detailed tabulated results for the experiments, at both ResNet-50 / Swin-T and ResNet-200 / Swin-B regimes. The ImageNet-1K top-1 ac-curacies and FLOPs for each step are shown in Table and . ResNet-50 regime experiments are run with 3 ran-dom seeds. mod-ernization 10 11 For ResNet-200, the initial number of blocks at each stage is (3, 24, 36, 3). We change it to Swin-B’s (3, 3, 27, 3) at the step of changing stage ratio. This drastically reduces the FLOPs, so at the same time, we also increase the width from 64 to 84 to keep the FLOPs at a similar level. After the step of adopting depthwise convolutions, we further increase the width to 128 (same as Swin-B’s) as a separate step. The observations on the ResNet-200 regime are mostly consistent with those on ResNet-50 as described in the main paper. One interesting difference is that inverting dimensions brings a larger improvement at ResNet-200 regime than at ResNet-50 regime (+0.79% . +0.14%). The performance gained by increasing kernel size also seems to saturate at kernel size 5 instead of 7. Using fewer normalization layers also has a bigger gain compared with the ResNet-50 regime (+0.46% . +0.14%). vs vs D. Detailed Architectures We present a detailed architecture comparison between ResNet-50, ConvNeXt-T and Swin-T in Table . For differ-ently sized ConvNeXts, only the number of blocks and the number of channels at each stage differ from ConvNeXt-T (see Section for details). ConvNeXts enjoy the simplic-ity of standard ConvNets, but compete favorably with Swin Transformers in visual recognition. 9 3 E. Benchmarking on A100 GPUs Following Swin Transformer [ ], the ImageNet models’ inference throughputs in Table are benchmarked using a V100 GPU, where ConvNeXt is slightly faster in inference than Swin Transformer with a similar number of parameters. We now benchmark them on the more advanced A100 GPUs, which support the TensorFloat32 (TF32) tensor cores. We employ PyTorch [ [ ] Versiyona 1.10 o‘z «Channel Last» memori layoutini kullani. ] for further speedup. 45 1 50 22 We present the results in Table . Swin Transformers and ConvNeXts both achieve faster inference throughput than V100 GPUs, but ConvNeXts’ advantage is now significantly greater, sometimes Bu preliminary studyada ConvNeXt, standard ConvNet modularlar və design simple, o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z modern hardware modularlar. 12 up to 49% faster F. Limitations Biz ConvNeXt, o‘z ConvNet modelini, o‘z bir Transformer vision jerarqik, o‘z image classification, object detection, instance and semantic segmentation tasks o‘z o‘z o‘z o‘z o‘z o‘z o‘z bir o‘z o‘z evaluation tasks o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z G. Societal Impact In the 2020s, research on visual representation learn-ing began to place enormous demands on computing re-sources. While larger models and datasets improve per-formance across the board, they also introduce a slew of challenges. ViT, Swin, and ConvNeXt all perform best with their huge model variants. Investigating those model designs inevitably results in an increase in carbon emissions. One important direction, and a motivation for our paper, is to strive for simplicity — with more sophisticated modules, the network’s design space expands enormously, obscuring critical components that contribute to the performance dif-ference. Additionally, large models and datasets present issues in terms of model robustness and fairness. Further investigation on the robustness behavior of ConvNeXt vs. Transformer will be an interesting research direction. In terms of data, our findings indicate that ConvNeXt models benefit from pre-training on large-scale datasets. While our method makes use of the publicly available ImageNet-22K dataset, individuals may wish to acquire their own data for pre-training. A more circumspect and responsible approach to data selection is required to avoid potential concerns with data biases. References [1] PyTorch Vision Models. . Accessed: 2021-10-01. https://pytorch.org/ vision/stable/models.html [2] GitHub repository: Swin transformer. . , 2021. https://github com/microsoft/Swin-Transformer [3] GitHub repository: Swin transformer for object detection. , 2021. https://github.com/SwinTransformer/Swin-Transformer-Object-Detection [4] Anonymous. Patches are all you need? 2021 yildan. Openreview [5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. , 2016. arXiv:1607.06450 [6] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training image transformers. , 2021. arXiv:2106.08254 [7] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. , 2021. NeurIPS [8] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In , 2019. ICCV [9] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delv-ing into high quality object detection. In 2018 yildan. CVPR [10] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tian-heng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. , 2019. arXiv:1906.07155 [11] François Chollet. Xception: Deep learning with deeply separable convolutions. 2017 yildan. CVPR [12] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christo-pher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In , 2020. ICLR MMSegmentation: Openmm-lab semantic segmentation toolbox and benchmark. https: 2020 yildan. //github.com/open- mmlab/mmsegmentation [14] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In , 2020. CVPR Workshops [15] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. 2021 yildan. NeurIPS [16] Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Mor-cos, Giulio Biroli, and Levent Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. , 2021. ICML [17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 yildan. CVPR o‘z [18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans-formers for language understanding. In , 2019. NAACL [19] Piotr Dollár, Serge Belongie, and Pietro Perona. The fastest pedestrian detector in the west. In 2010 yildan. BMVC o‘z [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In , 2021. ICLR-da [21] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. , 2021. ICCV Vitaly Fedyunin. Tutorial: Channel last memory format in PyTorch. , 2021. Accessed: 2021-10-01. https://pytorch.org/tutorials/ intermediate/memory_format_tutorial.html [23] Ross Girshick. Fast R-CNN. In , 2015. ICCV [24] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 yildan. CVPR [25] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Ji-aying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dy-namic weight. , 2021. arXiv:2106.04263 [26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. , 2021. arXiv:2111.06377 [27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-shick. Mask R-CNN. In 2017 yildan. ICCV [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. , 2016. CVPR [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In , 2016. ECCV o‘z [30] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada-vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust-ness: A critical analysis of out-of-distribution generalization. In , 2021. ICCV [31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In , 2018. ICLR Dan Hendrycks və Kevin Gimpel. Gaussian error linear units (gelus). 2016 yildan. ArXiv: 1606.08415 [33] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In , 2021. CVPR [34] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An-dreetto, and Hartwig Adam. MobileNets: Efficient con-volutional neural networks for mobile vision applications. , 2017. O‘zili: 1804-4861 [55] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation network. 2018 yildan. CVPR o‘z [36] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kil-ian Q Weinberger. Densely connected convolutional networks. In , 2017. CVPR o‘z [37] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In 2016 yildan. ECCV o‘z [38] Sergey Ioffe. batch renormalization: Min-batch dependency reducing in batch-normalized models. , 2017. NeurIPS [39] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. 2020 yildan. ECCV [40] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In 2012 yildan. Neuroza [41] Andrew Lavin and Scott Gray. Fast algorithms for convolu-tional neural networks. In , 2016. CVPR [42] Yann LeCun, Bernhard Boser, John S Denker, Donnie Hen-derson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Bu 1989. Neural kompjuter [43] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recogni-tion. 1998 yildan. IEEE proceduralar [44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In . 2014. ECCV Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformator: Hierarchical vision transformator using shifted windows. [46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In , 2019. ICLR [47] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. , 2021. arXiv preprint arXiv:2105.07926 [48] Eric Mintun, Alexander Kirillov, and Saining Xie. On in-teraction between augmentations and corruptions in natural corruption robustness. , 2021. NeurIPS [49] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In 2010 yildan. ICML Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: A imperative style, high-performance deep learning library. , 2019. Neuroza [51] Boris T Polyak and Anatoli B Juditsky. Stochastic approximation. , 1992. SIAM Journal on Control and Optimization [52] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. [53] Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In , 2019. ICCV [54] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaim-ing He, and Piotr Dollár. Network design spaces. 2020 yildan. CVPR [55] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. 2019 yildan. NeurIPS [56] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. , 2021. Neuroza [57] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. , 2015. NeurIPS [58] Henry A Rowley, Shumeet Baluja, and Takeo Kanade. Neural network-based facial detection. 1998 yildan. TPAMI Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San-jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, Li Fei-Fei. , 2015. IJCV [60] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In , 2016. NeurIPS [61] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh-moginov, Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 yildan. CVPR [62] Pierre Sermanet, David Eigen, Xiang Zhang, Michael Math-ieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In 2014 yildan. ICLR [63] Pierre Sermanet, Koray Kavukcuoglu, Soumith Chintala, and Yann LeCun. Pedestrian detection with unsupervised multi-stage feature learning. , 2013. CVPR o‘z [64] Karen Simonyan and Andrew Zisserman. Two-stream convo-lutional networks for action recognition in videos. , 2014. NeurIPS [65] Karen Simonyan and Andrew Zisserman. Very deep convolu-tional networks for large-scale image recognition. In 2015 yildan. ICLR [66] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. , 2021. CVPR o‘z [67] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. , 2021. arXiv preprint arXiv:2106.10270 [68] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 yildan. CVPR [69] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the incep-tion architecture for computer vision. In , 2016. CVPR [70] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In 2019 yildan. CVPR o‘z [71] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In , 2019. ICML [72] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. , 2021. ICML o‘z [73] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through atten-tion. , 2020. arXiv:2012.12877 o‘z [74] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. , 2021. ICCV bilan [75] Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky. In-stance normalization: The missing ingredient for fast styliza-tion. , 2016. ArXiv : 1607.08022 [76] Régis Vaillant, Christophe Monrocq, and Yann Le Cun. Orig-inal approach for the localisation of objects in images. , 1994. Vizyon, image və sinyall processing [77] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In 2017 yildan. NeurIPS [78] Haohan Wang, Songwei Ge, Eric P Xing, and Zachary C Lipton. Kilmizgan global representations o‘z lokalizing predictive power. 2019 yildan. Neuroza [79] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In 2018 yildan. CVPR o‘z [80] Ross Wightman. GitHub repository: Pytorch image mod-els. 2019 yildan. https://github.com/rwightman/pytorch-image-models [81] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet o‘z qaytaradi: A improved training procedure in hour. , 2021. arXiv:2110.00476 [82] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Convolutions to vision transformers. , 2021. ICCV bilan [83] Yuxin Wu and Kaiming He. Group normalization. In 2018 yildan. ECCV o‘z [84] Yuxin Wu və Justin Johnson. batch-normda "batch" qaytaradi. , 2021. O‘zili: 255.07576 [85] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. 2018 yildan. ECCV o‘z [86] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Hara konvolisyonlar transformlarni daha yaxşı baxadi. , 2021. Neuroza [87] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. 2017 yildan. CVPR o‘z [88] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. 2021 yildan. ICCV bilan [89] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu-larization strategy to train strong classifiers with localizable features. In 2019 yildan. ICCV bilan [90] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 2018 yildan. ICLR-da [91] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random deleting data augmentation. 2020 yildan. O‘z [92] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understand-ing of scenes through the ADE20K dataset. , 2019. IJCV Bu dokimi CC by 4.0 Deed (Attribution 4.0 International) lisenzonda arxiv qaytaradi. Bu dokimi CC by 4.0 Deed (Attribution 4.0 International) lisenzonda arxiv qaytaradi.