如何ConvNeXt证明经典AI模型仍然可以击败变形器

作者 : Zhuang Liu , Facebook AI 研究(FAIR); UC 伯克利 Hanzi Mao , Facebook AI 研究(FAIR) Chao-Yuan Wu, Facebook AI 研究(FAIR) 克里斯托夫·费赫滕霍夫(Christoph Feichtenhofer),Facebook AI Research(FAIR) 特雷沃·达雷尔, UC伯克利 Saining Xie, Facebook AI 研究(FAIR) 代码: https://github.com/facebookresearch/ConvNeXt 作者 : Zhuang Liu , Facebook AI 研究(FAIR); UC 伯克利 Hanzi Mao , Facebook AI 研究(FAIR) Chao-Yuan Wu, Facebook AI 研究(FAIR) 克里斯托夫·费赫滕霍夫(Christoph Feichtenhofer),Facebook AI Research(FAIR) 特雷沃·达雷尔, UC伯克利 Saining Xie, Facebook AI 研究(FAIR) 代码: https://github.com/facebookresearch/ConvNeXt 抽象 Transformability Roaring 20s 视觉识别的“Transformability Roaring 20s”部分始于Vision Transformers(ViTs)的引入,它迅速取代了ConvNets作为最先进的图像分类模型。另一方面,vanilla ViT在应用于一般计算机视觉任务(如对象检测和语义分类)时面临困难。然而,这种混合方法的有效性仍然在很大程度上归功于Transformers的本质优越性(例如,Swin Transformers),而不是对 ConvNet的原始偏见的重新引入,这使得Transformers作为一种通用视觉背骨几乎是可行的,并且在各种视觉任务上表现出惊人的表现。然而,这种混合方法的有效性仍然在很大程度上归功于Transformers的内在优越性,而 1、介绍 回顾到2010年代,这一十年被深度学习的巨大进步和影响所标记,主要驱动因素是神经网络的复兴,特别是神经网络(ConvNets)。在整个十年中,视觉识别领域成功地从工程功能转移到设计(ConvNet)架构。尽管背部传播训练的ConvNets的发明可以追溯到20世纪80年代。 直到2012年底,我们才看到其真正的视觉特征学习潜力。 42 亚历山大(Alexander)的介绍 ]推迟了“ImageNet 时刻” [ ],开启了计算机视觉的新时代. 该领域自此以快速的速度演变。 ) ,创意( ) , ], DenseNet [ ) , 移动网络( ) ) ,高效( ) )和雨( ) 专注于精度、效率和可扩展性的不同方面,并普及了许多有用的设计原则。 40 59 64 68 28 87 36 34 71 54 ConvNets在计算机视觉中完全占主导地位并非巧合:在许多应用场景中,一个“滑窗”策略是视觉处理的内在因素,尤其是在处理高分辨率图像时。ConvNets有几个内置的诱导偏见,使它们适合广泛的计算机视觉应用。最重要的是翻译等级,这对于反对检测等任务是可取的属性。 几十年来,这一直是ConvNets的默认使用,通常在有限的对象类别上,如数字。 面孔( ) , )和行人( ) , 进入2010年代,基于区域的探测器 [ , , , 进一步提升 ConvNets 到视觉识别系统的基本构建块的位置。 62 43 58 76 19 63 23 24 27 57 大约在同一时间,自然语言处理(NLP)神经网络设计的奥迪塞亚采取了非常不同的道路,因为变形器取代了重复的神经网络,成为主导的脊椎架构。尽管语言和视觉领域之间的兴趣任务差异,两条流在2020年惊人地汇聚,因为Vision Transformers(ViT)的引入完全改变了网络架构设计的景观:除了初始的“patchify”层,它将图像分成一个补丁序列,ViT没有引入图像特定的诱导偏见,并对原始NLP Transformers做出最小的变化。ViT的主要重点是扩展视觉行为:借助更大的模型和数据集大小,Transformers可以通过可接受的标准ResNets进行分类。这些图像分类任务的结果 基因组合的基因组合使用混合方法来弥合这种差距,例如“滑窗”策略( . attention within local windows) was reintroduced to Transformers, allowing them to behave more similarly to ConvNets. Swin Transformer [ 这是这个方向的一个里程碑,首次证明转换器可以被采用为通用视觉支柱,并在图像分类以外的各种计算机视觉任务中实现最先进的性能。 以G 45 在这种观点下,用于计算机视觉的Transformers的许多进步都旨在带来回转,然而,这些尝试会付出代价:滑动窗口自我关注的天真实施可能很昂贵。 );采用先进的方法,如循环转换( ) ],速度可以优化,但系统在设计上变得更加复杂,另一方面,它几乎讽刺的是,ConvNet已经满足了许多这些所需的属性,虽然在一个简单的,没有杂的方式。 55 45 在过去的10年里,在CovNets中,视觉转换器的采用逐渐改进,在最近的文献中,系统级比较( 猪变换器 ConvNets和等级视觉转换器在同一时间变得不同和相似:它们都配备了相似的诱导偏见,但在培训程序和宏观 / 微级架构设计方面有显著差异。 在这项工作中,我们调查ConvNets和Transformers之间的架构区别,并试图在比较网络性能时识别令人困惑的变量。我们的研究旨在弥合ConvNets的预ViT和后ViT时代之间的差距,以及测试纯 ConvNet可以实现的局限性。 以G 对 要做到这一点,我们开始使用标准的ResNet( 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 二、 我们的探索是由一个关键问题指导的: 我们发现了几个关键组成部分,这些组成部分在途中有助于实现性能差异,因此,我们提出了一系列 我们在各种视觉任务中评估ConvNeXt,例如ImageNet分类。 ],对象检测/细分在 COCO [ ],并在ADE20K上的语义细分 [ 令人惊讶的是,完全由标准ConvNet模块构建的ConvNeXts在所有主要基准的准确性、可扩展性和可靠性方面与Transformers竞争良好。 以G 以G Transformers 中的设计决策如何影响 ConvNets 的性能? 纯粹的约定 17 44 92 我们希望新的观察和讨论能够挑战一些共同的信念,并鼓励人们重新思考计算机视觉中的转变的重要性。 现代化ConvNet:一张路线图 在本节中,我们提供了一个从ResNet到ConvNet的轨迹,这种轨迹与Transformers有着相似之处,我们在FLOP方面考虑了两个模型大小,其中一个是ResNet-50 / Swin-T模式,FLOP大约为4*. .*0 × 109. 为了简化,我们将用 ResNet-50 / Swin-T 复杂性模型介绍结果. 高容量模型的结论一致,结果可以在附件中找到 . 5×109 和另一个是 ResNet-200 / Swin-B 模式,其 FLOP 约为 15 C 在高层次上,我们的探索旨在研究并遵循Swin Transformer的不同层次设计,同时保持网络的简单性作为标准ConvNet。 我们的起点是ResNet-50模型,我们首先用类似的训练技巧训练它,用来训练视觉变形器,并获得与原来的ResNet-50相比更好的结果,这将是我们的出发点,然后我们研究了一系列设计决策,我们总结了 1)宏观设计, 2)ResNeXt, 3)逆转瓶颈, 4)大内核大小,以及 5)各种层式微型设计。 ,我们展示了我们能够在“网络现代化”的每个步骤中实现的程序和结果,因为网络复杂性与最终性能密切相关,所以FLOP在探索过程中大致控制,尽管在中间步骤中,FLOP可能高于或低于参考模型。 2 2.1 培训技术 除了网络架构的设计之外,培训程序也影响了最终的性能,而视觉转换器不仅带来了一套新的模块和建筑设计决策,而且还引入了不同的培训技术(例如AdamW优化器)到视觉中。 , ] 证明一组现代训练技术可以显著提高简单的ResNet-50模型的性能. 在我们的研究中,我们使用了一种训练配方,接近DeiT的性能。 )和Swin的转换器( ) 该培训从 ResNets 的原始 90 个时代延伸到 300 个时代,我们使用了 AdamW 优化器。 ],数据增强技术如Mixup [ ( ) ( ) , 增量( ) ] , 随机删除 [ ],以及包括斯托加斯特深度在内的规范化方案 [ )和标签 Smoothing ( ) ].我们使用的完整的超级参数可在附件中找到 通过这种增强的训练配方,ResNet-50模型的性能提高了76.1%。 ] 到 78.8% (+2.7%),暗示传统ConvNets和视觉变形器之间的性能差异可能很大一部分是由于训练技术. 我们将在整个“现代化”过程中使用相同的超参数的固定训练配方。 7 81 73 45 46 90 89 14 91 36 69 A1的 1 2、宏观设计 现在我们分析了Swin Transformers的宏观网络设计。 , 使用多阶段设计,每个阶段都有不同的功能地图分辨率,有两个有趣的设计考虑:阶段计算比和“干细胞”结构。 28 65 ResNet 中的各个阶段的计算分布的原始设计在很大程度上是实证的。重量级的“res4”阶段旨在与下游任务兼容,例如对象检测,其中检测器头在 14×14 功能平面上运作。另一方面,Swin-T 遵循相同的原则,但阶段计算比例略有不同 1:1:3:1.对于较大的 Swin 变形机,比率为 1:1:9:1. Changing stage compute ratio. 标记,我们调整每个阶段的块数从(3, 4, 6, 3)在ResNet-50(3, 3, 9, 3),这也将FLOP与Swin-T对齐,这使模型精度从78.8%提高到79.4%。 , ],并且更优化的设计很可能存在。 53 54 从现在开始,我们将使用这个阶段的计算比率。 通常,干细胞设计关心输入图像在网络的开始时将如何处理。由于自然图像中固有的冗余性,一个常见干细胞将积极地将输入图像降示到适当的功能地图大小,既在标准ConvNets和视觉变形器中。在标准ResNet中,干细胞包含一个7×7的弯曲层与步骤2相继,其结果是输入图像的4次降示。在Transformers Vision中,一种更具侵略性的“patchify”策略被用作干细胞,这相当于一个大的内核大小(例如内核大小 = 14 或 16)和非弯曲的弯曲。Swin Transformer使用一个类似的“patchify”层,但具有更小的4个弯曲尺寸来适应架 Changing stem to “Patchify”. with a patchify layer implemented using a 4×4, stride 4 convolutional layer. The accuracy has changed from 79.4% to 这表明,ResNet中的干细胞可能被更简单的“patchify”层 à la ViT所取代,这将导致类似的性能。 × 我们将使用“patchify 支柱”(4) 4、网络中不重叠的合) 二、 在此部分,我们试图采用 ResNeXt 的想法 [ ],它具有更好的FLOPs/准确性交换比瓦尼拉ResNet. 核心组件是组合合,其中合过滤器分为不同的组。 在高层次上,ResNeXt的指导原则是“使用更多组,扩大宽度”。 更准确地说,ResNeXt在瓶颈中使用3×3合层的组合合。 87 在我们的案例中,我们使用深度缩,一个特殊的案例的聚合缩,其中群组的数量等于频道的数量。 )和Xception( ) ].我们注意到,深度弯曲类似于在自我注意中进行的重量总操作,该操作以每频道为基础。 34 11 基础, ., only mixing information in the spatial dimension. The combination of depthwise conv and 1 × 1 convs leads to a separation of spatial and channel mixing, a property shared by vision Transformers, where each operation either mixes information across spatial or channel dimension, but 一、e 使用深度旋转有效地减少了网络FLOP,以及预期的准确性. 按照ResNeXt中提出的策略,我们将网络宽度增加到与Swin-T的频道数量相同(从64到96)。 现在我们将使用ResNeXt设计。 四、逆转瓶颈 每个变压器块的一个重要设计是它创造了一个逆转的瓶颈。 ., the hidden dimension of the MLP block is four times wider than the input dimension (see Figure )。 一、e 4 有趣的是,这种转换器设计与逆转瓶颈设计相连,在ConvNets中使用的扩展比为4。 ],并随后在几个先进的ConvNet架构中获得了拉力。 , 》。 61 70 71 在这里,我们探索了逆转瓶颈设计。 (a) (b) 说明配置. 尽管深度弯曲层的FLOPs增加,但这种变化会使整个网络FLOPs减少到4.6G,因为下样本剩余块的缩短1×1弯曲层的FLOPs显著减少。 3 现在我们将使用逆转瓶颈。 5. 核心大小 在这个部分的探索中,我们专注于大圆形内核的行为。视觉转换器的最突出的方面之一是它们的非本地自我关注,这使每个层都有一个全球接收场。 , ],黄金标准(由VGGNet普及) ])是堆积在现代GPU上具有高效硬件实现的小型内核尺寸(3×3) conv层上。 虽然Swin Transformers重新引入了本地窗口,但窗口大小至少为7×7,显著大于3×3的ResNe(X)t内核大小。 40 68 65 41 要探索大型内核,一个前提是提升深度 conv 层的位置(图) (b)到(c))。这在变形机中也是一个显而易见的设计决定:MSA块放在MLP层之前。由于我们有一个逆转瓶颈块,这是一个自然的设计选择 - 复杂/低效率的模块(MSA,大内核)将有更少的渠道,而高效的密集的1×1层将进行重型升降。 Moving up depthwise conv layer. 3 通过所有这些准备,采用更大的内核大小转换的益处是显著的。我们试验了多个内核大小,包括3、5、7、9和11。网络的性能从79.9%(3×3)增加到80.6%(7×7),而网络的FLOP大致保持不变。 Increasing the kernel size. × 我们将使用7 7 深度 conv 在每个区块。 在这一点上,我们已经完成了对网络架构的宏观评估,令人兴奋的是,在Vision Transformer中所作的设计选择中,很大一部分可能被绘制到ConvNet实例中。 6、微型设计 在本节中,我们在微观尺度上探讨了其他几个建筑差异 - 这里的大多数探索都是在层级进行的,专注于激活函数和正常化层的具体选择。 NLP和视觉架构之间的一个差异是使用哪些激活函数的特征。随着时间的推移,许多激活函数已经开发,但直线单位(ReLU) 由于其简单性和效率,它仍然被广泛用于ConvNets。ReLU在原始的Transformer纸上也被用作激活函数。 Gaus-sian Error 线性单位,或 GELU [ ],可以被认为是ReLU的更平滑的变体,在最先进的变形器中使用,包括谷歌的BERT。 ] and OpenAI’s GPT-2 [ 我们发现,在我们的ConvNet中,ReLU也可以用GELU代替,尽管精度保持不变(80.6%)。 Replacing ReLU with GELU 49 77 32 18 52 转换器和ResNet块之间的一个小区别是,转换器具有较少的激活函数。考虑一个具有键/查询/值线性嵌入层的转换器块,投影层和MLP块中的两个线性层。 Fewer activation functions. 相比之下,我们通常会将激活函数附加到每个凝聚层中,包括1×1的转换。 ,我们从剩余块中删除所有GELU层,除了两个1×1层之间的一层,复制变压器块的风格. 这个过程提高了结果的0.7%到81.3%,几乎匹配Swin-T的性能。 4 我们现在将在每个块中使用一个GELU激活。 转换器块通常也有较少的正常化层,在这里我们删除两个BatchNorm(BN)层,在conv 1 × 1层之前只留下一个BN层。 请注意,我们每块有比Transformers更少的正常化层,因为我们实证地发现,在区块开始时添加一个额外的BN层不会改善性能。 Fewer normalization layers. 强化 布拉格( ) ]是ConvNets的一个必不可少的组成部分,因为它改善了融合并减少了过度装配。然而,BN也有许多复杂性,可能会对模型的性能产生不利影响。 已有许多尝试开发替代标准化( ) , , ] techniques, but BN has remained the preferred option in most vision tasks. On the other hand, the simpler Layer Normalization [ ](LN)已被用于变形器,结果在不同的应用场景中表现良好。 Substituting BN with LN. 38 84 60 75 83 5 在原始ResNet中直接替换LN为BN将导致低优性能。 随着网络架构和培训技术的所有修改,我们在这里重新审视了使用LN而不是BN的影响,我们观察到我们的ConvNet模型与LN训练没有任何困难;事实上,性能略有改善,获得了81.5%的准确性。 83 从现在开始,我们将使用一个LayerNorm作为每个剩余块的正常化选择。 In ResNet, the spatial downsampling is achieved by the residual block at the start of each stage, using 3×3 conv with stride 2 (and 1×1 conv with stride 2 at the shortcut connection). In Swin Transformers, a separate downsampling layer is added between stages. We explore a similar strategy in which we use 2×2 conv layers with stride 2 for spatial downsampling. This modification Separate downsampling layers. 令人惊讶的是,这导致了不同的训练。进一步的调查表明,在空间分辨率改变的地方添加正常化层可以帮助稳定训练。这些包括Swin Transformers中也使用的几个LN层:在每个采样层之前,一个在树干之后,一个在最终的全球平均聚合之后。 我们将使用单独的采样层,这将使我们进入我们的最终模型,我们称之为ConvNeXt。 ResNet、Swin 和 ConvNeXt 区块结构的比较可以在图 4 中找到:ResNet-50、Swin-T 和 ConvNeXt-T 的详细架构规格的比较可以在表 9 中找到。 4 9 We have finished our first “playthrough” and discovered ConvNeXt, a pure ConvNet, that can outperform the Swin Transformer for ImageNet-1K classification in this compute regime. It is worth noting that 此外, Our Con-vNeXt model has approximately the same FLOPs, #params., throughput, and memory use as the Swin Transformer, but does not require specialized modules such as shifted window attention or relative position biases. Closing remarks. 迄今为止讨论的所有设计选择都是从Vision Transformers中适应的。 即使在ConvNet文献中,这些设计也不是新鲜事物,这些设计在过去十年中都是单独研究的,但不是集体研究的。 这些发现令人鼓舞,但尚未完全令人信服 - 到目前为止,我们的探索仅限于小规模,但视觉变形器的扩展行为才是真正的区别。 此外,关于ConvNet是否能在下游任务中与Swin变形器竞争,如物体检测和语义细分,是计算机视觉专业人员的核心问题。 3. 在 ImageNet 上进行实证评估 我们构建不同的ConvNeXt变体,ConvNeXt-T/S/B/L,具有类似的复杂性为Swin-T/S/B/L。 ]. ConvNeXt-T/B is the end product of the “modernizing” pro-cedure on ResNet-50/200 regime, respectively. In addition, we build a larger ConvNeXt-XL to further test the scalabil-ity of ConvNeXt. The variants only differ in the number of channels ,以及块的数量 in each stage. Following both ResNets and Swin Transformers, the number of channels doubles at each new stage. We summarize the configurations below: 45 C B ConvNeXt-T: C = (96*,* 192*,* 384*,* 768), B = (3*,* 3*,* 9*,* 3) ConvNeXt-S: = (96*,* 192*,* 384*,* 768), = (3*,* 3*,* 27*,* 3) C B ConvNeXt-B: C = (128*,* 256*,* 512*,* 1024), B = (3*,* 3*,* 27*,* 3) ConvNeXt-L: C = (192*,* 384*,* 768*,* 1536),B = (3*,* 3*,* 27*,* 3) ConvNeXt-XL: = (256*,* 512*,* 1024*,* 2048), = (3*,* 3*,* 27*,* 3) C B 3.1. Settings ImageNet-1K 数据集由 1000 个对象类组成,其中有 1.2 万个训练图像。我们在验证集中报告 ImageNet-1K 的顶级 1 个精度。我们还对 ImageNet-22K 进行预训练,这是一组大于 21841 个类的数据集(1000 个 ImageNet-1K 类的超组),用于预训练使用 ∼14M 个图像,然后在 ImageNet-1K 上对预训练的模型进行精确调节以进行评估。 . A 我们使用 AdamW 训练 300 个时代的 ConvNeXts [ 以学习率为4e-3,随后有20个时代的线性加热和缩时间表。我们使用批量大小为4096个,重量下降为0.05个,对于数据增加,我们采用包括Mixup在内的常见方案。 ], Cutmix [ ( ) , 增量( ) ], and Random Erasing [ ].我们用 Stochastic 深度规范网络 [ ] and Label Smoothing [ ( ] of initial value 1e-6 is applied. We use Exponential Moving Average (EMA) [ ] as we find it alleviates larger models’ overfitting. Training on ImageNet-1K. 46 90 89 14 91 37 69 74 51 We pre-train ConvNeXts on ImageNet-22K for 90 epochs with a warmup of 5 epochs. We do not use EMA. Other settings follow ImageNet-1K. Pre-training on ImageNet-22K. 我们在 ImageNet-1K 上精心调整了 ImageNet-22K 预先训练的模型,使用了 30 个时代,我们使用了 AdamW,学习率为 5e-5,共生学习率时间表,层式学习率衰退。 , ],没有加热,批量大小为512,体重下降为1e-8.默认的预训练、精细调和测试分辨率为2242。 Fine-tuning on ImageNet-1K. 6 12 Compared with ViTs/Swin Transformers, ConvNeXts are simpler to fine-tune at different resolutions, as the network is fully-convolutional and there is no need to adjust the input patch size or interpolate absolute/relative position biases. 3.2. Results Table (上)显示了与最近的两种变形变形的结果比较,DeiT [ ] and Swin Transformers [ ], as well as two ConvNets from architecture search - RegNets [ 效率高( ) ] and EfficientNetsV2 [ ]. ConvNeXt competes favorably with two strong ConvNet baselines (RegNet [ )和高效网络( ) ])在精度计算的妥协方面,以及推断的通过。ConvNeXt还超过了类似复杂性的Swin Transformer。 sometimes with a substantial margin (e.g. 0.8% for ConvNeXt-T). Without specialized modules such as shifted windows or relative position bias, ConvNeXts also enjoy improved throughput compared to Swin Transformers. ImageNet-1K. 1 73 45 54 71 72 54 71 通过董事会, A highlight from the results is ConvNeXt-B at 3842 : it outperforms Swin-B by 0.6% (85.1% vs. 84.5%), but with 12.5% higher inference throughput (95.7 vs. 85.1 image/s). We note that the FLOPs/throughput advantage of ConvNeXt-B over Swin-B becomes larger when the resolution increases from 2242 to 3842 . Additionally, we observe an improved result of 85.5% when further scaling to ConvNeXt-L. We present results with models fine-tuned from ImageNet-22K pre-training at Table (lower). These experiments are important since a widely held view is that vision Transformers have fewer inductive biases thus can per-form better than ConvNets when pre-trained on a larger scale. Our results demonstrate that properly designed ConvNets are 通过大数据集进行预训练时,ConvNeXts 仍能与相似大小的Swin Transformers 相同或更好地表现,输出量略高一些,此外,我们的ConvNeXt-XL 模型获得了 87.8% 的精度 - 比 ConvNeXt-L 在 3842 获得了相当的提高,这表明 ConvNeXts 是可扩展的架构。 ImageNet-22K. 1 not On ImageNet-1K, EfficientNetV2-L, a searched architecture equipped with advanced modules (such as Squeeze-and-Excitation [ ]) and progressive training procedure achieves top performance. However, with ImageNet-22K pre-training, ConvNeXt is able to outperform EfficientNetV2, further demonstrating the importance of large-scale training. 35 In Appendix , we discuss robustness and out-of-domain generalization results for ConvNeXt. B 3.3 同性化合物 . ViT vs 在这个抽象中,我们检查我们的ConvNeXt块设计是否可以向ViT风格一般化。 ] isotropic architectures which have no downsampling layers and keep the same feature resolutions ( . 14×14) at all depths. We construct isotropic ConvNeXt-S/B/L using the same feature dimensions as ViT-S/B/L (384/768/1024). Depths are set at 18/18/36 to match the number of parameters and FLOPs. The block structure remains the same (Fig. ). We use the supervised training results from DeiT [ ] for ViT-S/B and MAE [ ] for ViT-L, as they employ improved training procedures over the original ViTs [ ]. ConvNeXt 模型以与以前相同的设置进行训练,但具有更长的加热时期。 图像网-1K 在 2242 分辨率的结果在表中 . We observe ConvNeXt can perform generally on par with ViT, showing that our ConvNeXt block design is competitive when used in non-hierarchical models. 20 e.g 4 73 26 20 2 4. Empirical Evaluation on Downstream Tasks We fine-tune Mask R-CNN [ ]和Cascade Mask R-CNN [ ) 在 COCO 数据集中使用 ConvNeXt 背骨。 Following Swin Transformer [ ], we use multi-scale training, AdamW optimizer, and a 3× schedule. Further details and hyperparameter settings can be found in Appendix . Object detection and segmentation on COCO. 27 9 45 A3 的 Table 显示对象检测和实例细分结果,比较 Swin Transformer、ConvNeXt 和传统 ConvNet 如 ResNeXt. 在不同的模型复杂性中,ConvNeXt 实现了与 Swin Transformer 相同或更好的性能。 ( . +1.0 AP) than Swin Transformers in terms of box and mask AP. 3 ConvNeXt is significantly better 以G We also evaluate ConvNeXt backbones on the ADE20K semantic segmentation task with UperNet [ ]. All model variants are trained for 160K iterations with a batch size of 16. Other experimental settings follow [ ] (see Appendix for more details). In Table ,我们通过多级测试报告验证mIoU。ConvNeXt模型可在不同的模型容量中实现竞争性能,进一步验证我们的架构设计的有效性。 Semantic segmentation on ADE20K. 85 6 A.3 4 Under similar FLOPs, models with depthwise convolutions are known to be slower and consume more memory than ConvNets with only dense convolutions. Remarks on model efficiency. 很自然地问 ConvNeXt 的设计是否会使其实际上效率低下。正如整个论文所示,ConvNeXts 的推断输出量与Swin Trans-formers的输出量相似或超过。 , 此外,我们注意到,训练 ConvNeXts 需要比训练 Swin Transformers 更少的内存。例如,使用 ConvNeXt-B 背骨训练 Cascade Mask-RCNN 消耗 17.4 GB 的顶级内存,每 GPU 批量大小为 2,而 Swin-B 的参照号为 18.5 GB。 与瓦尼拉 ViT 相比,ConvNeXt 和 Swin Transformer 由于本地计算而表现出更有利的精度FLOP 交换。 ,并且与视觉变形器中的自我注意机制没有直接关系。 1 3 ConvNet inductive bias 5、相关工作 In both the pre- and post-ViT eras, the hybrid model combining convolutions and self-attentions has been actively studied. Prior to ViT, the focus was on augmenting a ConvNet with self-attention/non-local modules [ , , , ] to capture long-range dependencies. The original ViT [ ] first studied a hybrid configuration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an ex-plicit [ , , , , , ] or implicit [ ) 时尚。 Hybrid models. 8 55 66 79 20 15 16 21 82 86 88 45 Han [ ] show that local Transformer attention is equivalent to inhomogeneous dynamic depthwise conv. The MSA block in Swin is then replaced with a dynamic or regular depthwise convolution, achieving comparable performance to Swin. A concurrent work, ConvMixer [ ] 表明,在小规模的设置中,深度旋转可以作为一个有前途的混合策略。ConvMixer使用较小的补丁尺寸来实现最佳结果,使输出比其他基线要低得多。 ] adopts Fast Fourier Transform (FFT) for token mixing. FFT is also a form of convolution, but with a global kernel size and circular padding. Unlike many recent Transformer or ConvNet designs, one primary goal of our study is to provide an in-depth look at the process of modernizing a standard ResNet and achieving state-of-the-art performance. Recent convolution-based approaches. et al. 25 4 56 6. Conclusions In the 2020s, vision Transformers, particularly hierarchical ones such as Swin Transformers, began to overtake ConvNets as the favored choice for generic vision backbones. The widely held belief is that vision Transformers are more accurate, efficient, and scalable than ConvNets. We propose ConvNeXts, a pure ConvNet model that can compete favorably with state-of-the-art hierarchical vision Transformers across multiple computer vision benchmarks, while retaining the simplicity and efficiency of standard ConvNets. In some ways, our observations are surprising while our ConvNeXt model itself is not completely new — many design choices have all been examined separately over the last decade, but not collectively. We hope that the new results reported in this study will challenge several widely held views and prompt people to rethink the importance of convolution in computer vision. We thank Kaiming He, Eric Mintun, Xingyi Zhou, Ross Girshick, and Yann LeCun for valuable discussions and feedback. Acknowledgments. Appendix 在本附件中,我们提供进一步的实验细节(§ ), robustness evaluation results (§ ), more modernization experiment results (§ )和一个详细的网络规格(§ ). We further benchmark model throughput on A100 GPUs (§ ). Finally, we discuss the limitations (§ ) and societal impact (§ )我们的工作。 A B C D E F G A. Experimental Settings A.1. ImageNet (Pre-)training We provide ConvNeXts’ ImageNet-1K training and ImageNet-22K pre-training settings in Table 设置用于我们在表中的主要结果 (一节) ). All ConvNeXt variants use the same setting, except the stochas-tic depth rate is customized for model variants. 5 1 3.2 For experiments in “modernizing a ConvNet” (Section ), we also use Table ’s setting for ImageNet-1K, except EMA is disabled, as we find using EMA severely hurts models with BatchNorm layers. 2 5 For isotropic ConvNeXts (Section ), the setting for ImageNet-1K in Table is also adopted, but warmup is ex-tended to 50 epochs, and layer scale is disabled for isotropic ConvNeXt-S/B. The stochastic depth rates are 0.1/0.2/0.5 for isotropic ConvNeXt-S/B/L. 3.3 A A.2. ImageNet Fine-tuning We list the settings for fine-tuning on ImageNet-1K in Table . The fine-tuning starts from the final model weights obtained in pre-training, without using the EMA weights, even if in pre-training EMA is used and EMA accuracy is reported. This is because we do not observe improvement if we fine-tune with the EMA weights (consistent with observa-tions in [ ]). The only exception is ConvNeXt-L pre-trained on ImageNet-1K, where the model accuracy is significantly lower than the EMA accuracy due to overfitting, and we select its best EMA model during pre-training as the starting point for fine-tuning. 6 73 In fine-tuning, we use layer-wise learning rate decay [ , ] with every 3 consecutive blocks forming a group. When the model is fine-tuned at 3842 resolution, we use a crop ratio of 1.0 (i.e., no cropping) during testing following [ , , ], 6 12 2 74 80 instead of 0.875 at 2242. A.3. Downstream Tasks For ADE20K and COCO experiments, we follow the training settings used in BEiT [ )和猪( ) ]. We also use MMDetection [ ] and MMSegmentation [ ] toolboxes. We use the final model weights (instead of EMA weights) from ImageNet pre-training as network initializations. 6 45 10 13 We conduct a lightweight sweep for COCO experiments including learning rate {1e-4, 2e-4}, layer-wise learning rate decay [ ] {0.7, 0.8, 0.9, 0.95}, and stochastic depth rate {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}. We fine-tune the ImageNet-22K pre-trained Swin-B/L on COCO using the same sweep. We use the official code and pre-trained model weights [ ]. 6 3 The hyperparameters we sweep for ADE20K experiments include learning rate {8e-5, 1e-4}, layer-wise learning rate decay {0.8, 0.9}, and stochastic depth rate {0.3, 0.4, 0.5}. We report validation mIoU results using multi-scale testing. Additional single-scale testing results are in Table . 7 B. Robustness Evaluation Additional robustness evaluation results for ConvNeXt models are presented in Table . We directly test our ImageNet-1K trained/fine-tuned classification models on sev-eral robustness benchmark datasets such as ImageNet-A [ ], ImageNet-R [ ], ImageNet-Sketch [ ] and ImageNet-C/C¯ [ , ] datasets. We report mean corruption error (mCE) for ImageNet-C, corruption error for ImageNet-C¯, and top-1 Accuracy for all other datasets. 8 33 30 78 31 48 ConvNeXt(特别是大型型号变体)显示出有前景的强度行为,优于最先进的强大变压机模型。 ] on several benchmarks. With extra ImageNet-22K data, ConvNeXt-XL demonstrates strong domain generalization capabilities ( . achieving 69.3%/68.2%/55.0% accuracy on ImageNet-A/R/Sketch benchmarks, respectively). We note that these ro-bustness evaluation results were acquired without using any specialized modules or additional fine-tuning procedures. 47 e.g C. Modernizing ResNets: detailed results Here we provide detailed tabulated results for the experiments, at both ResNet-50 / Swin-T and ResNet-200 / Swin-B regimes. The ImageNet-1K top-1 ac-curacies and FLOPs for each step are shown in Table and . ResNet-50 regime experiments are run with 3 ran-dom seeds. mod-ernization 10 11 For ResNet-200, the initial number of blocks at each stage is (3, 24, 36, 3). We change it to Swin-B’s (3, 3, 27, 3) at the step of changing stage ratio. This drastically reduces the FLOPs, so at the same time, we also increase the width from 64 to 84 to keep the FLOPs at a similar level. After the step of adopting depthwise convolutions, we further increase the width to 128 (same as Swin-B’s) as a separate step. The observations on the ResNet-200 regime are mostly consistent with those on ResNet-50 as described in the main paper. One interesting difference is that inverting dimensions brings a larger improvement at ResNet-200 regime than at ResNet-50 regime (+0.79% . +0.14%). The performance gained by increasing kernel size also seems to saturate at kernel size 5 instead of 7. Using fewer normalization layers also has a bigger gain compared with the ResNet-50 regime (+0.46% . +0.14%). vs vs D. Detailed Architectures We present a detailed architecture comparison between ResNet-50, ConvNeXt-T and Swin-T in Table . For differ-ently sized ConvNeXts, only the number of blocks and the number of channels at each stage differ from ConvNeXt-T (see Section ConvNeXts享受标准ConvNets的简单性,但在视觉识别方面与Swin Transformers有利竞争。 9 3 E. A100 GPU 的 Benchmarking Following Swin Transformer [ ], the ImageNet models’ inference throughputs in Table are benchmarked using a V100 GPU, where ConvNeXt is slightly faster in inference than Swin Transformer with a similar number of parameters. We now benchmark them on the more advanced A100 GPUs, which support the TensorFloat32 (TF32) tensor cores. We employ PyTorch [ ] version 1.10 to use the latest “Channel Last” memory layout [ ] for further speedup. 45 1 50 22 We present the results in Table . Swin Transformers and ConvNeXts both achieve faster inference throughput than V100 GPUs, but ConvNeXts’ advantage is now significantly greater, sometimes . This preliminary study shows promising signals that ConvNeXt, employed with standard ConvNet modules and simple in design, could be practically more efficient models on modern hardwares. 12 up to 49% faster F. Limitations 我们展示了 ConvNeXt,一个纯 ConvNet 模型,可以在图像分类、对象检测、实例和语义分类任务上表现得像一个等级视觉转换器一样好。 虽然我们的目标是提供广泛的评估任务,但我们认识到计算机视觉应用程序更为多样化。 ConvNeXt 可能更适合某些任务,而转换器可能更具灵活性。 一个案例是多模式学习,其中一个交叉关注模块可能更适合模拟跨多种模式的功能交互。 G. Societal Impact In the 2020s, research on visual representation learn-ing began to place enormous demands on computing re-sources. While larger models and datasets improve per-formance across the board, they also introduce a slew of challenges. ViT, Swin, and ConvNeXt all perform best with their huge model variants. Investigating those model designs inevitably results in an increase in carbon emissions. One important direction, and a motivation for our paper, is to strive for simplicity — with more sophisticated modules, the network’s design space expands enormously, obscuring critical components that contribute to the performance dif-ference. Additionally, large models and datasets present issues in terms of model robustness and fairness. Further investigation on the robustness behavior of ConvNeXt vs. Transformer will be an interesting research direction. In terms of data, our findings indicate that ConvNeXt models benefit from pre-training on large-scale datasets. While our method makes use of the publicly available ImageNet-22K dataset, individuals may wish to acquire their own data for pre-training. A more circumspect and responsible approach to data selection is required to avoid potential concerns with data biases. References [1] PyTorch Vision Models. . Accessed: 2021-10-01. https://pytorch.org/ 视力 / 稳定 / 模型.html [2] GitHub repository: Swin transformer. 是的 , 2021. https://github 首页 / 微软 / Swin-Transformer [3] GitHub repository: Swin transformer for object detection. , 2021. https://github.com/SwinTransformer/Swin-Transformer-Object-Detection [4] Anonymous. Patches are all you need? , 2021. 开放视野 [5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. , 2016. 图表: 1607.06450 [6] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. , 2021. arXiv:2106.08254 [7] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. , 2021. NeurIPS [8] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In ,2019年。 ICCV [9] Zhaowei Cai 和 Nuno Vasconcelos. Cascade R-CNN:深入到高质量的对象检测中。 , 2018. CVPR [10] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tian-heng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. , 2019. arXiv:1906.07155 [11] François Chollet. Xception: Deep learning with depthwise separable convolutions. In , 2017. CVPR 的 [12] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christo-pher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In , 2020. ICLR [13] MMSegmentation contributors. MMSegmentation: Openmm-lab semantic segmentation toolbox and benchmark. https: ,2020. //github.com/open- mmlab/mmsegmentation [14] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In , 2020. CVPR Workshops [15] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. , 2021. 神经 [16] Stéphane d’Ascoli,Hugo Touvron,Matthew Leavitt,Ari Mor-cos,Giulio Biroli和Levent Sagun。 , 2021. ICML 的 [17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In , 2009. CVPR [18]雅各布·德文,明威·昌,肯顿·李和克里斯蒂娜·图坦诺娃.伯特:深度双向变形器的预训练,以了解语言。 , 2019. NAACL [19] Piotr Dollár, Serge Belongie, and Pietro Perona. The fastest pedestrian detector in the west. In , 2010. BMVC [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ,2021年。 ICLR [21] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. , 2021. ICCV [22] Vitaly Fedyunin. Tutorial: Channel last memory format in PyTorch. , 2021. Accessed: 2021-10-01. https://pytorch.org/tutorials/ intermediate/memory_format_tutorial.html [23] Ross Girshick. Fast R-CNN. In , 2015. ICCV [24] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In , 2014. CVPR [25] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Ji-aying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dy-namic weight. , 2021. arXiv:2106.04263 [26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. , 2021. arXiv:2111.06377 [27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-shick. Mask R-CNN. In , 2017. ICCV [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In ,2016年。 CVPR [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ,2016年。 ECCV [30]Dan Hendrycks,Steven Basart,Norman Mu,Saurav Kada-vath,Frank Wang,Evan Dorundo,Rahul Desai,Tyler Zhu,Samyak Parajuli,Mike Guo等。 , 2021. ICCV [31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In , 2018. ICLR [32]丹·亨德里克斯和凯文·吉普尔.高斯错误的线性单位(gelus)。 , 2016. arXiv:1606.08415 [33] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In , 2021. CVPR [34] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An-dreetto, and Hartwig Adam. MobileNets: Efficient con-volutional neural networks for mobile vision applications. 2017年。 arXiv:1704.04861 [35] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In , 2018. CVPR [36] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kil-ian Q Weinberger. Densely connected convolutional networks. In , 2017. CVPR [37] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In , 2016. ECCV [38] Sergey Ioffe. Batch renormalization: Towards reducing mini-batch dependence in batch-normalized models. In , 2017. NeurIPS [39] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In , 2020. ECCV 的 [40] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In ,2012年。 神经 [41] Andrew Lavin and Scott Gray. Fast algorithms for convolu-tional neural networks. In , 2016. CVPR 的 [42] Yann LeCun, Bernhard Boser, John S Denker, Donnie Hen-derson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. , 1989. Neural computation [43] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recogni-tion. 1998年。 IEEE的程序 [44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In . 2014. ECCV 的 [45]Ze Liu,Yutong Lin,Yue Cao,Han Hu,Yixuan Wei,Zheng Zhang,Stephen Lin和Baining Guo. Swin转换器:使用转换窗户的等级视觉转换器。 [46] Ilya Loshchilov 和 Frank Hutter. 脱钩的体重衰减规范化。 , 2019. ICLR的 [47] Xiaofeng Mao,Gege Qi,Yuefeng Chen, Xiaodan Li,Ranjie Duan,Shaokai Ye,Yuan He和Hui Xue. 朝着强大的视觉转换器。 ,2021年。 arXiv 预印 arXiv:2105.07926 [48] Eric Mintun, Alexander Kirillov, and Saining Xie. On in-teraction between augmentations and corruptions in natural corruption robustness. ,2021年。 NeurIPS [49] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In , 2010. ICML [50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In ,2019年。 NeurIPS [51]鲍里斯·T·波利亚克和阿纳托利·B·朱迪茨基:通过平均化加速斯托哈斯特接近。 , 1992. SIAM Journal on Control and Optimization [52] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper-vised multitask learners. 2019. [53]Ilija Radosavovic,Justin Johnson,Saining Xie,Wan-Yen Lo和Piotr Dollár. 在视觉识别的网络设计空间中。 , 2019. ICCV [54]伊利亚·拉多萨沃维奇,拉杰·普拉特克·科萨拉胡,罗斯·吉尔希克,卡伊姆·恩·赫和皮奥特尔·多拉。设计网络设计空间。 ,2020年。 CVPR [55]Prajit Ramachandran,Niki Parmar,Ashish Vaswani,Irwan Bello,Anselm Levskaya和Jonathon Shlens。 ,2019年。 NeurIPS [56]Yongming Rao、Wenliang Zhao、Zheng Zhu、Jiwen Lu和Jie Zhou.全球过滤网络进行图像分类。 , 2021. NeurIPS [57] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In , 2015. NeurIPS [58] 亨利·A·罗利,Shumeet Baluja和Takeo Kanade. 基于神经网络的面部检测。 1998年。 TPAMI [59] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San-jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. , 2015. IJCV [60] Tim Salimans 和 Diederik P Kingma. 体重正常化:一个简单的再测量来加速深度神经网络的训练。 , 2016. 神经 [61] 马克·桑德勒,安德鲁·霍华德,孟long Zhu,安德烈·Zh-moginov和Liang-Chieh Chen. Mobilenetv2:逆转残留和线性瓶颈。 , 2018. CVPR [62]Pierre Sermanet、David Eigen、Xiang Zhang、Michael Math-ieu、Rob Fergus 和 Yann LeCun. Overbeat:集成识别、本地化和检测,使用突变网络。 , 2014. ICLR [63]Pierre Sermanet,Koray Kavukcuoglu,Soumith Chintala和Yann LeCun. 行人检测与无监督的多阶段特征学习。 ,2013年。 CVPR [64] Karen Simonyan and Andrew Zisserman. Two-stream convo-lutional networks for action recognition in videos. In ,2014年。 NeurIPS [65]Karen Simonyan和Andrew Zisserman. 非常深的共振网络,用于大规模的图像识别。 ,2015年。 ICLR [66] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck trans-formers for visual recognition. In ,2021年。 CVPR [67] 安德烈亚斯·施泰纳,亚历山大·科莱斯尼科夫, Xiaohua Zhai,罗斯·威特曼,雅各布·乌斯科里特和卢卡斯·贝耶尔. 如何在视觉转换器中训练你的视觉数据,增加和规律化。 , 2021. arXiv preprint arXiv:2106.10270 [68] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In , 2015. CVPR [69] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the incep-tion architecture for computer vision. In ,2016年。 CVPR 的 [70] Mingxing Tan, Bo Chen, Ruoming Pang,Vijay Vasudevan,Mark Sandler,Andrew Howard和Quoc V Le. Mnasnet:平台意识的神经元架构搜索移动。 , 2019. CVPR 的 [71] Mingxing Tan 和 Quoc Le. Efficientnet:重新思考神经网络的扩展模型。 ,2019年。 ICML 的 [72] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In ,2021年。 ICML [73] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through atten-tion. , 2020. arXiv:2012.12877 [74] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. ,2021年。 《ICCV》 [75] Dmitry Ulyanov,Andrea Vedaldi,Victor Lempitsky. 在状态的正常化:快速调节的缺失成分。 ,2016年。 arXiv:1607.08022 [76]Régis Vaillant,Christophe Monrocq和Yann Le Cun. 原始方法在图像中对象的定位。 ,1994年。 视觉、图像和信号处理 [77]阿什希·瓦斯瓦尼、诺亚姆·沙泽尔、尼基·帕尔马尔、雅各布·乌斯科·雷特、里昂·琼斯、艾丹·N·戈梅斯、卢卡斯·凯瑟和伊利亚·波洛苏金。注意力是你所需要的一切。 2017年。 NeurIPS [78] Haohan Wang,Songwei Ge,Eric P Xing和Zachary C Lipton. 通过惩罚本地预测力来学习强大的全球表现。 , 2019. 神经 [79] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In 2018年。 CVPR 的 [80] Ross Wightman. GitHub repository: Pytorch image mod-els. ,2019年。 https://github.com/rwightman/pytorch-image-models [81] Ross Wightman,Hugo Touvron和Hervé Jégou. Resnet 回击:一小时改进的训练程序。 ,2021年。 [82] Haiping Wu, Bin Xiao,Noel Codella,Mengchen Liu,Xiyang Dai,Lu Yuan和Lei Zhang。 ,2021年。 《ICCV》 [83] Yuxin Wu 和 Kaiming He. 群体正常化。 , 2018. ECCV 的 [84]尤金·吴和贾斯汀·约翰逊.在批量规范中重新思考“批量”。 , 2021. [85] Tete Xiao,Yingcheng Liu,Bolei Zhou,Yuning Jiang,和Jian Sun. 统一的感知分析,以了解场景。 , 2018. ECCV [86] Tete Xiao,Mannat Singh,Eric Mintun,Trevor Darrell,Piotr Dollár和Ross Girshick. 早期转换有助于转换者更好地看到。 , 2021. 神经 [87] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017年。 CVPR 的 [88] Weijian Xu、Yifan Xu、Tyler Chang 和 Zhuowen Tu. 同步规模调用注意图像变换器。 ,2021年。 《ICCV》 [89]Sangdoo Yun,Dongyoon Han,Seong Joon Oh,Sanghyuk Chun,Junsuk Choe和Youngjoon Yoo. Cutmix:Regu-larization策略以训练具有可定位功能的强大分类器。 , 2019. 《ICCV》 [90] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 2018年。 ICLR [91] Zhun Zhong、Liang Zheng、Guoliang Kang、Shaozi Li、Yi Yang. 随机删除数据增量。 ,2020年。 阿阿 [92] Bolei Zhou, Hang Zhao,Xavier Puig,Tete Xiao,Sanja Fidler,Adela Barriuso和Antonio Torralba.通过ADE20K数据集对场景的语义理解。 ,2019年。 本文已在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下获取。 本文已在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下获取。