Ideal Contribution Estimation: Taylor Expansion and Gradient Cache

Written by instancing | Published 2025/12/04
Tech Story Tags: deep-learning | computational-efficiency | data-contribution | first-order-taylor-expansion | lvis-test-set-sampling | gradient-cache | taylor-expansion | what-is-taylor-expansion

TLDRThe ideal contribution of generated data is approximated using a first-order Taylor expansion based on the gradient dot product. via the TL;DR App

Abstract and 1 Introduction

  1. Related work

    2.1. Generative Data Augmentation

    2.2. Active Learning and Data Analysis

  2. Preliminary

  3. Our method

    4.1. Estimation of Contribution in the Ideal Scenario

    4.2. Batched Streaming Generative Active Learning

  4. Experiments and 5.1. Offline Setting

    5.2. Online Setting

  5. Conclusion, Broader Impact, and References

A. Implementation Details

B. More ablations

C. Discussion

D. Visualization

4. Our method

4.1. Estimation of Contribution in the Ideal Scenario

Moreover, we can employ the classic first-order Taylor expansion to approximate ϕ(g, θ), which has also been widely used in previous work (Pruthi et al., 2020; He et al., 2023). Note that, it is possible to use more sophisticated methods here, e.g., Koh and Liang (2017).

Lemma 4.1. The loss of a network f on a dataset U can be approximated by a first-order approximation:

So the contribution of g to f can be approximated by:

Authors:

(1) Muzhi Zhu, with equal contribution from Zhejiang University, China;

(2) Chengxiang Fan, with equal contribution from Zhejiang University, China;

(3) Hao Chen, Zhejiang University, China ([email protected]);

(4) Yang Liu, Zhejiang University, China;

(5) Weian Mao, Zhejiang University, China and The University of Adelaide, Australia;

(6) Xiaogang Xu, Zhejiang University, China;

(7) Chunhua Shen, Zhejiang University, China ([email protected]).


This paper is available on arxiv under CC BY-NC-ND 4.0 Deed (Attribution-Noncommercial-Noderivs 4.0 International) license.


Written by instancing | Pioneering instance management, driving innovative solutions for efficient resource utilization, and enabling a more sus
Published by HackerNoon on 2025/12/04