Generating Private, High-Utility Tabular Data with Masked Language Models

Written by languagemodels | Published 2025/04/08
Tech Story Tags: masked-language-modeling-(mlm) | synthetic-data-generation | conditional-density-estimation | tabular-data | machine-learning-utility-(mlu) | non-parametric-estimation | histogram-based-methods | data-imputation

TLDRThis paper introduces MaCoDE, a method that reframes masked language modeling as conditional density estimation for generating synthetic tabular data. It achieves high machine learning utility, handles missing data, allows privacy control, and outperforms state-of-the-art methods on multiple real-world datasets.via the TL;DR App

Table of Links

  1. Abstract & Introduction

  2. Proposal

    1. Classification Target
    2. Masked Conditional Density Estimation (MaCoDE)
  3. Theoretical Results

    1. With Missing Data
  4. Experiments

  5. Results

    1. Related Works
    2. Conclusions and Limitations
    3. References
  6. A1 Proof of Theorem 1

    1. A2 Proof of Proposition 1
    2. A3 Dataset Descriptions
  7. A4 Missing Mechanism

    1. A5 Experimental Settings for Reproduction
  8. A6 Additional Experiments

  9. A7 Detailed Experimental Results

2.3 Theoretical Results

Remark 2. Since we constrain the values of continuous columns to the interval [0, 1], our proposed approach and Theorem 1 can be seen as similar to the copula density estimation method [22, 27, 9, 2]. However, given the heterogeneous nature of tabular data [3, 43], handling categorical variables is essential, and the categorical marginal distributions do not guarantee the uniqueness of the copula [44]. Therefore, instead of utilizing the copula density estimation method, we adopt a multi-class classification approach for density estimation.

2.4 With Missing Data

In (6), the missingness pattern is described by min(m, r), indicating that the missing data model is determined by the joint distribution p(m, r|x). Thus, as the masking vector defined in Definition 2 satisfies the conditions in Proposition 1, the missingness pattern of our proposed model also conforms to the MAR mechanism according to Proposition 1 if the given data is MAR. It implies that our inference strategies based on the observed dataset can be justified [34]. However, we empirically demonstrate in Section 3 that our proposed model is also applicable to other missing data scenarios.

Authors:

(1) Seunghwan An, Department of Statistical Data Science, University of Seoul, S. Korea ([email protected]);

(2) Gyeongdong Woo, Department of Statistical Data Science, University of Seoul, S. Korea ([email protected]);

(3) Jaesung Lim, Department of Statistical Data Science, University of Seoul, S. Korea ([email protected]);

(4) ChangHyun Kim, Department of Statistical Data Science, University of Seoul, S. Korea ([email protected]);

(5) Sungchul Hong, Department of Statistics, University of Seoul, S. Korea ([email protected]);

(6) Jong-June Jeon (corresponding author), Department of Statistics, University of Seoul, S. Korea ([email protected]).


This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license.


Written by languagemodels | Large Language Models (LLMs) ushered in a technological revolution. We breakdown how the most important models work.
Published by HackerNoon on 2025/04/08