From Images to Programs: A Denoising Diffusion Method for Inverse Graphics

Written by photosynthesis | Published 2025/09/24
Tech Story Tags: neural-networks | code-generation | denoising-diffusion-models | syntax-trees | tree-diffusion | program-synthesis | diffusion-models | boolean-operators

TLDRThis article presents a novel method for program synthesis using denoising diffusion models on syntax trees.via the TL;DR App

Table of Links

Abstract and 1. Introduction

  1. Background & Related Work

  2. Method

    3.1 Sampling Small Mutations

    3.2 Policy

    3.3 Value Network & Search

    3.4 Architecture

  3. Experiments

    4.1 Environments

    4.2 Baselines

    4.3 Ablations

  4. Conclusion, Acknowledgments and Disclosure of Funding, and References

Appendix

A. Mutation Algorithm

B. Context-Free Grammars

C. Sketch Simulation

D. Complexity Filtering

E. Tree Path Algorithm

F. Implementation Details

3 Method

The main idea behind our method is to develop a form of denoising diffusion models analogous to image diffusion models for syntax trees.

Consider the example task from Ellis et al. [11] of generating a constructive solid geometry (CSG2D) program from an image. In CSG2D, we can combine simple primitives like circles and quadrilaterals using boolean operations like addition and subtraction to create more complex shapes, with the context-free grammar (CFG),

In the following sections, we will first describe how “noise” is added to syntax trees. Then, we will detail how we train a neural network to reverse this noise. Finally, we will describe how we use this neural network for search.

Authors:

(1) Shreyas Kapur, University of California, Berkeley ([email protected]);

(2) Erik Jenner, University of California, Berkeley ([email protected]);

(3) Stuart Russell, University of California, Berkeley ([email protected]).


This paper is available on arxiv under CC BY-SA 4.0 DEED license.


Written by photosynthesis | Cultivating life through Photosynthesis, harnessing sunlight to nourish ecosystems and fuel a sustainable future.
Published by HackerNoon on 2025/09/24