Kamal Haasan starrer Indian 2 is one of the most anticipated sequels, set to release in 2024. Meanwhile, makers have now shared a new poster of the upcoming thriller.

Indian 2: Makers unveil new poster of Kamal Haasan as Senapathy; movie set to arrive in June 2024 | PINKVILLA

A new poster of the film was unveiled on Sunday on the occasion of the Tamil New Year, Puthandu.

Kamal Haasan’s Senapathy gets resurrected in new poster for Shankar’s ‘Indian 2’-Telangana Today

Director Muthaiya introduces son Vijay in &#x27;Sullan Sethu&#x27;, a crime drama set in Madurai. The intense first look poster was unveiled on Tamil New Year,

Director Muthaiya launches son with 'Sullan Sethu'; reveals first look poster | Tamil Movie News - Times of India

Kamal Haasan is gearing up for the release of his highly anticipated movie Indian 2. On the occasion of Tamil New Year Puthandu, the makers have unveiled a new poster today, April 14. This has come after the release month announcement which was made a few days ago.Indian 2 new poster unveiledThe ...

Indian 2 New Poster: Kamal Haasan Is Back As Senapathy With 'Zero Tolerance'- Republic World

Their latest poster elevated the excitement surrounding the release of &#x27;Indian 2&#x27;, the upcoming Tamil film. It was celebrated as

Kamal Haasan's 'Indian 2' Poster Unveiled: A Tamil New Year Treat for Fans! - Tamil News - IndiaGlitz.com

Check the date, history and significance of Ambedkar Jayanti 2024.

Ambedkar Jayanti 2024: Date, History, Significance, Quotes & Posters

Postjer Group

postjer

Welcome, everyone, to HackerNoon Startups of the week! Each week, the HackerNoon team highlights standout Startups from our Startups of the Year database. Each startup featured, has been nominated as the best in their respective technology category or region. Want to be nominated in HackerNoon’s Startups of the Year? Learn how here. This week, we’re thrilled to spotlight Postjer Group, Fast Remote Services, and Semillas Urbanas México. Meet the Startups of the Week Postjer Group Postjer Group is a multifaceted consultancy and innovation partner that specializes in sustainable product development, creative agency services, and startup investments, helping businesses thrive and turn visionary ideas into tangible success. Operating out of Brighton, UK, Postjer Group has been nominated as the top startup in the region, with additional nominations in IT services, Business Intelligence, and Software Development. Support the Postjer Group by voting here. Fast Remote Services Since 2020, Fast Remote Services has been on a mission to change how we learn and build technology. They’re a digital education and tech solutions company focused on building a community of next-gen tech leaders. Fast Remote Services has been nominated as one of the best startups in Nairobi, Kenya, with three additional nominations in the IT Services, Messaging & Communications, and Blogging categories. Vote for Fast Remote Services here. Semillas Urbanas México Semillas Urbanas México empowers the people of Guadalajara by offering fractional ownership in sustainable urban development projects, contributing to local growth, and allowing residents to actively shape their city’s future. They’ve earned a spot among the top startups in Guadalajara, Mexico, and also received nominations in  the Business Development, Fintech, and Crowdfunding categories. Suport Semillas Urbanas Mexico. Vote here. Feature Interview of the Week Once nominated, follow Semillas Urbanas México's lead and create your business page on HackerNoon to make the most of your free Startups of the Year interview. Choose from our list of startup interview templates, including the Cybersecurity Interview, which offers a tailored framework for cybersecurity startups to highlight their industry contributions. Share what inspired your entry into cybersecurity, the unique value you bring, and the challenges faced in safeguarding users from evolving threats. Discuss how you’ve overcome these hurdles and stayed ahead of emerging trends, then outline your long-term vision and goals to show voters why they should support your startup in the Startups of the Year awards. Here’s a quote from Semillas Urbanas México featured in their published interview, telling us why Startups of the Year is an important initiative: Being part of Startups of the Year is an opportunity to connect with fellow entrepreneurs and showcase our ambitious goals. Engaging with experts, entrepreneurs, and even potential customers through Hackernoon will enable us to reach a broader audience and gain valuable insights from the innovation and tech ecosystem, particularly regarding real estate crowdfunding. Want to be featured on HackerNoon’s Startups of the Week? Share your startup's story - use this interview template.
Here’s how the 2023 nominees did it: Meetanshi, Sniper.xyz, Wallet Factory, Grownu Workforce Management System, Motif, Lightrun Featured Startup Specials Package HackerNoon’s Startups of The Year offers an incredible branding opportunity for your startup. Whether you're looking for brand awareness or lead generation, we offer startup-friendly packages to boost your marketing efforts. Explore our  Lead Generation package, which delivers 1M impressions through the HackerNoon Newsletter! With this package, you’ll receive: 4 Total Newsletters Sent: 325,000 per send (85% of subscribers reside in the US)

Open Rate: 13% - 15%
Click-to-Open Rate: 13.61%
Sample: Here


Your Evergreen Tech Company News Page Learn more about this package here or book a meeting with us! That’s it for today. Until next time, keep innovating! The HackerNoon Team Welcome, everyone, to HackerNoon Startups of the week! Each week, the HackerNoon team highlights standout Startups from our Startups of the Year database. Each startup featured, has been nominated as the best in their respective technology category or region. Want to be nominated in HackerNoon’s Startups of the Year? Learn how here. Want to be nominated in HackerNoon’s Startups of the Year? Learn how here . Learn how here Learn how here This week, we’re thrilled to spotlight Postjer Group , Fast Remote Services , and Semillas Urbanas México. Postjer Group Postjer Group Fast Remote Services Fast Remote Services Semillas Urbanas México . Semillas Urbanas México Meet the Startups of the Week Postjer Group Postjer Group Postjer Group is a multifaceted consultancy and innovation partner that specializes in sustainable product development, creative agency services, and startup investments, helping businesses thrive and turn visionary ideas into tangible success. Operating out of Brighton, UK , Postjer Group has been nominated as the top startup in the region, with additional nominations in IT services , Business Intelligence , and Software Development . Brighton, UK IT services Business Intelligence Software Development Support the Postjer Group by voting here. Support the Postjer Group by voting here. here . here Fast Remote Services Fast Remote Services Since 2020, Fast Remote Services has been on a mission to change how we learn and build technology. They’re a digital education and tech solutions company focused on building a community of next-gen tech leaders. Fast Remote Services has been nominated as one of the best startups in Nairobi, Kenya , with three additional nominations in the IT Services , Messaging & Communications , and Blogging categories . Nairobi, Kenya IT Services Messaging & Communications Blogging categories Vote for Fast Remote Services here. Vote for Fast Remote Services here . here here Semillas Urbanas México Semillas Urbanas México Semillas Urbanas México empowers the people of Guadalajara by offering fractional ownership in sustainable urban development projects, contributing to local growth, and allowing residents to actively shape their city’s future. They’ve earned a spot among the top startups in Guadalajara, Mexico , and also received nominations in  the Business Development , Fintech , and Crowdfunding categories . Guadalajara, Mexico Business Development Fintech Crowdfunding categories Suport Semillas Urbanas Mexico. Vote here. Suport Semillas Urbanas Mexico. Vote here . Vote here Vote here Feature Interview of the Week Once nominated, follow Semillas Urbanas México 's lead and create your business page on HackerNoon to make the most of your free Startups of the Year interview. Semillas Urbanas México Semillas Urbanas México create your business page on HackerNoon create your business page on HackerNoon Startups of the Year Choose from our list of startup interview templates , including the Cybersecurity Interview , which offers a tailored framework for cybersecurity startups to highlight their industry contributions. Share what inspired your entry into cybersecurity, the unique value you bring, and the challenges faced in safeguarding users from evolving threats. Discuss how you’ve overcome these hurdles and stayed ahead of emerging trends, then outline your long-term vision and goals to show voters why they should support your startup in the Startups of the Year awards. list of startup interview templates list of startup interview templates Cybersecurity Interview Cybersecurity Interview Startups of the Year Here’s a quote from Semillas Urbanas México featured in their published interview, telling us why Startups of the Year is an important initiative: Here’s a quote from Semillas Urbanas México featured in their published interview , telling us why Startups of the Year is an important initiative: published interview Being part of Startups of the Year is an opportunity to connect with fellow entrepreneurs and showcase our ambitious goals. Engaging with experts, entrepreneurs, and even potential customers through Hackernoon will enable us to reach a broader audience and gain valuable insights from the innovation and tech ecosystem, particularly regarding real estate crowdfunding. Being part of Startups of the Year is an opportunity to connect with fellow entrepreneurs and showcase our ambitious goals. Engaging with experts, entrepreneurs, and even potential customers through Hackernoon will enable us to reach a broader audience and gain valuable insights from the innovation and tech ecosystem, particularly regarding real estate crowdfunding. Startups of the Year Want to be featured on HackerNoon’s Startups of the Week? Share your startup's story - use this interview template. Here’s how the 2023 nominees did it: Meetanshi, Sniper.xyz, Wallet Factory, Grownu Workforce Management System, Motif, Lightrun Want to be featured on HackerNoon’s Startups of the Week? Share your startup's story - use this interview template. Want to be featured on HackerNoon’s Startups of the Week? Share your startup's story - use this interview template . use this interview template use this interview template Here’s how the 2023 nominees did it: Meetanshi , Sniper.xyz , Wallet Factory , Grownu Workforce Management System , Motif , Lightrun Meetanshi Meetanshi Sniper.xyz Sniper.xyz Wallet Factory Wallet Factory Grownu Workforce Management System Grownu Workforce Management System Motif Motif Lightrun Lightrun Featured Startup Specials Package HackerNoon’s Startups of The Year offers an incredible branding opportunity for your startup. Whether you're looking for brand awareness or lead generation, we offer startup-friendly packages to boost your marketing efforts. startup-friendly packages Explore our Lead Generation package, which delivers 1M impressions through the HackerNoon Newsletter! Lead Generation With this package, you’ll receive: With this package, you’ll receive: 4 Total Newsletters Sent: 325,000 per send (85% of subscribers reside in the US)

Open Rate: 13% - 15%
Click-to-Open Rate: 13.61%
Sample: Here Your Evergreen Tech Company News Page 4 Total Newsletters Sent: 325,000 per send (85% of subscribers reside in the US) Open Rate: 13% - 15%
Click-to-Open Rate: 13.61%
Sample: Here Open Rate: 13% - 15% Click-to-Open Rate: 13.61% Sample: Here Open Rate: 13% - 15% Click-to-Open Rate: 13.61% Sample: Here Here Your Evergreen Tech Company News Page Evergreen Tech Company News Page Learn more about this package here or book a meeting with us! Learn more about this package here or book a meeting with us ! here here book a meeting with us book a meeting with us That’s it for today. Until next time, keep innovating! The HackerNoon Team

Meet Postjer Group, Fast Remote Services & Semillas Urbanas México: HackerNoon Startups of the Week

পোস্টজার গ্রুপ, ফাস্ট রিমোট সার্ভিস এবং সেমিলাস আরবানাস মেক্সিকো: হ্যাকারনুন স্টার্টআপ অফ দ্য উইকের সাথে দেখা করুন

Lernen Sie Postjer Group, Fast Remote Services und Semillas Urbanas México kennen: HackerNoon-Startups der Woche

Conoce Postjer Group, Fast Remote Services y Semillas Urbanas México: Startups de la semana de HackerNoon

Découvrez Postjer Group, Fast Remote Services et Semillas Urbanas México : les startups HackerNoon de la semaine

पोस्टजेर ग्रुप, फास्ट रिमोट सर्विसेज और सेमिलास अर्बानास मेक्सिको से मिलिए: हैकरनून स्टार्टअप्स ऑफ द वीक

Postjer Group、Fast Remote Services、Semillas Urbanas Mésico をご紹介します: HackerNoon の今週のスタートアップ

Postjer Group, Fast Remote Services 및 Semillas Urbanas México를 만나보세요: HackerNoon 주간 스타트업

Conheça o Postjer Group, Fast Remote Services e Semillas Urbanas México: HackerNoon Startups da Semana

Познакомьтесь с Postjer Group, Fast Remote Services и Semillas Urbanas México: стартапы недели HackerNoon

Postjer Group, Hızlı Uzaktan Hizmetler ve Semillas Urbanas México ile tanışın: Haftanın HackerNoon Girişimleri

Gặp gỡ Postjer Group, Fast Remote Services & Semillas Urbanas México: Các công ty khởi nghiệp của tuần của HackerNoon

结识 Postjer Group、Fast Remote Services 和 Semillas Urbanas México：HackerNoon 本周初创公司

Business Intelligence, AI - October 7, 2024 - HackerNoon, the independent technology publishing platform, is pleased to announce the nominees for its annual “Startups of The Year” award, featuring the most promising and innovative companies in the Business Intelligence startup ecosystem. The list, compiled by the HackerNoon team, showcases 30 startups making significant contributions and driving innovation in the Business Intelligence industry. https://youtu.be/XOpG6B4AKPU?embedable=true Nominations are open - learn more here. You can nominate your company as the Startup of The Year in the Business Intelligence industry here. The Startups in the Business Intelligence industry nominated for Startups of The Year 2024 include: Digital Fennec
AQUi9
Oronts
KROCKIO
OLabs - UX Research Labs in Vietnam
Co.Urban
Postjer Group
Slash Mark IT Startup
Cywer Learning
DesignerDudes
Glider Aerial Solutions
ByteCraft Studios
Kinda Solutions
Maitsys
UseCodify
U-Hertz
Neem Infosys
Humantek
Tecoreng
SupplyzPro
Vangwe
Tiyasha Software Private Limited
Sankadi
Altrium
Navicent Solutions
QOD Information Technology
FastServe
Ozeefy
HostXolo
MHS Planet “We are thrilled to recognize the top startups in the Business Intelligence industry and celebrate their achievements,” said David Smooke, HackerNoon Founder & CEO. “Starting and sustaining a company is difficult. We’re honored to showcase their work and impact.” Why It Pays Off to Be Nominated Startups of The Year 2024 is a 100% community-driven voting event recognizing startups transforming technology and the world for the better. Starting October 1st, 2024, the internet can nominate and vote for the best ventures in their city and industry. People can nominate their favorite startups until October 31st, and vote once per day per award for who they think should be Startup of The Year up to March 31st, 2025. Winners will be announced in April 2025, after a comprehensive evaluation by the HackerNoon team. Apart from validation from the world's greenest tech publication, all nominees get free interviews relevant to their region and industry. They will also receive curated startup-friendly packages and a free version of their very own Evergreen Tech Company News Page on HackerNoon. In sum, being nominated gives you major credibility as one of the top startups globally and an expert in your industry, meaning even without winning, you still gain tons of exposure. Nominations are open - learn more here. You can nominate your company as the Startup of The Year in the Business Intelligence industry here. About HackerNoon’s Startups of The Year Startups of The Year 2024 is HackerNoon’s flagship community-driven event celebrating startups, technology, and the spirit of innovation. Currently in its third iteration, the prestigious Internet award recognizes and celebrates tech startups of all shapes and sizes. This year, over 150,000 entities across 4200+ cities, 6 continents, and 100+ industries will participate in a bid to be crowned the best startup of the year! Millions of votes have been cast over the past few years, and many stories have been written about these daring and rising startups. The winners will get a free interview on HackerNoon and an Evergreen Tech Company News page. Visit our FAQ page to learn more. Download our design assets here. Check out the Startups of The Year Merch Shop here. HackerNoon’s Startups of The Year is a branding opportunity unlike any other. Whether your goal is brand awareness or lead generation, HackerNoon has curated startup-friendly packages to solve your marketing challenges. Meet our sponsors: Wellfound: Join the #1 global, startup-focused community. At Wellfound, we're not just a job board—we're the place where top startup talent and the world's most exciting companies connect to build the future. Notion: Notion is trusted and loved by thousands of startups as their connected workspace—from building product roadmaps to tracking fundraising. Try Notion with unlimited AI, FREE for up to 6 months, to build and scale your company with one powerful tool. Get your offer now! Hubspot: If you’re looking for a smart CRM platform that meets the needs of small businesses, look no further than HubSpot. Seamlessly connect your data, teams and customers in one easy-to-use scalable platform that grows with your business. Get started for free. Bright Data: Startups that leverage public web data can make faster, data-driven decisions, giving them a competitive edge. With Bright Data’s scalable web data collection, businesses can grow from a small operation to an enterprise by harnessing insights at every stage. Algolia: Algolia NeuralSearch is the world's only AI end-to-end Search and Discovery Platform combining powerful keyword and natural language processing in a single API. Business Intelligence, AI - October 7, 2024 - HackerNoon, the independent technology publishing platform, is pleased to announce the nominees for its annual “Startups of The Year” award, featuring the most promising and innovative companies in the Business Intelligence startup ecosystem. Business Intelligence, AI - October 7, 2024 - HackerNoon, the independent technology publishing platform, is pleased to announce the nominees for its annual “Startups of The Year” award, featuring the most promising and innovative companies in the Business Intelligence startup ecosystem. Business Intelligence , AI - October 7, 2024 Business Intelligence - HackerNoon, the independent technology publishing platform, is pleased to announce the nominees for its annual “ Startups of The Year ” award, featuring the most promising and innovative companies in the Startups of The Year startup ecosystem. The list, compiled by the HackerNoon team, showcases 30 startups making significant contributions and driving innovation in the Business Intelligence industry. https://youtu.be/XOpG6B4AKPU?embedable=true https://youtu.be/XOpG6B4AKPU?embedable=true Nominations are open - learn more here. You can nominate your company as the Startup of The Year in the Business Intelligence industry here. Nominations are open - learn more here . You can nominate your company as the Startup of The Year in the Business Intelligence industry here . here Startup of The Year in the Startup of The Year in the Business Intelligence industry here industry here The Startups in the Business Intelligence industry nominated for Startups of The Year 2024 include: Startups in Startups in Startups in the the Business Intelligence industry industry nominated Digital Fennec AQUi9 Oronts KROCKIO OLabs - UX Research Labs in Vietnam Co.Urban Postjer Group Slash Mark IT Startup Cywer Learning DesignerDudes Glider Aerial Solutions ByteCraft Studios Kinda Solutions Maitsys UseCodify U-Hertz Neem Infosys Humantek Tecoreng SupplyzPro Vangwe Tiyasha Software Private Limited Sankadi Altrium Navicent Solutions QOD Information Technology FastServe Ozeefy HostXolo MHS Planet Digital Fennec AQUi9 Oronts KROCKIO OLabs - UX Research Labs in Vietnam Co.Urban Postjer Group Slash Mark IT Startup Cywer Learning DesignerDudes Glider Aerial Solutions ByteCraft Studios Kinda Solutions Maitsys UseCodify U-Hertz Neem Infosys Humantek Tecoreng SupplyzPro Vangwe Tiyasha Software Private Limited Sankadi Altrium Navicent Solutions QOD Information Technology FastServe Ozeefy HostXolo MHS Planet “We are thrilled to recognize the top startups in the Business Intelligence industry and celebrate their achievements,” said David Smooke, HackerNoon Founder & CEO. “Starting and sustaining a company is difficult. We’re honored to showcase their work and impact.” “We are thrilled to recognize the top startups in the Business Intelligence industry and celebrate their achievements,” said David Smooke, HackerNoon Founder & CEO. “Starting and sustaining a company is difficult. We’re honored to showcase their work and impact.” “We are thrilled to recognize the top startups in and celebrate their achievements,” said David Smooke, HackerNoon Founder & CEO . “Starting and sustaining a company is difficult. We’re honored to showcase their work and impact.” David Smooke, HackerNoon Founder & CEO Why It Pays Off to Be Nominated Startups of The Year 2024 is a 100% community-driven voting event recognizing startups transforming technology and the world for the better. Starting October 1st, 2024, the internet can nominate and vote for the best ventures in their city and industry. People can nominate their favorite startups until October 31st , and vote once per day per award for who they think should be Startup of The Year up to March 31st, 2025. Winners will be announced in April 2025, after a comprehensive evaluation by the HackerNoon team. Startups of The Year 2024 nominate their favorite startups until October 31st nominate their favorite startups Apart from validation from the world's greenest tech publication, all nominees get free interviews relevant to their region and industry. They will also receive curated startup-friendly packages and a free version of their very own Evergreen Tech Company News Page on HackerNoon. get free interviews startup-friendly packages startup-friendly packages Evergreen Tech Company News Evergreen Tech Company News In sum, being nominated gives you major credibility as one of the top startups globally and an expert in your industry, meaning even without winning, you still gain tons of exposure. Nominations are open - learn more here. You can nominate your company as the Startup of The Year in the Business Intelligence industry here. Nominations are open - learn more here . You can nominate your company as the Startup of The Year in the Business Intelligence industry here . here Startup of The Year in the Startup of The Year in the Business Intelligence industry here industry here About HackerNoon’s Startups of The Year Startups of The Year 2024 is HackerNoon’s flagship community-driven event celebrating startups, technology, and the spirit of innovation. Currently in its third iteration, the prestigious Internet award recognizes and celebrates tech startups of all shapes and sizes. This year, over 150,000 entities across 4200+ cities, 6 continents, and 100+ industries will participate in a bid to be crowned the best startup of the year! Millions of votes have been cast over the past few years, and many stories have been written about these daring and rising startups. Startups of The Year 2024 Startups of The Year 2024 many stories many stories The winners will get a free interview on HackerNoon and an Evergreen Tech Company News page. free interview on HackerNoon Evergreen Tech Company News Evergreen Tech Company News Visit our FAQ page to learn more. FAQ FAQ Download our design assets here . here here Check out the Startups of The Year Merch Shop here . here here HackerNoon’s Startups of The Year is a branding opportunity unlike any other. Whether your goal is brand awareness or lead generation, HackerNoon has curated startup-friendly packages to solve your marketing challenges. startup-friendly packages startup-friendly packages Meet our sponsors: Meet our sponsors: Meet our sponsors: Wellfound: Join the #1 global, startup-focused community . At Wellfound, we're not just a job board—we're the place where top startup talent and the world's most exciting companies connect to build the future. Wellfound: Join the #1 global, startup-focused community Join the #1 global, startup-focused community Notion: Notion is trusted and loved by thousands of startups as their connected workspace—from building product roadmaps to tracking fundraising. Try Notion with unlimited AI, FREE for up to 6 months , to build and scale your company with one powerful tool. Get your offer now ! Notion: Try Notion with unlimited AI, FREE for up to 6 months Get your offer now ! Hubspot: If you’re looking for a smart CRM platform that meets the needs of small businesses, look no further than HubSpot. Seamlessly connect your data, teams and customers in one easy-to-use scalable platform that grows with your business. Get started for free . Hubspot: Get started for free Get started for free Bright Data: Startups that leverage public web data can make faster, data-driven decisions, giving them a competitive edge. With Bright Data’s scalable web data collection , businesses can grow from a small operation to an enterprise by harnessing insights at every stage. Bright Data: Bright Data’s scalable web data collection Bright Data’s scalable web data collection Algolia: Algolia NeuralSearch is the world's only AI end-to-end Search and Discovery Platform combining powerful keyword and natural language processing in a single API. Algolia: AI end-to-end Search and Discovery Platform AI end-to-end Search and Discovery Platform

Startups of The Year 2024: 23,222 Startups Nominated in the Business Intelligence Industry

2024-cü ilin startapları: Biznes intellekt sənayesində 23 222 startap namizəd oldu

Startups de l'any 2024: 23.222 startups nominades a la indústria de la intel·ligència empresarial

Startupy roku 2024: 23 222 nominovaných startupů v oboru Business Intelligence

Årets startups 2024: 23.222 startups nomineret i Business Intelligence Industry

Startups del año 2024: 23.222 startups nominadas en la industria de inteligencia empresarial

Startup Tahun 2024: 23.222 Startup Masuk Nominasi di Industri Business Intelligence

Startup dell'anno 2024: 23.222 startup nominate nel settore della Business Intelligence

2024 年のスタートアップ オブ ザ イヤー: ビジネス インテリジェンス業界でノミネートされたスタートアップ 23,222 社

Startups amin'ny taona 2024: 23,222 Startups voatendry amin'ny indostrian'ny faharanitan-tsaina

वर्ष २०२४ को स्टार्टअपहरू: बिजनेस इन्टेलिजेन्स उद्योगमा २३,२२२ स्टार्टअपहरू मनोनित

Стартапы года 2024: 23 222 стартапа, номинированных в сфере бизнес-аналитики

سال 2024 کے اسٹارٹ اپس: بزنس انٹیلی جنس انڈسٹری میں 23,222 اسٹارٹ اپ نامزد

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Experimental Datasets and Model

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Algorithms This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Algorithms

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Algorithms

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777.

Using Autodiff to Estimate Posterior Moments: Conclusion, Limitations, and References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Derivations Global Importance Sampling Massively Parallel Importance Sampling This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Derivations Global Importance Sampling Massively Parallel Importance Sampling

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Derivations

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Methods Of course, the contributions of this paper are not in computing the unbiased marginal likelihood estimator, which previously has been used in learning general probabilistic models, but instead our major contribution is a novel approach to computing key quantities of interest in Bayesian computation by applying the source term trick to the massively parallel marginal likelihood estimator. In particular, in the following sections, we outline in turn how to compute posterior expectations, marginals and samples. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors : Authors (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Methods Of course, the contributions of this paper are not in computing the unbiased marginal likelihood estimator, which previously has been used in learning general probabilistic models, but instead our major contribution is a novel approach to computing key quantities of interest in Bayesian computation by applying the source term trick to the massively parallel marginal likelihood estimator. In particular, in the following sections, we outline in turn how to compute posterior expectations, marginals and samples.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Methods

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Experiment We provide empirical results comparing the global and massively parallel importance weighting/sampling methods, as well as VI and HMC baselines. We considered two datasets: MovieLens100K (Fig. 1) and NYC Bus Breakdown (Fig. 2). MovieLens100K (Harper and Konstan 2015) contains 100k ratings of N=1682 films from among M=943 users. NYC Bus Breakdown describes the length of around 150,000 delays to New York school bus journeys, segregated by year, borough, bus company and journey type. We use hierarchical probabilistic graphical models for these datasets, which are described in more detail in Appendix section Experimental Datasets And Models. For global and massively parallel importance sampling, we use the prior as a proposal. For VI, we used a factorised approximate posterior and optimized using Adam with learning rates ranging from 10−2 to 10−4  (learning rates faster than 10−2 were unstable). For HMC, we used the NUTS (Hoffman, Gelman et al. 2014) implementation from PyMC (Salvatier, Wiecki, and Fonnesbeck 2016). We consider four quantities in Fig. 1 and Fig. 2. We begin by looking at the ELBO (Fig. 1a,2a). While massively parallel estimates of the ELBO can be computed using previous methods (e.g. Heap and Laurence 2023), it is nonetheless useful as it is a good measure of the quality of our approximate posteriors (as the ELBO can be written as the sum of the marginal likelihood and the KL-divergence between the true and approximate posterior Jordan et al. 1999; Kingma, Welling et al. 2019). Next, we plot the predictive log-likelihood, based on posterior samples obtained using the approach in the Methods section. In both of these plots, we see dramatic improvements against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other baselines such as VI and HMC (bottom row). Finally, we consider the quality of the posterior moment estimates computed using the approach in the Methods section. In particular we consider two measures of the quality of our posterior mean estimator. First, we consider the variance of the estimator (Fig. 1c,2c); in the ideal case, when we have exactly computed the posterior mean, this variance should be zero. While this variance can be zero in other cases (e.g. when the mean estimator always returns a constant), it does provide a lower-bound on the expected squared error between the estimator and the true value. Second, we directly considered the MSE between our posterior mean estimator and the true latent variable (Fig. 1d,2d). However, this quantity requires us to know the true value for the latent variable, which requires us to generate data from the model (we only generated data for d, columns a–c use real data). Again, in both of these cases, we see dramatic improvements for massively parallel against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other methods such as VI and HMC (bottom row). Note the extremely poor performance of VI and HMC is because these are iterative methods that take many steps, and hence a long time to reach good performance. HMC is especially problematic on these short timescales, because it needs many gradient steps before it returns even a single posterior sample (and this is ignoring burn-in and adaptation phases). In contrast, importance sampling gets a reasonable answer much more quickly, as it just a single, non-iterative computation. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Experiment We provide empirical results comparing the global and massively parallel importance weighting/sampling methods, as well as VI and HMC baselines. We considered two datasets: MovieLens100K (Fig. 1) and NYC Bus Breakdown (Fig. 2). MovieLens100K (Harper and Konstan 2015) contains 100k ratings of N=1682 films from among M=943 users. NYC Bus Breakdown describes the length of around 150,000 delays to New York school bus journeys, segregated by year, borough, bus company and journey type. We use hierarchical probabilistic graphical models for these datasets, which are described in more detail in Appendix section Experimental Datasets And Models. For global and massively parallel importance sampling, we use the prior as a proposal. For VI, we used a factorised approximate posterior and optimized using Adam with learning rates ranging from 10−2 to 10−4  (learning rates faster than 10−2 were unstable). For HMC, we used the NUTS (Hoffman, Gelman et al. 2014) implementation from PyMC (Salvatier, Wiecki, and Fonnesbeck 2016). We consider four quantities in Fig. 1 and Fig. 2. We begin by looking at the ELBO (Fig. 1a,2a). While massively parallel estimates of the ELBO can be computed using previous methods (e.g. Heap and Laurence 2023), it is nonetheless useful as it is a good measure of the quality of our approximate posteriors (as the ELBO can be written as the sum of the marginal likelihood and the KL-divergence between the true and approximate posterior Jordan et al. 1999; Kingma, Welling et al. 2019). Next, we plot the predictive log-likelihood, based on posterior samples obtained using the approach in the Methods section. In both of these plots, we see dramatic improvements against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other baselines such as VI and HMC (bottom row). Finally, we consider the quality of the posterior moment estimates computed using the approach in the Methods section. In particular we consider two measures of the quality of our posterior mean estimator. First, we consider the variance of the estimator (Fig. 1c,2c); in the ideal case, when we have exactly computed the posterior mean, this variance should be zero. While this variance can be zero in other cases (e.g. when the mean estimator always returns a constant), it does provide a lower-bound on the expected squared error between the estimator and the true value. Second, we directly considered the MSE between our posterior mean estimator and the true latent variable (Fig. 1d,2d). However, this quantity requires us to know the true value for the latent variable, which requires us to generate data from the model (we only generated data for d, columns a–c use real data). Again, in both of these cases, we see dramatic improvements for massively parallel against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other methods such as VI and HMC (bottom row). Note the extremely poor performance of VI and HMC is because these are iterative methods that take many steps, and hence a long time to reach good performance. HMC is especially problematic on these short timescales, because it needs many gradient steps before it returns even a single posterior sample (and this is ignoring burn-in and adaptation phases). In contrast, importance sampling gets a reasonable answer much more quickly, as it just a single, non-iterative computation.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Experiments

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Background This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Background

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Background

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Related work There is a considerable body of work in the discrete graphical model setting that computes posterior expectations, marginals and samples (Dawid 1992; Pfeffer 2005; Bidyuk and Dechter 2007; Geldenhuys, Dwyer, and Visser 2012; Gogate and Dechter 2012; Claret et al. 2013; Sankaranarayanan, Chakarov, and Gulwani 2013; Goodman and Stuhlmuller 2014; Gehr, Misailovic, and Vechev 2016; ¨ Narayanan et al. 2016; Albarghouthi et al. 2017; Dechter et al. 2018; Wang, Hoffmann, and Reps 2018; Obermeyer et al. 2019; Holtzen, Van den Broeck, and Millstein 2020). Our work differs in two respects. First, our massively parallel methods are not restricted to discrete graphical mod els, but can operate with arbitrary continuous latent variables and graphs with a mixture of continuous and discrete latent variables. Second, this prior work involves complex implementations that, in one sense or another, “proceed by recording an adjoint compute graph alongside the forward computation and then traversing the adjoint graph backwards starting from the final result of the forward computation” (Obermeyer et al. 2019). The forward computation is reasonably straightforward: it is just a big tensor product that can be computed efficiently using pre-existing libraries such as opt-einsum, and results in (an estimate of) the marginal likelihood. However, the backward traversal is much more complex, if for no other reason than the need to implement separate traversals for each operation of interest (computing posterior expectations, marginals and samples). Additionally, these traversals need to correctly handle all special cases, including optimized implementations of plates and timeseries. Importantly, optimizing the forward computation is usually quite straightforward while implementing an optimized backward traversal is far more complex. For instance, the forward computation for a timeseries involves a product of T matrices arranged in a chain. Naively computing this product on GPUs is very slow, as it requires T separate matrix multiplications. However, it is possible to massively optimize this forward computation, converting O(T) to O(log(T)) tensor operations by multiplying adjacent pairs of matrices in a single batched matrix multiplication operation. This optimization is straightforward in the forward computation. However, applying this optimization as part of the backward computation is far more complex (see Corenflos, Chopin, and Sarkk ¨ a 2022 for details). This ¨ complexity (along with similar complexity for other important optimizations such as plates) was prohibitive for academic teams implementing e.g. new probabilistic programming languages. Our key contribution is thus to provide a much simpler approach to directly compute posterior expectations, marginals and samples by differentiating through the forward computation, without having to hand-write and hand-optimize backward traversals. There is work on fitting importance weighted autoencoders (IWAE; Burda, Grosse, and Salakhutdinov 2015) and reweighted wake-sleep (RWS; Bornschein and Bengio 2014; Le et al. 2020) in the massively parallel setting (Aitchison 2019; Geffner and Domke 2022; Heap and Laurence 2023) for general probabilistic models. However, this work only provides methods for performing massively parallel updates to approximate posteriors (e.g. by optimizing a massively parallel ELBO). This work does not provide a method to individually reweight the samples to provide accurate posterior expectations, marginals and samples. Instead, this previous work simply takes the learned approximate posterior as an estimate of the true posterior, and does not attempt to correct for inevitable biases. Critically, our key contribution is not the massively parallel importance sampling method itself, which we acknowledge does bear similarities to e.g. particle filtering/SMC methods (Gordon, Salmond, and Smith 1993; Doucet, Johansen et al. 2009; Andrieu, Doucet, and Holenstein 2010; Maddison et al. 2017; Le et al. 2017; Lindsten et al. 2017; Naesseth et al. 2018; Kuntz, Crucinio, and Johansen 2023; Lai, Domke, and Sheldon 2022; Crucinio and Johansen 2023) that have been generalised to arbitrary graphical models and where the resampling step has been eliminated. Instead, our key contribution is the simple method for computing posterior expectations, marginals and samples without requiring the implementation of complex backwards traversals, and this has not appeared in past work. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Related work There is a considerable body of work in the discrete graphical model setting that computes posterior expectations, marginals and samples (Dawid 1992; Pfeffer 2005; Bidyuk and Dechter 2007; Geldenhuys, Dwyer, and Visser 2012; Gogate and Dechter 2012; Claret et al. 2013; Sankaranarayanan, Chakarov, and Gulwani 2013; Goodman and Stuhlmuller 2014; Gehr, Misailovic, and Vechev 2016; ¨ Narayanan et al. 2016; Albarghouthi et al. 2017; Dechter et al. 2018; Wang, Hoffmann, and Reps 2018; Obermeyer et al. 2019; Holtzen, Van den Broeck, and Millstein 2020). Our work differs in two respects. First, our massively parallel methods are not restricted to discrete graphical mod els, but can operate with arbitrary continuous latent variables and graphs with a mixture of continuous and discrete latent variables. Second, this prior work involves complex implementations that, in one sense or another, “proceed by recording an adjoint compute graph alongside the forward computation and then traversing the adjoint graph backwards starting from the final result of the forward computation” (Obermeyer et al. 2019). The forward computation is reasonably straightforward: it is just a big tensor product that can be computed efficiently using pre-existing libraries such as opt-einsum, and results in (an estimate of) the marginal likelihood. However, the backward traversal is much more complex, if for no other reason than the need to implement separate traversals for each operation of interest (computing posterior expectations, marginals and samples). Additionally, these traversals need to correctly handle all special cases, including optimized implementations of plates and timeseries. Importantly, optimizing the forward computation is usually quite straightforward while implementing an optimized backward traversal is far more complex. For instance, the forward computation for a timeseries involves a product of T matrices arranged in a chain. Naively computing this product on GPUs is very slow, as it requires T separate matrix multiplications. However, it is possible to massively optimize this forward computation, converting O(T) to O(log(T)) tensor operations by multiplying adjacent pairs of matrices in a single batched matrix multiplication operation. This optimization is straightforward in the forward computation. However, applying this optimization as part of the backward computation is far more complex (see Corenflos, Chopin, and Sarkk ¨ a 2022 for details). This ¨ complexity (along with similar complexity for other important optimizations such as plates) was prohibitive for academic teams implementing e.g. new probabilistic programming languages. Our key contribution is thus to provide a much simpler approach to directly compute posterior expectations, marginals and samples by differentiating through the forward computation, without having to hand-write and hand-optimize backward traversals. There is work on fitting importance weighted autoencoders (IWAE; Burda, Grosse, and Salakhutdinov 2015) and reweighted wake-sleep (RWS; Bornschein and Bengio 2014; Le et al. 2020) in the massively parallel setting (Aitchison 2019; Geffner and Domke 2022; Heap and Laurence 2023) for general probabilistic models. However, this work only provides methods for performing massively parallel updates to approximate posteriors (e.g. by optimizing a massively parallel ELBO). This work does not provide a method to individually reweight the samples to provide accurate posterior expectations, marginals and samples. Instead, this previous work simply takes the learned approximate posterior as an estimate of the true posterior, and does not attempt to correct for inevitable biases. Critically, our key contribution is not the massively parallel importance sampling method itself, which we acknowledge does bear similarities to e.g. particle filtering/SMC methods (Gordon, Salmond, and Smith 1993; Doucet, Johansen et al. 2009; Andrieu, Doucet, and Holenstein 2010; Maddison et al. 2017; Le et al. 2017; Lindsten et al. 2017; Naesseth et al. 2018; Kuntz, Crucinio, and Johansen 2023; Lai, Domke, and Sheldon 2022; Crucinio and Johansen 2023) that have been generalised to arbitrary graphical models and where the resampling step has been eliminated. Instead, our key contribution is the simple method for computing posterior expectations, marginals and samples without requiring the implementation of complex backwards traversals, and this has not appeared in past work.

Stories, News & Mentions about POSTJER

Postjer’s Stories on HackerNoon

Postjer's News & Mentions