PostEra is a biotechnology company backed by YCombinator (YC W20)

PostEra

postera

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Experimental Datasets and Model

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Algorithms This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Algorithms

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Algorithms

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777.

Using Autodiff to Estimate Posterior Moments: Conclusion, Limitations, and References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Derivations Global Importance Sampling Massively Parallel Importance Sampling This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Derivations Global Importance Sampling Massively Parallel Importance Sampling

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Derivations

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Methods Of course, the contributions of this paper are not in computing the unbiased marginal likelihood estimator, which previously has been used in learning general probabilistic models, but instead our major contribution is a novel approach to computing key quantities of interest in Bayesian computation by applying the source term trick to the massively parallel marginal likelihood estimator. In particular, in the following sections, we outline in turn how to compute posterior expectations, marginals and samples. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors : Authors (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Methods Of course, the contributions of this paper are not in computing the unbiased marginal likelihood estimator, which previously has been used in learning general probabilistic models, but instead our major contribution is a novel approach to computing key quantities of interest in Bayesian computation by applying the source term trick to the massively parallel marginal likelihood estimator. In particular, in the following sections, we outline in turn how to compute posterior expectations, marginals and samples.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Methods

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Experiment We provide empirical results comparing the global and massively parallel importance weighting/sampling methods, as well as VI and HMC baselines. We considered two datasets: MovieLens100K (Fig. 1) and NYC Bus Breakdown (Fig. 2). MovieLens100K (Harper and Konstan 2015) contains 100k ratings of N=1682 films from among M=943 users. NYC Bus Breakdown describes the length of around 150,000 delays to New York school bus journeys, segregated by year, borough, bus company and journey type. We use hierarchical probabilistic graphical models for these datasets, which are described in more detail in Appendix section Experimental Datasets And Models. For global and massively parallel importance sampling, we use the prior as a proposal. For VI, we used a factorised approximate posterior and optimized using Adam with learning rates ranging from 10−2 to 10−4  (learning rates faster than 10−2 were unstable). For HMC, we used the NUTS (Hoffman, Gelman et al. 2014) implementation from PyMC (Salvatier, Wiecki, and Fonnesbeck 2016). We consider four quantities in Fig. 1 and Fig. 2. We begin by looking at the ELBO (Fig. 1a,2a). While massively parallel estimates of the ELBO can be computed using previous methods (e.g. Heap and Laurence 2023), it is nonetheless useful as it is a good measure of the quality of our approximate posteriors (as the ELBO can be written as the sum of the marginal likelihood and the KL-divergence between the true and approximate posterior Jordan et al. 1999; Kingma, Welling et al. 2019). Next, we plot the predictive log-likelihood, based on posterior samples obtained using the approach in the Methods section. In both of these plots, we see dramatic improvements against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other baselines such as VI and HMC (bottom row). Finally, we consider the quality of the posterior moment estimates computed using the approach in the Methods section. In particular we consider two measures of the quality of our posterior mean estimator. First, we consider the variance of the estimator (Fig. 1c,2c); in the ideal case, when we have exactly computed the posterior mean, this variance should be zero. While this variance can be zero in other cases (e.g. when the mean estimator always returns a constant), it does provide a lower-bound on the expected squared error between the estimator and the true value. Second, we directly considered the MSE between our posterior mean estimator and the true latent variable (Fig. 1d,2d). However, this quantity requires us to know the true value for the latent variable, which requires us to generate data from the model (we only generated data for d, columns a–c use real data). Again, in both of these cases, we see dramatic improvements for massively parallel against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other methods such as VI and HMC (bottom row). Note the extremely poor performance of VI and HMC is because these are iterative methods that take many steps, and hence a long time to reach good performance. HMC is especially problematic on these short timescales, because it needs many gradient steps before it returns even a single posterior sample (and this is ignoring burn-in and adaptation phases). In contrast, importance sampling gets a reasonable answer much more quickly, as it just a single, non-iterative computation. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Experiment We provide empirical results comparing the global and massively parallel importance weighting/sampling methods, as well as VI and HMC baselines. We considered two datasets: MovieLens100K (Fig. 1) and NYC Bus Breakdown (Fig. 2). MovieLens100K (Harper and Konstan 2015) contains 100k ratings of N=1682 films from among M=943 users. NYC Bus Breakdown describes the length of around 150,000 delays to New York school bus journeys, segregated by year, borough, bus company and journey type. We use hierarchical probabilistic graphical models for these datasets, which are described in more detail in Appendix section Experimental Datasets And Models. For global and massively parallel importance sampling, we use the prior as a proposal. For VI, we used a factorised approximate posterior and optimized using Adam with learning rates ranging from 10−2 to 10−4  (learning rates faster than 10−2 were unstable). For HMC, we used the NUTS (Hoffman, Gelman et al. 2014) implementation from PyMC (Salvatier, Wiecki, and Fonnesbeck 2016). We consider four quantities in Fig. 1 and Fig. 2. We begin by looking at the ELBO (Fig. 1a,2a). While massively parallel estimates of the ELBO can be computed using previous methods (e.g. Heap and Laurence 2023), it is nonetheless useful as it is a good measure of the quality of our approximate posteriors (as the ELBO can be written as the sum of the marginal likelihood and the KL-divergence between the true and approximate posterior Jordan et al. 1999; Kingma, Welling et al. 2019). Next, we plot the predictive log-likelihood, based on posterior samples obtained using the approach in the Methods section. In both of these plots, we see dramatic improvements against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other baselines such as VI and HMC (bottom row). Finally, we consider the quality of the posterior moment estimates computed using the approach in the Methods section. In particular we consider two measures of the quality of our posterior mean estimator. First, we consider the variance of the estimator (Fig. 1c,2c); in the ideal case, when we have exactly computed the posterior mean, this variance should be zero. While this variance can be zero in other cases (e.g. when the mean estimator always returns a constant), it does provide a lower-bound on the expected squared error between the estimator and the true value. Second, we directly considered the MSE between our posterior mean estimator and the true latent variable (Fig. 1d,2d). However, this quantity requires us to know the true value for the latent variable, which requires us to generate data from the model (we only generated data for d, columns a–c use real data). Again, in both of these cases, we see dramatic improvements for massively parallel against standard “global” importance sampling for a fixed K (top row), and for a fixed time against other methods such as VI and HMC (bottom row). Note the extremely poor performance of VI and HMC is because these are iterative methods that take many steps, and hence a long time to reach good performance. HMC is especially problematic on these short timescales, because it needs many gradient steps before it returns even a single posterior sample (and this is ignoring burn-in and adaptation phases). In contrast, importance sampling gets a reasonable answer much more quickly, as it just a single, non-iterative computation.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Experiments

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Background This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Background

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Background

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Related work There is a considerable body of work in the discrete graphical model setting that computes posterior expectations, marginals and samples (Dawid 1992; Pfeffer 2005; Bidyuk and Dechter 2007; Geldenhuys, Dwyer, and Visser 2012; Gogate and Dechter 2012; Claret et al. 2013; Sankaranarayanan, Chakarov, and Gulwani 2013; Goodman and Stuhlmuller 2014; Gehr, Misailovic, and Vechev 2016; ¨ Narayanan et al. 2016; Albarghouthi et al. 2017; Dechter et al. 2018; Wang, Hoffmann, and Reps 2018; Obermeyer et al. 2019; Holtzen, Van den Broeck, and Millstein 2020). Our work differs in two respects. First, our massively parallel methods are not restricted to discrete graphical mod els, but can operate with arbitrary continuous latent variables and graphs with a mixture of continuous and discrete latent variables. Second, this prior work involves complex implementations that, in one sense or another, “proceed by recording an adjoint compute graph alongside the forward computation and then traversing the adjoint graph backwards starting from the final result of the forward computation” (Obermeyer et al. 2019). The forward computation is reasonably straightforward: it is just a big tensor product that can be computed efficiently using pre-existing libraries such as opt-einsum, and results in (an estimate of) the marginal likelihood. However, the backward traversal is much more complex, if for no other reason than the need to implement separate traversals for each operation of interest (computing posterior expectations, marginals and samples). Additionally, these traversals need to correctly handle all special cases, including optimized implementations of plates and timeseries. Importantly, optimizing the forward computation is usually quite straightforward while implementing an optimized backward traversal is far more complex. For instance, the forward computation for a timeseries involves a product of T matrices arranged in a chain. Naively computing this product on GPUs is very slow, as it requires T separate matrix multiplications. However, it is possible to massively optimize this forward computation, converting O(T) to O(log(T)) tensor operations by multiplying adjacent pairs of matrices in a single batched matrix multiplication operation. This optimization is straightforward in the forward computation. However, applying this optimization as part of the backward computation is far more complex (see Corenflos, Chopin, and Sarkk ¨ a 2022 for details). This ¨ complexity (along with similar complexity for other important optimizations such as plates) was prohibitive for academic teams implementing e.g. new probabilistic programming languages. Our key contribution is thus to provide a much simpler approach to directly compute posterior expectations, marginals and samples by differentiating through the forward computation, without having to hand-write and hand-optimize backward traversals. There is work on fitting importance weighted autoencoders (IWAE; Burda, Grosse, and Salakhutdinov 2015) and reweighted wake-sleep (RWS; Bornschein and Bengio 2014; Le et al. 2020) in the massively parallel setting (Aitchison 2019; Geffner and Domke 2022; Heap and Laurence 2023) for general probabilistic models. However, this work only provides methods for performing massively parallel updates to approximate posteriors (e.g. by optimizing a massively parallel ELBO). This work does not provide a method to individually reweight the samples to provide accurate posterior expectations, marginals and samples. Instead, this previous work simply takes the learned approximate posterior as an estimate of the true posterior, and does not attempt to correct for inevitable biases. Critically, our key contribution is not the massively parallel importance sampling method itself, which we acknowledge does bear similarities to e.g. particle filtering/SMC methods (Gordon, Salmond, and Smith 1993; Doucet, Johansen et al. 2009; Andrieu, Doucet, and Holenstein 2010; Maddison et al. 2017; Le et al. 2017; Lindsten et al. 2017; Naesseth et al. 2018; Kuntz, Crucinio, and Johansen 2023; Lai, Domke, and Sheldon 2022; Crucinio and Johansen 2023) that have been generalised to arbitrary graphical models and where the resampling step has been eliminated. Instead, our key contribution is the simple method for computing posterior expectations, marginals and samples without requiring the implementation of complex backwards traversals, and this has not appeared in past work. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Related work There is a considerable body of work in the discrete graphical model setting that computes posterior expectations, marginals and samples (Dawid 1992; Pfeffer 2005; Bidyuk and Dechter 2007; Geldenhuys, Dwyer, and Visser 2012; Gogate and Dechter 2012; Claret et al. 2013; Sankaranarayanan, Chakarov, and Gulwani 2013; Goodman and Stuhlmuller 2014; Gehr, Misailovic, and Vechev 2016; ¨ Narayanan et al. 2016; Albarghouthi et al. 2017; Dechter et al. 2018; Wang, Hoffmann, and Reps 2018; Obermeyer et al. 2019; Holtzen, Van den Broeck, and Millstein 2020). Our work differs in two respects. First, our massively parallel methods are not restricted to discrete graphical mod els, but can operate with arbitrary continuous latent variables and graphs with a mixture of continuous and discrete latent variables. Second, this prior work involves complex implementations that, in one sense or another, “proceed by recording an adjoint compute graph alongside the forward computation and then traversing the adjoint graph backwards starting from the final result of the forward computation” (Obermeyer et al. 2019). The forward computation is reasonably straightforward: it is just a big tensor product that can be computed efficiently using pre-existing libraries such as opt-einsum, and results in (an estimate of) the marginal likelihood. However, the backward traversal is much more complex, if for no other reason than the need to implement separate traversals for each operation of interest (computing posterior expectations, marginals and samples). Additionally, these traversals need to correctly handle all special cases, including optimized implementations of plates and timeseries. Importantly, optimizing the forward computation is usually quite straightforward while implementing an optimized backward traversal is far more complex. For instance, the forward computation for a timeseries involves a product of T matrices arranged in a chain. Naively computing this product on GPUs is very slow, as it requires T separate matrix multiplications. However, it is possible to massively optimize this forward computation, converting O(T) to O(log(T)) tensor operations by multiplying adjacent pairs of matrices in a single batched matrix multiplication operation. This optimization is straightforward in the forward computation. However, applying this optimization as part of the backward computation is far more complex (see Corenflos, Chopin, and Sarkk ¨ a 2022 for details). This ¨ complexity (along with similar complexity for other important optimizations such as plates) was prohibitive for academic teams implementing e.g. new probabilistic programming languages. Our key contribution is thus to provide a much simpler approach to directly compute posterior expectations, marginals and samples by differentiating through the forward computation, without having to hand-write and hand-optimize backward traversals. There is work on fitting importance weighted autoencoders (IWAE; Burda, Grosse, and Salakhutdinov 2015) and reweighted wake-sleep (RWS; Bornschein and Bengio 2014; Le et al. 2020) in the massively parallel setting (Aitchison 2019; Geffner and Domke 2022; Heap and Laurence 2023) for general probabilistic models. However, this work only provides methods for performing massively parallel updates to approximate posteriors (e.g. by optimizing a massively parallel ELBO). This work does not provide a method to individually reweight the samples to provide accurate posterior expectations, marginals and samples. Instead, this previous work simply takes the learned approximate posterior as an estimate of the true posterior, and does not attempt to correct for inevitable biases. Critically, our key contribution is not the massively parallel importance sampling method itself, which we acknowledge does bear similarities to e.g. particle filtering/SMC methods (Gordon, Salmond, and Smith 1993; Doucet, Johansen et al. 2009; Andrieu, Doucet, and Holenstein 2010; Maddison et al. 2017; Le et al. 2017; Lindsten et al. 2017; Naesseth et al. 2018; Kuntz, Crucinio, and Johansen 2023; Lai, Domke, and Sheldon 2022; Crucinio and Johansen 2023) that have been generalised to arbitrary graphical models and where the resampling step has been eliminated. Instead, our key contribution is the simple method for computing posterior expectations, marginals and samples without requiring the implementation of complex backwards traversals, and this has not appeared in past work.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Related Work

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Abstract Importance sampling is a popular technique in Bayesian inference: by reweighting samples drawn from a proposal distribution we are able to obtain samples and moment estimates from a Bayesian posterior over some n latent variables. Recent work, however, indicates that importance sampling scales poorly — in order to accurately approximate the true posterior, the required number of importance samples grows is exponential in the number of latent variables (Chatterjee and Diaconis 2018). Massively parallel importance sampling works around this issue by drawing K samples for each of the n latent variables and reasoning about all Kn combinations of latent samples. In principle, we can reason efficiently over Kn combinations of samples by exploiting conditional independencies in the generative model. However, in practice this requires complex algorithms that traverse backwards through the graphical model, and we need separate backward traversals for each computation (posterior expectations, marginals and samples). Our contribution is to exploit the source term trick from physics to entirely avoid the need to hand-write backward traversals. Instead, we demonstrate how to simply and easily compute all the required quantities — posterior expectations, marginals and samples — by differentiating through a slightly modified marginal likelihood estimator. Introduction Importance weighting allows us to reweight samples drawn from a proposal in order to compute expectations of a different distribution, such as a Bayesian posterior. However, importance weighting breaks down in larger models. Chatterjee and Diaconis (2018) showed that the number of samples required to accurately approximate the true posterior scales as  exp (DKL (P (z|x)||Q (z))), where P (z|x) is the true posterior over latent variables, z, given data x, and Q (z) is the proposal. Problematically, we expect the KL divergence to scale with n, the number of latent variables. Indeed, if z is composed of n latent variables, and P (z|x) and Q (z) are IID over those n latent variables, then the KL-divergence is exactly proportional to n. Thus, we expect the required number of importance samples to be exponential in the number of latent variables, and hence we expect accurate importance sampling to be intractable in larger models. To resolve this issue we use a massively parallel importance sampling scheme that in effect uses an exponential number of samples to compute posterior expectations, marginals and samples (Kuntz, Crucinio, and Johansen 2023; Heap and Laurence 2023). This involves drawing K samples of each of the n latent variables from the proposal, then individually reweighting all Kn combinations of all samples of all latent variables. While reasoning about all Kn combinations of samples might seem intractable, we should in principle be able to perform efficient computations by exploiting conditional independencies in the underlying graphical model. However, many computations that are possible in principle are extremely complex in practice, and that turns out to be the case here. We noticed that we could perhaps perform this reasoning over Kn latent variables using methods from the discrete graphical model literature. This turned out to be less helpful than we had hoped because these algorithms involve highly complex backward traversals of the generative model. Worse, different traversals are needed for computing posterior expectations, marginals and samples, making a general implementation challenging. Our contribution is to develop a much simpler approach to computing posterior expectations, marginals and samples, which entirely avoids the need to explicitly write backwards computations. Specifically, we show that posterior expectations, marginals and samples can be obtained simply by differentiating through (a slightly modified) forward computation that produces an estimate of the marginal likelihood. The required gradients can be computed straightforwardly using modern autodiff, and the resulting implicit backward computations automatically inherit potentially complex optimizations from the forward pass. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Abstract Importance sampling is a popular technique in Bayesian inference: by reweighting samples drawn from a proposal distribution we are able to obtain samples and moment estimates from a Bayesian posterior over some n latent variables. Recent work, however, indicates that importance sampling scales poorly — in order to accurately approximate the true posterior, the required number of importance samples grows is exponential in the number of latent variables (Chatterjee and Diaconis 2018). Massively parallel importance sampling works around this issue by drawing K samples for each of the n latent variables and reasoning about all Kn combinations of latent samples. In principle, we can reason efficiently over Kn combinations of samples by exploiting conditional independencies in the generative model. However, in practice this requires complex algorithms that traverse backwards through the graphical model, and we need separate backward traversals for each computation (posterior expectations, marginals and samples). Our contribution is to exploit the source term trick from physics to entirely avoid the need to hand-write backward traversals. Instead, we demonstrate how to simply and easily compute all the required quantities — posterior expectations, marginals and samples — by differentiating through a slightly modified marginal likelihood estimator. n Introduction Importance weighting allows us to reweight samples drawn from a proposal in order to compute expectations of a different distribution, such as a Bayesian posterior. However, importance weighting breaks down in larger models. Chatterjee and Diaconis (2018) showed that the number of samples required to accurately approximate the true posterior scales as  exp (DKL (P (z|x)||Q (z))), where P (z|x) is the true posterior over latent variables, z, given data x, and Q (z) is the proposal. Problematically, we expect the KL divergence to scale with n, the number of latent variables. Indeed, if z is composed of n latent variables, and P (z|x) and Q (z) are IID over those n latent variables, then the KL-divergence is exactly proportional to n. Thus, we expect the required number of importance samples to be exponential in the number of latent variables, and hence we expect accurate importance sampling to be intractable in larger models. To resolve this issue we use a massively parallel importance sampling scheme that in effect uses an exponential number of samples to compute posterior expectations, marginals and samples (Kuntz, Crucinio, and Johansen 2023; Heap and Laurence 2023). This involves drawing K samples of each of the n latent variables from the proposal, then individually reweighting all Kn combinations of all samples of all latent variables. While reasoning about all Kn combinations of samples might seem intractable, we should in principle be able to perform efficient computations by exploiting conditional independencies in the underlying graphical model. However, many computations that are possible in principle are extremely complex in practice, and that turns out to be the case here. We noticed that we could perhaps perform this reasoning over Kn latent variables using methods from the discrete graphical model literature. This turned out to be less helpful than we had hoped because these algorithms involve highly complex backward traversals of the generative model. Worse, different traversals are needed for computing posterior expectations, marginals and samples, making a general implementation challenging. Our contribution is to develop a much simpler approach to computing posterior expectations, marginals and samples, which entirely avoids the need to explicitly write backwards computations. Specifically, we show that posterior expectations, marginals and samples can be obtained simply by differentiating through (a slightly modified) forward computation that produces an estimate of the marginal likelihood. The required gradients can be computed straightforwardly using modern autodiff, and the resulting implicit backward computations automatically inherit potentially complex optimizations from the forward pass.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Abstract & Introduction

— Scientific American, Volume XXXVI., No. 8, February 24, 1877 by Various, is part of the HackerNoon Books Series. You can jump to any chapter in this book . For Posterity here A Suggestion FOR POSTERITY—A SUGGESTION. The Irish gentleman who declined to aid an enterprise for the benefit of posterity, remarking that posterity had never done anything for him, was, after all the sport made of him, no unfair representative of the bulk of mankind. There is talk enough about doing great things for the advantage of future ages, but the real motive is apt to be something very different. To perpetuate their own name or fame, men or nations often set up lasting monuments, and sometimes unintentionally convey thereby to after times a few more or less instructive indications of the artistic or industrial skill of their day and generation. To further their own immediate ends, or to secure some benefit to their immediate descendants, men frequently undertake great material enterprises, and sometimes the work so done remains for ages the source of perennial good. But very rarely, if ever, can it be said that any work of man was undertaken solely, or even chiefly, for the benefit of posterity—more rarely still, for remote posterity. Hence it happens that we owe far more to accident, to fire, rapine, volcanic outbursts, and the protecting care of desolation, for the knowledge we have of times long past, than to any intentional legacies of art or learning left us by the men of those times. The lost and abandoned tools, weapons, and ornaments of the stone age are all that we have to tell us of the childhood of humanity. Had no fiery disasters ever overtaken the pile-dwellers of the Swiss lakes, we should probably have never heard of such a people. To the mud and ashes of Vesuvius, rather than to the historians of the Roman Empire, we owe the best of our knowledge of how Roman cities looked and Roman citizens lived eighteen hundred years ago. In the fragments of a library, buried in the ruins of a royal palace, we find almost our only records of the arts and sciences of ancient Assyria. Under the ash heaps of a forgotten age, in Cyprus, Cesnola finds the only known vestiges of a primitive civilization, reaching far back into the domain of mythology. Thanks to the destroyers of Troy and Mycenæ, and the protective care of temporary oblivion, Schliemann is now able to verify tradition and lay before an astonished and delighted world numerous precious relics of heroic ages hitherto remembered only in song. terra cotta Who can estimate the value of these and similar findings to us—the value of the revelations they bring of man's condition in those remote ages? Who can say how many or how few the ages will be ere the time comes when the antiquaries of the future will be rejoicing over equally fragmentary vestiges of the doings and possessions of our day? On the other hand, who can estimate the value of the knowledge lost beyond hope of recovery, or the checks to human progress experienced, in the repeated wiping out, so to speak, of the higher races and the civilizations they embodied? And who can say that similar disasters may not come again and again to humanity? Suppose a pestilence peculiarly fatal to the white race should fall upon the world to-day, crippling, perhaps exterminating, the now dominant civilized nations; how long would the material elements of our science and art or general culture remain with power to enlighten the barbarous tribes that would inherit the earth? Human progress has more than once been set back for centuries by such natural or unnatural causes, leaving the sites of once splendid civilizations to be overrun with barbaric hordes knowing nothing of the better times that went before. Suppose, again, that, by one of those geologic changes so numerous in the history of our unstable globe, the existing continents should sink a thousand feet. Every center of modern civilization would be submerged. The great social and political organizations of humanity would be broken up, and in the wreck of nations that would ensue very little of the glory and culture of the race could survive. The earth is dotted with vestiges of lost and forgotten empires. Can we reasonably assume, in the face of such facts, that the nations of to-day are immortal? The question is: Shall we continue to trust to chance, as all other civilizations have, for the preservation of the conquests we have made among the forces and secrets of nature; or shall we do something positive for posterity, and leave the ages to come some certain and abiding legacy of our treasures of art and learning? It may be that human progress will go on and on to the end of time without a break; that in the course of centuries mankind will surpass us in civilization, knowledge and power, as much as we surpass the earliest and rudest men we have yet found traces of: maybe infinitely more. In such a case, what would not the scholars of, say the year 5000 A.D., or any other future age, be willing to give for a comprehensive picture of humanity as it exists to-day—for a reasonably complete library of our literature, science, and art? We may safely assume that nothing of the sort will be possible if matters are left to take their natural course. By that time every structure, every machine, every book, every work of art, now in use or stored away in our libraries and galleries of art, will have disappeared, a prey to time, the elements, or the more destructive violence of man. On the other hand, it may be that, through repeated disasters of one sort or another, mankind, three thousand years hence, will have lost all the knowledge men ever possessed, and be slowly struggling upward for the hundredth time from inherited barbarism. In such a case, what enormous benefits might not accrue to man from a fortunate opening up of the wealth of knowledge we possess! In any supposable case between these extremes of progress or degradation, a legacy of art and learning, such as we might easily set apart for remote posterity, would certainly be acceptable, perhaps extremely useful. Besides, it might be possible for us to set such a worthy example to those who shall come after us that, come what might, humanity would never be left absolutely void of the means of instruction, nor any worthy human achievement be absolutely lost or forgotten. The experience of these later years has demonstrated the value of such legacies even when unintentional, unselected, and wretchedly fragmentary. It has made clear also how a legacy deliberately made may be indefinitely preserved. Roughly outlined, the carrying out of such a truly philanthropic enterprise would involve nothing more difficult than— The construction of a practically indestructible treasure chamber in some secure place; and First. The preparation of a library well calculated to withstand the corroding tooth of time. Second. Two kinds of structures would meet the first demand—massive pyramids of covered earth or of solid masonry, or chambers hewn from the heart of some granitic hill. In low latitudes, where glacial action is not to be feared, the pyramidal form might be preferable: in more northern regions the rock-cut chamber would probably be at once cheaper and more durable. In either case, an elevated site should be chosen as a safeguard against submergence. To secure the permanence of the records would be more difficult. Ordinary books and papers would clearly be unsuitable for long keeping; though for comparatively limited periods they might answer if securely packed in airtight waterproof cases. Nothing liable to spontaneous decay should be admitted. Stereotype plates of metal would be even more open to objection than printed sheets. The noble metals would be too costly, the baser would corrode; and with either the value of the plates as metal would be a standing danger to the deposit. The material basis of the library must be, as nearly as possible, worthless for other uses (to insure them against the natural greed of man), yet such as will hold the records sharply and faithfully under all circumstances. The tablets of ancient Assyria are instructive in this connection. Possibly plates of artificial stone, or sheets of a -like preparation of asbestos, might be less bulky and equally durable. terra cotta papier-maché Having determined this point, and dug from the solid rock a chamber for the reception of our legacy, the next step would be the selection of its contents. Obviously the books to be preserved should embrace first of all lexicons and grammars of every known form of speech, since it is impossible to tell which of the dialects of to-day will be the parents of the dominant tongue of any distant future time; while we may be practically certain that some one or more of the languages of to-day will furnish a key to any language that men will ever use. Next in order would come encyclopædias, the most comprehensive and complete that there might be room for. The sacred books of all nations might come next; then the works of the great poets, historians and novelists; after them, the best obtainable records of art, science, the various industries, and so on, with specimens of the best and most typical of our works of art, manufacture, and the like. The spaces between the various articles should be filled in with some insoluble and neutral substance, to prevent corrosion, or the infiltration of water and consequent damage to the plates. Then, the entrance to the chamber being securely sealed, permanent records should be made in many places and in various ways, setting forth the purpose of the deposit, its exact location, and the nature of its contents. Among such records not the least valuable would be deeply cut polyglot inscriptions on natural cliffs in different parts of the world, observation having shown that such records may remain to challenge human curiosity for ages after all other records of their time have disappeared. Even a single deposit of this sort might prove of enormous value to the race at some critical period of its history. But the probability is that the good work would not end with one deposit. From age to age this and other nations might repeat the experiment, commemorating in this way important epochs in their history. The fashion once set might easily become a permanent feature of all great national celebrations. The cost would be comparatively small: a penny contribution from each of the visitors to the Philadelphia Exhibition, for example, would have been quite sufficient to provide for a memorial of our first Centennial year that would have carried an imperishable picture of the civilization of the day to the end of—our first millennium, at least; and we may safely infer that, whatever may be the condition of the world at that not very remote epoch, a memorial of that sort would be something worth having. As we have intimated, the custom might easily become general, so that in the course of ages the earth would become dotted with such repositories of art and learning. Then, come what might to humanity—whatever might be the ups and downs of nations—whatever moral, social, or intellectual advances mankind might make—whatever lapses or disasters might befall them—it could hardly happen that a knowledge of any considerable period of human history, or the advantage of any worthy human achievement, could ever be permanently blotted out and lost. It is true that "posterity" has never done anything of the sort for us. It is true that "posterity" may have no valid claim on us for such a legacy. But we might venture to make "posterity" a present! It would not cost us much, and it might turn out to be immensely valuable and useful to some far future age. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Various (2006). Scientific American, Volume XXXVI., No. 8, February 24, 1877. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/19406/pg19406-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at www.gutenberg.org https://www.gutenberg.org/policy/license.html

FOR POSTERITY—A SUGGESTION

BLEISTEIN v. DONALDSON LITHOGRAPHING CO. 239 Court Filing, retrieved on September 29, 2023 is part of . You can jump to any part in this filing . This part is 1 of 4. HackerNoon’s Legal PDF Series here Argument for Plaintiffs in Error Chromolithographs representing actual groups of persons and things, which have been designed from hints or descriptions of the scenes represented, and which are to be used as advertisements for a circus are "pictorial illustrations" within the meaning of Rev. Stat. § 4952, allowing a copyright to the "author, designer, or proprietor ... of any engraving, cut, print... . or chromo" as affected by the act of 1874, chap. 301, § 3, 18 Stat. 78, 79. And on complying with all the statutory requirements the proprietors are entitled to the protection of the copyright laws. The case is stated in the opinion of the court. r von Briesen for plaintiffs in error: Mr. Ansley Wilcox and Mr. Arthu This action comes here upon writ of error to the Circuit Court of Appeals for the Sixth Circuit, which court heard it on writ of error directed to the United States Circuit Court for the District of Kentucky. The Circuit Court, at the close of plaintiffs' case, instructed the jury to find a verdict for defendant, which was done and judgment entered thereon. The Circuit Court of Appeals affirmed said judgment. There were three causes of action which were all based upon sec. 4965 of the Revised Statutes, quoted on page 60. By order of the Circuit Court, dated June 10, 1899, the marshal seized 10,590 eight-page prints and 13,205 four-page prints, described in the writ, and also five metal electrotype plates, all of which he found in the defendant's possession (page 13). The action was tried at Covington, Kentucky, on December 12 and 13, 1899, before Hon. Walter Evans, sitting as Circuit Judge, and a jury. At the outset of the trial, during the direct examination of the first witness, the court anticipated the question upon which it afterwards took the case away from the jury and decided it, by the following remark: "The real controversy will able whether this is a subject of copyright, whether it comes within the Copyright law." At the close of the plaintiffs' case, defendant moved for "peremptory instructions for the defendant." The court said, " State why, in a word," to which defendant's counsel answered: "In the first place I want to say with reference to the Statuary Exhibit. . . . It is alleged in the petition, and is in fact copyrighted on the 18th of April, and the publication plainly shows it was prior to that. That is a specific objection to that one upon that ground specifically-that is the Statuary. "The Court: Now as to the other two. "Counsel: The specific objection to this one, the Ballet, is that it is an immoral picture. "And the general objection that I make to them all is that they are none of them subject matter of copyright. They are all mere matter of advertising." The next day the court delivered a written opinion which concludes as follows: "The case must turn upon the others (other questions), and especially upon the general proposition that the things copyrighted in this case were by no means such as either the Constitution or the legislation of Congress intended to protect by the privilege of copyright. The court cannot bring its mind to yield to the conclusion that such tawdry pictures as these were ever meant to be given the enormous protection of not only the exclusive right to print them, but the additional protection of a penalty of a dollar each for reprints from them. "As previously stated, they are neither' pictorial illustrations' nor' works connected with the fine arts' within the meaning of section 4952. Not being so, there was no authority to grant the copyrights, whether the Constitution authorizes Congress to promote the fine arts or not. "The judgment of the court is, that the plaintiffs, on their own showing, are not entitled to recover, and for that reason the motion of defendant will be granted, and I will instruct the jury to find a verdict for it." The jury, in accordance with said instruction, returned a verdict for the defendant. There is no question as to the fact of infringement. The sheets in evidence, made by defendant, contain reproductions by means of cheap .electrotype plates of each of the plaintiffs' designs. These reproductions are not in colors. The principal questions are: First. Whether on the question of artistic merit or value of these lithographic prints or chromos, the Circuit Court was justified in taking the case from the jury, and condemning them entirely as not being fit subjects for copyright. Second. Whether the copyrights were obtained for these prints in accordance with the Constitution and laws of the United States, and are valid copyrights. The second question involves the inquiries: Whether the copyrights were properly taken out by the plaintiffs, in their trade names of "The Courier Co." and "The Courier Lithographing Co.," and, incidentally, whether plaintiffs have the right to sue in their individual names for infringement of these copyrights; and whether the Statuary Act Design was copyrighted before it was published. YOJ. CLXXXVIII-16. The three pictures in question are show-bills or circus bills, also called posters and, more definitely, picture-posters. They are colored lithographs or chromolithographs, commonly called " chromos." They were designed primarily to be sold to the proprietors of circuses and other shows, and by them to be used for advertising; but they could be sold to any one, or used for any purpose for which they were adapted. They were made in the plaintiffs' lithographing establishment under a special contract with the proprietor of a circus, by which the plaintiffs agreed to design and get up certain representations of scenes supposed to be exhibited at the show, the plaintiffs reserving rights of design and of copyright, and with the usual understanding that so long as the proprietor of the circus used these designs he had the right to them, but if he ceased to use any of them, the plaintiffs could sell the design or the pictures which embodied it, to any one. The fundamental question of the right to copyright such show-bills or posters, is a question of great importance, involving the protection of an immense industry. The foundation of the copyright law is in the provision of the Constitution (art. 1, sec. 8), which authorizes Congress- "To promote the progress of science and useful arts, by securing for limited times, to authors and inventors, the exclusive right to their respective writings and discoveries." It is settled that the words "authors" and " writings," in this section, are not confined to literary writers and their works, but include, among others, designers, engravers and lithographers, as well as photographers. Burrow-Giles Litho. Co. v. Sarony, 111 U. S. 53; Trade Mark Cases, 100 U. S. 82. Picture-posters or show bills, such as these chromolithographs were, are not designed for close inspection or long-continued study, like an oil painting, a steel or wood engraving, or an etching, and they are not to be judged by the same standards. They are intended to catch the eye of the passer on the street, or any one who merely glances at them, and to challenge his attention,-if possible to compel him to look again, so that he will observe what is the subject of the poster and have this forced upon his mind, and will be attracted by it. Their function is to illustrate something, and to advertise it by appealing quickly to the imagination, and conveying instantly a strong and favorable impression. Thus, to be successful, they require artistic ability, and above all things creativeness or originality of a high order, but peculiar. They must be designed boldly, and executed on broad lines, with not much attention to detail, so that the spirit of the picture will stand out at once, and almost leap at you, and will not be lost in a mass of details and minor features. Such is the ideal picture-poster, a special and peculiar branch of pictorial art, and one into which many gifted artists, highly successful in other fields, have ventured with greater or less success. Charles Hiatt's work entitled "Picture Posters," published in 1895 by George Bell & Sons, London ; "The Modern Poster," by Alexandre and others, published in 1895 by Charles Scribner's Sons. Certainly it does not lie in the mouth of the pirate, who has stolen and copied them at some expense and considerable risk, to deny that they have merit and value. I. The designs were proper subjects of copyright and each of these picture-posters was a proper subject of copyright, within the language and the spirit of the copyright law. There was abundant evidence of originality of design, of artistic merit, and of practical value and usefulness, as to each of the pictures. If any of these qualities was seriously questioned by the defence, it became the duty of the court to send the case to the jury. All of the pictures are new and original designs and involve new and original conceptions and creations. There was enough evidence on this subject to require the case to be submitted to the jury if any question was raised about it, citing, and in some instances distinguishing, as to definition of author, writings, etc., The Trade Mark Cases, 100 U. S. 82; Lithograph Co. v. Sarony, 111 U. S. 53 ; Nottage v. Jackson, 11 Q. B. Div. 627 ; Brightly v. Littleton, 37 Fed. Rep. 103 ; Carlisle v. Colusa County, 57 Fed. Rep. 979; Drury v. Ewing, Fed. Cases, No. 4095. If any one of the pictures was sufficiently proved to be new and original, this was enough to carry the case to the jury upon this question ; they were all proved to be new and original. II. As to artistic merit and value. The pictures being original designs, we maintain that they are of sufficient artistic merit and of sufficient value and usefulness to be entitled to copyright. At least there was enough evidence of this to require the case to be submitted to the jury, if any question was raised about it,-and furthermore no such question was raised by the defence. "If a copyrighted article has merit and value enough to be the object of piracy, it should also be of sufficient importance to be entitled to protection." Drone on Copyright, p. 212, cited with approval in Henderson v. Tomkins, 60 Fed. Rep. 758, 765 ; Church v. Linton, 25 Ont. Rep. 121; Hegeman v. Springer, 110 Fed. Rep. 374; Bolles v. Outing Co., 77 Fed. Rep. 966; 175 U. S. 262; Richardson v. Miller, Fed. Cases, No. 11,791. We have nothing to do with cases involving attempts to copyright mere catalogues or price lists, or labels, sometimes containing pictures, reproduced by photographic or other mechanical processes, of articles intended for sale, but which obviously have no artistic merit or originality. These decisions, whether condemning or upholding such copyrights, do not touch the questions involved in the case at bar. Distinguishing Mott Iron Works v. Clow, 82 Fed. Rep. 216; also citing Yuengling v. Scltile, 12 Fed. Rep. 97, 101; Sehurmaker v. Schwencke, 25 Fed. Rep. 466 ; 1amb v. Grand Ra )ids School Furniture Co., 39 Fed. Rep. 474; Drone on Copyright, 164, 165; Grace v. Newman, L. R. 19 Eq. Cases, 623 ; Maple v. Junior Army &, .Navy Stores, L. R. 21 Ch. Div. 369; Church v. Linton, 25 Ont. Rep. 131; Carlisle v. Colusa, County, 57 Fed Rep. 979. "The degree of merit of the copyrighted matter the law is not concerned with. Any is legally enough. To use it or not use it, is voluntary on the part of the public." III. The copyrights were properly taken out by the plaintiffs in their trade names of "The Courier Co." and "The Courier Litho. Co.," and the plaintiffs have the right to sue in their individual names for infringement of these copyrights. That copartners in business, who are the proprietors of a  copyrighted article, may take out a copyright in either of their copartnership or trade names, is well settled. Scribner v. Clark, 50 Fed. Rep. 473; affirmed as Belford v. Scribner, 144 U. S. 488; Callaghaan v. Myers, 128 U. S. 617; Scribner v. Allen Co., 49 Fed. Rep. 854; Werckmeister v. Springer Lith. Co., 63 Fed. Rep. 808; Rock v. Lazarus, Law Rep. 15 Eq. Cases, 104; Weldon v. Dicks, Law Rep. 10 Ch. Div. 247; Fruit-Cleaning Co. v. Fresno) Home Packing Co., 94 Fed. Rep. 845. Finally, the plaintiffs were the "proprietors" of each of the copyrighted prints, and as such were authorized to take out the copyrights by the express language of the copyright law, Rev. Stat., sec. 4952, which includes "proprietors" with "authors, inventors (and) designers." Colliery Eng. Co. v. United etc., Co., 94 Fed. Rep. 152. No formal assignment of the right to a copyright is necessary. Consent is sufficient to constitute one the proprietor. Carte v. Evans, 27 Fed. Rep. 861. See also Schumacher v. Schwencke, 25 Fed. Rep. 466; Little v. Gould, Fed. Cases, No. 8395 ; Lawrence v. Dana, Fed. Cases, No. 8136 ; Sweet v. Benning, 81 Eng. Com. Law Rep. 459; 16 Com. Bench Rep. 459; Gill v. United States, 160 U. S. 426, 435. All of the pictures, and particularly the Statuary Act Design, were copyrighted before publication. The law is well settled that there was no publication of these prints when they were shipped from Buffalo on April 11, or when they were received by MHr. Wallace at Peru, Indiana, on or about April 15. There was no publication until they were exposed to the general public, so that the public, without discrimination as to persons, might enjoy them. This must have been some time after April 15, when the last copyright was surely completed. Publication is a legal conclusion which follows from certain acts. Drone on Copyright, p. 291 ; Jewelers -Mere. Agency v: Jewelers Pub. Co., 84 Hun (N. Y. Sup. Ct.), 12, 16; Callag7ian v. Myers, 128 U. S. 617; Black v. Henry G. Allen Co., 56 Fed. Rep. 764; Belford v. Scribner, 144 U. S. 488; Garland v. Gemmill, 14 Canada Sup. Ct. Rep. 321; Prince Albert v. Strange, 2 De Gex & Smale, 652; 1 MacNaghten & Gorden; 47 Eng. Ch. Rep. 25. The representation of a play upon the stage regularly at a theatre, does not constitute a publication. Tompkins v. Halleck, 133 Massachusetts, 32; Palmer v. De Witt, 47 NI. Y. 532; Boucicault v. Hart, Fed. Cases, No. 1692. The use by a teacher of his manuscript and allowing pupils to make copies for the purpose of obtaining his instruction, does not amount to a publication. Bartlett v. Crittenden, Fed. Cases, Nos. 1076 and 1082. The printing of copies of an operetta and distributing them to artists, for private use only in learning their parts, and the representing of the operetta on the stage, is not a publication. French v. Kreling, 63 Fed. Rep. 621; Reed v. Carusi, Fed. Cases, No. 11,642; Blume v. Spear, 30 Fed. Rep. 629; Exch. Tel. Co. v. Cent. News, Law Rep. 2 Oh. Div. 48. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case 2023-0409 retrieved on September 29, 2023, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. tile.loc.gov

Circus Posters and the History of Copyrighting Adverts 

is a video streaming service emphasizing recommendations and deep personalization. We offer thousands of movies and TV series to our users and, among other things, personalize titles’ posters. IVI For such personalization, one has to create multiple diverse posters for each title. While the best posters are hand-crafted by designers, the process is too time-consuming. That’s why we created a tool called Parker, which generates high-quality title previews and posters automatically. Though this approach saves time, it also contains a major flaw – Parker can produce posters with awkward-looking facial expressions, which makes the resulting images unusable. Not seeing an effective solution and racing against the clock, we turned to — a data-labeling service — and 4 days later, the issue was put to rest. Here’s how. Toloka The Nitty-Gritty of the Problem Our software generates facial images from the platform’s media catalog that can at times look quite goofy and awkward when caught mid-frame. What’s even worse, such facial shots often have no recognizable emotions. When our moderators were sifting through the images by hand, that proved to be too labor-intensive and time-consuming. This is why we decided to use Machine Learning (ML) to train an algorithm to do this work instead and hopefully win in the long haul. This isn’t, in and of itself, a super challenging task; in fact, a simple binary classification will do just fine – 0 for a normal face and 1 for an awkward one. Easy enough. All that’s needed is a training model and the actual labeled data. Finding a suitable model is again not an issue since dozens are available on the market – from Xception to Efficient. All of them can be used on Keras, which is a common tool for constructing neural networks. The most common ML models that can be utilized on Keras. But the data is a different story altogether. Going by several datasets used to classify human emotions, we worked out that we would need approximately 72,000 labeled images to reliably train our algorithm. As a result, we extracted high-quality images of faces from 18 films, with each actor having no fewer than 10 (to understand the person’s expressions) and no more than 100 images (to not let the algorithm get used to one face). The raw data was there. What about the labeling? The Solution We estimated that our in-house team would need around 1,5 months to complete the project, and it would cost us around $3000. This is when Toloka stepped in, a company that uses minimal efforts of crowd performers from around the globe along with its vast IT infrastructure to label the data. The labeling assignment would contain 5 key stages: Decomposition Instructions Task interface Quality control Aggregation and cost analysis Decomposition In a binary classification task, performers are expected to answer a simple yes-no question, in this case: “Is this a normal face or not?” The task included 2 phases: Confirming that each image in the dataset was acceptable, i.e. it actually contained a face. Answering the task’s main question. Each one, in turn, contained an additional 2 subphases: Training and performer screening. Real-life labeling. Instructions Two sets of instructions with examples were given to Tolokers (Toloka’s crowd performers) – one for each phase of the task. In the first phase, an image was considered acceptable if it contained only one face with clearly visible facial features that covered more than half the image. The viewer also had to be able to tell whether the eyes and mouth were open or closed. Tolokers were asked to judge whether the images were “acceptable” or not. “Acceptable” images needed to have only one face, visible facial features, and appropriate size in relation to the rest of the image. Unacceptable images were the ones that did not meet these criteria and, in most cases, contained no face or more than one face. Images that contained no face or multiple faces were labeled as “unacceptable” and excluded from the 2nd phase. In the second phase that included only acceptable images, a face was deemed awkward-looking if it contained a half-open mouth or eyes that did not form a recognizable facial expression. “Acceptable” images that had faces with half-open mouths or eyes and no easily recognizable expressions were labeled as “awkward-looking” in the 2nd phase. On the other hand, a face was deemed normal if the mouth and eyes weren’t half-open, and if they were, a distinct expression could still be read. “Acceptable” images that had faces without half-open mouths or eyes were labeled as “normal” in the 2nd phase. “Acceptable” images that contained faces with half-open mouths or eyes but still had easily recognizable expressions were also labeled as “normal” in the 2nd phase. Task Interface We decided to use Toloka’s ready image classification template for all phases and subphases of the task. Each page contained 8-10 images and 3 radio buttons, with an option to utilize the keyboard in addition to the mouse. An example of the task’s interface on Toloka with a selection of 8 images that Tolokers had to label as “acceptable/unacceptable” in the 1st phase and “normal/awkward-looking” in the 2nd phase of the assignment. Quality control To minimize errors and exclude bots or unscrupulous performers, 4 quality control mechanisms were put in place: Training The interface provided hints and explained incorrect answers during training. Only Tolokers who passed the training session with 80% accuracy were allowed to complete the assignment. Hidden tests During the assignment, 10% of the questions/answers were known to us – we incorporated them on purpose. This way, any Toloker with a fail rate that exceeded 25% was dropped. Overlap As is the case with most crowdsourcing tasks, each question was given to more than one performer, ending with majority voting. Every question was answered by 3 performers in the 1st phase (face/no face) and 5 performers in the 2nd phase (normal/awkward-looking face). Assurance tools In addition, 4 quality assurance methods were built into the assignment: Language restrictions (only those with proven Russian skills could participate). Duration (overly fast responses were flagged). Daily task limits (each Toloker could complete only a certain number of tasks in a row). Captchas (Turing challenge-response tests were spread throughout). Aggregation Using a common technique, we aggregated the labeled data to get the final results. All in all, the assignment took 4 days  – 11 times faster than the estimated in-house expenses. The accuracy varied between 75 and 90%, approaching the latter figure towards the project’s completion. Final Word Having tried crowdsourcing for the first time, we were very pleased; in fact, so much so that we’re now preparing a new project: a new improvement to Parker teaching it to recognize and prioritize good-looking posters and previews over average ones.

Streaming Platform Uses ML to Filter Out Awkward-Looking Faces in Auto-Generated Movie Posters

Have you ever seen an ad campaign that just made you groan? Mismatched graphics, awkward font choices, and just amateur--- everything? Yes, me too. But, let’s be honest, that ad could have been mine a year ago. Professional marketing isn’t a cakewalk. More often than not, advertising is a huge challenge for small to medium-sized businesses. There’s more than one reason for it. One is money. As a small business owner, I have minimal capital, and there are never more than ten spare cents in my operating budget.  Adding a professional graphic designer to a small business team is very rarely financially feasible: the only time it will happen is if you’ve just won a big round of startup funding. Even then, you’ll need to make sure your advertising plan pays very high dividends before you make a hire. But if adding a professional designer and marketer to the team is not happening, what is the alternative? Well, you could design every ad yourself. In my case, the problem with that would be lack of skill: I’m good at what I do, but it just happens to not be graphic design. Traditional design software has steep learning curves, and even if I spent days focused on nothing but that, I’d come up with nothing better than that ad we were just talking about. The one that made you groan. So—what about outsourcing? Still not happening, even if I could find someone well qualified who was willing to be paid in peanuts. There’s time. I have very little free time, and it’s the same way for . Short-staffed with a hundred demands on every spare moment, there’s no way I can spend precious hours communicating my needs to someone outside the company: someone who starts with no knowledge of what we’re about and will need to have everything explicitly explained to them. every other SMB owner I know Hunting for a Solution So, that was what my problem was one year ago. Our problem, because I know I wasn’t alone.  What would your ideal solution look like? Here’s mine: a graphic design program that has absolutely no learning curve and makes complete sense to an absolute newbie who has never seen it before. An ideal program should take me minutes to produce professional-looking ads that will resound with my prospective clientele and community and won’t look like they were built with a cookie-cutter but will instead carry my branding and my unique signature. Oh, one more thing. It has to be free, or at least almost free because of the budget we talked about. Once our advertising has successfully raked in more clients, we might be able to pay something for that advertising. Before? Not much chance. Easy Small Business Marketing Well as hard to believe as it is, I have actually found what I was looking for. It’s called and it’s an online graphic design software that’s so easy to use that it hardly seems to be a software at all. Imagine an interface that guides you through what you need: sit at the computer with no inspiration whatsoever, and by the time you’re through the process you know exactly what you need, and as a happy bonus, you’ve got it, too, all ready to print or publish. And double bonus: that entire process, from inspiration to finished process, might just have taken ten minutes with no drain on the budget. PosterMyWall Part of the offering is a wealth of design templates, intuitively organized by and . If I’ve got a theme in mind, I’m sure to find something that’s a good fit for it. If I am beginning with a blank slate and no idea what I need, I’ll develop a good sense of what should work for me after I’ve scrolled through a few templates in the gallery. category size When I’ve selected one that fits our needs, everything begins to happen really quickly. Customizing the information on the templates so it advertises what I want it to takes maybe 5 minutes in the easy to use . editor Placing my logo and brand name for consistent branding in my marketing graphics is another minute or two. For the sake of maintaining a consistent brand image, I can switch the colors and typography used in the design to my brand’s colors and fonts. If I want, I can even switch up the visuals by uploading my own image or video or choosing one from the stock libraries right there in the editor. And then I’ve got it; the stunning social media files or posters or web graphics or whatever is the need of the hour to bring my small business into the spotlight. One of my favorite features is the way I can create one perfect design and then resize it for free to fit any of my social media channels or other platforms like my website or blog. But wait, there’s more... Another feature I like about PosterMyWall is the way there’s room to grow as your needs change. I currently use the free version: unlimited free social media templates without any watermark and tonnes of other design image and video templates with a small, easy to miss watermark that you can pay a small fee to remove. Higher quality, pay per piece downloads are also available at affordable prices. When our budget gets a little roomier, we can spring for a plan to get unlimited downloads of high-resolution images and videos. Premium Videos seem to be key in digital marketing strategy for startups these days, and with PosterMyWall I can prepare high-quality , and more to get our message out in a way that can’t be missed.  The promo video maker is as easy to use as the rest of the website, and there are more than 2,000 available. social media videos digital signage small business video templates In the US, I can also order banners, posters, and flyers at fairly competitive prices. Not having to download, upload, and haggle with local print shops is a definite bonus: with this service, everything is streamlined. Wrapping it Up So that’s where we are. Utilizing web-based, affordable solutions like is one of my top marketing tips for small businesses because it just changes the scene so much. I don’t need to buy fancy new computers or graphic editing software, and I don’t even need to hire a new team member. With online graphic software, I get to skip multiple tools to get to an end-to-end solution. I find myself able to reach my audience effectively, without putting a strain on my budget. That’s small business productivity right there. Try it yourself, and tell me what you think of it. PosterMyWall

PosterMyWall: A Better Way to Approach Marketing

Adaptive Graph Neural Networks for Cosmological Data Generalization: Acknowledgements and Disclosure

BiFi -- “pieces of meat” (Lusoga and Luganda, Ugandan). Imagine a world in which WiFi is not just data signals but can also be free Bitcoin satoshis! Call it BiFi (Bitcoin WiFi) World. I speculate below on how it would work and why it would be a pretty good idea. This Is Not UBI, Really. Universal Basic Income (UBI) is a scheme that hopes to give people some free money be it dollars or shillings, to sustain their livelihood every month. Some argue it is not a good free market idea, as free things for some means leeching the hard work of others. I agree. Moreover, UBI is a dollar scheme, and dollars are printed by some people. Hence, people will be spending debt, while the demands of living skyrocket necessitating that they be given even more debt to spend. Arguments for how AI will unleash so much abundance on Earth for a minority owning these AI systems, seem to make UBI possible and to actually warrant that it be implemented. After all, why should anybody be making billions of dollars every year while other people starve? However, the reality is that AI systems are nowhere near the level of autonomy needed to make massive profits without considerable human input. A billion dollars in profit, if made mostly by human labor, will not be made when these are given away freely to non-working people. It is Psychology 101. I'd rather earn free money than sweat it out at a job. My idea is different. My idea is more economic unity effected through unconstrained egalitarian sharing in a scarce digital commodity. The Future of Charity Is Free Sats for All The world of BiFi or Bitcoin WiFi would be a generous world. Certainly far from 1984. Instead of corporate surveillance as done with free information packages, companies give you real goodies as a way to effect corporate social responsibility. And it can be done. States could jump aboard too. Unlike UBI, states shall not be trying to feed people with these satoshis. They will be like candy for children and adults alike. Good candy for creating unity, peace, and encouraging people to dive deeper into the arcane world of cryptographic protocols. A most decentralized way of encouraging innovation in society as a whole. Instead of taking away autonomy, you give it to people. Instead of states banning cryptography like happened with Phil Zimmermann and his PGP, they could encourage it by spreading some value. The ethos of sharing satoshis is deeply rooted in the Bitcoin community. However, the problem with current implementations is that it requires sitting somebody down and convincing them about this great technological idea called Bitcoin. People are afraid of scammers. Especially when it comes to finance, people would rather they keep ignorant than run into something they do not quite understand. And there is a lot to unpack. Scarcity
Digital money
Cryptography
Cryptographic keys
Signing
P2P
Austrian economics
Etc It is quite a lot. Just Give Anonymous People Sats, and Let Them Explore 1 bitcoin has climbed back to $59,000 as of this morning, EAT. Divide this into 100 million, and each satoshi still goes for only $0.00059. To share sats freely and without requiring KYC (it doesn't require KYC), people need to run, at the very least, lighting network/LN nodes. But these are expensive. So, the next best alternative is to use LN apps which charge a few fees. Now, imagine a poster with something like an SSID or a QR code for a WiFi network, except it gives out some free satoshis every, say 10 minutes. Say, 100 sats. This isn't life-changing money. But it is free, and if legally authorized (as has to be for any crypto project today), it could be something done as a 24/7 autonomous charity system. After downloading the required BiFi app, scan the QR code or input the SSID, and voila, enjoy you receive 100 sats. No need to enter your name, no need for an ID, nothing. Just like WiFi. You get the password, and you enjoy. What Could Happen? The Bitcoin community is not growing quickly because very few people are getting some sats on their phones. There is the orange-pilling arc, and I am lucky to have dived in because of it. But I needed to be sent some sats by a Machankura ambassador. And being a content writer perhaps, I later found a way to earn more sats, hence, participating more. The rest is history. A lot of people might be like me. Others, might only need to touch some sats and see how uncomplicated holding some can be. And by extension, all crypto tokens can be WiFi-d. But Bitcoin is my thing. Because it is scarce. So, this will be true charity work. Circa 2025, there might be a “bank run” on the Bitcoin blockchain. The price will shoot up from $70k to $200k. Demand will be through the roof! Don't Airdrop It, Wi-Fi It. No need to shill anybody online. No marketing campaign needed. Simply, give it to people, and see what they do. Yes, you could later put out a data collection form for those who have been orange-pilled. Later, not before. It will go a long way to spread this goodness everywhere. Flaws and all. BiFi -- “pieces of meat” (Lusoga and Luganda, Ugandan). BiFi -- “pieces of meat” (Lusoga and Luganda, Ugandan). Imagine a world in which WiFi is not just data signals but can also be free Bitcoin satoshis! Call it BiFi (Bitcoin WiFi) World. I speculate below on how it would work and why it would be a pretty good idea. This Is Not UBI, Really. Universal Basic Income (UBI) is a scheme that hopes to give people some free money be it dollars or shillings, to sustain their livelihood every month. Some argue it is not a good free market idea, as free things for some means leeching the hard work of others. I agree. Moreover, UBI is a dollar scheme, and dollars are printed by some people. Hence, people will be spending debt, while the demands of living skyrocket necessitating that they be given even more debt to spend. Arguments for how AI will unleash so much abundance on Earth for a minority owning these AI systems, seem to make UBI possible and to actually warrant that it be implemented. After all, why should anybody be making billions of dollars every year while other people starve? However, the reality is that AI systems are nowhere near the level of autonomy needed to make massive profits without considerable human input. A billion dollars in profit, if made mostly by human labor, will not be made when these are given away freely to non-working people. It is Psychology 101. I'd rather earn free money than sweat it out at a job. My idea is different. My idea is more economic unity effected through unconstrained egalitarian sharing in a scarce digital commodity. The Future of Charity Is Free Sats for All The world of BiFi or Bitcoin WiFi would be a generous world. Certainly far from 1984. Instead of corporate surveillance as done with free information packages, companies give you real goodies as a way to effect corporate social responsibility. And it can be done. States could jump aboard too. Unlike UBI, states shall not be trying to feed people with these satoshis. They will be like candy for children and adults alike. Good candy for creating unity, peace, and encouraging people to dive deeper into the arcane world of cryptographic protocols. A most decentralized way of encouraging innovation in society as a whole. Instead of taking away autonomy, you give it to people. Instead of states banning cryptography like happened with Phil Zimmermann and his PGP, they could encourage it by spreading some value. The ethos of sharing satoshis is deeply rooted in the Bitcoin community. However, the problem with current implementations is that it requires sitting somebody down and convincing them about this great technological idea called Bitcoin. People are afraid of scammers. Especially when it comes to finance, people would rather they keep ignorant than run into something they do not quite understand. And there is a lot to unpack. Scarcity Digital money Cryptography Cryptographic keys Signing P2P Austrian economics Etc Scarcity Digital money Cryptography Cryptographic keys Signing P2P Austrian economics Etc It is quite a lot. Just Give Anonymous People Sats, and Let Them Explore 1 bitcoin has climbed back to $59,000 as of this morning, EAT. Divide this into 100 million, and each satoshi still goes for only $0.00059. To share sats freely and without requiring KYC (it doesn't require KYC), people need to run, at the very least, lighting network/LN nodes. But these are expensive. So, the next best alternative is to use LN apps which charge a few fees. Now, imagine a poster with something like an SSID or a QR code for a WiFi network, except it gives out some free satoshis every, say 10 minutes. Say, 100 sats. This isn't life-changing money. But it is free, and if legally authorized (as has to be for any crypto project today), it could be something done as a 24/7 autonomous charity system. After downloading the required BiFi app, scan the QR code or input the SSID, and voila, enjoy you receive 100 sats. No need to enter your name, no need for an ID, nothing. Just like WiFi. You get the password, and you enjoy. What Could Happen? The Bitcoin community is not growing quickly because very few people are getting some sats on their phones. There is the orange-pilling arc, and I am lucky to have dived in because of it. But I needed to be sent some sats by a Machankura ambassador. And being a content writer perhaps, I later found a way to earn more sats, hence, participating more. The rest is history. A lot of people might be like me. Others, might only need to touch some sats and see how uncomplicated holding some can be. And by extension, all crypto tokens can be WiFi-d. But Bitcoin is my thing. Because it is scarce. So, this will be true charity work. Circa 2025, there might be a “bank run” on the Bitcoin blockchain. The price will shoot up from $70k to $200k. Demand will be through the roof! Don't Airdrop It, Wi-Fi It. No need to shill anybody online. No marketing campaign needed. Simply, give it to people, and see what they do. Yes, you could later put out a data collection form for those who have been orange-pilled. Later, not before. It will go a long way to spread this goodness everywhere. Flaws and all.

BiFi - Bitcoin Sats Given as Freely as WiFi

In this session of our "Behind the Startup" series, we sit down with Zad, the visionary founder of OpenCred, to explore the revolutionary concept of tokenizing trust in the Web3 environment. As a Stanford-certified expert in blockchain and cryptography, Zad discusses the inception of OpenCred, its unique approach to digital identity through eSFT tokens, and its ambitious mission to reshape social and economic interactions online. Ishan Pandey: Hi Zad, great to have you here for our "Innovators in Web3" series. Can you tell us about your background and what led you to create OpenCred? Zad: As a postgraduate in Blockchain Strategy and Stanford-certified cryptography expert, I have spent years conducting cutting-edge research in asset tokenization, DID, and cybersecurity. Trust is actually one of the most difficult things to measure and quantify, no matter “IRL” or online. Now with blockchain and tokens we finally have the tools to tackle this problem in a novel way. I founded OpenCred to bring tokenized trust to the Web3 industry, unleash the hidden potential that lies in social relationships, and eventually enable the establishment of a decentralized society. Ishan Pandey: Could you start by explaining what OpenCred is and what specific gap in the Web3 space it aims to fill? Zad: OpenCred is a decentralized Web3 credit protocol that applies a new token standard called “eSFT” to tokenize trust relationships into tangible and quantifiable economic assets. Our vision is to empower users to monetize their online activity while guaranteeing data privacy and ownership. By providing a blockchain-based solution for establishing and measuring trust relationships, OpenCred fills a crucial gap in the Web3 ecosystem: enabling trustworthy interactions and collaboration without relying on centralized intermediaries or KYC. Ishan Pandey: In the deck, you mention that OpenCred introduces a groundbreaking trust value (credit) quantification for Web3. Can you elaborate on how this works and why it's important? Zad: OpenCred aggregates both on-chain (e.g., DeFi TVL, smart contract interactions) and off-chain data (e.g., social media followers and interactions) to tokenize trust in the form of CRED tokens. This means users can increase their personal trust value by increasing their online activity and footprint. Each CRED is unique to the user and contains a number of fungible Score Tokens (ST) that determine the value of a user’s CRED. By thus providing a measurable and objective way to assess trustworthiness, OpenCred enables secure, traceable, and efficient trust-based interactions in various Web3 applications, such as social media, e-commerce, DeFi, and gaming. This groundbreaking mechanism is crucial for fostering a trustworthy environment in the digital space and enabling countless new use cases in many areas from B2B to B2C to C2B to C2C. Ishan Pandey: The CRED token seems to be a key component of the OpenCred ecosystem. Can you explain its dual nature as a semi-fungible token and how it represents both identity and trustworthiness? And how does this design benefit users of your platform? Zad: CRED tokens are a groundbreaking innovation of OpenCred, creating a new token standard we call embedded semi-fungible token (eSFT) that combines both non-fungible (unique) and fungible (interchangeable) properties. The non-fungible part of the token represents a user's unique identity within the OpenCred protocol, ensuring that each user has a distinct and verifiable presence. On the other hand, the fungible part of the token quantifies the user's trustworthiness, which is determined by their interactions (on-chain & off-chain) and the amount of CRED tokens they receive from others. This design benefits users by empowering them to monetize their online activity and social relationships. Moreover, by tokenizing trust and making it quantifiable and tangible, we are creating a new kind of asset that simply has not existed before, thereby creating huge value. Ishan Pandey: How does sending CRED between users facilitate the building of a trustable and quantifiable social graph in Web3? Zad: When a user sends CRED tokens to another user, they are essentially establishing a trust relationship that is recorded on the blockchain. This transaction signifies that the sender trusts the recipient to a certain degree, based on their interactions, reputation, or other factors. As users continue to interact and send CRED tokens to each other, a dynamic, evolving on-chain social graph emerges. The social graph is quantifiable and traceable. This quantification of trust enables users to make informed decisions when interacting with others, as they can assess the trustworthiness of potential collaborators, partners, or service providers based on their position in the social graph. Ishan Pandey: With OpenCred positioning itself as a provider of "Trust as a Service," how do you ensure the reliability and security of the trust metrics generated by your protocol? Zad: There is a clear friction between incentivizing users on the one hand and ensuring the authenticity of trust relationships on the other hand. We are aware of this and will tackle this problem in multiple ways. First of all, we will connect the sending of CRED with some actions such as “follow”, “like”, etc., so that the transfer of CRED corresponds with the establishment of a meaningful trust relationship. Secondly, we will implement punishments for sybil attackers and other malicious behavior. Additionally, OpenCred implements a multi-layered governance model that involves community participation and decentralized decision-making. This approach helps to maintain the integrity and fairness of the trust metrics, as the community can collectively monitor and flag any suspicious activities or attempts to manipulate trust scores. To further enhance the reliability of trust metrics, OpenCred incorporates a reputation staking mechanism, where users are required to stake a portion of our governance tokens to participate in trust-building activities. OpenCred places a strong emphasis on user education and awareness. By providing clear guidelines, tutorials, and best practices for using the trust protocol. Ishan Pandey: How do you plan to drive adoption and grow the OpenCred ecosystem, particularly in terms of attracting both users and developers? Zad: OpenCred actively seeks partnerships and collaborations with established Web3 projects across various sectors, such as social media, e-commerce, gaming, and decentralized finance (DeFi). We currently have established partnerships with over 10 Web3 projects with close to 1 million daily active users ready to onboard our protocol once it is live. To attract developers, OpenCred will provide a range of tools, SDKs, and documentation to facilitate the integration of its trust protocol into various applications. The protocol will offer developer grants, hackathons, and bounty programs to incentivize the creation of innovative trust-based solutions. By fostering a vibrant developer community and providing the necessary resources and support, OpenCred aims to accelerate the development of a diverse range of dApps that leverage its trust infrastructure. OpenCred's tokenomics model is designed to incentivize user participation and ecosystem growth. The governance token serves as the primary means of interaction within the OpenCred ecosystem, used for establishing trust relationships, staking, and accessing trust-based services. The protocol will implement a reward system that incentivizes users to actively participate in trust-building activities, such as providing accurate trust assessments, staking tokens, and contributing to the network's security. These incentives will drive user adoption and encourage long-term engagement with the OpenCred ecosystem. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. In this session of our "Behind the Startup" series, we sit down with Zad, the visionary founder of OpenCred, to explore the revolutionary concept of tokenizing trust in the Web3 environment. As a Stanford-certified expert in blockchain and cryptography, Zad discusses the inception of OpenCred, its unique approach to digital identity through eSFT tokens, and its ambitious mission to reshape social and economic interactions online. Ishan Pandey: Hi Zad, great to have you here for our "Innovators in Web3" series. Can you tell us about your background and what led you to create OpenCred? Ishan Pandey: Zad: As a postgraduate in Blockchain Strategy and Stanford-certified cryptography expert, I have spent years conducting cutting-edge research in asset tokenization, DID, and cybersecurity. Trust is actually one of the most difficult things to measure and quantify, no matter “IRL” or online. Now with blockchain and tokens we finally have the tools to tackle this problem in a novel way. I founded OpenCred to bring tokenized trust to the Web3 industry, unleash the hidden potential that lies in social relationships, and eventually enable the establishment of a decentralized society. Zad: Ishan Pandey: Could you start by explaining what OpenCred is and what specific gap in the Web3 space it aims to fill? Ishan Pandey: Zad: OpenCred is a decentralized Web3 credit protocol that applies a new token standard called “eSFT” to tokenize trust relationships into tangible and quantifiable economic assets. Our vision is to empower users to monetize their online activity while guaranteeing data privacy and ownership. By providing a blockchain-based solution for establishing and measuring trust relationships, OpenCred fills a crucial gap in the Web3 ecosystem: enabling trustworthy interactions and collaboration without relying on centralized intermediaries or KYC. Zad: Ishan Pandey: In the deck, you mention that OpenCred introduces a groundbreaking trust value (credit) quantification for Web3. Can you elaborate on how this works and why it's important? Ishan Pandey: Zad: OpenCred aggregates both on-chain (e.g., DeFi TVL, smart contract interactions) and off-chain data (e.g., social media followers and interactions) to tokenize trust in the form of CRED tokens. This means users can increase their personal trust value by increasing their online activity and footprint. Each CRED is unique to the user and contains a number of fungible Score Tokens (ST) that determine the value of a user’s CRED. By thus providing a measurable and objective way to assess trustworthiness, OpenCred enables secure, traceable, and efficient trust-based interactions in various Web3 applications, such as social media, e-commerce, DeFi, and gaming. This groundbreaking mechanism is crucial for fostering a trustworthy environment in the digital space and enabling countless new use cases in many areas from B2B to B2C to C2B to C2C. Zad: Ishan Pandey: The CRED token seems to be a key component of the OpenCred ecosystem. Can you explain its dual nature as a semi-fungible token and how it represents both identity and trustworthiness? And how does this design benefit users of your platform? Ishan Pandey: Zad: CRED tokens are a groundbreaking innovation of OpenCred, creating a new token standard we call embedded semi-fungible token (eSFT) that combines both non-fungible (unique) and fungible (interchangeable) properties. The non-fungible part of the token represents a user's unique identity within the OpenCred protocol, ensuring that each user has a distinct and verifiable presence. On the other hand, the fungible part of the token quantifies the user's trustworthiness, which is determined by their interactions (on-chain & off-chain) and the amount of CRED tokens they receive from others. This design benefits users by empowering them to monetize their online activity and social relationships. Moreover, by tokenizing trust and making it quantifiable and tangible, we are creating a new kind of asset that simply has not existed before, thereby creating huge value. Zad: Ishan Pandey: How does sending CRED between users facilitate the building of a trustable and quantifiable social graph in Web3? Ishan Pandey: Zad: When a user sends CRED tokens to another user, they are essentially establishing a trust relationship that is recorded on the blockchain. This transaction signifies that the sender trusts the recipient to a certain degree, based on their interactions, reputation, or other factors. As users continue to interact and send CRED tokens to each other, a dynamic, evolving on-chain social graph emerges. The social graph is quantifiable and traceable. This quantification of trust enables users to make informed decisions when interacting with others, as they can assess the trustworthiness of potential collaborators, partners, or service providers based on their position in the social graph. Zad: Ishan Pandey: With OpenCred positioning itself as a provider of "Trust as a Service," how do you ensure the reliability and security of the trust metrics generated by your protocol? Ishan Pandey: Zad: There is a clear friction between incentivizing users on the one hand and ensuring the authenticity of trust relationships on the other hand. We are aware of this and will tackle this problem in multiple ways. First of all, we will connect the sending of CRED with some actions such as “follow”, “like”, etc., so that the transfer of CRED corresponds with the establishment of a meaningful trust relationship. Secondly, we will implement punishments for sybil attackers and other malicious behavior. Additionally, OpenCred implements a multi-layered governance model that involves community participation and decentralized decision-making. This approach helps to maintain the integrity and fairness of the trust metrics, as the community can collectively monitor and flag any suspicious activities or attempts to manipulate trust scores. To further enhance the reliability of trust metrics, OpenCred incorporates a reputation staking mechanism, where users are required to stake a portion of our governance tokens to participate in trust-building activities. OpenCred places a strong emphasis on user education and awareness. By providing clear guidelines, tutorials, and best practices for using the trust protocol. Zad: Ishan Pandey: How do you plan to drive adoption and grow the OpenCred ecosystem, particularly in terms of attracting both users and developers? Ishan Pandey: Zad: OpenCred actively seeks partnerships and collaborations with established Web3 projects across various sectors, such as social media, e-commerce, gaming, and decentralized finance (DeFi). We currently have established partnerships with over 10 Web3 projects with close to 1 million daily active users ready to onboard our protocol once it is live. To attract developers, OpenCred will provide a range of tools, SDKs, and documentation to facilitate the integration of its trust protocol into various applications. The protocol will offer developer grants, hackathons, and bounty programs to incentivize the creation of innovative trust-based solutions. Zad: By fostering a vibrant developer community and providing the necessary resources and support, OpenCred aims to accelerate the development of a diverse range of dApps that leverage its trust infrastructure. OpenCred's tokenomics model is designed to incentivize user participation and ecosystem growth. The governance token serves as the primary means of interaction within the OpenCred ecosystem, used for establishing trust relationships, staking, and accessing trust-based services. The protocol will implement a reward system that incentivizes users to actively participate in trust-building activities, such as providing accurate trust assessments, staking tokens, and contributing to the network's security. These incentives will drive user adoption and encourage long-term engagement with the OpenCred ecosystem. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program . HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: business blogging program business blogging program

Stories, News & Mentions about POSTERA

Postera’s Stories on HackerNoon