Chatteris in the IoT Applications Industry

Chatter

The Absence and Return of Mr. Harry Chatteris

Since our last post about Crypto news, Coinfeeds has evolved from a bot on reddit to a crypto news site. Artificial intelligence is used to identify coins mentioned in news headlines and tweets, so that you can see a feed about each specific coin. This makes me wonder, what are the most talked about coins on Crypto Twitter? So we are publishing this into a daily Twitter chatter page. Demo for a day Top coins of the day Here’s a demo for today, May 19, 2021. Here is the absolute ranking of the most talked about coins today. Bitcoin is at the top of the list, like most days. Polygon ($MATIC) is number 2 - they are having a great run right now. You can click on the coin and see some of the most popular tweets about it. For Polygon, it is: Polkadot moved +15 in ranking - they are having an event, Polkadot decoded tomorrow. Sushi and Aave made the top 10 after their recent great performance. Aave yesterday tweeted a preview of Aave for institutions, and moved 40 ranks to #8, and today moved down -2. Trending coins of the day It is always interesting to check the coins with very large changes in ranking vs the day before. The coins rising the most in ranking today are: Huobi moved 108 ranks. They are doing an NFT giveaway with Thundercore, and just listed $RLC Bancor moved +96 ranks - they are offering liquidity mining for $MATIC with an attractive APR. Here's the tweet: Weekly Trending Coins We can also compare a coin's trend with last week's data. Here are the coins rising the most compared to a week ago: BOSAGORA is doing an event with Binance. Aavegotchi and Livepeer are listing on Kraken, and Wootrade just supported $NEO. How is this created? First we use the Twitter API to listen to over 800 Twitter accounts related to cryptocurrencies. This includes accounts by crypto projects, accounts that belong to crypto teams and founders, and individuals who frequently talk about cryptocurrencies. Next, we created an AI model that identifies when a tweet is talking about a coin, and then resolves that to a single coin in a knowledge base. Finally, we aggregate the tweets based on coins mentioned. Where can I find the this? The Twitter chatter page will be hosted here: https://coinfeeds.io/discover, and updated daily. Hope you give it a try!

Get The Latest Trends on Crypto via Twitter Chatter Analysis

Ever sit down to do something and suddenly your mind is off on a trail like a dog after a bunny? Which would be great if you stayed on that track, but instead: SQUIRREL!!! Then it’s off in another direction. You know if you could just focus, you could get things done, but your brain just will not SHUT UP. Instead it starts reminding you that your friend’s birthday was last week and you never sent a card, and when did cards get so expensive anyway, and oh yeah, at the last party you went to David did that cool card trick — how did he do that? Sometimes, you just need the brain chatter to STOP so you can get to the calm and quiet needed to enjoy your life. The good news is that you have a secret magic tool to achieve peace which costs less than a dollar. It is wooden and yellow. Has a handy pink eraser. And frequently comes with a №2 stamped on the side. Here are 9 ways a can quickly and easily silence brain chatter: humble pencil Press pause on the pings. Your brain knows when it has committed to a task that remains unfinished. In fact, it will ping you endlessly with reminders. After all, part of the brain’s job is to keep track of things until they get done. Taking the time to let your brain spill those unfinished tasks onto a piece of paper in list format, gives it the confidence that the task will be completed. With relief, it releases the worrisome reminders. Just by having a list, you assure your brain “I’ve got this.” David Allen, author of , frames it as closing open loops. And while Allen has a great system to increase , if you are really overwhelmed with brain chatter, the best place to start is by creating a simple to-do list. Besides, there is a unique kind of high in crossing things off once you’ve done them. Getting Things Done productivity Use loose leaf notebook paper for your brain, like you use tissue for your nose. Sometimes, our brain chatter is so congested that we can’t even get to a list. We need to blow that congestion out. In her book, , Henriette Anne Klauser, shares: “I encourage a form of getting ideas out which I call Rapidwriting. It means to write fast without stopping to consider, to edit, to rearrange, or critique. Use Rapidwriting when you cover the page, ploughing through objections, mowing down the inner voice that criticizes your every word. Write whatever is on your mind. Tell the truth. It’s okay to burn up your Rapidwriting pages ranting and raving.” Write it Down, Make it Happen The important thing about Rapidwriting is that you cannot use a filter. Normally, when we write things down, we pause to think and compose in our head. Skip that step. Go straight from brain to paper. Think of it like Kleenex. You can throw the pages out afterward. If your brain feels like a popcorn popper, mimic that pattern on paper. When our brains are in popcorn-mode, it can be difficult to find the clarity. An action-plan eludes us. Spilling ideas popcorn-style onto paper, then drawing connections between related items is called Mindmapping. This form of planning is especially helpful when your brain is overwhelmed with ideas. Drawing lines between related ideas creates structure — and frequently creates flashes of insight. There are many Mindmapping programs that allow you to do this digitally, but using a pencil and paper can be more effective in silencing the brain — even if the finished product doesn’t look as neat. Find your zen, with Zentangles Zentangle is a method of drawing structured, repetitive patterns in a free-flow from your hand to the page. The drawing bypasses your usual planning functions, so you can enjoy the process without worrying about the result. Drawing Zentangles is meditative and it allows your brain to relax as your hands play. It doesn’t take long to go from ricocheting thoughts, to a calm and peaceful brain. Best of all, the method is simple to learn. You draw a shape (or big, loopy scribble), then you divide the shape into sections. Each section is given its own pattern. Simple or complex — whatever you desire. To learn more about this method, check out . https://zentangle.com/ If your brain chatter centers around being stuck, start an imaginary correspondence. Sometimes our chatter is a function of not being sure exactly what we want. Our thoughts become loops trying to solve our lives like a problem, rather than filled with the enjoyment of living them. Inspired by an exercise in Henriette Anne Klauser’s, , I began writing letters to my new job at a time when I was at a plateau. I purchased a composition notebook and wrote entries like a diary. “Dear new job, I am so excited that I get to spend so much of every day writing.” And, “Dear new job, I’m so grateful for how many opportunities I get to speak to groups.” Write it Down, Make it Happen The fun part is that writing these letters helped me get very clear on what I wanted — which prompted me to make inspired choices with my time. Funny thing about this process, I didn’t wind up with a new job. What happened was that my current job began to look like what I most wanted it to be. Leverage a schoolroom punishment to convince your brain to run on a more positive track. Did a teacher ever punish you by making you write 100 sentences? It was the worst, right? And chances are, those words were burned into your psyche. You remember exactly what you wrote. Despite the negative connotations most of us have around writing sentences, the practice can be used to create positive, new, chatter-free tracks for our brains to run on. The repetitive, kinesthetic task of writing sentences can be leveraged to help us reprogram our subconscious mind. Choose an affirmation, a goal, a belief you want to reinforce, or a piece of scripture, then pick up that pencil and start writing. You will find that not only does your brain begin to calm down, but that whatever you are writing becomes part of your natural thought patterns. Replaying an awkward interaction? Rewrite the script for a better outcome. Sometimes the chatter in our heads is just us rehashing dialogue. We are either saying what we wish we had said, or revising what we did say, over…and over…and over. The next time that happens, get it out of your head and onto paper. And don’t constrain yourself to what the other person actually said. Rewrite the script in exactly the way you wish it had happened. Your brain will stop the replay because you’ve captured it in print — even if you did take creative license by giving it a Hollywood ending. Did you know the pencil doesn’t have to touch the paper at all to quiet your brain? Repetitive, rhythmic, motor patterns are calming. It’s the reason we fidget when we are feeling anxious. So if writing things down doesn’t appeal to you for calming a chatty brain, keep that pencil lead in the air and learn to spin it around your thumb. WikiHow has a tutorial: https://www.wikihow.com/Spin-a-Pencil-Around-Your-Thumb It turns your pencil into an old school fidget spinner, drawing just enough energy from your brain’s reserves to keep it from running off after that squirrel. Want a long-term solution to brain chatter? Proactively drain it before it even starts. One of the best long-term practices to retrain a chatter-filled brain is Julia Cameron’s daily practice of “Morning Pages.” In her book, Cameron writes: “In order to retrieve your creativity, you need to find it. I ask you to do this by an apparently pointless process I call the morning pages…the morning pages are three pages of longhand writing, strictly stream-of-consciousness: ‘Oh, god, another morning. I have NOTHING to say. I need to wash the curtains. Did I get my laundry yesterday? Blah, blah, blah…’ They might also, more ingloriously be called brain drain, since this is one of their main functions.”  Cameron asserts that there is no wrong way to do the morning pages. It’s a simple process of writing–longhand–whatever comes to mind. Cameron says, “although occasionally colorful, the morning pages are often negative, frequently fragmented, often self-pitying, repetitive, stilted or babyish, angry or bland–even silly sounding.” Over time, the practice of morning pages retrains the brain to hang out in its quiet state, because you start to see the chatter for what it is. Just chatter. The Artist’s Way Imagine yourself with a peaceful brain. The worst part of brain chatter is that it makes us tired. It’s like having a roommate who stands right next to us all the time talking in incomplete sentences who never — ever — shuts up. Imagine the next time that internal roommate is going on-and-on, how delightful it will be to evict them. The best part is that it takes very little effort to make this happen. You just pull a pencil out of the drawer with one of these techniques and in a few moments the brain gets quiet. Hear — and feel — the silence. Inside and outside. Even that squirrel is quiet. // Want to learn to silence your brain? Get your free and templates. Visual Journaling Quick Start Guide

9 Ways a Humble Pencil Can Silence Brain Chatter

Just about everyone these days uses Promises in one form or another. I use them enough that I created a snippet that produces this boilerplate: VS Code Many times, you have some async to call and want it to return a brand new promise on each call. However, there are some important use cases where you only need to execute the promise’s logic once per page load. Consider a user profile retrieval function. Most of the time, when users hit an app, you need to retrieve their user profile information from some backend data store (often via HTTP) to provide a personalized experience by accounting for their recorded preferences, show their name and that sort of thing. function Let’s assume for a moment that the user’s profile doesn’t change between page loads (I’ll come back this assumption shortly). This is pretty common in my experience. In this case, there’s no need to go out and retrieve the user’s profile each time a component on the page needs it to do its job. You can, of course, and many first-cut implementations do this. This will make your app unnecessarily “chatty” — multiple distinct components on the page, each finding the user’s profile in order to give that personalized touch. You can cut down on that chatter by relying upon Promises’ built-in memoization ( by ). You can call the same promise multiple times. If it has already resolved()’d with a particular user profile record, subsequent calls resolve with the same user profile record without re-running its logic. In other words — fewer HTTP calls, less chatter. read about this important technique here if you aren’t familiar with the concept Divyanshu Maithani This works best if you funnel all your http calls like this through a re-usable service type component such as an Angular service, a redux action or your favorite technique for implementing a data I/O layer. If your various UI components are making their own HTTP calls, then it’s more difficult to leverage this capability efficiently. As a result of this, my expanded example now looks more like this: If the getUserProfilePromise class property isn’t initialized, the getUserProfile method creates a new promise. If it has been initialized, it just returns the existing promise. In the end, the Promise’s logic (and expensive HTTP call) executes just once. What if the assumption is wrong, though? What if the user’s profile usually, but not *always*, stays the same over the course of a single page load? This usually arises in my single page apps when a user edits their profile and that edit transaction doesn’t trigger a page reload. It’s easy to handle on the service— just null out the variable holding on to the promise. That will force the service code to re-execute the Promise’s logic next time it’s invoked. I usually create a “reRead” type method just to make this clear in the code: I’ve been using user profile as an example, but you can imagine there’s a class of data that only has to be read once per page load. In the app I’m currently working on, there are a handful of master tables that rarely change at all and if they do change while the user is on page, it’s OK if we wait on a page refresh before getting the latest. That includes, for example, a country master, a region master, menu options and things of that nature. As always, I hope this is useful! Leave a comment with your thoughts and experiences! </end>

Use Promises to Cut Down on Chatter and Improve Performance

Machine Learning can reproduce TEDtalks, Obama speeches and Death Metal band logos. Could it also replicate my husband’s ramblings? About a month ago, I graduated from a at General Assembly (SG). It was a wonderfully gruelling and intense process, and I now feel a tiny bit like Neo from the Matrix. It’s really cool to look at the source of things and be able to make sense of what’s going on. 12-week coding bootcamp Since then, between code katas and job interviews, I’ve had a lot of fun working on . Here’s a story about one of my favourites. little side-projects It all began when I attended a . I had just spent 12 weeks learning some rather practical skills, and I was eager to see how they could be used for mischief. The theme was Machine Learning. Creative Coding meetup There were 2 speakers, and their talks were fascinating. One of them, , wrote about how : Rob Peart he used neural networks to generate death metal band logos Delightfully ominous. (Image: ) Rob Peart Sounded fun — I wanted to play! Getting started (dry run) I followed the installation instructions on , and referred to when I ran into trouble. Like Rob said, training on Shakespeare gets boring pretty quick. I didn’t really want to use images either, so I started looking around. Jeff Thompson’s tutorial Assad Udin’s installation guide Finding other examples to learn from I discovered two projects I really loved — ’s (trained on over 4 million words!) and (>700k words). samim TED-RNN Obama-RNN Now, it just so happens that my favourite human being in the world is 66.2% through . That’s at least 662k words — good enough for what I wanted to do. a project to write a million words Getting started… for real this time First, I exported the blog from Wordpress, getting a XML file with over 60,000 lines. Next, I spent about 2 hours cleaning it up. Removing the metadata was easy, removing formatting from within posts was harder. I got lots of regexp practice! Once that was done, it was training time. Training the Shakespeare data set took about an hour, so I wasn’t expecting this to take much longer… Boy, was I wrong. It took 5 full hours! At long last… Creating my first sample was super quick… and then I got a bunch of disappointing rubbish. “Play I tencid a plistance of good how from a bit of tirely problems we imagine from that 4n.” This didn’t look anything like the results from Obama-RNN… Time to actually read the documentation. Digging into the details In Karpathy’s article on , he explains that setting the temperature changes how a sample is created. RNN-effectiveness Lower temperature will cause the model to make more likely, but also more boring and conservative predictions. Higher temperatures cause the model to take more chances and increase diversity of results, but at a cost of more mistakes. That made a lot of sense. The samples below are 6000-character long chunks of text starting with “Let’s talk about procrastination”, at every temperature setting. Displaying the information in a way that makes sense I wanted a way to visualise what was happening so… word clouds! (No need to reinvent the wheel here, is free and does exactly what it says on the tin.) wordclouds.com temperature 0.1 to temperature 0.3 The conservative samples (1 and 2) were repetitive and boring. The results were long chunks of text, with no punctuation, sentences or paragraphs. In the first sample, the word ‘world’ appeared over 120 times! Sample 3 was the first one to return made-up words (‘confriend’ and ‘boathe’). temperature 0.4 & 0.5 Sample 4 was the first one with paragraphs! There were only 2, and they were ridiculously long 3000-character sentences. There were also about 10 semi-gibberish words, but they still look familiar (windom =wisdom, priviled = privilege, speaked = speak). As we get less conservative, the generator tries more ‘risky’ things. best results at temperature 0.6 & word cloud of the original text (>600k words) Sample 6 is, in my opinion, the best result I got. It has bits that actually sound like a Lorem Ipsum version of Visa. Some of my favourites lines: think about it, I was a teenager and the fact that it is to do things and drink the guy out of the conversations and off the same thing So I suck on things that I wanted to read out of down at great me to that I wanted to get the optimal starts of how to be an only that I should be a conversation in the days of external to truly write a lot of the reality of my playing something outside for the most is a break with seemingly the more less to the end of the real. better in a sense of thinking and building so I feel like I should be writing everything that I love having my responsible of anybody would actually interesting to me I feel like I’ve present for it– and then I wanted to do the same than a communicate early and learn and the problem of realize that you know thinking. I think it’s not really feel good right. some interesting for our curiosity to the world that I want to figure and think about this makes me this possibly love what they get the idea temperatures 0.7–0.9 temperature 1.0 & the resulting output Samples 7–10 are increasingly garbage, and pretty hard to read With 9 & 10, sentences became shorter, choppy and abrupt Punctuation seems a bit random, with question marks and dashes showing up There are more and more gibberish-words — the word clouds really help visualise this! What I learnt, what I loved & what I would do better “Gasp. Language!” I love, love, love that swear words appeared. I’m not sure what this says about me. (“fuck” appeared at temperature 0.3 and “bitch” appeared at 0.5.) I think the results here aren’t as great as TED-RNN and Obama-RNN because those examples learnt from a much more consistent format. , on the other hand, uses this blog to write anything and everything he wants. On retrospect, it might’ve done a better job if I picked certain categories of posts rather than all of them. Visakan I laughed way too hard at this. To be honest, I still can’t entirely explain what a neural is, or how exactly machine works. But I feel like I’m somewhere in the ballpark now, rather than a total outsider wondering what everyone’s on about. network learning Multi-layer Recurrent Neural Networks! Sequential processing! Optimization algorithms! Corpus! My little toy probably isn’t going to be writing blogposts for anytime soon, but it sure was a lot of fun to build and tinker with! Visakan You can check out the complete text samples here and my portfolio here .

visakanv-RNN — machine-generated husband chatter

A Sparse Bayesian Learning for Diagnosis of Nonstationary and Spatially Correlated Faults:References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777.

Using Autodiff to Estimate Posterior Moments: Conclusion, Limitations, and References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Abstract Importance sampling is a popular technique in Bayesian inference: by reweighting samples drawn from a proposal distribution we are able to obtain samples and moment estimates from a Bayesian posterior over some n latent variables. Recent work, however, indicates that importance sampling scales poorly — in order to accurately approximate the true posterior, the required number of importance samples grows is exponential in the number of latent variables (Chatterjee and Diaconis 2018). Massively parallel importance sampling works around this issue by drawing K samples for each of the n latent variables and reasoning about all Kn combinations of latent samples. In principle, we can reason efficiently over Kn combinations of samples by exploiting conditional independencies in the generative model. However, in practice this requires complex algorithms that traverse backwards through the graphical model, and we need separate backward traversals for each computation (posterior expectations, marginals and samples). Our contribution is to exploit the source term trick from physics to entirely avoid the need to hand-write backward traversals. Instead, we demonstrate how to simply and easily compute all the required quantities — posterior expectations, marginals and samples — by differentiating through a slightly modified marginal likelihood estimator. Introduction Importance weighting allows us to reweight samples drawn from a proposal in order to compute expectations of a different distribution, such as a Bayesian posterior. However, importance weighting breaks down in larger models. Chatterjee and Diaconis (2018) showed that the number of samples required to accurately approximate the true posterior scales as  exp (DKL (P (z|x)||Q (z))), where P (z|x) is the true posterior over latent variables, z, given data x, and Q (z) is the proposal. Problematically, we expect the KL divergence to scale with n, the number of latent variables. Indeed, if z is composed of n latent variables, and P (z|x) and Q (z) are IID over those n latent variables, then the KL-divergence is exactly proportional to n. Thus, we expect the required number of importance samples to be exponential in the number of latent variables, and hence we expect accurate importance sampling to be intractable in larger models. To resolve this issue we use a massively parallel importance sampling scheme that in effect uses an exponential number of samples to compute posterior expectations, marginals and samples (Kuntz, Crucinio, and Johansen 2023; Heap and Laurence 2023). This involves drawing K samples of each of the n latent variables from the proposal, then individually reweighting all Kn combinations of all samples of all latent variables. While reasoning about all Kn combinations of samples might seem intractable, we should in principle be able to perform efficient computations by exploiting conditional independencies in the underlying graphical model. However, many computations that are possible in principle are extremely complex in practice, and that turns out to be the case here. We noticed that we could perhaps perform this reasoning over Kn latent variables using methods from the discrete graphical model literature. This turned out to be less helpful than we had hoped because these algorithms involve highly complex backward traversals of the generative model. Worse, different traversals are needed for computing posterior expectations, marginals and samples, making a general implementation challenging. Our contribution is to develop a much simpler approach to computing posterior expectations, marginals and samples, which entirely avoids the need to explicitly write backwards computations. Specifically, we show that posterior expectations, marginals and samples can be obtained simply by differentiating through (a slightly modified) forward computation that produces an estimate of the marginal likelihood. The required gradients can be computed straightforwardly using modern autodiff, and the resulting implicit backward computations automatically inherit potentially complex optimizations from the forward pass. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Abstract Importance sampling is a popular technique in Bayesian inference: by reweighting samples drawn from a proposal distribution we are able to obtain samples and moment estimates from a Bayesian posterior over some n latent variables. Recent work, however, indicates that importance sampling scales poorly — in order to accurately approximate the true posterior, the required number of importance samples grows is exponential in the number of latent variables (Chatterjee and Diaconis 2018). Massively parallel importance sampling works around this issue by drawing K samples for each of the n latent variables and reasoning about all Kn combinations of latent samples. In principle, we can reason efficiently over Kn combinations of samples by exploiting conditional independencies in the generative model. However, in practice this requires complex algorithms that traverse backwards through the graphical model, and we need separate backward traversals for each computation (posterior expectations, marginals and samples). Our contribution is to exploit the source term trick from physics to entirely avoid the need to hand-write backward traversals. Instead, we demonstrate how to simply and easily compute all the required quantities — posterior expectations, marginals and samples — by differentiating through a slightly modified marginal likelihood estimator. n Introduction Importance weighting allows us to reweight samples drawn from a proposal in order to compute expectations of a different distribution, such as a Bayesian posterior. However, importance weighting breaks down in larger models. Chatterjee and Diaconis (2018) showed that the number of samples required to accurately approximate the true posterior scales as  exp (DKL (P (z|x)||Q (z))), where P (z|x) is the true posterior over latent variables, z, given data x, and Q (z) is the proposal. Problematically, we expect the KL divergence to scale with n, the number of latent variables. Indeed, if z is composed of n latent variables, and P (z|x) and Q (z) are IID over those n latent variables, then the KL-divergence is exactly proportional to n. Thus, we expect the required number of importance samples to be exponential in the number of latent variables, and hence we expect accurate importance sampling to be intractable in larger models. To resolve this issue we use a massively parallel importance sampling scheme that in effect uses an exponential number of samples to compute posterior expectations, marginals and samples (Kuntz, Crucinio, and Johansen 2023; Heap and Laurence 2023). This involves drawing K samples of each of the n latent variables from the proposal, then individually reweighting all Kn combinations of all samples of all latent variables. While reasoning about all Kn combinations of samples might seem intractable, we should in principle be able to perform efficient computations by exploiting conditional independencies in the underlying graphical model. However, many computations that are possible in principle are extremely complex in practice, and that turns out to be the case here. We noticed that we could perhaps perform this reasoning over Kn latent variables using methods from the discrete graphical model literature. This turned out to be less helpful than we had hoped because these algorithms involve highly complex backward traversals of the generative model. Worse, different traversals are needed for computing posterior expectations, marginals and samples, making a general implementation challenging. Our contribution is to develop a much simpler approach to computing posterior expectations, marginals and samples, which entirely avoids the need to explicitly write backwards computations. Specifically, we show that posterior expectations, marginals and samples can be obtained simply by differentiating through (a slightly modified) forward computation that produces an estimate of the marginal likelihood. The required gradients can be computed straightforwardly using modern autodiff, and the resulting implicit backward computations automatically inherit potentially complex optimizations from the forward pass.

CHATTER

chatter.dev

-- employees

Since n.d.

COMPANY RANKING

CHATTER

EVERGREEN INDEX #2298

Representing this company? Click here to claim and customize this page!

Chatter WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

CHATTER

chatter.dev

-- employees

Since n.d.

COMPANY RANKING

CHATTER

EVERGREEN INDEX #2298

RELATED COMPANIESRANK

STORIES ABOUT CHATTER

Representing this company? Click here to claim and customize this page!

Chatter WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!