Finthesisis in the Finance-Accounting Tech Industry

Finthesis

finthesis

Realistic Text-to-Speech Voice Synthesis: Comparing Tortoise and Bark

First and Last Things by H. G. Wells, is part of the HackerNoon Books Series. You can jump to any chapter in this book here . THE SYNTHESIS. THE SYNTHESIS. It seems to me that the whole living creation may be regarded as walking in its sleep, as walking in the sleep of instinct and individualized illusion, and that now out of it all rises man, beginning to perceive his larger self, his universal brotherhood and a collective synthetic purpose to increase Power and realize Beauty... I write this down. It is the form of my belief, and that unanalyzable something called Beauty is the light that falls upon that form. It is only by such images, it is only by the use of what are practically parables, that I can in any way express these things in my mind. These two things, I say, are the two aspects of my belief; one is the form and the other the light. The former places me as it were in a scheme, the latter illuminates and inspires me. I am a member in that great being, and my function is, I take it, to develop my capacity for beauty and convey the perception of it to my fellows, to gather and store experience and increase the racial consciousness. I hazard no whys nor wherefores. That is how I see things; that is how the universe, in response to my demand for a synthesizing aspect, presents itself to me. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. H. G. Wells (2009). First and Last Things: A Confession of Faith and Rule of Life. Urbana, Illinois: Project Gutenberg. Retrieved October 2022, https://www.gutenberg.org/files/4225/4225-h/4225-h.htm This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org , located at https://www.gutenberg.org/policy/license.html .

THE SYNTHESIS

We’ve heard of , we’ve heard of NeRFs, and we’ve seen these kinds of applications allowing you to recreate someone’s face and pretty much make him or her say whatever you want. deepfakes What you might not know is how inefficient those methods are and how much computing and time they require. Plus, we only see the best results. Keep in mind that what we see online is the results associated with the faces we could find most examples of, so basically, internet personalities and the models producing those results are trained using lots of computing, meaning expensive resources like many graphics cards. Still, the results are really impressive and only getting better. Fortunately, some people like Jiaxian Tang and colleagues are working on making those methods more available and effective with a new model called RAD-NeRF. From a single video, they can synthesize the person talking for pretty much any word or sentence in real time with better quality. You can animate a talking head following any audio track in real-time. This is both so cool and so scary at the same time... Learn more in the video References ►Tang, J., Wang, K., Zhou, H., Chen, X., He, D., Hu, T., Liu, J., Zeng, G. and Wang, J., 2022. Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition. . ►Results/project page: ​​ arXiv preprint arXiv:2211.12368 https://me.kiui.moe/radnerf/ Video Transcript 0:02 [Music] 0:07 we've heard of deep fakes we've heard of 0:09 Nerfs and we've seen these kinds of 0:11 applications allowing you to recreate 0:13 someone's face and pretty much make him 0:15 say whatever you want what you might not 0:17 know is how inefficient those methods 0:20 are and how much Computing and time they 0:22 require plus we only see the best 0:24 results keep in mind that what we see 0:26 online is the results associated with 0:29 the faces we could find most examples of 0:31 so basically internet personalities and 0:34 the models producing those results are 0:36 trained using lots of computing meaning 0:38 expensive resources like many graphics 0:41 cards still the results are really 0:43 impressive and only getting better 0:45 fortunately some people like Jackson 0:47 tang and colleagues are working on 0:49 making those methods more available and 0:52 effective with a new model called red 0:54 Nerf but let's hear that from their own 0:57 model hello thanks for watching the 0:59 supplementary video for our paper 1:00 real-time neural Radiance talking head 1:03 synthesis via decomposed audio spatial 1:05 encoding 1:06 our method is person-specific and only 1:08 needs a three to five minutes monocular 1:10 video for training 1:11 after training the model can synthesize 1:14 realistic Talking Heads driven by 1:15 arbitrary audio in real time while 1:17 keeping comparable or better rendering 1:19 quality compared to previous methods so 1:21 you heard that right from a single video 1:23 they can synthesize the person talking 1:26 for pretty much any word or sentence in 1:28 real time with better quality you can 1:30 animate a talking head following any 1:33 audio track in real time this is both so 1:36 cool and so scary at the same time just 1:39 imagine what could be done if we could 1:40 make you say anything at least they 1:43 still need access to a video of you 1:45 speaking in front of the camera for 5 1:47 minutes so it's hard to achieve that 1:48 without you knowing still as soon as you 1:51 appear online anyone will be able to use 1:53 such a model and create infinite videos 1:56 of you talking about anything they want 1:58 they can even host live streams with 2:00 this method which is even more dangerous 2:03 and makes it even harder to say wetsuit 2:05 or not anyways even though this is 2:08 interesting and I'd love to hear your 2:10 thoughts in the comments and keep the 2:11 discussion question going here I wanted 2:13 to cover something that is only positive 2:15 and exciting science more precisely how 2:19 did they achieve to animate Talking 2:20 Heads in real time from any audio using 2:23 only a video of the face as they State 2:26 their red Nerf model can run 500 times 2:29 faster than the previous works with 2:31 better rendering quality and more 2:33 control you may ask how is that possible 2:36 we usually trade quality for efficiency 2:39 yet they achieve to improve both 2:41 incredibly these immense improvements 2:43 are possible thanks to three main points 2:46 the first two are related to the 2:48 architecture of the model more 2:50 specifically how they adapted the Nerf 2:52 approach to make it more efficient and 2:54 with improved motions of the Torso and 2:57 head the first step is to make nerves 2:59 more efficient I won't dive into how 3:02 Nerfs work since we covered it numerous 3:04 time basically it's an approach based on 3:06 neural networks to reconstruct 3D 3:09 volumetric scenes from a bunch of 2D in 3:11 images which means regular images this 3:14 is why they will take a video as input 3:17 as it basically gives you a lot of 3:19 images of a person from many different 3:21 angles so it usually uses a network to 3:24 predict all pixels colors and densities 3:26 from the camera Viewpoint you are 3:28 visualizing and does that for all 3:31 viewpoints you want to show when 3:32 rotating around the subject which is 3:34 extremely computation hungry as you are 3:37 predicting multiple parameters for each 3:39 coordinate in the image every time and 3:41 you are learning to predict all of them 3:43 Plus in their case it isn't only a Nerf 3:46 producing or 3D scene it also has to 3:49 match an audio input and fit the lips 3:51 mouth eyes and movements with what the 3:53 person says instead of predicting all 3:56 pixels densities and colors matching the 3:58 audio for a specific frame they will 4:00 work with two separate new and condensed 4:03 spaces called grid spaces or grid-based 4:06 Nerf they will translate their 4:08 coordinates into a smaller 3D grid space 4:11 trans laid their audio into a smaller 2D 4:13 grid space and then send them to render 4:16 the head this means they never merge the 4:19 audio data with the spatial data which 4:22 will increase the size exponentially 4:23 adding two dimensional inputs to each 4:26 coordinate so reducing the size of the 4:29 audio features along with keeping the 4:31 audio and spatial features separate is 4:34 what makes the approach so much more 4:36 efficient but how can the results be 4:38 better if they use condensed spaces that 4:40 have less information adding a few 4:42 controllable features like an eye 4:44 blinking control to our grid Nerf the 4:47 model will learn more realistic 4:48 behaviors for the eyes compared to 4:51 previous approaches something really 4:53 important for realism the second 4:55 Improvement they've done is to model the 4:57 Torso with another Nerf using the same 5:00 approach instead of trying to model it 5:02 with the same Nerf used further head 5:04 which will require much fewer parameters 5:07 and different needs as the goal here is 5:09 to animate moving heads and not whole 5:12 bodies since the Torso is pretty much 5:14 static in these cases they use a much 5:16 simpler and more efficient Nerf based 5:18 module that only works in 2D working in 5:21 the image space directly instead of 5:24 using camera arrays as we usually do 5:26 with Nerf to generate many different 5:28 angles which aren't needed for a torso 5:30 so it is basically much more efficient 5:32 because they modified the approach for 5:35 this very specific use case of the rigid 5:37 torso and moving head videos they then 5:40 recompose the head with the Torso to 5:42 produce the final video and voila this 5:45 is how you produce talking head videos 5:47 over any audio input super efficiently 5:50 of course this was just an overview of 5:53 this new exciting research publication 5:55 and they do other modifications during 5:57 the training of their algorithm to make 5:59 it more efficient which is the third 6:01 point I mentioned at the beginning of 6:03 the video if you were wondering I invite 6:05 you to read their paper for more 6:07 information the link is in the 6:09 description below before you leave I 6:10 just wanted to thank the people who 6:12 recently supported this channel through 6:14 patreon this is not necessary and 6:16 strictly to support the work I do here 6:18 huge thanks to artem vladiken Leopoldo 6:22 Alta Murano J Cole Michael carichao 6:25 Daniel gimness and a few Anonymous 6:28 generous donors it will be greatly 6:30 appreciated if you also want and can 6:33 afford to support my work financially 6:35 the link to my patreon page is in the 6:37 description below as well but no worries 6:39 if not a sincere comment below this 6:42 video is all I need to be happier I hope 6:45 you've enjoyed this video and I will see 6:47 you next week with another amazing paper 6:51 [Music]

Efficient NeRFs for Real-Time Portrait Synthesis (RAD-NeRF)

NeRF eficientes para la síntesis de retratos en tiempo real (RAD-NeRF)

रीयल-टाइम पोर्ट्रेट सिंथेसिस (आरएडी-एनईआरएफ) के लिए कुशल एनईआरएफ

リアルタイム ポートレート合成 (RAD-NeRF) のための効率的な NeRF

NeRFs eficientes para síntese de retrato em tempo real (RAD-NeRF)

NeRF hiệu quả để tổng hợp chân dung thời gian thực (RAD-NeRF)

用于实时人像合成的高效 NeRF (RAD-NeRF)

eDiffi, NVIDIA's most recent model, generates better-looking and more accurate images than all previous approaches like DALLE 2 or Stable Diffusion. eDiffi better understands the text you send and is more customizable, adding a feature we saw in a previous paper from NVIDIA: the painter tool. Learn more in the video... References ►Read the full article: ► Balaji, Y. et al., 2022, eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, ►Project page: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/ediffi/ https://arxiv.org/abs/2211.01324 https://deepimagination.cc/eDiffi/ https://www.louisbouchard.ai/newsletter/ Video Transcript 0:06 the new state-of-the-art approach for 0:08 image synthesis it generates better 0:10 looking and more accurate images than 0:13 all previous approaches like Delhi 2 or 0:15 stable diffusion either if he better 0:17 understands the text you send and is 0:19 more customizable adding a new feature 0:21 we saw in a previous paper from Nvidia 0:23 the painter tool as they see you can 0:26 paint with words in short this means you 0:29 can enter a few subjects and paint in 0:32 the image what should appear here and 0:34 there allowing you to create much more 0:36 customized images compared to a random 0:39 generation following a prompt this is 0:41 the next level enabling you to pretty 0:43 much get the exact image you have in 0:45 mind by simply drawing a horrible quick 0:47 sketch something even I can do as I 0:50 mentioned the results are not only Sota 0:52 and better looking than stable diffusion 0:55 but they are also way more controllable 0:57 of course it's a different use case as 0:59 it needs a bit more work and a clearer 1:02 ID in mind for creating such a draft but 1:04 it's definitely super very exciting and 1:06 interesting it's also why I wanted to 1:08 cover it on my channel since it's not 1:11 merely a better model but also a 1:13 different approach with much more 1:15 control over the output the tool isn't 1:17 available yet unfortunately but I sure 1:19 hope it will be soon by the way you 1:22 should definitely subscribe to the 1:23 channel and follow me on Twitter at what 1:25 say hi if you like this kind of video 1:27 and would like to have access to easily 1:30 digestible news on this heavily 1:32 complicated field another win which they 1:34 allow you to have more control in this 1:37 new model is by using the same feature 1:39 we saw but differently indeed the model 1:42 generates images Guided by a sentence 1:44 but it can also be influenced using a 1:47 quick sketch so it basically takes an 1:49 image and a text as inputs this means 1:52 you can do other stuff as it understands 1:54 images here they leverage this 1:56 capability by developing a style 1:58 transfer approach where you can 2:00 influence the style of the image 2:02 generation process giving an image with 2:04 a particular style well along with your 2:06 text input this is super cool and just 2:09 look at the results they speak for 2:11 themselves it's incredible beating both 2:14 Sota style transfer models and image 2:16 synthesis models with a single approach 2:18 now the question is how could Nvidia 2:22 develop a model that creates better 2:23 looking images enable more control over 2:26 both the style and the image structure 2:29 as well as better understanding and 2:31 representing what you actually want in 2:34 your text well they change the typical 2:36 diffusion architecture in two ways first 2:39 they encode the text using two different 2:41 approaches that I already covered on the 2:43 channel which we refer to as clip and T5 2:46 encoders this means they will use 2:48 pre-trained models to take text and 2:50 create various embeddings focusing on 2:52 different features as they are trained 2:55 and behaved differently and meanings are 2:57 just representations maximizing what the 3:00 sentence actually means for the 3:01 algorithm or the machine to understand 3:04 it regarding the input image they just 3:06 use the clip embeddings as well 3:08 basically encoding the image so that the 3:11 model can understand it which you can 3:13 learn more about in my other videos 3:14 covering generative models as they are 3:16 pretty much all built on clip this is 3:19 what allows them to have more control 3:21 over the output as well as processed 3:23 text and images rather than only text 3:25 the second modification is using a 3:28 Cascade of diffusion models instead of 3:31 reusing the same iteratively as we 3:33 usually do with diffusion based models 3:35 here the use models trained for the 3:38 specific part of the generative process 3:39 meaning that each model does not have to 3:42 be as general as the regular diffusion 3:44 denoiser since each model has to focus 3:46 on a specific part of the process it can 3:49 be much better at it they use this 3:51 approach because they observed that the 3:52 denoising models seemed to use the text 3:55 embeddings a lot more to orient its 3:57 generation towards the beginning of the 3:59 process and then use it less and less to 4:02 focus on output quality and Fidelity the 4:05 this naturally brings the hypothesis 4:07 that reusing the same denoising model 4:09 throughout the whole process might not 4:11 be the best ID since it automatically 4:13 focuses on different tasks and we know 4:15 that a generalist is far from the expert 4:18 level at all tasks why not use a few 4:20 experts instead of one generalist to get 4:23 much better results so this is what they 4:25 did and why they call them denoising 4:28 experts and the main reason for this 4:30 improves performance in quality and 4:32 faithfulness the rest of the 4:34 architecture is pretty similar to other 4:36 approaches of scaling the final results 4:38 with other models to get a high 4:40 definition final image the image and 4:43 video synthesis fields are just getting 4:45 crazy nowadays and we are seeing 4:47 impressive results coming out every week 4:49 I am super excited for the next releases 4:51 and I love to see different approaches 4:53 with both innovative ways of tackling 4:55 the problem and also going for different 4:57 use cases as a great person once said 5:01 what a time to be alive I hope you like 5:04 this quick overview of the approach a 5:06 bit more high level than what I usually 5:08 do as it takes most Parts I already 5:10 covered in numerous videos and changed 5:12 them to act differently I invite you to 5:15 watch my stable diffusion video to learn 5:17 a bit more about the diffusion approach 5:19 itself and read the nvidia's paper to 5:21 learn more about this specific approach 5:23 and its implementation I will see you 5:26 next week with another amazing paper 5:32 foreign 5:36 [Music]

An Intro to eDiffi: NVIDIA's New SOTA Image Synthesis Model

Una introducción a eDiffi: el nuevo modelo de síntesis de imágenes SOTA de NVIDIA

Une introduction à eDiffi : le nouveau modèle de synthèse d'images SOTA de NVIDIA

eDiffi का एक परिचय: NVIDIA का नया SOTA छवि संश्लेषण मॉडल

eDiffi の紹介: NVIDIA の新しい SOTA 画像合成モデル

Uma introdução ao eDiffi: o novo modelo de síntese de imagem SOTA da NVIDIA

Giới thiệu về eDiffi: Mô hình tổng hợp hình ảnh SOTA mới của NVIDIA

eDiffi 简介：NVIDIA 的新 SOTA 图像合成模型

The Larger Synthesis

Have you ever dreamed of taking the style of a picture, like this cool TikTok drawing style on the left, and applying it to a new picture of your choice? Well, I did, and it has never been easier to do. In fact, you can even achieve that from only text and can try it right now with this new method and their Google Colab notebook available for everyone (see references). Simply take a picture of the style you want to copy, enter the text you want to generate, and this algorithm will generate a new picture out of it! Just look back at the results above, such a big step forward! The results are extremely impressive, especially if you consider that they were made from a single line of text! If that sounds interesting, watch the video and learn more! Watch the video References ►Read the full article: ►CLIPDraw: Frans, K., Soros, L.B. and Witkowski, O., 2021. CLIPDraw: exploring text-to-drawing synthesis through language-image encoders. ►StyleCLIPDraw: Schaldenbrand, P., Liu, Z. and Oh, J., 2021. StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis. ►CLIPDraw Colab notebook: ►StyleCLIPDraw code: ►StyleCLIPDraw Colab notebook: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/clipdraw/ https://arxiv.org/abs/2106.14843 https://arxiv.org/abs/2111.03133 https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb https://github.com/pschaldenbrand/StyleCLIPDraw https://colab.research.google.com/github/pschaldenbrand/StyleCLIPDraw/blob/master/Style_ClipDraw_1_0_Refactored.ipynb https://www.louisbouchard.ai/newsletter/ Video Transcript 00:00 have you ever dreamed of taking a 00:01 picture like this cool tick tock drawing 00:03 style and applying it to a new picture 00:06 of your choice well i did and it has 00:08 never been easier to do in fact you can 00:10 even achieve that from only text and you 00:13 can try it right now with this new 00:15 method and their google collab notebook 00:17 available for everyone simply take a 00:19 picture of the style you want to copy 00:21 enter the text you want to generate and 00:23 this algorithm will generate a new 00:25 picture out of it look at that such a 00:28 big step forward the results are 00:30 extremely impressive especially if you 00:31 consider that they were made from a 00:33 single line of text here i tried 00:35 imitating the same style with another 00:37 text input to be honest sometimes it may 00:40 look a bit all over the place especially 00:42 if you select a more complicated or 00:44 messy drawing style like this one 00:46 speaking of something messy if you are 00:47 like me and your model versioning and 00:49 resource tracking looks like this you 00:51 may be the perfect candidate to try the 00:53 sponsor of two days video which is none 00:55 other than weights and biases i always 00:57 assumed i could stack folders like this 00:59 and simply add old v1 v2 v3 and so on to 01:03 my file names without any problem until 01:06 i had to work with someone while it may 01:07 be easy for me to find my old tests it 01:10 was impossible to explain my thought 01:12 process behind this mess and was my 01:14 teammate's nightmare if you care about 01:15 your teammates and reproducibility don't 01:18 do like i did and give weights and 01:20 biases a shot no more notebooks or 01:22 results saved everywhere as it creates a 01:24 super friendly user dashboard for you 01:26 and your team to track your experiments 01:28 and it's super easy to set up and use 01:30 it's the first link in the description 01:32 and i promise within a month you will be 01:34 completely dependent 01:37 as we said this new model by peter 01:39 schaldenbrunn ethel called style clip 01:42 draw which is an improvement upon clip 01:44 draw by kevin franz at all takes an 01:46 image and takes as inputs and can 01:48 generate a new image based on your text 01:50 and following the style in the image so 01:52 the model has to both understand what's 01:54 in the text and the image to correctly 01:56 copy its style as you may suspect this 01:59 is incredibly challenging but we are 02:01 fortunate enough to have a lot of 02:02 researchers working on so many different 02:04 challenges like trying to link text with 02:07 images which is what clip can do quickly 02:10 clip is a model developed by openai that 02:12 can basically associate a line of text 02:14 with an image both the text and images 02:17 will be encoded similarly so that they 02:19 will be very close to each other in the 02:21 new space they are encoded in if they 02:23 both mean the same thing using clip the 02:25 researchers could understand the text 02:27 from the user input and generate an 02:29 image out of it if you are not familiar 02:31 with clip yet i would recommend watching 02:33 a video i made about it together with 02:35 dolly earlier this year but then how did 02:38 they apply a new style to it clip is 02:40 just linking existing images to texts it 02:43 cannot create a new image indeed we also 02:46 need something else to capture the style 02:48 of the image sent in both the textures 02:50 and shapes well the image generation 02:52 process is quite unique it won't simply 02:55 generate an image right away rather it 02:57 will draw on a canvas and get better and 02:59 better over time it will just draw 03:01 random lines at first and create an 03:03 initial image this new image is then 03:06 sent back to the algorithm and compared 03:08 with both the style image and the text 03:10 which will generate another version this 03:12 is one iteration at each iteration we 03:15 draw random curves again oriented by the 03:17 two losses we'll see in a second this 03:19 random process is quite cool since it 03:22 will allow each new test to look 03:24 different so using the same image and 03:26 same text as inputs you will end up with 03:29 different results that may look even 03:31 better here you can see a very important 03:33 step called image augmentation it will 03:35 basically create multiple variations of 03:38 the image and allow the model to 03:39 converge on results that look right to 03:42 humans and not simply on the right 03:44 numerical values for the machine this 03:46 simple process is repeated until we are 03:49 satisfied with the results so this whole 03:51 model learns on the fly over many 03:54 iterations optimizing two losses we see 03:56 here one for aligning the content of the 03:59 image with the text sent and the other 04:01 further style here you can see the first 04:03 lust is based on how close the clip 04:06 encodings are as we said earlier where 04:08 clip is basically judging the results 04:11 and its decision will orient the next 04:12 generation the second one is also very 04:15 simple we send both images into a 04:18 pre-trained convolutional neural network 04:20 like vgg which will encode the images 04:22 similarly to clip we then compare these 04:24 encodings to measure how close they are 04:26 to each other this will be our second 04:29 judge that will orient the next 04:30 generation as well this way using both 04:33 judges we can get closer to the text and 04:35 the wanted style at the same time in the 04:37 next generation if you are not familiar 04:39 with convolutional neural networks and 04:41 encodings i will strongly recommend 04:43 watching the video i made explaining 04:45 them in simple terms this iterative 04:47 process makes the model a bit slow to 04:49 generate a beautiful image but after a 04:51 few hundred iterations or in other words 04:53 after a few minutes you have your new 04:55 image and i promise it's worth the wait 04:58 it also means that it doesn't require 05:00 any other training which is pretty cool 05:02 now the interesting part you've been 05:04 waiting for indeed you can use it right 05:06 now for free or at least pretty cheaply 05:08 using the collab notebook linked in the 05:10 description below i had some problems 05:12 running it and i would recommend buying 05:14 the pro version of collab if you'd like 05:16 to play with it without any issues 05:19 otherwise feel free to ask me any 05:21 questions in the comments if you 05:22 encounter any problems i pretty much 05:24 went through all of them myself to use 05:27 it you simply run all cells like that 05:29 and that's it you can now enter a new 05:31 text for the generation or send a new 05:33 image for the style from a link and 05:35 voila now tweak the parameters and see 05:38 what you can do if you play with it 05:40 please send me the results on twitter 05:42 and tag me i'd love to see them as they 05:44 state in the paper the results will have 05:46 the same biases as the models they use 05:49 such as clip which you should consider 05:51 if you play with it of course this was a 05:53 simple overview of the paper and i 05:55 strongly invite you to read both clip 05:57 draw and style clip draw for more 05:58 technical details and try their collab 06:01 notebook both are linked in the 06:02 description below thank you once again 06:05 weights and biases for sponsoring this 06:07 video and huge thanks to you for 06:09 watching until the end i hope you 06:11 enjoyed this week's video let me know 06:13 what you think and how you will use this 06:15 new model 06:17 [Music]

StyleCLIPDraw: Text-to-Drawing Synthesis with Artistic Control

AI Image Synthesis: What The Future Holds

Comparative Study of Different Adversarial Text to Image Methods Introduction synthesis of images from text has become popular with deep and neural network architectures to aid in learning discriminative text feature representations. Automatic realistic convolutional recurrent power and strong properties of representations even though attractive, its a complex process and requires -specific knowledge. Over the years the techniques have evolved as auto-adversarial networks in space of machine learning algorithms continue to evolve. Discriminative generalization attribute domain In comparison, offers an easy, general, and flexible plugin that can be used to identify and describing objects across multiple domains by means of visual categories. natural language The best thing is to combine the generality of text descriptions with the discriminative power of attributes. This blog addresses different text to image synthesis algorithms using that aims to directly map words and characters to image pixels with GAN (Generative Adversarial Network) natural language representation and image synthesis techniques. The featured algorithms . learn a text feature representation that captures the important visual details and then use these features to synthesize a compelling image that a human might mistake for real Generative Adversarial Text to Image Synthesis This image synthesis mechanism uses to learn a correspondence function with images by conditioning the model conditions on instead of class labels. deep convolutional and recurrent text encoders text descriptions An effective approach that enables text-based image synthesis using a character-level text encoder and . The purpose of the GAN is to class-conditional GAN view (text, image) pairs as joint observations and train the discriminator to judge pairs as real or fake . Equipped with a manifold (regularization procedure which encourages outputs to appear more realistic) for the GAN generator that significantly improves the quality of generated samples. interpolation regularizer interpolated The objective of GAN is to view (text, image) and train the discriminator to judge pairs as or . pairs as joint observations real fake Both the and the perform has been trained to enable conditioning tightly only on textual features. generator network G discriminator network D feed-forward learning and inference by e, LICENSE- Apache 2.0 Sourc , has several layers of with followed by Discriminator D stride2 convolution spatial batch normalization leaky ReLU . The GAN is trained in with mini-batches SGD (Stochastic Gradient Descent). In addition to the real/fake inputs to the discriminator during training, it is also fed with the third type of input consisting of real images with mismatched text, which aids the discriminator to score it as fake. The below figure illustrates text to image generation samples of different types of birds. — (Open Source Apache 2.0 License) Source Library and Usage git clone https://github.com/zsdonghao/text-to-image.git [TensorFlow 1.0+, TensorLayer 1.4+, NLTK : for tokenizer]
python downloads.py [download Oxford-102 flower dataset and caption files(run this first)]
python data_loader.py [load data for further processing]
python train_txt2im.py [train a text to image model]
python utils.py  [helper functions]
python models.py [models] 2. Multi-Scale Gradient GAN for Stable Image Synthesis ) is responsible for handling in gradients passing from the discriminator to the generator that become , due to a learning imbalance during training. It uses an effective technique that allows the flow of gradients from the discriminator to the generator at multiple scales helping to generate Multi-Scale Gradient Generative Adversarial Network (MSG-GAN instability uninformative synchronized multi-scale images. The discriminator not only looks at the final output (highest resolution) of the generator but also at the outputs of the as illustrated in the below figure. As a result, the discriminator becomes a function of of the generator (by using concatenation operations) and importantly, passes to all the simultaneously. The architecture of MSG-GAN for generating synchronized multi-scale images. intermediate layers multiple scale outputs gradients scales — (Open Source MIT License) Source The architecture of MSG-GAN for generating synchronized multi-scale images. — (Open Source MIT License) Source MSG-GAN is robust to changes in the learning rate and has a more consistent increase in when compared to MSG-GAN shows the same trait and consistency for all the resolutions and images generated at higher resolution maintain the symmetry of certain features such as the same color for both eyes, or earrings in both ears. Moreover, the training phase allows a better understanding of image properties (e.g., and ). image quality progressive growth (Pro-GAN). convergence quality diversity Library and Usage git clone https://github.com/akanimax/BMSG-GAN.git [PyTorch]
python train.py --depth=7 \ 
                  --latent_size=512 \
                  --images_dir=<path to images> \
                  --sample_dir=samples/exp_1 \
                  --model_dir=models/exp_1 3 . T2F-text-to-face-generation-using-deep-learning ( StackGAN ++ and ProGAN ) In the architecture, works on the principle of adding new layers that model increasingly fine details as training progresses. Here both the generator and discriminator start by images of low resolution and adds images’ in-depth details in steps. It helps in a more and training process. ProGAN creating subsequent stable faster architecture consists of and The different branches of the tree represent images of varying scales, all belonging to the same scene. StackGAN has been known for yielding different types of approximate distributions. These multiple related distributions include multi-scale image distributions and joint conditional and unconditional image distributions. StackGAN multiple generators discriminators in a tree-like structure . uses a combined architecture of . is the principle working methodology. The textual description is encoded into a summary vector using an . The summary vector i.e. Embedding as illustrated in the below diagram is passed through the to obtain the textual part of the latent vector (uses like technique) for the GAN as input. T2F ProGAN and StackGAN ProGAN is known for the synthesis of facial images , while StackGAN is known for text encoding , where conditioning augmentation LSTM network Conditioning Augmentation block (a single linear layer ) VAE parameterization The second part of the latent vector is The latent vector yielded is then fed to the generator part of the . The embedding thus formed is finally fed to the final layer of the discriminator for . The training of the GAN proceeds layer by layer. Every next layer adds spatial resolutions at an increasing level. random Gaussian noise . GAN conditional distribution matching is used to introduce any new layer. This step helps to remember and restore previously learned information.T2F architecture for generating face from textual descriptions. The fade-in technique , LICENSE-MIT Source The below figure illustrates mechanism of facial image generation from textual captions for each of them. Source — , LICENSE-MIT https://github.com/akanimax/T2F.git Library and Usage git clone https://github.com/akanimax/T2F.gitpip install -r requirements.txtmkdir training_runsmkdir training_runs/generated_samples training_runs/losses training_runs/saved_modelstrain_network.py --config=configs/11.comf 4. Object-driven Text-to-Image Synthesis via Adversarial Training AttnGAN LICENSE — MIT Source performs fine-grained text-to-image synthesis. Such occurs in two steps. At first, a semantic layout (class labels, bounding boxes, shapes of salient objects) is generated and then the generating images are synthesized by a . Object-driven Attentive GAN (Obj-GAN) in-depth granular image synthesis de-convolutional image generator However is accomplished with the sentence being served as input to Obj-GAN. This facilitates the to generate a sequence of objects specified by their bounding boxes (with class labels) and shapes. semantic layout generation Obj-GAN The is trained as an to generate a sequence of bounding boxes, followed by a shape generator to . box generator attentive seq2seq model predict and generate the shape of each object in its bounding box In the image generation step, the and are designed to enable image generation conditioned on the semantic layout generated in the first step. The generator concentrates on synthesizing the image region within a by focusing on words that are most relevant to the object in that bounding box. object-driven attentive generator object-wise discriminator bounding box vectors serve as an important tool encode information from the words that are most relevant to that image region. This is accomplished with the help of both -wise and -wise context vectors for defined image regions. Attention-driven context patch object A is also used. It is able to offer rich object-wise signals. These signals help to determine whether the synthesized object matches the text description and the pre-generated layout. Fast R-CNN based object-wise discriminator discrimination (paying to most and performs better than traditional grid attention, capable of . Object-driven attention attention relevant words pre-generated class labels) generates complex scenes in high quality The open-source code for Obj-GAN from Microsoft is not available yet. - (License-OpenSource) Source 5. MirrorGan MirrorGAN is built to emphasize features. It helps in the -preserving text-to-image-to-text framework. global-local attentive semantic MirrorGAN is equipped to learn text-to-image generation by re-description. It is composed of three modules: “a semantic text embedding module ( ), a global-local for cascaded image generation ( ), and a semantic text regeneration and module ( )”. STEM collaborative attentive module GLAM alignment STREAM STEM generates using recurrent neural network ( ) to embed the given text description into local and . word-and sentence-level embeddings RNN word-level features global sentence-level features GLAM has a . It is designed by three image generation networks sequentially for generating target images from coarse to fine scales. During target image generation, it leverages both and This helps to the diversity and of the generated images. multi-stage cascaded generator stacking local word attention global sentence. progressively enhance semantic consistency STREAM purposes to the text description from the generated image. The image aligns with the given text description. regenerate semantically Word-level attention model takes in related words. This helps to generate an attentive word-context feature. Word embedding and the visual feature is taken as the input in each stage. The word embedding is first converted into an underlying common semantic space of visual features by a and with the to obtain the . Finally, the attentive word-context feature is obtained by calculating the inner product between the and r along with neighboring contextual high perception layer multiplied visual feature attention score attention score perception laye word embedding. MirrorGAN’s two most important components semantic text regeneration and alignment module maintains overall sync between input text and output image. These two modules help to the from the The output finally semantically aligns with the given text description. In addition, an encoder decoder-based image caption framework is used to generate captions in the architecture. The encoder is a and the is an . regenerate text description generated image. convolutional neural network (CNN) decoder RNN performs better than at all settings by a large margin, demonstrating the of the proposed text-to-image-to-text framework and the since MirrorGAN generated high-quality images with semantics consistent with the input text descriptions. MirrorGAN AttnGAN superiority global-local collaborative attentive module The following figure illustrates how a generates images modifying some words of the text descriptions and the corresponding top-2 attention maps in the last stage preserving the semantic similarity. Mirror GAN Source Library and Usage git clone git@github.com:komiya-m/MirrorGAN.git [python 3.6.8, keras 2.2.4, tensor-flow 1.12.0]
Dependencies : easydict, pandas, tqdm
python main_clevr.py
cd MirrorGAN
python pretrain_STREAM.py
python train.py 6. StoryGAN Story visualization takes as input a and generates at its output , one for each sentence. multi-sentence paragraph sequence of images Story visualization task is a sequential conditional generation problem where it the with the . jointly considers current input sentence contextual information Story GAN gives less focus on the continuity in generated images ( ), but more on the global consistency across and . frames dynamic scenes characters Relies on the component in the , where the Context Encoder the story flow in addition to providing the image generator with both local and . Text2Gist Context Encoder dynamically tracks global conditional information and the structure on the inputs help to enhance the image quality and ensure consistency across the generated images and the story to be visualized. Two-level discriminator recurrent The below figure illustrates a architecture. The variables represented in gray solid circles serves as an input story S and individual sentences with random noise . The generator network is built using specific customized components – There are two discriminators on top that actively serve its primary task to and each is real or fake. StoryGAN s1, . . . , sT 1, . . . , T Story Encoder, Context Encoder and Image Generator . discriminate each image sentence pair image-sequence-story pair The framework of StoryGAN, - LICENSE-MIT Source The architecture is capable of with the of the images/sentences in the story when they are concatenated. The product of image and text features is to have a that serves as an input to a fully connected layer. The is employed with a to predict whether it is a fake or real story pair. Story GAN distinguishing real/fake stories feature vectors embedded compact feature representation fully connected layer sigmoid non-linearity Library and Usage git clone https://github.com/yitong91/StoryGAN.git   [Python 2.7, PyTorch, cv2]python main_clevr.py 7. Keras-text-to-image : Sample DCGAN Architecture to generate 64×64 RGB pixel images from the LSUN dataset, , License -MIT In Keras text to image translation is achieved using and as well as . Source GAN Word2Vec recurrent neural networks It uses which has been a breakthrough in GAN research as it introduces major architectural changes to tackle problems like training instability, mode collapse, and internal covariate shift. ( ) DCGan Deep Convolutional Generative Adversarial Network Sample DCGAN Architecture to generate 64x64 RGB pixel images from the LSUN dataset, , License -MIT Source Library and Usage git clone https://github.com/chen0040/keras-text-to-image.git
import os 
import sys 
import numpy as np
from random import shuffle


def train_DCGan_text_image():
    seed = 42

    np.random.seed(seed)
    
    current_dir = os.path.dirname(__file__)
    # add the keras_text_to_image module to the system path
    sys.path.append(os.path.join(current_dir, '..'))
    current_dir = current_dir if current_dir is not '' else '.'

    img_dir_path = current_dir + '/data/pokemon/img'
    txt_dir_path = current_dir + '/data/pokemon/txt'
    model_dir_path = current_dir + '/models'

    img_width = 32
    img_height = 32
    img_channels = 3
    
    from keras_text_to_image.library.dcgan import DCGan
    from keras_text_to_image.library.utility.img_cap_loader import load_normalized_img_and_its_text

    image_label_pairs = load_normalized_img_and_its_text(img_dir_path, txt_dir_path, img_width=img_width, img_height=img_height)

    shuffle(image_label_pairs)

    gan = DCGan()
    gan.img_width = img_width
    gan.img_height = img_height
    gan.img_channels = img_channels
    gan.random_input_dim = 200
    gan.glove_source_dir_path = './very_large_data'

    batch_size = 16
    epochs = 1000
    gan.fit(model_dir_path=model_dir_path, image_label_pairs=image_label_pairs,
            snapshot_dir_path=current_dir + '/data/snapshots',
            snapshot_interval=100,
            batch_size=batch_size,
            epochs=epochs)
def load_generate_image_DCGaN():
    seed = 42
    np.random.seed(seed)

    current_dir = os.path.dirname(__file__)
    sys.path.append(os.path.join(current_dir, '..'))
    current_dir = current_dir if current_dir is not '' else '.'
    
    img_dir_path = current_dir + '/data/pokemon/img'
    txt_dir_path = current_dir + '/data/pokemon/txt'
    model_dir_path = current_dir + '/models'

    img_width = 32
    img_height = 32
    
    from keras_text_to_image.library.dcgan import DCGan
    from keras_text_to_image.library.utility.image_utils import img_from_normalized_img
    from keras_text_to_image.library.utility.img_cap_loader import load_normalized_img_and_its_text

    image_label_pairs = load_normalized_img_and_its_text(img_dir_path, txt_dir_path, img_width=img_width, img_height=img_height)

    shuffle(image_label_pairs)

    gan = DCGan()
    gan.load_model(model_dir_path)

    for i in range(3):
        image_label_pair = image_label_pairs[i]
        normalized_image = image_label_pair[0]
        text = image_label_pair[1]

        image = img_from_normalized_img(normalized_image)
        image.save(current_dir + '/data/outputs/' + DCGan.model_name + '-generated-' + str(i) + '-0.png')
        for j in range(3):
            generated_image = gan.generate_image_from_text(text)
            generated_image.save(current_dir + '/data/outputs/' + DCGan.model_name + '-generated-' + str(i) + '-' + str(j) + '.png') Conclusion Here I have presented some of the popular techniques for generating images from text. You can explore more on some more techniques at . Happy Coding!! https://github.com/topics/text-to-image

Summarizing Most Popular Text-to-Image Synthesis Methods With Python

I was writing a chat bot where a user interacts with a machine learning powered bot, then I wanted to write a general example application for anybody to use it. In this application, there will not be any intelligence. The bot will simply recite what it heard so that anyone can implement his/her own logic. I used Angular & reactive approach and the main logic is mainly the service.  Demo can be found here . A small catch for now this application will only work on Chrome browsers as it's the only browser supports Web Speech API currently. Since I will be using reactive approach, I created an interface with an optional payload parameter named . All our actions will implement this interface. Components will be subscribing to these actions and react to them. Action Action {
	payload?: ;
} SpeakingStarted Action {} SpeakingEnded Action {} ListeningStarted Action {} ListeningEnded Action {} RecognizedTextAction Action { ( ) {}
} SpeakAction Action { ( ) {}
} export interface any export class implements export class implements export class implements export class implements export class implements constructor payload: public string export class implements constructor payload: public string In the constructor, we will inject because web speech API lives outside of Angular, we need NgZone to bring it into the Angular realm. NgZone ( ) { .window = ( unknown) IWindow; .createSpeaker(); .createListener(); .subscriptions();
	} constructor zone: NgZone private this window as as this this this One challenge is that synthesizer & recognizer are separate objects. When synthesizer is speaking, the recognizer picks up and it goes into a loop. So, we need to pause one or the other. As you can see in the snippet above, we are creating speaker and listener separately and then we will set up subscriptions to prevent issues in between them. For example: We are reacting to action (as shown below), when it's received, we stop listening and when or we start listening again. SpeakingStarted SpeakingEnded ListeningEnded subscriptions() { .getType(SpeakingStarted)
    .pipe( tap( .stopListening()),
      takeUntil( .destroy$)
    )
    .subscribe();

  merge( .getType(SpeakingEnded), .getType(ListeningEnded))
    .pipe(
      filter( ! .isSpeaking), tap( .startListening()),
      takeUntil( .destroy$)
    )
    .subscribe(); .getType(SpeakAction)
    .pipe(
      tap( ._speak(text)),
      takeUntil( .destroy$)
    )
    .subscribe();
} this // tap(()=> console.log('will stop recognition')), => () this this this this => () this // tap(()=> console.log('will start recognition')), => () this this this ( ) => text this this We need a couple observables that application can subscribe to. Speaker & listener will use the observable for dispatch their actions. action$ is a utility method so that consumer can just pass the type of action it wants and gets only those events. getType such as getType(SpeakerStarted) . _voices$ = BehaviorSubject( );
voices$ = ._voices$.asObservable(); _activeVoice$ = BehaviorSubject( );
activeVoice$ = ._activeVoice$.asObservable(); _action$ = Subject<Action>();
action$ = ._action$.asObservable();

getType(action: Action | ) { .action$.pipe(
    filter( i action),
    map( i.payload),
    takeUntil( .destroy$)
  );
} private new null this private new null this private new this any return this ( ) => i instanceof ( ) => i this Based on the API, we are creating an instance and setting up the parameters and attaching functions that we are interested in. and are the functions we need in our case. SpeechSynthesisUtterance onstart onend We are assigning functions so that whenever invoked, it will trigger an action. We are wrapping them in so that actions actually work in Angular. Lastly, we are loading voices. zone.run createSpeaker() { .speaker = SpeechSynthesisUtterance(); .speaker.lang = .language; .speaker.onstart = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingStarted());
    });
  }; .speaker.onend = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingEnded());
    });
  }; .loadVoices();
} private this new this this this => () this => () this true this new this => () this => () this false this new this To load voices, we need to add function to object on the window (This is not the speaker object). Similarly, we are emitting an event after receiving voices and remove this function after the first run. The reason is function may be invoked more than once during the lifetime of our service. (We could also check if the voices are changed but we don't need such a feature for now). onvoiceschanged speechSynthesis onvoiceschanged loadVoices() { .window.speechSynthesis.onvoiceschanged = { .zone.run( { voices = .window.speechSynthesis.getVoices(); .voices = voices; ._voices$.next(voices); voice_us = voices.find( { i.name.indexOf( .defaultVoiceName) > ;
      }); .onVoiceSelected(voice_us, );
    }); .window.speechSynthesis.onvoiceschanged = ;
  };
} private this => () this => () const this this this const ( ) => i // console.log(i.name); return this -1 this false // we are removing the function after its called, // as we will not need this to be called any more. this null Similar to speaker, we are instantiating our listener, setting up parameters and actions. Documentation can be found for the parameters. here When listener resulted in some value, it will invoke function with possible results. We call method to get the text out of it and dispatch RecognizedTextAction with the actual recognized text. Any component subscribed to this action can get the actual value without dealing with details. onresult extractText When listener ended, we are restarting the listener so it can start over. Full Service: { Observable, merge, Subject, BehaviorSubject } ; { Injectable, NgZone } ; { map, filter, tap, takeUntil } ; IWindow Window {
	webkitSpeechRecognition: ;
	SpeechRecognition: ;
	SpeechSynthesisUtterance: ;
} RecognizedText {
	term: ;
	confidence: ;
	isFinal: ;
} Action {
	payload?: ;
} SpeakingStarted Action {} SpeakingEnded Action {} ListeningStarted Action {} ListeningEnded Action {} RecognizedTextAction Action { ( ) {}
} SpeakAction Action { ( ) {}
} ({
	providedIn: ,
}) SenseService { defaultVoiceName = ; language = ;

	destroy$ = Subject(); : IWindow;
	listener: ;
	speaker: ;

	isAllowed = ;

	voices: [] = ; _voices$ = BehaviorSubject( );
	voices$ = ._voices$.asObservable(); _activeVoice$ = BehaviorSubject( );
	activeVoice$ = ._activeVoice$.asObservable(); _action$ = Subject<Action>();
	action$ = ._action$.asObservable(); isSpeaking(val: ) { .speaker._isSpeaking = val;
	} isSpeaking(): { !! .speaker._isSpeaking;
	} isListening(val: ) { .listener._isListening = val;
	} isListening(): { !! .listener._isListening;
	} ( ) { .window = ( unknown) IWindow; .createSpeaker(); .createListener(); .subscriptions();
	}

	subscriptions() { .getType(SpeakingStarted)
			.pipe( tap( .stopListening()),
				takeUntil( .destroy$)
			)
			.subscribe();

		merge( .getType(SpeakingEnded), .getType(ListeningEnded))
			.pipe(
				filter( ! .isSpeaking), tap( .startListening()),
				takeUntil( .destroy$)
			)
			.subscribe(); .getType(SpeakAction)
			.pipe(
				tap( ._speak(text)),
				takeUntil( .destroy$)
			)
			.subscribe();
	}

	getType(action: Action | ): Observable< > { .action$.pipe(
			filter( i action),
			map( i.payload),
			takeUntil( .destroy$)
		);
	} createSpeaker() { key = ; ( .window[key]) { .log( ); .speaker = .window[key]; ;
		} .speaker = SpeechSynthesisUtterance(); .window[key] = .speaker; .speaker.lang = .language; .speaker.onstart = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingStarted());
			});
		}; .speaker.onend = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingEnded());
			});
		}; .loadVoices();
	} loadVoices() { .window.speechSynthesis.onvoiceschanged = { .zone.run( { voices = .window.speechSynthesis.getVoices(); .voices = voices; ._voices$.next(voices); voice_us = voices.find( { i.name.indexOf( .defaultVoiceName) > ;
				}); .onVoiceSelected(voice_us, );
			}); .window.speechSynthesis.onvoiceschanged = ;
		};
	} createListener() { key = ; ( .window[key]) { .log( ); .listener = .window[key]; .startListening(); ;
		} webkitSpeechRecognition = .window.webkitSpeechRecognition; .listener = webkitSpeechRecognition(); .window[key] = .listener; .listener.continuous = ; .listener.interimResults = ; .listener.lang = .language; .listener.maxAlternatives = ; .listener.maxResults = ; .listener.onstart = { .zone.run( { .isListening = ; ._action$.next( ListeningStarted());
			});
		}; .listener.onresult = { (speech.results) { term: RecognizedText;
				term = .extractText(speech); (term.isFinal) { .zone.run( { ._action$.next( RecognizedTextAction(term.term));
					});
				}
			}
		}; .listener.onerror = { (error.error === ) {
			} (error.error === ) { .isAllowed = ;
			} { .error(error.error);
			}
		}; .listener.onend = { .zone.run( { .isListening = ; ._action$.next( ListeningEnded());
			});
		}; .startListening();
	} stopListening() { .listener.stop();
	} startListening() { (! .startListening) { ;
		} { .log( ); (! .isAllowed) { ;
			} .listener.start();
		} {}
	}

	onVoiceSelected(voice: , speak = ) { .speaker.voice = voice; ._activeVoice$.next(voice); (speak) ._speak( );
	}

	extractText(speech: ): RecognizedText { term = ; result = speech.results[speech.resultIndex]; transcript = result[ ].transcript; confidence = result[ ].confidence; (result.isFinal) { (result[ ].confidence < ) { } {
				term = transcript.trim(); }
		} { (result[ ].confidence > ) {
				term = transcript.trim();
			}
		} <RecognizedText>{
			term,
			confidence,
			isFinal: result.isFinal,
		};
	}

	speak(text: ) { ._action$.next( SpeakAction(text));
	} _speak(text: ): { .log( ); .speaker.text = text; .window.speechSynthesis.speak( .speaker);
	}
} import from 'rxjs' import from '@angular/core' import from 'rxjs/operators' interface extends any any any export interface string number boolean export interface any export class implements export class implements export class implements export class implements export class implements constructor payload: public string export class implements constructor payload: public string @Injectable 'root' export class private 'Google US English' private 'en-US' new window any any true any null private new null this private new null this private new this set boolean this get boolean return this set boolean this get boolean return this constructor zone: NgZone private this window as as this this this this // tap(()=> console.log('will stop recognition')), => () this this this this => () this // tap(()=> console.log('will start recognition')), => () this this this ( ) => text this this any any return this ( ) => i instanceof ( ) => i this private const '_ms_Speaker' if this console 'speaker found' this this return this new this this // this.speaker.voiceURI = 'native'; // this.speaker.volume = 1; // 0 to 1 // this.speaker.rate = 1; // 0.1 to 10 // this.speaker.pitch = 0; //0 to 2 // this.speaker.text = 'Hello World'; this this this => () this => () this true this new this => () this => () this false this new this private this => () this => () const this this this const ( ) => i // console.log(i.name); return this -1 this false // we are removing the function after its called, // as we will not need this to be called any more. this null private const '_ms_Listener' if this console 'recognition found' this this this return const this this new this this this true this true this this this 1 this 25 this => () this => () this true this new this ( ) => speech if let this // console.log(term) if this => () this new this ( ) => error if 'no-speech' else if 'not-allowed' this false else console this => () this => () // console.log('recognition onend'); this false this new this private this private if this return try console 'recognition started' if this return this catch any true this this if this 'Hello' any let '' let let 0 let 0 if if 0 0.3 // console.log('Not recognized'); else // console.log(term); else if 0 0.6 // return term; return string this new private string void console 'speaking...' this this this Usage on any component/service: .senseService.getType(RecognizedTextAction).subscribe( .log(text)); .senseService.speaker( ); .senseService
.getType(RecognizedTextAction)
.pipe(
  debounceTime( ),
  tap( { .senseService.speak( );
  }, takeUntil( .destroy$))
)
.subscribe(); this => text console this 'test speak' // or this 200 ( ) => msg // process ... this `response....` this Demo can be found here .

Speech Recognition And Speech Synthesis on Angular

As put forth countless times before I, there are indeed an untold number of abysmal projects and cash-grabs currently in the cryptocurrency sector (see: ). These serve the purpose of lining the investors’ and team’s pockets. However, Raiden does not fit this mold, rather the Raiden Network shapes one of its own: it has the potential to become one of the most ambitious projects in the next decade as scaling becomes a tenuous issue. Are ICOs a Scam blockchain Currently the amount of transactions on the Ethereum network has been increasing . This is a strong harbinger for the crypto community as people and institutions view the Ethereum network as the best way to conduct their transactions. But as was demonstrated with the advent of CryptoKitties, a more pressing issue is that the network does not have sufficient throughput to enable near-immediate and cheap transactions at scale: the gas for a kitty costs around $2. So how does Raiden help alleviate this issue? exponentially If they succeed, Raiden could provide up to a million transactions per second on the Ethereum network, all the while boasting incredibly modest fees. In a nutshell, Raiden aims to discover the fastest and shortest way to conduct transactions without relying on the blockchain, called off-chain for this reason, except for creating the chain and closing the payment channel. If William wants to send a transfer to Sally, he simply opens a payment channel with a smart contract, secures it with a deposit of his tokens held in escrow — keeping the transfers trustworthy — and then Sally can close the smart contract when she is satisfied. The tokens are then removed from escrow and sent to Sally. In an applied sense, Raiden could be vital to the rise of micropayments on the Ethereum network as on-chain transaction costs are almost invariant with the amount sent. Therefore, microtransactions would be better accomplished over Raiden’s instead of paying large fees consistently. Micro Raiden — uRaiden — was crafted for this purpose. Internet technologies like Web APIs and data processing are all priced as a function of use unlike the current Ethereum network. Yet, Micro Raiden allows for an elastic pricing model so different types of customers can pay for exactly what they need. An exciting feature is the immediate confirmation of payment one receives from Raiden’s off-chain quality: the wait for confirmations and furiously checking to see if the transaction has succeeded is a great friction for widespread adoption. network Our world is tending toward a future where micropayments will be ubiquitous: developments within the internet of things (IoT) sector are making headway and our devices will harness micropayments when they interact with one another. One can easily picture a self-driving car topping up its battery by purchasing kilowatts of electricity on Ethereum’s blockchain; the Raiden network could make this a reality sooner than we think. Consensys’ Grid Plus is aiming to utilize Raiden to do just that by connecting consumers who want to buy and sell solar power with near-zero fees. If 2018 is remotely similar to 2017, the desire and usage of cryptocurrency is only the beginning. Indeed, at the current rate of adoption, Ethereum desperately requires a scaling solution. Raiden vis-à-vis their past development of uRaiden and their roadmap, stands at the precipice of kick-starting Ethereum’s prowess into the next decade. If you’re still mildly interested in what I have to say, consider following me on Twitter

The Synthesis of Raiden and Ethereum

That’s David Logan writing in . Tribal Leadership At we try to bring together the big ideas of the , the mental models, and combine them to help us make better decisions, avoid stupidity, and live a more meaningful . Farnam Street world life Want to explore more? Check out Einstein on The Essential Feature of Productive Thought Seneca on Gathering Ideas And Combinatorial Creativity Commonplace Books as a Source for Networked Knowledge and Combinatorial Creativity Steve Jobs on Creativity If you’re looking to successfully diverge and work smarter not harder, sign up for my weekly Brain Food digest . You can follow Shane on Twitter and Facebook .

Big Ideas Emerge By Synthesis

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 6 Conclusion The results we present here are in line with previous research (for example [Lo and Hew, 2017], [Buckley and Doyle, 2016], [Dahlstrøm, 2012]) and show a stark increase in student engagement with pre-lecture activities on a course taught in flipped mode. As the pre-lecture activities of interest were quiz-based - with points awarded for attempting the quizzes on time rather than quiz score - we were able to investigate whether the increased engagement was meaningful or the result of students ‘gaming’ the leaderboard. In our study, there was no evidence of the latter. It is thought that the two statistics courses considered here are the only time the students came across the Level Up! plugin or other form of gamification. It is therefore not known whether the increased engagement is due to the novelty of a leaderboard and whether gamification would be as successful if implemented widely across multiple courses. The significant increase in student engagement after the adoption of a gamification strategy is certainly worthy of further research. Though the results from the gamification intervention were very positive, student engagement is still far from perfect. Diversity in teaching and learning approaches is surely the key to engaging students across their courses, with gamification a valuable strategy in a teacher’s toolbox. 7 References [Alomari et al., 2019] Alomari, I., Al-Samarraie, H., and Yousef, R. (2019). The role of gamification techniques in promoting student learning: A review and synthesis: A review and synthesis. Journal of Information Technology Education:Research, 18:395–417. [Blair, 2017] Blair, A. (2017). Understanding first-year students’ transition to university: A pilot study with implications for student engagement, assessment, and feedback. Politics, 37.2:215–228. [Bredow et al., 2021] Bredow, C. A., Roehling, P. V., Knorp, A. J., and Sweet, A. M. (2021). To flip or not to flip? a meta-analysis of the efficacy of flipped learning in higher education. Review of educational research, 91(6):878–918. [Buckley and Doyle, 2016] Buckley, P. and Doyle, E. (2016). Gamification and student motivation. Interactive Learning Environments, 24(6):1162–1175. [Cronhjort et al., 2017] Cronhjort, M., Filipsson, L., and Weurlander, M. (2017). Improved engagement and learning in flipped-classroom calculus. Teaching Mathematics and its Applications: An International Journal of the IMA, 37(3):113–121. [Dahlstrøm, 2012] Dahlstrøm, C. (2012). Impacts of gamification on intrinsic motivation. Education and Humanities Research, pages 1–11. [Deterding et al., 2011] Deterding, S., Khaled, R., Nacke, L. E., Dixon, D., et al. (2011). Gamification: Toward a definition. In CHI 2011 gamification workshop proceedings, volume 12, pages 1–79. Vancouver, Canadá. [Ding et al., 2018] Ding, L., Er, E., and Orey, M. (2018). An exploratory study of student engagement in gamified online discussions. Computers & Education, 120:213–226. [Domínguez et al., 2013] Domínguez, A., de Navarrete, J. S., de Marcos, L., Fernández-Sanz, L., Pagés, C., and Martínez-Herráiz, J.-J. (2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63:380–392. [Dong et al., 2012] Dong, T., Dontcheva, M., Joseph, D., Karahalios, K., Newman, M., and Ackerman, M. (2012). Discovery-based games for learning software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, page 2083–2086, New York, NY, USA. Association for Computing Machinery. [Ekici, 2021] Ekici, M. (2021). A systematic review of the use of gamification in flipped learning. Education and Information Technologies, 26(3):3327–3346. [Hakulinen et al., 2013] Hakulinen, L., Auvinen, T., and Korhonen, A. (2013). Empirical study on the effect of achievement badges in trakla2 online learning environment. In 2013 Learning and Teaching in Computing and Engineering, pages 47–54. [Hartikainen et al., 2019] Hartikainen, S., Rintala, H., Pylväs, L., and Nokelainen, P. (2019). The concept of active learning and the measurement of learning outcomes: A review of research in engineering higher education. Education Sciences, 9(4):276. [Heilbrunn et al., 2017] Heilbrunn, B., Herzig, P., and Schill, A. (2017). Gamification Analytics—Methods and Tools for Monitoring and Adapting Gamification Designs, pages 31–47. Springer International Publishing, Cham. [Huang and Hew, 2018] Huang, B. and Hew, K. F. (2018). Implementing a theory-driven gamification model in higher education flipped courses: Effects on out-of-class activity completion and quality of artifacts. Computers & Education, 125:254–272. [Ireland and Mouthaan, 2020] Ireland, J. and Mouthaan, M. (2020). Perspectives on curriculum design: comparing the spiral and the network models. Research Matters. [Kim and Castelli, 2021] Kim, J. and Castelli, D. M. (2021). Effects of gamification on behavioral change in education: A meta-analysis. International Journal of Environmental Research and Public Health, 18(7). [Leitner et al., 2017] Leitner, P., Khalil, M., and Ebner, M. (2017). Learning Analytics in Higher Education—A Literature Review, volume 94, pages 1–23. Springer International Publishing. [Lo and Hew, 2017] Lo, C. K. and Hew, K. F. (2017). A critical review of flipped classroom challenges in k-12 education: possible solutions and recommendations for future research. Research and Practice in Technology Enhanced Learning, 12(1):4. [Lo et al., 2017] Lo, C. K., Hew, K. F., and Chen, G. (2017). Toward a set of design principles for mathematics flipped classrooms: A synthesis of research in mathematics education. Educational Research Review, 22:50–73. [Marcey, 2014] Marcey, D. J. (2014). The lecture hall as an arena of inquiry: Using cinematic lectures and inverted classes (clic) to flip an introductory biology lecture course. Archive of the Academic Commons. [Owens et al., 2020] Owens, D. C., Sadler, T. D., Barlow, A. T., and Smith-Walters, C. (2020). Student motivation from and resistance to active learning rooted in essential science practices. Research in Science Education, 50(1):253–277. [Stone, 2012] Stone, B. B. (2012). Flip your classroom to increase active learning and student engagement. In Proceedings from 28th Annual Conference on Distance Teaching and Learning, Madison, Wisconsin, USA. [Touchton, 2015] Touchton, M. (2015). Flipping the classroom and student performance in advanced statistics: Evidence from a quasi-experiment. Journal of Political Science Education, 11(1):28–44. [Wester et al., 2021] Wester, E. R., Walsh, L. L., Arango-Caro, S., and Callis-Duehl, K. L. (2021). Student engagement declines in stem undergraduates during covid-19–driven remote learning. Journal of Microbiology & Biology Education, 22(1):10.1128/jmbe.v22i1.2385. [Wilson, 2013] Wilson, S. G. (2013). The flipped class: A method to address the challenges of an undergraduate statistics course. Teaching of Psychology, 40(3):193–199. [Wood, 2023] Wood, A. (2023). Effective Teaching in Large STEM Classes. IOP Series in Physics Education. IOP Publishing. Publisher Copyright: © IOP Publishing Ltd 2023. All rights reserved. [Zaric et al., 2017] Zaric, N., Scepanovic, S., Markovic, T., Ljucovic, J., and Davcev, D. (2017). The model for gamification of e-learning in higher education based on learning styles. In International Conference on ICT Innovations, pages 265–273. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 6 Conclusion The results we present here are in line with previous research (for example [Lo and Hew, 2017], [Buckley and Doyle, 2016], [Dahlstrøm, 2012]) and show a stark increase in student engagement with pre-lecture activities on a course taught in flipped mode. As the pre-lecture activities of interest were quiz-based - with points awarded for attempting the quizzes on time rather than quiz score - we were able to investigate whether the increased engagement was meaningful or the result of students ‘gaming’ the leaderboard. In our study, there was no evidence of the latter. It is thought that the two statistics courses considered here are the only time the students came across the Level Up! plugin or other form of gamification. It is therefore not known whether the increased engagement is due to the novelty of a leaderboard and whether gamification would be as successful if implemented widely across multiple courses. The significant increase in student engagement after the adoption of a gamification strategy is certainly worthy of further research. Though the results from the gamification intervention were very positive, student engagement is still far from perfect. Diversity in teaching and learning approaches is surely the key to engaging students across their courses, with gamification a valuable strategy in a teacher’s toolbox. 7 References [Alomari et al., 2019] Alomari, I., Al-Samarraie, H., and Yousef, R. (2019). The role of gamification techniques in promoting student learning: A review and synthesis: A review and synthesis. Journal of Information Technology Education:Research, 18:395–417. [Blair, 2017] Blair, A. (2017). Understanding first-year students’ transition to university: A pilot study with implications for student engagement, assessment, and feedback. Politics, 37.2:215–228. [Bredow et al., 2021] Bredow, C. A., Roehling, P. V., Knorp, A. J., and Sweet, A. M. (2021). To flip or not to flip? a meta-analysis of the efficacy of flipped learning in higher education. Review of educational research, 91(6):878–918. [Buckley and Doyle, 2016] Buckley, P. and Doyle, E. (2016). Gamification and student motivation. Interactive Learning Environments, 24(6):1162–1175. [Cronhjort et al., 2017] Cronhjort, M., Filipsson, L., and Weurlander, M. (2017). Improved engagement and learning in flipped-classroom calculus. Teaching Mathematics and its Applications: An International Journal of the IMA, 37(3):113–121. [Dahlstrøm, 2012] Dahlstrøm, C. (2012). Impacts of gamification on intrinsic motivation. Education and Humanities Research, pages 1–11. [Deterding et al., 2011] Deterding, S., Khaled, R., Nacke, L. E., Dixon, D., et al. (2011). Gamification: Toward a definition. In CHI 2011 gamification workshop proceedings, volume 12, pages 1–79. Vancouver, Canadá. [Ding et al., 2018] Ding, L., Er, E., and Orey, M. (2018). An exploratory study of student engagement in gamified online discussions. Computers & Education, 120:213–226. [Domínguez et al., 2013] Domínguez, A., de Navarrete, J. S., de Marcos, L., Fernández-Sanz, L., Pagés, C., and Martínez-Herráiz, J.-J. (2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63:380–392. [Dong et al., 2012] Dong, T., Dontcheva, M., Joseph, D., Karahalios, K., Newman, M., and Ackerman, M. (2012). Discovery-based games for learning software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, page 2083–2086, New York, NY, USA. Association for Computing Machinery. [Ekici, 2021] Ekici, M. (2021). A systematic review of the use of gamification in flipped learning. Education and Information Technologies, 26(3):3327–3346. [Hakulinen et al., 2013] Hakulinen, L., Auvinen, T., and Korhonen, A. (2013). Empirical study on the effect of achievement badges in trakla2 online learning environment. In 2013 Learning and Teaching in Computing and Engineering, pages 47–54. [Hartikainen et al., 2019] Hartikainen, S., Rintala, H., Pylväs, L., and Nokelainen, P. (2019). The concept of active learning and the measurement of learning outcomes: A review of research in engineering higher education. Education Sciences, 9(4):276. [Heilbrunn et al., 2017] Heilbrunn, B., Herzig, P., and Schill, A. (2017). Gamification Analytics—Methods and Tools for Monitoring and Adapting Gamification Designs, pages 31–47. Springer International Publishing, Cham. [Huang and Hew, 2018] Huang, B. and Hew, K. F. (2018). Implementing a theory-driven gamification model in higher education flipped courses: Effects on out-of-class activity completion and quality of artifacts. Computers & Education, 125:254–272. [Ireland and Mouthaan, 2020] Ireland, J. and Mouthaan, M. (2020). Perspectives on curriculum design: comparing the spiral and the network models. Research Matters. [Kim and Castelli, 2021] Kim, J. and Castelli, D. M. (2021). Effects of gamification on behavioral change in education: A meta-analysis. International Journal of Environmental Research and Public Health, 18(7). [Leitner et al., 2017] Leitner, P., Khalil, M., and Ebner, M. (2017). Learning Analytics in Higher Education—A Literature Review, volume 94, pages 1–23. Springer International Publishing. [Lo and Hew, 2017] Lo, C. K. and Hew, K. F. (2017). A critical review of flipped classroom challenges in k-12 education: possible solutions and recommendations for future research. Research and Practice in Technology Enhanced Learning, 12(1):4. [Lo et al., 2017] Lo, C. K., Hew, K. F., and Chen, G. (2017). Toward a set of design principles for mathematics flipped classrooms: A synthesis of research in mathematics education. Educational Research Review, 22:50–73. [Marcey, 2014] Marcey, D. J. (2014). The lecture hall as an arena of inquiry: Using cinematic lectures and inverted classes (clic) to flip an introductory biology lecture course. Archive of the Academic Commons. [Owens et al., 2020] Owens, D. C., Sadler, T. D., Barlow, A. T., and Smith-Walters, C. (2020). Student motivation from and resistance to active learning rooted in essential science practices. Research in Science Education, 50(1):253–277. [Stone, 2012] Stone, B. B. (2012). Flip your classroom to increase active learning and student engagement. In Proceedings from 28th Annual Conference on Distance Teaching and Learning, Madison, Wisconsin, USA. [Touchton, 2015] Touchton, M. (2015). Flipping the classroom and student performance in advanced statistics: Evidence from a quasi-experiment. Journal of Political Science Education, 11(1):28–44. [Wester et al., 2021] Wester, E. R., Walsh, L. L., Arango-Caro, S., and Callis-Duehl, K. L. (2021). Student engagement declines in stem undergraduates during covid-19–driven remote learning. Journal of Microbiology & Biology Education, 22(1):10.1128/jmbe.v22i1.2385. [Wilson, 2013] Wilson, S. G. (2013). The flipped class: A method to address the challenges of an undergraduate statistics course. Teaching of Psychology, 40(3):193–199. [Wood, 2023] Wood, A. (2023). Effective Teaching in Large STEM Classes. IOP Series in Physics Education. IOP Publishing. Publisher Copyright: © IOP Publishing Ltd 2023. All rights reserved. [Zaric et al., 2017] Zaric, N., Scepanovic, S., Markovic, T., Ljucovic, J., and Davcev, D. (2017). The model for gamification of e-learning in higher education based on learning styles. In International Conference on ICT Innovations, pages 265–273. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Exploring the Impact of Gamification in Flipped Classrooms: Conclusion and References

Generative Artificial Intelligence for Software Engineering: Background

Recommendations by Concise User Profiles from Review Text: Ethics Statement and References

Biden-⁠Harris Administration Commends Federal Agencies for Completing Key AI Actions in 180 Days

বিডেন-হ্যারিস প্রশাসন 180 দিনের মধ্যে মূল এআই অ্যাকশনগুলি সম্পূর্ণ করার জন্য ফেডারেল এজেন্সিগুলির প্রশংসা করেছে

Biden-Harris-Regierung lobt Bundesbehörden für die Umsetzung wichtiger KI-Maßnahmen in 180 Tagen

La administración Biden-⁠Harris elogia a las agencias federales por completar acciones clave de IA en 180 días

L’administration Biden-⁠Harris félicite les agences fédérales pour avoir mené à bien des actions clés en matière d’IA en 180 jours

बिडेन-हैरिस प्रशासन ने 180 दिनों में प्रमुख एआई कार्रवाइयों को पूरा करने के लिए संघीय एजेंसियों की सराहना की

バイデン・ハリス政権、連邦政府機関が180日以内に主要なAI対策を完了したことを称賛

Biden-⁠해리스 행정부, 180일 만에 핵심 AI 작업을 완료한 연방 기관 칭찬

Administração Biden-⁠Harris elogia agências federais por concluírem ações importantes de IA em 180 dias

Администрация Байдена-Харриса благодарит федеральные агентства за завершение ключевых действий в области искусственного интеллекта за 180 дней

Biden-⁠Harris Yönetimi, Temel Yapay Zeka Eylemlerini 180 Günde Tamamlayan Federal Kurumları Övdü

Chính quyền Biden-⁠Harris khen ngợi các cơ quan liên bang đã hoàn thành các hành động quan trọng về AI trong 180 ngày

拜登-哈里斯政府赞扬联邦机构在 180 天内完成关键人工智能行动

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk;
(2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com;
(3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk;
(4) Jener Rasmussen, jener@jener.dk;
(5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com;
(6) Asger G. Weirsøe, asger@weircon.dk;
(7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk;
(8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. Table of Links Introduction Definitions Model Checking Tool Overview Evaluation Conclusion and References 6 Conclusion In this paper, we present CGAAL, our model checker of alternating-time temporal logic properties in concurrent games. CGAAL checks such properties by encoding the problem as an extended dependency graph and then computes the satisfaction relation using the distributed local on-the-fly CERTAINZERO algorithm by Dalsgaard et al. [4]. We provide multiple novel search strategies for the algorithm and allow concurrent games to be expressed in our language LCGS. Our experiments show that the local on the-fly algorithm outperforms the global algorithm in the majority of cases. We also find that CGAAL outperforms the state-of-the-art tool PRISM-games by being up to two orders of magnitude faster, especially in models where synchronisations affect the internal state of modules and whenever we are not required to compute the entire fixed point. However, this comparison is unfair, since our test only uses a fraction of PRISM-games feature set. CGAAL is still in early development and much work is needed before it competes with PRISM in terms of capabilities. However, dependency graphs have been used for encoding various model checking problems, and we intend to incorporate these techniques into CGAAL. References [1] Rajeev Alur, Thomas A. Henzinger & Orna Kupferman (2002): Alternating-Time Temporal Logic. J. ACM 49(5), p. 672–713, doi:10.1145/585265.585270. [2] Edmund M. Clarke & E. Allen Emerson (1982): Design and synthesis of synchronization skeletons using branching time temporal logic. In Dexter Kozen, editor: Logics of Programs, Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 52–71, doi:10.1007/BFb0025774. [3] Edmund M Clarke, Thomas A Henzinger, Helmut Veith, Roderick Bloem et al. (2018): Handbook of model checking. 10, Springer, doi:10.1007/978-3-319-10575-8. [4] Dalsgaard, Andreas E. and Enevoldsen, Søren and Fogh, Peter and Jensen, Lasse S. and Jepsen, Tobias S. and Kaufmann, Isabella and Larsen, Kim G. and Nielsen, Søren M. and Olesen, Mads Chr. and Pastva, Samuel and Srba, Jiˇrí (2017): Extended Dependency Graphs and Efficient Distributed Fixed-Point Computation. In Wil van der Aalst & Eike Best, editors: Application and Theory of Petri Nets and Concurrency, Springer International Publishing, Cham, pp. 139–158, doi:10.1007/978-3-319-57861-3_10. [5] Søren Enevoldsen, Mathias Claus Jensen, Kim Guldstrand Larsen, Anders Mariegaard & Jiˇrí Srba (2020): Verification of Multiplayer Stochastic Games via Abstract Dependency Graphs. LOPSTR2020, doi:10.1007/978-3-030-68446-4_13. [6] Søren Enevoldsen, Kim Guldstrand Larsen & Jirí Srba (2022): Extended abstract dependency graphs. Int. J. Softw. Tools Technol. Transf. 24(1), pp. 49–65, doi:10.1007/s10009-021-00638-8. [7] Glen Jeh & Jennifer Widom (2002): SimRank: a measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, ACM, pp. 538–543, doi:10.1145/775047.775126. [8] Jonas F. Jensen, Thomas Nielsen, Lars K. Oestergaard & Jiˇrí Srba (2016): TAPAAL and Reachability Analysis of P/T Nets, pp. 307–318. Springer Berlin Heidelberg, Berlin, Heidelberg, doi:10.1007/978-3-662-53401-4_16. [9] Jonas Finnemann Jensen, Kim Guldstrand Larsen, Jiri Srba & Lars Kaerlund Østergaard (2016): Efficient model-checking of weighted CTL with upper-bound constraints. International Journal on Software Tools for Technology Transfer 18(4), pp. 409–426, doi:10.1007/s10009-014-0359-5. [10] Mathias Claus Jensen, Anders Mariegaard & Kim Guldstrand Larsen (2019): Symbolic model checking of weighted PCTL using dependency graphs. NASA Formal Methods Symposium, pp. 298–315, doi:10.1007/978-3-030-20652-9_20. [11] M. Kwiatkowska, G. Norman & D. Parker (2011): PRISM 4.0: Verification of Probabilistic Real-time Systems. In G. Gopalakrishnan & S. Qadeer, editors: Proc. 23rd International Conference on Computer Aided Verification (CAV’11), LNCS 6806, Springer, pp. 585–591, doi:10.1007/978-3-642-22110-1_47. [12] M. Kwiatkowska, G. Norman, D. Parker & G. Santos (2020): PRISM-games 3.0: Stochastic Game Verification with Concurrency, Equilibria and Time. In: Proc. 32nd International Conference on Computer Aided Verification (CAV’20), LNCS 12225, Springer, pp. 475–487, doi:10.1007/978-3-030-53291-8_25. [13] Sangkeun Lee, Minsuk Kahng & Sang goo Lee (2015): Constructing compact and effective graphs for recommender systems via node and edge aggregations. Expert Systems with Applications 42(7), pp. 3396–3409, doi:10.1016/j.eswa.2014.11.062. [14] Xinxin Liu & Scott A. Smolka (1998): Simple Linear-Time Algorithms for Minimal Fixed Points (Extended Abstract). In Kim Guldstrand Larsen, Sven Skyum & Glynn Winskel, editors: Automata, Languages and Programming, 25th International Colloquium, ICALP’98, Aalborg, Denmark, July 13-17, 1998, Proceedings, Lecture Notes in Computer Science 1443, Springer, pp. 53–66, doi:10.1007/BFb0055040. [15] Lawrence Page, Sergey Brin, Rajeev Motwani & Terry Winograd (1998): The PageRank Citation Ranking: Bringing Order to the Web. Technical Report, Stanford Digital Library Technologies Project. Available at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.1768. This paper is available on Arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. Authors: (1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk; (2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com; (3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk; (4) Jener Rasmussen, jener@jener.dk; (5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com; (6) Asger G. Weirsøe, asger@weircon.dk; (7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk; (8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk; (2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com; (3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk; (4) Jener Rasmussen, jener@jener.dk; (5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com; (6) Asger G. Weirsøe, asger@weircon.dk; (7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk; (8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. Table of Links Introduction Introduction Definitions Definitions Model Checking Model Checking Tool Overview Tool Overview Evaluation Evaluation Conclusion and References Conclusion and References 6 Conclusion In this paper, we present CGAAL, our model checker of alternating-time temporal logic properties in concurrent games. CGAAL checks such properties by encoding the problem as an extended dependency graph and then computes the satisfaction relation using the distributed local on-the-fly CERTAINZERO algorithm by Dalsgaard et al. [4]. We provide multiple novel search strategies for the algorithm and allow concurrent games to be expressed in our language LCGS. Our experiments show that the local on the-fly algorithm outperforms the global algorithm in the majority of cases. We also find that CGAAL outperforms the state-of-the-art tool PRISM-games by being up to two orders of magnitude faster, especially in models where synchronisations affect the internal state of modules and whenever we are not required to compute the entire fixed point. However, this comparison is unfair, since our test only uses a fraction of PRISM-games feature set. CGAAL is still in early development and much work is needed before it competes with PRISM in terms of capabilities. However, dependency graphs have been used for encoding various model checking problems, and we intend to incorporate these techniques into CGAAL. References [1] Rajeev Alur, Thomas A. Henzinger & Orna Kupferman (2002): Alternating-Time Temporal Logic. J. ACM 49(5), p. 672–713, doi:10.1145/585265.585270. [2] Edmund M. Clarke & E. Allen Emerson (1982): Design and synthesis of synchronization skeletons using branching time temporal logic. In Dexter Kozen, editor: Logics of Programs, Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 52–71, doi:10.1007/BFb0025774. [3] Edmund M Clarke, Thomas A Henzinger, Helmut Veith, Roderick Bloem et al. (2018): Handbook of model checking. 10, Springer, doi:10.1007/978-3-319-10575-8. [4] Dalsgaard, Andreas E. and Enevoldsen, Søren and Fogh, Peter and Jensen, Lasse S. and Jepsen, Tobias S. and Kaufmann, Isabella and Larsen, Kim G. and Nielsen, Søren M. and Olesen, Mads Chr. and Pastva, Samuel and Srba, Jiˇrí (2017): Extended Dependency Graphs and Efficient Distributed Fixed-Point Computation. In Wil van der Aalst & Eike Best, editors: Application and Theory of Petri Nets and Concurrency, Springer International Publishing, Cham, pp. 139–158, doi:10.1007/978-3-319-57861-3_10. [5] Søren Enevoldsen, Mathias Claus Jensen, Kim Guldstrand Larsen, Anders Mariegaard & Jiˇrí Srba (2020): Verification of Multiplayer Stochastic Games via Abstract Dependency Graphs. LOPSTR2020, doi:10.1007/978-3-030-68446-4_13. [6] Søren Enevoldsen, Kim Guldstrand Larsen & Jirí Srba (2022): Extended abstract dependency graphs. Int. J. Softw. Tools Technol. Transf. 24(1), pp. 49–65, doi:10.1007/s10009-021-00638-8. [7] Glen Jeh & Jennifer Widom (2002): SimRank: a measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, ACM, pp. 538–543, doi:10.1145/775047.775126. [8] Jonas F. Jensen, Thomas Nielsen, Lars K. Oestergaard & Jiˇrí Srba (2016): TAPAAL and Reachability Analysis of P/T Nets, pp. 307–318. Springer Berlin Heidelberg, Berlin, Heidelberg, doi:10.1007/978-3-662-53401-4_16. [9] Jonas Finnemann Jensen, Kim Guldstrand Larsen, Jiri Srba & Lars Kaerlund Østergaard (2016): Efficient model-checking of weighted CTL with upper-bound constraints. International Journal on Software Tools for Technology Transfer 18(4), pp. 409–426, doi:10.1007/s10009-014-0359-5. [10] Mathias Claus Jensen, Anders Mariegaard & Kim Guldstrand Larsen (2019): Symbolic model checking of weighted PCTL using dependency graphs. NASA Formal Methods Symposium, pp. 298–315, doi:10.1007/978-3-030-20652-9_20. [11] M. Kwiatkowska, G. Norman & D. Parker (2011): PRISM 4.0: Verification of Probabilistic Real-time Systems. In G. Gopalakrishnan & S. Qadeer, editors: Proc. 23rd International Conference on Computer Aided Verification (CAV’11), LNCS 6806, Springer, pp. 585–591, doi:10.1007/978-3-642-22110-1_47. [12] M. Kwiatkowska, G. Norman, D. Parker & G. Santos (2020): PRISM-games 3.0: Stochastic Game Verification with Concurrency, Equilibria and Time. In: Proc. 32nd International Conference on Computer Aided Verification (CAV’20), LNCS 12225, Springer, pp. 475–487, doi:10.1007/978-3-030-53291-8_25. [13] Sangkeun Lee, Minsuk Kahng & Sang goo Lee (2015): Constructing compact and effective graphs for recommender systems via node and edge aggregations. Expert Systems with Applications 42(7), pp. 3396–3409, doi:10.1016/j.eswa.2014.11.062. [14] Xinxin Liu & Scott A. Smolka (1998): Simple Linear-Time Algorithms for Minimal Fixed Points (Extended Abstract). In Kim Guldstrand Larsen, Sven Skyum & Glynn Winskel, editors: Automata, Languages and Programming, 25th International Colloquium, ICALP’98, Aalborg, Denmark, July 13-17, 1998, Proceedings, Lecture Notes in Computer Science 1443, Springer, pp. 53–66, doi:10.1007/BFb0055040. [15] Lawrence Page, Sergey Brin, Rajeev Motwani & Terry Winograd (1998): The PageRank Citation Ranking: Bringing Order to the Web. Technical Report, Stanford Digital Library Technologies Project. Available at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.1768. This paper is available on Arxiv under CC 4.0 license. Arxiv

CGAAL: Wrapping It Up and What Comes Next

This paper is under CC 4.0 license. available on arxiv : Authors (1) Ryen W. White, Microsoft Research, Redmond, WA, USA. Table of Links Abstract and Taking Search to task AI Copilots Challenges Opportunities The Undiscovered Country and References 3 CHALLENGES Despite the promise of copilots, there are significant challenges that should be acknowledged and we must find ways to overcome. Those include issues with the copilot output shown in response to searcher requests, the impacts that the copilots can have on searchers, and shifts in the degree of agency that humans have in the search process that result from the introduction of copilots. • : Searchers rely a lot on the answers from copilots,but those answers can be erroneous or non-sensical. So-called“hallucination” is a well-studied problem in foundation models [24]. Copilots can hallucinate for many reasons. One of the main reasons being gaps in the training data. RAG, discussed earlier, is a way to help address this by ensuring that the copilot has access to up-to-date, relevant information at inference time to help ground its responses. Injection of knowledge from other external sources, such as knowledge graphs and Wikipedia, can also help improve the accuracy of copilot responses. An issue related to copilots surfacing misinformation is toxicity (i.e., offensive or harmful content), which can also be present in the copilot output, and must be mitigated before answers are shown to searchers. Hallucinations • : Biases in the training data, e.g., social biases and stereotypes [31], affect the output of foundation models and hence the answers provided by copilots. Synthesis of content from different sources can amplify biases in this data. As with hallucinations, this is a well-studied problem [6]. Copilots are also subject to biases from learning from their own or other AI generated content (via feedback loops); biased historical sequences lead to biased downstream models. Copilots may also amplify existing cognitive biases, such as confirmation bias, by favoring responses that are aligned with searchers’ existing beliefs and values, and by providing responses that keep searchers engaged with the copilot, regardless of the ramifications for the searcher. Biases Learning may be affected/interrupted by the use of AI copilots since they remove the need for searchers to engage as fully with the search system and the information retrieved. Learning is already a core part of the search process [32, 37, 54]. Both exploratory search and search as learning involve considerable time and effort in finding and examining relevant content. While this could be viewed as a cost, this deep exposure to content also helps people learn. As mentioned earlier, copilot users can ask richer questions (allowing them to specify their tasks and goals more fully) but they then receive synthesized answers generated by the copilot, creating fewer, new, or simply different learning opportunities for humans that must be understood. • Human learning: Supporting search requires considering the degree of searcher involvement in the search process, which varies depending on the search task [2]. Copilots enable more strategic, • Human control: higher-order actions (higher up the “task tree” from Figure 1 than typical interactions with search systems). It is clear that searchers want control over the search process. They want to know what information is/not being included and why. This helps them understand and trust system output. As things stand, copilot users delegate full control of answer generation to the AI, but the rest is mixed, i.e., less control of search mechanics (queries, etc.) but more control of task specifications (via natural language and dialog). There is more than just a basic tension between automation and control. In reality, it is not a zero sum game. Designers of copilots need to ensure human control while increasing automation [44]. New frameworks for task completion are moving in this direction. For example, AutoGen [67], uses multiple specialized assistive AI copilots that engage with humans and with each other directly to help complete complex tasks, with humans staying informed and in control throughout. Overall, these are just a few of the challenges that affect the viability of copilots. There are other challenges, such as deeply ingrained search habits that may be a barrier to the adoption of new search functionality, despite the clear benefits to searchers.

Navigating Complex Search Tasks with AI Copilots: Challenges

This paper is available on Arxiv under CC 4.0 license.
Authors:
(1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk;
(2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com;
(3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk;
(4) Jener Rasmussen, jener@jener.dk;
(5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com;
(6) Asger G. Weirsøe, asger@weircon.dk;
(7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk;
(8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. Table of Links Introduction Definitions Model Checking Tool Overview Evaluation Conclusion and References We present CGAAL, our efficient on-the-fly model checker for alternating-time temporal logic (ATL) on concurrent game structures (CGS). We present how our tool encodes ATL as extended dependency graphs with negation edges and employs the distributed on-the-fly algorithm by Dalsgaard et al. Our tool offers multiple novel search strategies for the algorithm, including DHS which is inspired by PageRank and uses the in-degree of configurations as a heuristic, IHS which estimates instability of assignment values, and LPS which estimates the distance to a state satisfying the constituent property using linear programming. CGS are input using our modelling language LCGS, where composition and synchronisation are easily described. We prove the correctness of our encoding, and our experiments show that our tool CGAAL is often one to three orders of magnitude faster than the popular tool PRISM-games on case studies from PRISM’s documentation and among case studies we have developed. In our evaluation, we also compare and evaluate our search strategies, and find that our custom search strategies are often significantly faster than the usual breadth-first and depth-first search strategies. 1 Introduction Software plays a large role in our everyday lives, making decisions, enabling efficient communication, ensuring safety, and many more critical tasks. Furthermore, the complexity of the software and the decisions they have to make are ever-increasing due the to interconnectivity and reactive nature of modern systems. These software systems mutually depend on, communicate with, and guide each other based on their collective and internal states. Even a few interconnected systems that are not necessarily themselves too complex can give rise to an extremely complex system as a whole. Safety-critical software that operates in contexts where errors could lead to human casualties or significant capital losses requires methods to verify their correctness. Unfortunately, such methods are difficult to implement due to the sheer state-space explosion that arises from the inherent parallel composition of systems. In this paper, we consider systems that can be expressed as discrete multiplayer games in which the actors can perform actions concurrently. That is, in each configuration of the game, each player simultaneously chooses an action they wish to perform and the resulting decision vector then deterministically results in the next configuration of the game. Such concurrent games are extremely expressive and can be used to model a variety of complex systems and lends themselves inherently, by their concurrent nature, to describing multi-actor systems. The properties that we wish to verify on these systems are those that can be expressed by Alternating-time Temporal Logic (ATL) [1], a game theoretic extension of Computation Tree Logic (CTL) [2, 3], that like CTL can be used to describe safety and liveness properties. ATL extends CTL by replacing the existential and universal path quantifiers with so-called coalition quantifiers. These coalition quantifiers describe properties of the game in which a coalition of players can either enforce some desired outcome or cannot avoid an outcome. Of course, proof of satisfaction of such property is done by finding a strategy showing that the players can indeed enforce said property or by showing that no matter what strategy the players follow they cannot avoid said property. One existing tool that can model check concurrent (stochastic) games is PRISM-games [12], an extension of the probabilistic model checker PRISM [11]. PRISM excels at stochastic models such as discreteand continuous-time Markov chains, Markov decision processes, and (priced) probabilistic timed automata, for which it can verify various properties described in LTL , CSL, and probabilistic CTL*. There are multiple verification engines in PRISM, both symbolic and explicit state. The PRISM-games extension focuses on stochastic games in which game-theoretic approaches are needed. PRISM-games can among other things synthesise strategies and reason about equilibria-based properties, where players may have distinct, but not necessarily conflicting objectives. These properties are typically described with probabilistic ATL with rewards. In version 3.0 [12], PRISM-games added support for concurrent stochastic games. In [14], Liu and Smolka present a global and a local algorithm to compute fixed-point boolean vertex assignments in dependency graphs with directed hyper-edges representing dependencies between vertex values. The global algorithm has a better worst-case running time of the two, but the on-the-fly local algorithm only explores the graph as needed to compute the fixed-point assignment. Many problems have since been encoded as dependency graphs. Notably, model-checking problems for CTL can be encoded as dependency graphs and the algorithms by Liu and Smolka can be used to compute the satisfaction relation enabling model-checking using dependency graphs. J. F. Jensen et al. [9] show that weighted CTL can be encoded in dependency graphs as well and introduce symbolic edges to handle weights efficiently. In [10], M. C. Jensen et al. introduce symbolic dependency graphs, an encoding of probabilistic weighted CTL, and an implementation of a local and global algorithm. A. Dalsgaard et al. [4] present a distributed extension to the local algorithm to boost the performance and show how to model Petri Nets problems using CTL and dependency graphs. Furthermore, they extend dependency graphs by adding components and negation edges to solve formulae with negations. An abstract dependency graph framework is presented by S. Enevoldsen et al. in [6] where they generalise the various extensions of the dependency graphs with a general algorithm for any Noetherian partial order domain. In the closely related work of [5], S. Enevoldsen et at. uses abstract dependency graphs to model check probabilistic ATL for weighted stochastic games. In multiple works [4, 10] the local algorithm is found to be faster more often than the global algorithm in practice. In this paper, we introduce the tool CGAAL (Concurrent Games AALborg), a model checker implemented in Rust that allows for the verification of ATL properties on concurrent games. CGAAL works by encoding the model checking problem as an extended dependency graph and by implementing the on-the-fly local distributed algorithm by Dalsgaard et al. [4]. The concurrent games are described and inputted to the tool using LCGS, our PRISM-language inspired model language, and for specific ATL formulae, we can output a strategy describing how to ensure satisfaction given a positive result. Furthermore, CGAAL implements a variety of strategies for searching the problem state space: a heuristic inspired by PageRank [15] and two heuristic search strategies that exploit the state vector representation of our model, alongside the usual breadth-first and depth-first approach. We conduct experiments comparing our different search strategies with each other on several case studies and show that having more available compute threads often leads to speed-ups of 1-2 orders of magnitude. We also perform experiments comparing our tool on a selection of case studies to the state-of-the-art PRISM-games. Though while PRISM-games is a tool specialised for probabilistic games we find that our implementation often outperforms PRISM-games and especially whenever we are not required to compute the entire fixed point in which case we are often 1-3 orders of magnitude faster. Again, we emphasise that PRISM games is meant for probabilistic games and as such this is not a completely fair one-to-one comparison. Outline. This paper is structured as follows. Section 2 introduces the formal definitions of concurrent games and alternating-time temporal logic. In Section 3 we present how model checking of ATL properties in concurrent games can be encoded in extended dependency graphs. We also give a short explanation of the CERTAINZERO algorithm and our search strategies. In Section 4 we give a short example of how to use CGAAL and the LCGS language. An evaluation of our tool and a comparison with PRISM can be found in Section 5, and we conclude on our findings in Section 6. This paper is available on Arxiv under CC 4.0 license. This paper is available on Arxiv under CC 4.0 license. Authors: (1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk; (2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com; (3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk; (4) Jener Rasmussen, jener@jener.dk; (5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com; (6) Asger G. Weirsøe, asger@weircon.dk; (7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk; (8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. This paper is available on Arxiv under CC 4.0 license. Authors: Authors: (1) Falke B. Ø. Carlsen, Department of Computer Science, Aalborg University, Denmark & falkeboc@cs.aau.dk; (2) Lars Bo P. Frydenskov, Department of Computer Science, Aalborg University, Denmark & larsbopark@gmail.com; (3) Nicolaj Ø. Jensen, Department of Computer Science, Aalborg University, Denmark & noje@cs.aau.dk; (4) Jener Rasmussen, jener@jener.dk; (5) Mathias M. Sørensen, mathiasmehlsoerensen@gmail.com; (6) Asger G. Weirsøe, asger@weircon.dk; (7) Mathias C. Jensen, Department of Computer Science, Aalborg University, Denmark & mcje@cs.aau.dk; (8) Kim G. Larsen, Department of Computer Science, Aalborg University, Denmark & kgl@cs.aau.dk. Table of Links Introduction Introduction Definitions Definitions Model Checking Model Checking Tool Overview Tool Overview Evaluation Evaluation Conclusion and References Conclusion and References We present CGAAL, our efficient on-the-fly model checker for alternating-time temporal logic (ATL) on concurrent game structures (CGS). We present how our tool encodes ATL as extended dependency graphs with negation edges and employs the distributed on-the-fly algorithm by Dalsgaard et al. Our tool offers multiple novel search strategies for the algorithm, including DHS which is inspired by PageRank and uses the in-degree of configurations as a heuristic, IHS which estimates instability of assignment values, and LPS which estimates the distance to a state satisfying the constituent property using linear programming. CGS are input using our modelling language LCGS, where composition and synchronisation are easily described. We prove the correctness of our encoding, and our experiments show that our tool CGAAL is often one to three orders of magnitude faster than the popular tool PRISM-games on case studies from PRISM’s documentation and among case studies we have developed. In our evaluation, we also compare and evaluate our search strategies, and find that our custom search strategies are often significantly faster than the usual breadth-first and depth-first search strategies. 1 Introduction Software plays a large role in our everyday lives, making decisions, enabling efficient communication, ensuring safety, and many more critical tasks. Furthermore, the complexity of the software and the decisions they have to make are ever-increasing due the to interconnectivity and reactive nature of modern systems. These software systems mutually depend on, communicate with, and guide each other based on their collective and internal states. Even a few interconnected systems that are not necessarily themselves too complex can give rise to an extremely complex system as a whole. Safety-critical software that operates in contexts where errors could lead to human casualties or significant capital losses requires methods to verify their correctness. Unfortunately, such methods are difficult to implement due to the sheer state-space explosion that arises from the inherent parallel composition of systems. In this paper, we consider systems that can be expressed as discrete multiplayer games in which the actors can perform actions concurrently. That is, in each configuration of the game, each player simultaneously chooses an action they wish to perform and the resulting decision vector then deterministically results in the next configuration of the game. Such concurrent games are extremely expressive and can be used to model a variety of complex systems and lends themselves inherently, by their concurrent nature, to describing multi-actor systems. The properties that we wish to verify on these systems are those that can be expressed by Alternating-time Temporal Logic (ATL) [1], a game theoretic extension of Computation Tree Logic (CTL) [2, 3], that like CTL can be used to describe safety and liveness properties. ATL extends CTL by replacing the existential and universal path quantifiers with so-called coalition quantifiers. These coalition quantifiers describe properties of the game in which a coalition of players can either enforce some desired outcome or cannot avoid an outcome. Of course, proof of satisfaction of such property is done by finding a strategy showing that the players can indeed enforce said property or by showing that no matter what strategy the players follow they cannot avoid said property. One existing tool that can model check concurrent (stochastic) games is PRISM-games [12], an extension of the probabilistic model checker PRISM [11]. PRISM excels at stochastic models such as discreteand continuous-time Markov chains, Markov decision processes, and (priced) probabilistic timed automata, for which it can verify various properties described in LTL , CSL, and probabilistic CTL*. There are multiple verification engines in PRISM, both symbolic and explicit state. The PRISM-games extension focuses on stochastic games in which game-theoretic approaches are needed. PRISM-games can among other things synthesise strategies and reason about equilibria-based properties, where players may have distinct, but not necessarily conflicting objectives. These properties are typically described with probabilistic ATL with rewards. In version 3.0 [12], PRISM-games added support for concurrent stochastic games. In [14], Liu and Smolka present a global and a local algorithm to compute fixed-point boolean vertex assignments in dependency graphs with directed hyper-edges representing dependencies between vertex values. The global algorithm has a better worst-case running time of the two, but the on-the-fly local algorithm only explores the graph as needed to compute the fixed-point assignment. Many problems have since been encoded as dependency graphs. Notably, model-checking problems for CTL can be encoded as dependency graphs and the algorithms by Liu and Smolka can be used to compute the satisfaction relation enabling model-checking using dependency graphs. J. F. Jensen et al. [9] show that weighted CTL can be encoded in dependency graphs as well and introduce symbolic edges to handle weights efficiently. In [10], M. C. Jensen et al. introduce symbolic dependency graphs, an encoding of probabilistic weighted CTL, and an implementation of a local and global algorithm. A. Dalsgaard et al. [4] present a distributed extension to the local algorithm to boost the performance and show how to model Petri Nets problems using CTL and dependency graphs. Furthermore, they extend dependency graphs by adding components and negation edges to solve formulae with negations. An abstract dependency graph framework is presented by S. Enevoldsen et al. in [6] where they generalise the various extensions of the dependency graphs with a general algorithm for any Noetherian partial order domain. In the closely related work of [5], S. Enevoldsen et at. uses abstract dependency graphs to model check probabilistic ATL for weighted stochastic games. In multiple works [4, 10] the local algorithm is found to be faster more often than the global algorithm in practice. In this paper, we introduce the tool CGAAL (Concurrent Games AALborg), a model checker implemented in Rust that allows for the verification of ATL properties on concurrent games. CGAAL works by encoding the model checking problem as an extended dependency graph and by implementing the on-the-fly local distributed algorithm by Dalsgaard et al. [4]. The concurrent games are described and inputted to the tool using LCGS, our PRISM-language inspired model language, and for specific ATL formulae, we can output a strategy describing how to ensure satisfaction given a positive result. Furthermore, CGAAL implements a variety of strategies for searching the problem state space: a heuristic inspired by PageRank [15] and two heuristic search strategies that exploit the state vector representation of our model, alongside the usual breadth-first and depth-first approach. We conduct experiments comparing our different search strategies with each other on several case studies and show that having more available compute threads often leads to speed-ups of 1-2 orders of magnitude. We also perform experiments comparing our tool on a selection of case studies to the state-of-the-art PRISM-games. Though while PRISM-games is a tool specialised for probabilistic games we find that our implementation often outperforms PRISM-games and especially whenever we are not required to compute the entire fixed point in which case we are often 1-3 orders of magnitude faster. Again, we emphasise that PRISM games is meant for probabilistic games and as such this is not a completely fair one-to-one comparison. Outline . This paper is structured as follows. Section 2 introduces the formal definitions of concurrent games and alternating-time temporal logic. In Section 3 we present how model checking of ATL properties in concurrent games can be encoded in extended dependency graphs. We also give a short explanation of the CERTAINZERO algorithm and our search strategies. Outline In Section 4 we give a short example of how to use CGAAL and the LCGS language. An evaluation of our tool and a comparison with PRISM can be found in Section 5, and we conclude on our findings in Section 6. This paper is available on Arxiv under CC 4.0 license. This paper is available on Arxiv under CC 4.0 license. Arxiv

Stories, News & Mentions about FINTHESIS

Finthesis’s Stories on HackerNoon