Our powerful phishing mitigation platform will help to protect your organization from the harmful effects of online phishing scams. Unlike other solutions, C9LAB's unique approach fully integrates simulated phishing campaigns with education and training materials to equip your users with the knowledge they need to avoid actual attacks. At the same time, our domain protection service ensures that malicious actors are intercepted while they attempt to register domain names similar to yours. This creates a safe environment for you and your organization. Best of all, our solution includes an up-to-date feed of verified URLs, IP addresses, and domains that contain potential active threats so you can stay ahead of the curve as new schemes emerge. With C9LAB as your trusted partner, you can be sure that your defense against cybercrime is always airtight!

C9lab.com

Large Language Models (LLMs) like GPT-4, Llama-2, and Claude have carved out a significant niche, offering unprecedented capabilities in natural language processing and generation. These models, though seemingly similar, harbor distinct characteristics that influence their interaction with our language, a phenomenon rooted in the core of their design - tokenization, embeddings, and prompting. Tokenization is the process where human language is dissected into manageable pieces, called tokens, which serve as the building blocks of comprehension for these models. Each employs a unique tokenizer, translating English or any other human language into a numerical language that the model understands. This translation is not universal; it's as diverse as the human languages we speak. A word or phrase might be tokenized differently by GPT-4 than by Llama-2, leading to varied interpretations and responses. LLM Prompting, on the other hand, is an art of communication with these models. It’s akin to having a conversation where the quality of the response is heavily influenced by the nature of the question asked. Mastering the art of prompting is essential to harness the full potential of LLMs, and it’s a skill that evolves with practice and understanding. In this video, we dive deep into the world of LLMs, offering insights into their distinct behaviors and delving into the nuances of tokenization and prompting. We explore advanced techniques and tools that enhance the efficacy of these interactions, ensuring that the conversation between humans and AI is not just coherent but also rich in context and relevance. I also share a new prompting technique that is particularly useful for summarization! Lastly, I cover the importance of parameter tuning when using those LLMs (not just when training them!), such as adapting the temperature, frequency penalty, and the other parameters influencing your outputs. Why should you watch this video? Because in the realm of AI, understanding is empowerment. This is a guided journey that equips you with the knowledge to navigate the complex terrains of LLMs effectively. It’s an exploration of the subtle, yet profound differences in how models like GPT-4, Llama-2, and Claude perceive and respond to human language. Let’s demystify the intricate dance between prompts, tokenization, embeddings, the most important model parameters, and their responses, and learn to control LLM outputs better: https://youtu.be/AOeKFlSLMOA?embedable=true&transcript=true References ►Amazing Prompting resource: learnprompting.org ►Chain of Density Prompting paper from Adams et al., https://arxiv.org/pdf/2309.04269.pdf ►Colab notebook of the tokenizer example: https://colab.research.google.com/drive/1IVQyGmj1t9R12oajT4OMjMWzwZk3jWgm?usp=sharing ►Tokenization in Machine Learning Explained: https://vaclavkosar.com/ml/Tokenization-in-Machine-Learning-Explained ►Twitter: https://twitter.com/Whats_AI ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/ ►Support me on Patreon: https://www.patreon.com/whatsai ►Join Our AI Discord: https://discord.gg/learnaitogether

GPT-4, Llama-2, Claude: How Different Language Models React to Prompts

Large language models. You must’ve heard these words before. They represent a specific type of machine learning-based algorithm that understand and can generate language, a field often called or NLP. natural language processing You’ve certainly heard of the most known and powerful language model: . GPT-3 GPT-3, as I’ve described in the video covering it is able to take language, understand it and generate language in return. But be careful here; it doesn’t really understand it. In fact, it’s far from understanding. GPT-3 and other language-based models merely use what we call dictionaries of words to represent them as numbers, remember their positions in the sentence, and that’s it. Let's dive into those powerful machine learning models and try to understand what they see instead of words, called word embeddings, and how to produce them with an example provided by Cohere. Learn more in the video... References ►Read the full article: ►BERT Word Embeddings Tutorial: ►Cohere's Notebook from the code example: ►Cohere Repos focused on embeddings: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/text-embedding/ https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/#why-bert-embeddings https://colab.research.google.com/github/cohere-ai/notebooks/blob/main/notebooks/Basic_Semantic_Search.ipynb https://github.com/cohere-ai/notebooks https://www.louisbouchard.ai/newsletter/ Video Transcript 0:07 language models you must have heard 0:10 these words before they represent a 0:13 specific type of machine learning 0:14 algorithms that understand and can 0:16 generate language a field often called 0:19 natural language processing or NLP 0:22 you've certainly heard of the most known 0:24 and Powerful language models like gpt3 0:26 gpt3 as I've described in the video 0:28 covering it is able to take language 0:30 understand it and generate language in 0:33 return but be careful here it doesn't 0:35 really understand it in fact it's far 0:38 from understanding gbd3 and other 0:41 language-based models merely use what we 0:44 call dictionaries of words to represent 0:46 them as numbers remember their positions 0:49 in the sentence and that's it using a 0:52 few numbers and positional numbers 0:53 called embeddings they are able to 0:55 regroup similar sentences which also 0:58 means that they are able to kind of 1:00 understand sentences by comparing them 1:02 to known sentences like our data set 1:05 it's the same process for image sentence 1:07 models that take your sentence to 1:10 generate an image they do not really 1:11 understand it but they can compare it to 1:13 similar images producing some sort of 1:16 understanding of the concepts in your 1:18 sentence in this video we will have a 1:20 look at what those powerful machine 1:22 learning models see instead of words 1:24 called word embeddings and how to 1:27 produce them with an example provided by 1:29 the sponsor of this video a great 1:31 company in the NLP field cohere which I 1:35 will talk about at the end of the video 1:36 as they have a fantastic platform for 1:39 NLP we've talked about embeddings and 1:42 gpt3 but what's the link between the two 1:44 emittings are what is seen by the models 1:47 and how they process the words we know 1:50 and why use embeddings well because as 1:53 of now machines cannot process words and 1:56 we need numbers in order to train those 1:59 large models thanks to our carefully 2:01 built data set we can use mathematics to 2:04 measure the distance between embeddings 2:06 and correct our Network based on this 2:08 distance iteratively getting our 2:10 predictions closer to the real meaning 2:12 and improving the results and meetings 2:15 are also what the models like clip 2:17 stable diffusion or Dali used to 2:19 understand sentences and generate images 2:21 this is done by comparing both images 2:24 and text in the same embedding space 2:26 meaning that the model does not 2:28 understand either text or images but it 2:31 can understand if an image is similar to 2:33 a specific text or not so if we find 2:36 enough image caption pairs we can train 2:38 a huge and Powerful model like Dali to 2:41 take a sentence embed it find its 2:43 nearest image clone and generate it in 2:46 return so machine learning with text is 2:48 all about comparing embeddings but how 2:51 do we get those embeddings we get them 2:53 using another model trained to find the 2:56 best way to generate similar embeddings 2:58 for similar sentences while keeping the 3:01 differences in meaning for similar words 3:03 compared to using a straight one for one 3:06 dictionary the sentences are usually 3:08 represented with special tokens marking 3:10 the beginning and end of our text then 3:13 as I said we have our poses from all 3:15 embeddings which indicate the position 3:17 of each word relative to each other 3:19 often using sinusoidal functions I 3:22 linked a great article about this in the 3:25 description if you'd like to learn more 3:26 finally we have our word embeddings we 3:29 start with all our words being split 3:31 into an array just like a table of words 3:34 starting now there are no longer words 3:36 they are just tokens or numbers from the 3:40 whole English dictionary you can see 3:42 here that all the words now are 3:44 represented by a number indicating where 3:46 they are in the dictionary thus having 3:49 the same number for the word Bank even 3:51 though their meaning are different in 3:53 the sentence we have now we need to add 3:56 a little bit of intelligence to that but 3:58 not too much this is done thanks to a 4:00 model trained to take this new list of 4:03 numbers and further encode it into 4:05 another list of numbers that better 4:08 represent the sentence for example it 4:10 will no longer have the same embedding 4:13 for the two words bank here this is 4:15 possible because the model used to do 4:17 that has been trained on a lot of 4:19 annotated Text data and learned to 4:21 encode similar meaning sentences next to 4:24 each other and opposite sentences far 4:27 from each other thus allowing our 4:29 embeddings to be less biased by our 4:31 choice of words then the initial simple 4:34 one for one word embedding we initially 4:37 had here's what using imagings looks 4:39 like in a very short NLP example there 4:42 are more links below to learn more about 4:44 embeddings and how to code it yourself 4:46 here we will take some Hacker News posts 4:49 and build a model label to retrieve the 4:51 most similar post of a new input 4:53 sentence to start we need a data set in 4:56 this case it is a pre-embedded set of 4:58 3000 Hacker News posts that have already 5:01 been emitted into numbers then we build 5:04 a memory saving all those embeddings for 5:07 future comparison we basically just 5:09 saved these embeddings in an efficient 5:11 way when a new query is done for example 5:13 here asking what is your most profound 5:16 life inside you can generate its 5:18 embedding using the same embedding 5:20 Network usually it is bird or a version 5:23 of it and we compare the distance 5:25 between the embedding space to all other 5:27 Hacker News posts in our memory note 5:30 that it's really important here to 5:32 always use the same network whether for 5:34 generating your data set or for querying 5:36 it as I said there is no real 5:38 intelligence here nor that it actually 5:40 understands the words it just has been 5:42 trained to embed similar sentences 5:45 nearby in the unmanning space nothing 5:47 more if you send your sentence to a 5:50 different network to generate an 5:51 embedding and compare the embedding to 5:53 the ones you had from another Network 5:55 nothing will work it will just be like 5:58 the nice people that try to talk to me 5:59 in Hebrew at eccv last week it just 6:02 wasn't in an embedding space my brain 6:04 could understand fortunately for us our 6:06 brain can learn to transfer from one 6:08 embedding space to another as I can with 6:11 French and English but it requires a lot 6:13 of work and practice and it's the same 6:16 for machines anyways coming back to our 6:18 problem we could find the most similar 6:21 posts that's pretty cool but how could 6:23 we achieve this as I mentioned it's 6:25 because of the network birth in this 6:28 case it learns to create similar 6:30 embeddings from similar sentences we can 6:32 even visualize it in two Dimensions like 6:35 this where you can see how two similar 6:37 points represent similar subjects you 6:39 can do many other things once you have 6:41 those embeddings like extracting 6:43 keywords performing a semantic search 6:45 doing sentiment analysis or even 6:47 generating images as we said and 6:49 demonstrated in previous videos I have a 6:52 lot of videos covering those and listed 6:55 a few interesting notebooks to learn to 6:57 play with encodings thanks to the cohere 6:59 team now let me talk a little bit about 7:02 kohilu as they are highly relevant to 7:05 this video cook here provides a 7:07 everything you need if you are working 7:09 in the NLP field including a super 7:11 simple way to use embedding models in 7:14 your application literally with just an 7:16 API call you can embed the text without 7:18 knowing anything about how the embedding 7:21 models work the API does it for you in 7:23 the background here you can see the 7:25 semantic search notebook that uses 7:27 cohere API to create embeddings of an 7:30 archive of questions and question 7:32 queries to later perform search of 7:34 similar questions using cook here you 7:37 can easily do anything related to text 7:39 generate categorize and organize at 7:42 pretty much any scale you can integrate 7:44 large language models trained on 7:46 billions of words with a few lines of 7:48 code and it works in any Library you 7:51 don't even need machine learning skills 7:53 to get started they even have learning 7:55 resources like the recent cohere for 7:57 ai's colors program that I really like 8:00 this program is an incredible 8:01 opportunity for emerging talent in NLP 8:04 research around the world if selected 8:06 you will will work alongside their team 8:08 and have access to a large-scale 8:10 experimental framework and cohere 8:12 experts which is pretty cool I also 8:15 invite you to join their great Discord 8:17 Community ingeniously called Co Unity I 8:21 hope you've enjoyed this video and will 8:23 try out cohere for yourself with the 8:25 first link below I am sure you will 8:27 benefit from it thank you very much for 8:29 watching the whole video and thanks to 8:31 anyone supporting my work by leaving a 8:33 like comment or trying out our sponsors 8:36 that I carefully select for these videos

Text Embedding Explained: How AI Understands Words

Incrustación de texto explicada: cómo la IA entiende las palabras

Explication de l'intégration de texte : comment l'IA comprend les mots

पाठ एंबेडिंग समझाया: एआई शब्दों को कैसे समझता है

テキスト埋め込みの説明: AI が単語を理解する方法

Explicação da incorporação de texto: como a IA entende as palavras

Giải thích về nhúng văn bản: Cách AI hiểu các từ

文本嵌入解释：AI 如何理解单词

Have you ever dreamed of a good transcription tool that would accurately understand what you say and write it down? Not like the automatic YouTube translation tools… I mean, they are good but far from perfect. Just try it out and turn the feature on for the video, and you’ll see what I’m talking about. Luckily, OpenAI just released and open-sourced a pretty powerful AI model just for that: Whisper. It understands stuff I can’t even comprehend, not being a native English speaker (listen in the video) and it works for language translation too! Learn more in the video below... References ►Read the full article: ► Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C. and Sutskever, I., Robust Speech Recognition via Large-Scale Weak Supervision. ►Project link: ►Code: ►Google Colab notebook: ►YouTube Whisperer app: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/whisper/ https://openai.com/blog/whisper/ https://github.com/openai/whisper https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb https://huggingface.co/spaces/jeffistyping/Youtube-Whisperer https://www.louisbouchard.ai/newsletter/ Video Transcript 0:00 have you ever dreamed of a good 0:01 transcription tool that will accurately 0:03 understand what you see and write it 0:05 down not like the automatic YouTube 0:07 translation tools I mean they are good 0:09 but far from perfect just try it out and 0:12 turn the feature on for this video and 0:14 you'll see what I'm talking about well 0:16 open AI just released an open sourced 0:18 and a pretty powerful AI model just for 0:21 that whisper it even understands stuff I 0:24 can't even comprehend not being a native 0:26 English speaker 0:28 this is the micro machine man presenting 0:29 the most miniature motorcade of 0:30 micro machine each one has dramatic 0:32 details terrific trim Precision page Art 0:33 Plus incredible micro machine pocket 0:34 place that says a PlayStation fire it 0:36 works for language translation too 0:38 whisper a consistent the liquid it 0:41 sounds automatically alternator 0:48 the results and precision are incredible 0:51 but what's even cooler is how it works 0:53 let's dive into it but first let me 0:56 introduce this episode's sponsor that is 0:58 highly related to This research assembly 1:00 AI assembly AI is the API platform for 1:03 state-of-the-art AI models from startups 1:06 to Fortune 500 companies developers and 1:08 product teams around the world leverage 1:10 assembly AI to build better ai-based 1:13 products and features if you are 1:15 building a meeting summarizer podcast 1:17 analyzer or really anything related to 1:19 audio or video and want to leverage AI 1:22 to power transcription or insights at 1:24 scale definitely check out their API 1:26 platform more specifically I wanted to 1:29 share their summarization model which I 1:31 find really cool as the name says with 1:34 this model you can build tools that 1:36 automatically summarize your audio and 1:38 video files the model is flexible to fit 1:41 your use case and can be customized to 1:44 different summary types bullets 1:46 paragraphs headlines or adjust it all 1:48 Works through simple API calls and you 1:51 can find all the information you need 1:53 for the summarization model and assembly 1:55 AI with the first link below 1:59 when it comes to the model itself 2:01 whisper is pretty classic it is built on 2:04 the Transformer architecture stacking 2:06 encoder blocks and decoder blocks with 2:08 the attention mechanism propagating 2:10 information between both it will take 2:13 the audio recording split it into 30 2:16 second chunks and process them one by 2:18 one for each 30 second recordings it 2:21 will encode the audio using the encoder 2:23 section and save the position of each 2:25 word said and leverage this encoded 2:28 information to find what was said using 2:30 the decoder the decoder will predict 2:33 what we call tokens from all this 2:34 information which are basically each 2:37 words being said then it will repeat 2:39 this process for the next word using all 2:41 the same information as well as the 2:43 predicted previous word helping it to 2:46 guess the next one that will make more 2:48 sense as I said the overall architecture 2:50 is a classic encoder and decoder and I 2:53 covered it in multiple videos similar to 2:55 gpt3 and other language models which I 2:58 invite you to check for more 3:00 architectural details this works as it 3:02 was trained on more than 600 000 hours 3:05 of multilingual and multitask supervised 3:08 data collected from the web meaning that 3:11 they trained their Audio model in a 3:12 similar way as gpt3 with data available 3:15 on the internet making it a large and 3:18 general Audio model it also makes the 3:20 model way more robust than others in 3:23 fact they mentioned that whisper 3:24 approaches human level robustness due to 3:27 being trained on such a diverse set of 3:29 data ranging from Clips TED Talks 3:32 podcasts interviews and more which all 3:34 represent real world-like data with some 3:36 of them transcribed using machine 3:38 learning based models and not humans 3:40 using such imperfect data certainly 3:43 reduces the possible Precision but I 3:45 will argue it helps for robustness when 3:47 used as sparsely compared to Pure human 3:49 curated audio data sets with perfect 3:52 transcriptions having such a general 3:54 model isn't very powerful in itself as 3:57 it will be beaten at most tasks by 3:58 smaller and more specific models adapted 4:01 to the task at hand but it has other 4:03 benefits you can use this kind of 4:05 pre-trained models and fine-tune them on 4:08 your task meaning that you will take 4:10 this powerful model and retrain a part 4:13 of it or the entire thing with your own 4:15 data this technique has been shown to 4:17 produce much better models than starting 4:19 training from scratch with your data and 4:21 what's even cooler is that openai open 4:24 sourced their code and everything 4:25 instead of an API so you can use whisper 4:28 as a pre-trained foundation architecture 4:30 to build upon and create more powerful 4:33 models for yourself some people have 4:35 already released the tools like the 4:37 YouTube Whisperer unhugging face by Jeff 4:39 is typing taking a YouTube link and 4:42 generating transcriptions which I found 4:44 thanks to Yannick kilter they also 4:46 released a Google collab notebook to 4:48 play with right away while something 4:50 competition is key I'm glad openai is 4:53 releasing some of its work to the public 4:54 I'm convinced such collaborations are 4:57 the best way to advance in our field let 5:00 me know what you think if you'd like to 5:01 see more public releases of openai or if 5:04 you like the final products they build 5:06 like dally as always you can find more 5:08 information about whisper in the paper 5:11 and code linked below and I hope you've 5:13 enjoyed this video I will see you next 5:15 week with another amazing paper

What is OpenAI's Whisper Model?

¿Qué es el modelo de susurro de OpenAI?

Qu'est-ce que le modèle Whisper d'OpenAI ?

OpenAI का व्हिस्पर मॉडल क्या है?

OpenAI のウィスパー モデルとは?

O que é o modelo Whisper da OpenAI?

Mô hình thì thầm của OpenAI là gì?

OpenAI 的耳语模型是什么？

BlobGAN allows for unreal manipulation of images, made super easily controlling simple blobs. All these small blobs represent an object, and you can move them around or make them bigger, smaller, or even remove them, and it will have the same effect on the object it represents in the image. This is so cool! As the authors shared in their results, you can even create novel images by duplicating blobs, creating unseen images in the dataset ! Correct me if I’m wrong, but I believe it is one of, if not the first, paper to make the modification of images as simple as moving blobs around and allowing for edits that were unseen in the training dataset. like a room with two ceiling fans And you can actually play with this one compared to some companies we all know! They shared their code publicly and a Colab Demo you can try right away. Even more exciting is how BlobGAN works. Learn more in the video! Watch the video References ►Read the full article: ►Epstein, D., Park, T., Zhang, R., Shechtman, E. and Efros, A.A., 2022. BlobGAN: Spatially Disentangled Scene Representations. arXiv preprint arXiv:2205.02837. ►Project link: ►Code: ►Colab Demo: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/blobgan/ https://dave.ml/blobgan/ https://github.com/dave-epstein/blobgan https://colab.research.google.com/drive/1clvh28Yds5CvKsYYENGLS3iIIrlZK4xO?usp=sharing#scrollTo=0QuVIyVplOKu https://www.louisbouchard.ai/newsletter/ Video Transcript 0:00 if you think that the progress with guns 0:02 was over you couldn't be more wrong 0:04 here's blob gun and this new paper is 0:07 just incredible blob gun allows for 0:09 unreal manipulation of images made super 0:12 easily controlling simple blobs all 0:14 these small blobs represent an object 0:17 and you can move them around make them 0:19 bigger smaller or even remove them and 0:22 it will have the same effect on the 0:24 object it represents in the image this 0:26 is so cool as the authors shared in 0:29 their results you can even create novel 0:31 images by duplicating blubs creating 0:34 unseen images in the data set like this 0:37 room with two ceiling fans correct me if 0:40 i'm wrong but i believe it's one of if 0:42 not the first paper to make the 0:44 modification of images as simple as 0:46 moving blobs around and allowing for 0:49 edits that were unseen in the training 0:51 dataset and you can actually play with 0:53 this one compared to other companies we 0:55 all know they shared are called publicly 0:58 and a collab demo you can try right away 1:00 even more exciting is how bloggian works 1:03 which we'll dive into in a few seconds 1:05 to publish an excellent paper like 1:07 blobgun the researchers needed to run 1:09 many experiments on multiple machines 1:12 those who played with guns know how long 1:14 and painful this process can be plus 1:16 their code is available on github and 1:18 google collab this means their code has 1:21 to be reproducible funnily enough this 1:24 is also a really strong point of this 1:26 episode's sponsor weights and biases 1:28 weights and biases change my life as a 1:30 researcher it tracks everything you need 1:32 for your code to be reproducible the 1:34 hyper parameters the github commit 1:36 hardware usage metrics and the python 1:38 version leaving you without headaches ok 1:41 some might still appear because of 1:43 deadlines or bugs but none from trying 1:45 to reproduce experiments weights and 1:47 biases is also super helpful when 1:49 sharing your experiment results with 1:51 your colleagues a great tool for that is 1:53 reports they can act as dashboards for 1:56 supervisor pis or managers to check how 1:59 experimentation is going meaning more 2:01 time for research while improving your 2:03 feedback's quality please don't be like 2:06 most researchers who keep their code a 2:08 secret and try using weights and biases 2:10 with the first link below 2:13 now let's get back to our paper blub gun 2:16 spatially disentangled scene 2:18 representations the title says it ital 2:21 blovkian uses blobs to disentangle 2:23 objects in a scene meaning that the 2:25 model learns to associate each blob with 2:28 a specific object in the scene like a 2:30 bed window or ceiling fan once trained 2:33 you can move the blobs and objects 2:35 around individually make them bigger or 2:37 smaller duplicate them or even remove 2:40 them from the picture of course the 2:42 results are not entirely realistic but 2:44 as a great person would say just imagine 2:47 the potential of this approach two more 2:49 papers down the line 2:51 what's even cooler is that this training 2:53 occurs in an unsupervised scheme this 2:55 means that you do not need every single 2:57 image example to train it as you would 3:00 in supervised learning a quick example 3:02 is that supervised training will require 3:05 you to have all the desired 3:06 manipulations in your image that are set 3:08 to teach blobs to learn those 3:10 transformations whereas in unsupervised 3:13 learning you do not need this extensive 3:15 data and the model will learn to achieve 3:17 this task by itself associating bluffs 3:20 to objects on its own without explicit 3:22 labels we train the model with a 3:24 generator and a discriminator in a gun 3:27 fashion i will simply do a quick 3:28 overview as i've covered guns in 3:30 numerous videos before as always in guns 3:33 the discriminator's responsibility is to 3:35 train the generator to create realistic 3:38 images the most important part of the 3:40 architecture is the generator with our 3:42 blobs and a style gun 2 like decoder i 3:45 also covered style gun based generators 3:48 in other videos if you are curious about 3:50 how it works but in short we first 3:52 create our blobs this is done by taking 3:55 random noise as in most generator 3:57 networks and mapping it into blobs using 4:00 a first neural network this will be 4:02 learned during training then you need to 4:05 do the impossible take this blob 4:07 representation and create a real image 4:10 out of it this is where the gan magic 4:12 happens since you are still listening 4:14 please consider subscribing to the 4:16 channel and liking the video it means a 4:18 lot and supports my work for free also 4:21 we have a community called learn ai 4:23 together on discord to learn exchange 4:26 with fellow ai enthusiasts i'm convinced 4:28 you'll love it there and i will be glad 4:30 to meet you 4:32 we need a star gun like architecture to 4:34 create our images from these blobs of 4:37 course we added the architecture to take 4:39 the blobs we just created as inputs 4:41 instead of the usual random noise 4:43 then we turn our model using the 4:45 discriminator to learn to generate 4:47 realistic images once we have good 4:50 enough results it means our model can 4:52 take on blob representation instead of 4:54 noise and generate images but we still 4:57 have a problem how can we disentangle 4:59 those blobs and make them match objects 5:02 well this is the beauty of our 5:04 unsupervised approach the model will 5:06 iteratively improve and create realistic 5:08 results while also learning how to 5:11 represent these images in the form of a 5:13 fixed number of blobs you can see here 5:15 how blubs are often used to represent 5:17 the same objects or very similar objects 5:20 in the scene here you can also see how 5:22 the same gloves are used to represent 5:24 either a window or a painting which 5:26 makes a lot of sense likewise you can 5:29 see that light is almost always 5:31 represented in the fort blub similarly 5:33 you can see how blubs are often 5:35 representing the same regions in the 5:37 scene most certainly leads you to the 5:39 similarities of images in the dataset 5:42 used for this experiment and voila this 5:45 is how blobgan learns to manipulate 5:47 scenes using a very intuitive blob 5:50 representation i'm excited to see the 5:52 realism of the results improve keeping a 5:54 similar approach using such a technique 5:57 we could design simple interactive apps 5:59 to allow designers and anyone to 6:01 manipulate images easily which is quite 6:04 exciting of course this was just an 6:06 overview of this new paper and i 6:08 strongly recommend reading their paper 6:10 for a better understanding and a lot 6:12 more detail on their approach 6:13 implementation and tests they did as i 6:16 said earlier in the video they also 6:18 shared their code publicly and a color 6:20 demo you can try right away all the 6:22 links are in the description below 6:24 thank you for watching until the end and 6:27 i will see you next week with another 6:28 amazing paper [Music]

BlobGAN: A BIG step for GANs

BlobGAN: Un GRAN paso para las GAN

BlobGAN : un GRAND pas pour les GAN

BlobGAN: GANs के लिए एक बड़ा कदम

BlobGAN: GAN の大きな一歩

BlobGAN: um GRANDE passo para as GANs

BlobGAN: Một bước LỚN cho GAN

BlobGAN：GAN 迈出的一大步

Learn how this algorithm can understand images and automatically remove the undesired object or person and save your future Instagram post! You’ve most certainly experienced this situation once: You take a great picture with your friend, and someone is photobombing behind you, ruining your future Instagram post. Well, that’s no longer an issue. Either it is a person or a trashcan you forgot to remove before taking your selfie that’s ruining your picture. This AI will just automatically remove the undesired object or person in the image and save your post. It’s just like a professional photoshop designer in your pocket, and with a simple click! This task of removing part of an image and replacing it with what should appear behind has been tackled by many AI researchers for a long time. It is called image inpainting, and it’s extremely challenging. Learn more in the video! Watch the video References ► Complete article: ► Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K. and Lempitsky, V., 2022. Resolution-robust Large Mask Inpainting with Fourier Convolutions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2149-2159). ► Code: ► Colab Demo: Product using LaMa: https://cleanup.pictures/ ► Fourier Domain explained by the great : ► Great in-depth explanation of LaMa with the authors by : ► My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/lama/ https://github.com/saic-mdal/lama https://colab.research.google.com/github/saic-mdal/lama/blob/master/colab/LaMa_inpainting.ipynb @3Blue1Brown https://youtu.be/spUNpyF58BY @Yannic Kilcher https://youtu.be/Lg97gWXsiQ4 https://www.louisbouchard.ai/newsletter/ Video Transcript 00:00 you've most certainly experienced this 00:01 situation once you take a great picture 00:04 with your friend and someone is 00:05 photobombing behind you ruining your 00:07 future instagram post well that's no 00:10 longer an issue either it's a person or 00:12 a trash can you forgot to remove before 00:14 taking your selfie that's ruining your 00:16 picture this ai will automatically 00:18 remove the undesired object or person in 00:21 the image and save your post it's just 00:23 like a professional photoshop designer 00:25 in your pocket with a simple click this 00:27 task of removing part of an image and 00:29 replacing it with what should appear 00:31 behind it has been tackled by many ai 00:34 researchers for a long time it's called 00:36 image and painting and it's extremely 00:38 challenging as you will see the paper i 00:40 want to show you achieves it with 00:41 incredible results and can do it easily 00:43 in high definition unlike previous 00:45 approaches you may have heard of before 00:48 you definitely want to stay until the 00:49 end of the video to see that you won't 00:51 believe how great and realistic it looks 00:54 for something produced in a split second 00:56 by an algorithm as i said this task of 00:59 imaging painting is basically you 01:00 removing unwanted objects from your 01:02 images you should be doing the same in 01:05 your work life and remove any friction 01:07 your next step as an ar professional or 01:09 student to do that should be to do like 01:11 me and try the sponsor of today's 01:13 episode weights and biases if you run a 01:16 lot of experiments you should be using 01:18 weights and vices it will remove all 01:20 painful steps from hyperparameter tuning 01:22 to results analysis with a handful of 01:25 lines of code added and it's entirely 01:27 free for personal usage it takes not 01:29 even five minutes to set up and you 01:31 don't have anything else to do forever 01:34 talking about removing friction points i 01:36 don't think you can do better than that 01:38 weight and biases has everything you 01:39 need for your code to be reproducible 01:42 without you even trying for your 01:44 well-being do like me and give weights 01:46 and biases a try for free with the first 01:48 link below to remove an object from an 01:50 image the machine needs to understand 01:52 what should appear behind the subject 01:54 and to do this will require having a 01:56 three-dimensional understanding of the 01:58 world as humans do but it doesn't have 02:00 that it just has access to a few pixels 02:03 in an image which is why it's so 02:05 complicated whereas it looks quite 02:06 simple for us that can simply imagine 02:09 the depth and guess that there should be 02:11 the rest of the wall here the window and 02:13 etc we basically need to teach the 02:15 machine how the world typically looks 02:17 like 02:18 so we will do that using a lot of 02:20 examples of real world images so that it 02:23 can have an idea of what our world looks 02:25 like in the two-dimensional picture 02:27 world which is not a perfect approach 02:29 but does the job then another problem 02:32 comes with the computational cost of 02:34 using real-world images with way too 02:36 many pixels to fix that most current 02:38 approaches work with low quality images 02:41 so a downsized version of the image that 02:43 is manageable for our computers and 02:46 upscale the inpainted part at the end to 02:48 replace it in the original image making 02:50 the final results look worse than it 02:53 could be or at least they won't look 02:55 great enough to be shared on instagram 02:57 and have all the likes you deserve you 02:59 can't really feel it high quality images 03:01 directly as it will take way too much 03:03 time to process and train or can you 03:06 well these are the main problems the 03:08 researchers attacked in this paper and 03:10 here's how roman suvarov ital from 03:13 samsung research introduced a new 03:15 network called llama that is quite 03:17 particular as you can see in image and 03:19 painting you will typically send the 03:21 initial image as well as what you'd like 03:23 to remove from it this is called a mask 03:26 and will cover the image as you can see 03:28 here and the network won't have access 03:30 to this information anymore as it needs 03:32 to fill in the pixels then it has to 03:35 understand the image and try to fill in 03:37 the same pixels it thinks should fit 03:39 best so in this case they start like any 03:41 other network and downscale the image 03:44 but don't worry their technique will 03:45 allow them to keep the same quality as a 03:47 high resolution image this is because 03:50 here in the processing of the image they 03:52 use something a bit different than usual 03:54 typically we can see different networks 03:56 here in the middle mostly convolutional 03:58 neural networks such networks are often 04:01 used on images due to how convolutions 04:03 work which i explained in other videos 04:05 like the one appearing on the top right 04:07 of your screen if you are interested in 04:09 how it works in short the network will 04:11 work in two steps first it will compress 04:14 the image and try to only save relevant 04:16 information the network will end up 04:18 conserving mostly the general 04:20 information about the image like its 04:22 color overall style or general object 04:24 appearing but not precise details then 04:27 it will try to reconstruct the image 04:29 using the same principles but backward 04:32 we use some tricks like skip connections 04:34 that will save information from the 04:35 first few layers of the network and pass 04:38 it along the second step so that it can 04:40 orient it towards the right objects in 04:42 short it easily knows that there's a 04:44 tower with a blue sky and trees called 04:47 global information but it needs the skip 04:49 connections to know that it's the eiffel 04:51 tower in the middle of the screen that 04:53 there are clouds here and there the 04:55 trees have these colors etc all the fine 04:58 grained details which we call local 05:00 information following a long training 05:02 with many examples we will expect our 05:04 network to reconstruct the image or at 05:06 least a very similar image that contains 05:09 the same kind of objects and be very 05:11 similar if not identical to the initial 05:14 image but remember in this case we are 05:16 working with low quality images that we 05:18 need to upscale which will hurt the 05:20 quality of the results the particularity 05:22 here is that instead of using 05:24 convolutions as in regular convolutional 05:26 networks and skip connections to keep 05:28 local knowledge it uses what we call the 05:31 fast fourier convolution or ffc this 05:34 means that the network will work in both 05:36 the spatial and frequency domains and 05:39 doesn't need to get back to the early 05:40 layers to understand the context of the 05:42 image each layer will work with 05:44 convolutions in the spatial domain to 05:46 process local features and use 4g 05:49 convolutions in the frequency domain to 05:51 analyze global features the frequency 05:53 domain is a bit special and i linked a 05:55 great video covering it in the 05:57 description below if you are curious it 05:58 will basically transform your image into 06:00 all possible frequencies just like sound 06:03 waves and tell you how much of each 06:05 frequency the image contains so each new 06:09 pixel of this newly created image will 06:11 represent a frequency covering the whole 06:13 spatial image and how much it is present 06:16 instead of colors the frequencies here 06:19 are just the repeated patterns at 06:21 different scales for example one of 06:23 these frequency pixels could be highly 06:25 activated by the vertical lines at a 06:27 specific distance from each other in 06:29 this case it could be the same distance 06:31 as the length of a brick so it will be 06:33 highly activated if there is a brick 06:35 wall in the image from this you'd 06:37 understand that there's probably a brick 06:39 wall and the size proportional to how 06:41 much it is activated and you can repeat 06:43 this for all pixels being activated for 06:45 similar patterns giving you good hints 06:48 of the overall aspect of the image but 06:50 nothing about the object themselves or 06:52 the colors the spatial domain will take 06:54 charge of this so doing convolutions on 06:57 this new 4d image allows you to work 06:59 with the whole image at each step of the 07:01 convolution process so it has access to 07:04 a much better global understanding of 07:05 the image even at early layers without 07:08 much computational cost which is 07:10 impossible to achieve with regular 07:11 convolutions in the spatial domain then 07:14 both global and local results are saved 07:17 and sent to the next layer which will 07:19 repeat these steps you will end up with 07:21 the final image that you can upscale 07:23 back the use of the fourier domain is 07:25 what makes it scalable to bigger images 07:27 as their image resolution doesn't affect 07:29 the fourier domain since it uses 07:31 frequencies over the whole image instead 07:34 of colors and the repeated pattern it's 07:36 looking for will be the same whatever 07:38 the size of the image meaning that even 07:40 with training this network with small 07:42 images you will be able to feed it much 07:44 larger images afterward and get amazing 07:47 results 07:54 as you can see the results are not 07:55 perfect but they are quite impressive 07:57 and i'm excited to see what they will do 07:59 next to improve them of course this was 08:01 just a simple overview of this new model 08:03 and you can find more detail about the 08:05 implementation in the paper linked in 08:07 the description below you can also 08:09 implement it yourself with the code link 08:11 below as well i hope you enjoyed the 08:13 video and if so please take a second to 08:16 share it with a friend that could find 08:17 this interesting it will mean a lot and 08:20 help this channel grow 08:21 thank you for watching 08:24 [Music]

This AI Removes Unwanted Objects From Your Images!

We’ve seen AI generate images from other images using GANs. Then, there were models able to generate questionable images using text. In early 2021, was published, beating all previous attempts to generate images from text input using CLIP, a model that links images with text as a guide. A very similar task called image captioning may sound really simple but is, in fact, just as complex. It is the ability of a machine to generate a natural description of an image. DALL-E It’s easy to simply tag the objects you see in the image but it is quite another challenge to understand what’s happening in a single 2-dimensional picture, and this new model does it extremely well! Watch the video References ►Read the full article: ►Paper: Mokady, R., Hertz, A. and Bermano, A.H., 2021. ClipCap: CLIP Prefix for Image Captioning. ►Code: ►Colab Demo: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/clipcap/ https://arxiv.org/abs/2111.09734 https://github.com/rmokady/CLIP_prefix_caption https://colab.research.google.com/drive/1tuoAC5F4sC7qid56Z0ap-stR3rwdk0ZV?usp=sharing https://www.louisbouchard.ai/newsletter/ Video Transcript 00:00 we've seen ai generate images from other 00:02 images using guns then there were models 00:05 able to generate questionable images 00:07 using text in early 2021 dolly was 00:10 published beating all previous attempts 00:12 to generate images from text input using 00:14 clip a model that links images with text 00:16 as a guide a very similar task called 00:19 image captioning may sound really simple 00:21 but is in fact just as complex it's the 00:23 ability of a machine to generate a 00:25 natural description of an image indeed 00:28 it's almost as difficult as the machine 00:30 needs to understand the image and the 00:32 text it generates just like in text to 00:34 image synthesis it's easy to simply tag 00:36 the objects you see in the image this 00:38 can be done using a regular 00:39 classification model but it's quite 00:41 another challenge to understand what's 00:43 happening in a single two-dimensional 00:45 picture humans can do it quite easily 00:47 since we can interpolate from our past 00:49 experience and we can even put ourselves 00:51 in the place of the person in the 00:52 picture and quickly get what's going on 00:55 this is a whole other challenge for a 00:56 machine that only sees pixels yet these 00:59 researchers published an amazing new 01:01 model that does this extremely well in 01:03 order to publish such a great paper 01:05 about image captioning the researchers 01:07 needed to run many many experiments plus 01:10 their code is fully available on github 01:12 which means it is reproducible these are 01:14 two of the strong points of this episode 01:16 sponsor weights and biases if you want 01:18 to publish papers in big conferences or 01:20 journals and do not want to be part of 01:22 the 75 of the researchers that do not 01:25 share their code i'd strongly suggest 01:27 using weights and biases it changed my 01:29 life as a researcher and my work in my 01:31 company weights and biases will 01:32 automatically track each run the hyper 01:34 parameters the github version hardware 01:37 and osu's the python version packages 01:39 install and training script everything 01:41 you need for your code to be 01:42 reproducible without you even trying it 01:45 just needs a line of code to tell what 01:47 to track once and that's it please don't 01:50 be like most researchers that keep their 01:52 code a secret i assume mostly because it 01:54 is hardly reproducible and try out 01:56 weights and biases with the first link 01:58 below as the researchers explicitly said 02:00 image captioning is a fundamental task 02:02 in vision language understanding and i 02:05 entirely agree the results are fantastic 02:07 but what's even cooler is how it works 02:09 so let's dive into the model and it's in 02:12 our working a little before doing so 02:14 let's quickly review what image 02:16 captioning is image captioning is where 02:18 an algorithm will predict a textual 02:20 description of a scene inside an image 02:22 here it will be done by a machine and in 02:25 this case it will be a machine learning 02:27 algorithm this algorithm will only have 02:29 access to the image as input and will 02:31 need to output such a textual 02:33 description of what is happening in the 02:36 image in this case the researchers used 02:38 clip to achieve this task if you are not 02:40 familiar with how clip works or why it's 02:42 so amazing i'd strongly invite you to 02:45 watch one of the many videos i made 02:47 covering it in short clip links images 02:49 to text by encoding both types of data 02:52 into one similar representation where 02:54 they can be compared this is just like 02:56 comparing movies with books using a 02:58 short summary of the piece given only 03:00 such a summary you can tell what's it 03:02 about and compare both but you have no 03:04 idea whether it's a movie or a book in 03:07 this case the movies are images and the 03:09 books are text descriptions then clip 03:11 creates its own summary to allow simple 03:14 comparisons between both pieces using 03:16 distance calculation on bit differences 03:19 you can already see how clips seems 03:21 perfect for this task but it requires a 03:23 bit more work to fit our needs here clip 03:26 will simply be used as a tool to compare 03:28 text inputs with images inputs so we 03:30 still need to generate such a text that 03:32 could potentially describe the image 03:34 instead of comparing the text to images 03:36 using clips encoding they will simply 03:39 encode the image using clips network and 03:41 use the generated encoded information as 03:44 a way to guide a future text generation 03:46 process using another model such a task 03:49 can be performed by any language model 03:51 like gpt3 which could improve their 03:53 results but the researchers opted for 03:54 its predecessor gpd2 a smaller and more 03:57 intuitive version of the powerful openai 04:00 model they are basically conditioning 04:02 the text generation from gpt2 using 04:04 clips encoding so clips model is already 04:07 trained and they also used a pre-trained 04:10 version of gpd2 that they will further 04:12 train using the clips encoding as a 04:14 guide to orient the text generation it's 04:17 not that simple since they still need to 04:19 adapt the clips encoding to a 04:21 representation that gpt2 can understand 04:23 but it isn't that complicated either it 04:25 will simply learn to transfer the clips 04:27 encoding into multiple vectors with the 04:30 same dimensions as a typical word 04:32 embedding this step of learning how to 04:34 match clips outputs to gpd2's inputs is 04:37 the step that will be thought during 04:39 training as both gpt2 and clip are 04:41 already trained and they are powerful 04:43 models to do their respective tasks so 04:45 you can see this as a third model called 04:47 a mapping network with the sole 04:49 responsibility of translating one's 04:51 language into the other which is still a 04:53 challenging task if you are curious 04:55 about the actual architecture of such a 04:57 mapping network they tried with both a 04:59 simple multi-layer perceptron or mlp and 05:02 a transformer architecture confirming 05:04 that the latter is more powerful to 05:06 learn a meticulous set of embeddings 05:08 that will be more appropriate for the 05:09 task when using powerful pre-trained 05:11 language models if you are not familiar 05:14 with transformers you should take five 05:15 minutes to watch the video i made 05:17 covering them as you will only more 05:19 often stumble upon this type of network 05:21 in the near future the model is very 05:23 simple and extremely powerful just 05:26 imagine having clip merge with gpt3 in 05:28 such a way we could use such a model to 05:30 describe movies automatically or create 05:32 better applications for blind and 05:34 visually impaired people that's 05:36 extremely exciting for real world 05:38 applications of course this was just an 05:40 overview of this new model and you can 05:42 find more detail about the 05:44 implementation in the paper linked in 05:46 the description below i hope you enjoyed 05:48 the video and if so please take a second 05:51 to share it with a friend that could 05:52 find this interesting it will mean a lot 05:55 and help this channel grow thank you for 05:57 watching and stay tuned for my next 05:59 video the last one of the year and quite 06:01 an exciting one 06:06 [Music]

New SOTA Image Captioning: ClipCap

Have you ever dreamed of taking the style of a picture, like this cool TikTok drawing style on the left, and applying it to a new picture of your choice? Well, I did, and it has never been easier to do. In fact, you can even achieve that from only text and can try it right now with this new method and their Google Colab notebook available for everyone (see references). Simply take a picture of the style you want to copy, enter the text you want to generate, and this algorithm will generate a new picture out of it! Just look back at the results above, such a big step forward! The results are extremely impressive, especially if you consider that they were made from a single line of text! If that sounds interesting, watch the video and learn more! Watch the video References ►Read the full article: ►CLIPDraw: Frans, K., Soros, L.B. and Witkowski, O., 2021. CLIPDraw: exploring text-to-drawing synthesis through language-image encoders. ►StyleCLIPDraw: Schaldenbrand, P., Liu, Z. and Oh, J., 2021. StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis. ►CLIPDraw Colab notebook: ►StyleCLIPDraw code: ►StyleCLIPDraw Colab notebook: ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/clipdraw/ https://arxiv.org/abs/2106.14843 https://arxiv.org/abs/2111.03133 https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb https://github.com/pschaldenbrand/StyleCLIPDraw https://colab.research.google.com/github/pschaldenbrand/StyleCLIPDraw/blob/master/Style_ClipDraw_1_0_Refactored.ipynb https://www.louisbouchard.ai/newsletter/ Video Transcript 00:00 have you ever dreamed of taking a 00:01 picture like this cool tick tock drawing 00:03 style and applying it to a new picture 00:06 of your choice well i did and it has 00:08 never been easier to do in fact you can 00:10 even achieve that from only text and you 00:13 can try it right now with this new 00:15 method and their google collab notebook 00:17 available for everyone simply take a 00:19 picture of the style you want to copy 00:21 enter the text you want to generate and 00:23 this algorithm will generate a new 00:25 picture out of it look at that such a 00:28 big step forward the results are 00:30 extremely impressive especially if you 00:31 consider that they were made from a 00:33 single line of text here i tried 00:35 imitating the same style with another 00:37 text input to be honest sometimes it may 00:40 look a bit all over the place especially 00:42 if you select a more complicated or 00:44 messy drawing style like this one 00:46 speaking of something messy if you are 00:47 like me and your model versioning and 00:49 resource tracking looks like this you 00:51 may be the perfect candidate to try the 00:53 sponsor of two days video which is none 00:55 other than weights and biases i always 00:57 assumed i could stack folders like this 00:59 and simply add old v1 v2 v3 and so on to 01:03 my file names without any problem until 01:06 i had to work with someone while it may 01:07 be easy for me to find my old tests it 01:10 was impossible to explain my thought 01:12 process behind this mess and was my 01:14 teammate's nightmare if you care about 01:15 your teammates and reproducibility don't 01:18 do like i did and give weights and 01:20 biases a shot no more notebooks or 01:22 results saved everywhere as it creates a 01:24 super friendly user dashboard for you 01:26 and your team to track your experiments 01:28 and it's super easy to set up and use 01:30 it's the first link in the description 01:32 and i promise within a month you will be 01:34 completely dependent 01:37 as we said this new model by peter 01:39 schaldenbrunn ethel called style clip 01:42 draw which is an improvement upon clip 01:44 draw by kevin franz at all takes an 01:46 image and takes as inputs and can 01:48 generate a new image based on your text 01:50 and following the style in the image so 01:52 the model has to both understand what's 01:54 in the text and the image to correctly 01:56 copy its style as you may suspect this 01:59 is incredibly challenging but we are 02:01 fortunate enough to have a lot of 02:02 researchers working on so many different 02:04 challenges like trying to link text with 02:07 images which is what clip can do quickly 02:10 clip is a model developed by openai that 02:12 can basically associate a line of text 02:14 with an image both the text and images 02:17 will be encoded similarly so that they 02:19 will be very close to each other in the 02:21 new space they are encoded in if they 02:23 both mean the same thing using clip the 02:25 researchers could understand the text 02:27 from the user input and generate an 02:29 image out of it if you are not familiar 02:31 with clip yet i would recommend watching 02:33 a video i made about it together with 02:35 dolly earlier this year but then how did 02:38 they apply a new style to it clip is 02:40 just linking existing images to texts it 02:43 cannot create a new image indeed we also 02:46 need something else to capture the style 02:48 of the image sent in both the textures 02:50 and shapes well the image generation 02:52 process is quite unique it won't simply 02:55 generate an image right away rather it 02:57 will draw on a canvas and get better and 02:59 better over time it will just draw 03:01 random lines at first and create an 03:03 initial image this new image is then 03:06 sent back to the algorithm and compared 03:08 with both the style image and the text 03:10 which will generate another version this 03:12 is one iteration at each iteration we 03:15 draw random curves again oriented by the 03:17 two losses we'll see in a second this 03:19 random process is quite cool since it 03:22 will allow each new test to look 03:24 different so using the same image and 03:26 same text as inputs you will end up with 03:29 different results that may look even 03:31 better here you can see a very important 03:33 step called image augmentation it will 03:35 basically create multiple variations of 03:38 the image and allow the model to 03:39 converge on results that look right to 03:42 humans and not simply on the right 03:44 numerical values for the machine this 03:46 simple process is repeated until we are 03:49 satisfied with the results so this whole 03:51 model learns on the fly over many 03:54 iterations optimizing two losses we see 03:56 here one for aligning the content of the 03:59 image with the text sent and the other 04:01 further style here you can see the first 04:03 lust is based on how close the clip 04:06 encodings are as we said earlier where 04:08 clip is basically judging the results 04:11 and its decision will orient the next 04:12 generation the second one is also very 04:15 simple we send both images into a 04:18 pre-trained convolutional neural network 04:20 like vgg which will encode the images 04:22 similarly to clip we then compare these 04:24 encodings to measure how close they are 04:26 to each other this will be our second 04:29 judge that will orient the next 04:30 generation as well this way using both 04:33 judges we can get closer to the text and 04:35 the wanted style at the same time in the 04:37 next generation if you are not familiar 04:39 with convolutional neural networks and 04:41 encodings i will strongly recommend 04:43 watching the video i made explaining 04:45 them in simple terms this iterative 04:47 process makes the model a bit slow to 04:49 generate a beautiful image but after a 04:51 few hundred iterations or in other words 04:53 after a few minutes you have your new 04:55 image and i promise it's worth the wait 04:58 it also means that it doesn't require 05:00 any other training which is pretty cool 05:02 now the interesting part you've been 05:04 waiting for indeed you can use it right 05:06 now for free or at least pretty cheaply 05:08 using the collab notebook linked in the 05:10 description below i had some problems 05:12 running it and i would recommend buying 05:14 the pro version of collab if you'd like 05:16 to play with it without any issues 05:19 otherwise feel free to ask me any 05:21 questions in the comments if you 05:22 encounter any problems i pretty much 05:24 went through all of them myself to use 05:27 it you simply run all cells like that 05:29 and that's it you can now enter a new 05:31 text for the generation or send a new 05:33 image for the style from a link and 05:35 voila now tweak the parameters and see 05:38 what you can do if you play with it 05:40 please send me the results on twitter 05:42 and tag me i'd love to see them as they 05:44 state in the paper the results will have 05:46 the same biases as the models they use 05:49 such as clip which you should consider 05:51 if you play with it of course this was a 05:53 simple overview of the paper and i 05:55 strongly invite you to read both clip 05:57 draw and style clip draw for more 05:58 technical details and try their collab 06:01 notebook both are linked in the 06:02 description below thank you once again 06:05 weights and biases for sponsoring this 06:07 video and huge thanks to you for 06:09 watching until the end i hope you 06:11 enjoyed this week's video let me know 06:13 what you think and how you will use this 06:15 new model 06:17 [Music]

StyleCLIPDraw: Text-to-Drawing Synthesis with Artistic Control

Researchers used AI to generate images. Then, they leveraged it to take an image and edit it following a specific style, like changing it into a cartoon character or transforming any face into a smiling face. This needed a lot of tweaking and model engineering and many trials and errors before achieving something realistic. There have been many advances in this field, mainly StyleGAN, which has the incredible ability to generate realistic images in pretty much any domain; real-life humans, cartoons, sketches, etc. StyleGAN is amazing, but it still needs quite a lot of work to make the results look as intended, which is why many people are trying to understand how these images are made, and especially how to control them. This is extremely complicated as the representation in which we edit the images is not human-friendly. Instead of being regular images with three dimensions, red, green, and blue, it is extremely dense in information and therefore contains hundreds of dimensions with information about all the features the image may contain. This is why understanding and localizing the features we want to change to generate a new version of the same image requires so much work. The keywords here are “of the same image.” The challenge is to edit only the wanted parts and keep everything else the same. If we change the colors of the eyes, we want all other facial features to stay the same. I recently covered various techniques where the researchers tried to make this control much easier for the user by using or quick sketches of what we want to achieve. only a few image examples Now, you can do that using only text! Learn more in the video… Watch the video References ►The full article: ►My Newsletter (A new AI application explained weekly to your emails!): ►Patashnik, Or, et al., (2021), "Styleclip: Text-driven manipulation of stylegan imagery.", ►Code (use with local GUI or colab notebook): ►Demo: ►OpenAI's Distill article for CLIP: Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, , 2021. https://www.louisbouchard.ai/styleclip/ https://www.louisbouchard.ai/newsletter/ https://arxiv.org/abs/2103.17249 https://github.com/orpatashnik/StyleCLIP https://colab.research.google.com/github/orpatashnik/StyleCLIP/blob/main/notebooks/StyleCLIP_global.ipynb https://distill.pub/2021/multimodal-neurons/ Video Transcript 00:00 Researchers used AI to generate images. 00:02 Then, they could leverage it to take an image and edit it following a specific style, like 00:07 changing it into a cartoon character or transforming any face into a smiling face. 00:12 This needed a lot of tweaking and model engineering and many trials and errors before achieving 00:18 something realistic. 00:19 There have been many advances in this field, mainly StyleGAN, which has the incredible 00:24 ability to generate realistic images in pretty much any domain; real-life humans, cartoons, 00:30 sketches, etc. 00:31 StyleGAN is amazing, but it still needs quite a lot of work to make the results look as 00:36 intended, which is why many people are trying to understand how these images are made, and 00:41 especially how to control them. 00:43 This is extremely complicated as the representation in which we edit the images is not human-friendly. 00:49 Instead of being regular images with three dimensions, red, greed, and blue, it is extremely 00:54 dense in information and therefore contains hundreds of dimensions with information about 00:59 all the features the image may contain. 01:02 This is why understanding and localizing the features we want to change to generate a new 01:06 version of the same image requires so much work. 01:09 The keywords here are "of the same image." 01:12 The challenge is to edit only the wanted parts and keep everything else the same. 01:17 If we change the colors of the eyes, we want all other facial features to stay the same. 01:21 I recently covered various techniques where the researchers tried to make this control 01:26 much easier for the user by using only a few image examples or quick sketches of what we 01:31 want to achieve. 01:32 Now, you can do that using only text. 01:35 In this new paper, Or Patashnik et al. created a model able to control the image generation 01:40 process through simple text input. 01:43 You can send it pretty much any face transformation and using StyleGAN and CLIP. 01:48 It will understand what you want and change it. 01:51 Then, you can tweak some parameters to have the best result possible, and it takes less 01:55 than a second. 01:56 photo: paper + video GUI gif example 01:57 I mentioned StyleGAN. 01:58 StyleGAN is NVIDIA's state-of-the-art GAN architecture for image synthesis or image 02:03 generation. 02:04 I made a lot of videos covering it in various applications that you should definitely watch 02:08 if you are not familiar with it. 02:10 Before entering in the details, the only thing left to cover is the other model I talked 02:15 about that StyleGAN is combined with, which is CLIP. 02:18 Quickly, CLIP is a powerful language to image model recently published by OpenAI. 02:24 As we will see, this model is the one in charge of controlling the modifications to the image 02:28 using only our image and text input. 02:32 It was trained on a lot of image-text pairs from the Web and can basically understand 02:36 what appears in an image. 02:38 Since CLIP was trained on such image-text pairs, it can efficiently match a text description 02:42 to an existing image. 02:44 Thus, we can use this same principle in our current model to orient the StyleGAN-generated 02:49 image to the desired text transformation. 02:52 You should read OpenAI's Distill article if you'd like to learn more about CLIP. 02:56 It is linked in the description below. 02:58 It has been used to search specific images on Unsplash from text input and other very 03:03 cool applications. 03:04 It will become very clear soon how CLIP can be useful in this case. 03:08 By the way, if you find this interesting, take a second to share the fun and send this 03:12 video to a friend. 03:14 It helps a lot! 03:15 As I said, the researchers used both these already trained models, StyleGAN and CLIP, 03:20 to make this happen. 03:21 Here's how... 03:22 It takes an input image, such as a human face in this case. 03:26 But it can also be a horse, a cat or a car... 03:29 Anything that you can find a StyleGAN model trained on such images with sufficient data. 03:34 Then, this image is encoded into a latent code using an encoder, just like this, here 03:40 called w. 03:41 This latent code is just a condensed representation of the image produced by a convolutional neural 03:47 network. 03:48 It contains the most useful information about the image which have been identified during 03:52 the training of the model. 03:54 If this is already too complicated, I'd strongly recommend pausing the video and watch the 03:58 short 1-minute video I made covering GANs where I explain how the encoder part typically 04:04 works. 04:05 This latent code, or new image representation, is then sent into three mapper networks that 04:11 are trained to manipulate the desired attributes of the image while preserving the other features. 04:16 Each of these networks is in charge of learning how to map a specific level of detail, from 04:21 coarse to fine, which is decided when extracting the information from the encoder at different 04:26 depths in the network, as I explained in my GAN video. 04:30 This way, they can manipulate general or fine features individually. 04:34 This is where the CLIP model is used to manipulate these mappings. 04:38 Because of the training, the mappings will learn to move accordingly to the text input 04:42 as the CLIP model understands the content of the images and encode the text in the same 04:48 way as the image is encoded. 04:49 Thus, CLIP can understand the translations made from a text to another, like "a neutral 04:55 face" to "a surprised face," and tell the mapping networks how to apply this same transformation 05:01 to the image mappings. 05:03 This transformation is the delta vector here that is controlled by CLIP and applies this 05:08 same relative translations and rotations to the latent code w, as what happened for the 05:13 text. 05:14 Then, this modified latent code is sent in the StyleGAN generator to create our transformed 05:20 image. 05:21 In summary, the CLIP model understands the changes happening in a sentence, like "a neutral 05:26 face" to "a surprised face," and they apply the same transformation to the encoded image 05:32 representation. 05:33 This new transformed latent code is then sent to the StyleGAN generator to generate the 05:37 new image. 05:38 And Voilà! 05:39 This is how you can send an image and change it based on a simple sentence with this new 05:44 model. 05:45 They also made a google colab and a local GUI to test it for yourself with any image 05:50 and play with it easily using sliders to control the modifications intuitively. 05:55 Of course, the code is available on GitHub as well. 05:59 If you found this interesting, please give this video a like, comment on what you think 06:03 about this research below, and share it with a friend. 06:06 It helps A LOT! 06:07 The only limitation for this is that you have to train the mapping networks, but they also 06:12 attacked this issue in their paper. 06:14 For a deeper understanding of how it works and to see these two other techniques they 06:18 introduced to control image generation with CLIP without any training needed. 06:22 I'd strongly recommend reading their paper. 06:25 It is worth the time! 06:26 All the links are in the description below. 06:28 Thank you for watching!

C9lab.com

c9lab.com

11-50 employees

Since 2022

COMPANY RANKING

C9LAB.COM

EVERGREEN INDEX #1853

C9lab.com WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

C9lab.com

c9lab.com

11-50 employees

Since 2022

COMPANY RANKING

C9LAB.COM

EVERGREEN INDEX #1853

RELATED COMPANIESRANK

STORIES ABOUT C9LAB.COM

C9lab.com WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!