Tanso

tanso

Text-to-image generation is not a new idea. Notably, the (GAN) architecture, a once-popular deep-learning computer vision algorithm had . Generative Adversarial Network generated birds and flowers from text After more improvements to the generative image algorithms like and , we now have a deluge of commercial text to image generators, ranging from , and . hyperealistic generation of human faces GLIDE diffusion model Google’s Imagen Tiktok’s Greenscreen OpenAI Dall-E And here comes the new kid on the block, Stable Diffusion Before moving ahead, for those who wants to understand the nitty-gritties of diffusion models, I strongly encourage readers to go through this awesome blogpost by Lilian Weng, and . https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ this tweet by Tom Goldstein The rest of the article would be an exercise of using the stable diffusion model to generate images given “ ”. There is no main purpose for doing so, other than satisfying my curiosity. my name Narcissistic as it sounds, I want to generate an image with my name. Be honest, don’t we all from time to time? And thus, I went to and typed in . “google your own name” https://huggingface.co/spaces/stabilityai/stable-diffusion Liling Tan Before I show the generated results, I would like to clarify some specifics about my first and last name. In general, it is more probable that a English romanization of my Chinese character name refers to a female individual than a male. Next (陳) is a common Southern Chinese name that originates from the , and more commonly you would expect the English translation/romanization of Mandarin Chinese, . With K-Pop being a global phenomenon, the top-ranked Google result of would point you to the .  Also, since (my last name), has the same spelling as the color, the top Google search will end up with results pointing to “ ”. Liling Tan Min language Chen Chen Korean Singer from the K-pop band EXO Tan tan a yellowish-brown color For reference, here are the control experiments results, if I googled “liling tan” And now, the results… Okay, that’s definitely nowhere close to how I look. I kind of expected the image to show an Asian female but the first image was kinda weird. Now, what happens if I lowercase and re-run the generation? Alright, the model seems hell-bent on some facial features and sorta generating one older and another younger version of “ ” liling tan Interestingly, the model has some internal mechanisms to block some "unsafe” content and output this error message. This Image was not displayed because our detection model detected . Unsafe content Then I got really curious, do the two versions of “ ” exist IRL (in real life)? liling tan So I do the natural thing to reverse image search with https://lens.google.com/ Image Search Result 1 Image Search Result 2 Hmmm, no results, lets extend the frame to beyond the face… And of course, what was I expecting, surely Google Shopping will take the chance to advertise and sell me something -_- Maybe the older version generated by the model is more grounded to someone IRL? Image Search Result 3 Hmmm, no results, so the model kinda generated two unique people that doesn’t exist IRL? But, we see two separate dots that indicates two other results exists, let’s see the first one at the bottom. Of course, it will try to sell me something again… What was I expecting? @_@ Let’s try the results other dot. Now this is interesting, it’s trying to promote a inspired piece. Girl with a Pearl Earring (ca. 1665) by Johannes Vermeer But what about “Find image source”? Does it really find the source of the images that the model use to slice, dice and “diffuse”? It’s hard to say: Find the image source - Result 1 Find the image source - Result 2 Find the image source - Result 3 Đợi tí (Wait a minute),does that mean that we don’t know which image the model has been trained on or used to splice before generating the results? Now that we find the underbelly from the results, lets backtrack the OG paper listed on the original source code https://github.com/CompVis/stable-diffusion Robin Rombach*, Andreas Blattmann*, Dominik Lorenz, Patrick Esser, Björn Ommer. In CVPR '22. High-Resolution Image Synthesis with Latent Diffusion Models. According to the paper, the model is pre-trained on the LAION database and the Conceptual Captions dataset . https://laion.ai/blog/laion-5b/ https://ai.google.com/research/ConceptualCaptions/ Our model is pretrained on the LAION [73] database and finetuned on the Conceptual Captions [74] dataset. What if, we build a reverse image search based on the datasets? Perhaps if we can find the approximate nearest neighbor images to the outputs generated by the model, then it might be possible to “explain away” which images the model had deemed to be “salient” enough to diffuse and generate the outputs from my name. Voila! Here’s a search engine to probe the dataset used to train Stable Diffusion, https://waxy.org/2022/08/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator/ And here’s the list of . results of searching for Liling in the LAION database Ah ha, that is one image that the model must have “chosen to diffuse” It is highly possible that the model somehow and generated an image similar to that image. “ focused ” an image in the training data with “Liling” in the caption Why did you bother to “ ”? diffuse <your_name> To conclude the post, this exercise is purely out of curiosity. I wanted to know what the model would generate. But this exercise also highlights some sort of bias when using generative models. While “Liling Tan” isn’t to generate anything close to me or any other top-ranked “Liling Tan” search results, the model seems to be more stable in . stable enough generating  famous people, (e.g. John Oliver) To end this article, here’s a result from generating images with my online handle as the prompt… alvations Hopefully, after reading, you would also try to “ ”! diffuse <your_name>

Stable Diffusion, Unstable Me: Text-to-image Generation

Photo of from gwanghwamu pixelbay So yesterday I found this paper “ ” from and implemented . But then something hit me, Res Net and High Way net are built in a way that allows direction connection between the input data X and the transformed data X`. Dilated Recurrent Neural Networks NIPS 2017 here Why can’t we do the exact same for back propagation as well? Connect the gradient from previous layers to deeper layers….. I mean, if you are not using frameworks to perform auto differentiation, why don’t we connect the gradients from latest layer to more deeper layer, and just see how that goes? In this post, we’ll do exactly that, also lets go one step further and compare with a model that applies . Google Brain’s Gradient Noise Since I got the inspiration after reading the Dilated RNN, I’ll just call this Dilated Back Propagation, however if anyone knows other papers where they performed back propagation in this fashion, please let me know in the comment section. Also, I will assume you already have read my Blog post about Implementing Dilated RNN, if not please click here. Network Architecture (Feed Forward Direction) Side View Front View → Final Output of the Network, a (1*10) vector for One hot encoding of predicted number Red Circle → Hidden State 0 for Layer 2 → Hidden States for Layer 2 Brown Circle Lime Circle → Hidden State 0 for Layer 1 → Hidden State for Layer 1 Pink Circle Black Circle → Input for each Time Stamp (Please of this since I am going to use this knowledge to explain the Training / Test Data) Blue Numbers 1,2,3,4,5 TAKE NOTE → Direction of Feed Forward Operation, Pinkish? Arrow As seen above, the network architecture is exactly the same as the previous post. However there is one thing I changed and that is the input data for each time stamp. Training Data / Test Data → Input at Time Stamp 1 (Vectorized 14*14 Pixel Image) → Input at Time Stamp 2 (Vectorized 14*14 Pixel Image) → Input at Time Stamp 3 (Vectorized 14*14 Pixel Image) → Input at Time Stamp 4 (Vectorized 14*14 Pixel Image) → Input at Time Stamp 5 (Vectorized 14*14 Pixel Image) Pink Box Yellow Box Blue Box Purple Box Green Box Despite some images looking bigger then other, all of them are (14*14) pixel images. And each of the image are made by applying different kind of pooling operation to the original image which is (28*28) pixel image. Each of the pooling operations are described below. → Mean Pooling Using function → Variance Pooling Using function → Max Pooling Using function → Standard Deviation Pooling Using function → Median Pooling Using function Pink Box np.mean Yellow Box np.var Blue Box np.max Purple Box np.std Green Box np.median Below is the code of achieving this. And with that in mind, lets take a look at other training data. Finally, the reason I did this, is simply put, I wanted to. Image of 0 Image of 3 Case 1: Normal Back Propagation → Standard Direction of Gradient Flow Purple Arrow Above is the normal (or standard) back propagation, where we compute the gradient in each layer, pass them on to the next layer. And each of weights at different time stamps uses them to update their weights and the gradient flow continuous on. Case 2: Google Brain Gradient Noise → Standard Direction of Gradient Flow → Added Gradient Noise Purple Arrow Yellow Arrow Again, the purple arrow represent standard gradient flow, however this time before updating each weights we are going to add some Noise to the Gradient. Below is the screen shot of how we can achieve this. Case 3: Dilated Back Propagation → Standard Direction of Gradient Flow → Dilated Back Propagation where we pass on some portion of the gradient to the previous layers, which are not directly connected. Purple Arrow Black Arrows Now, here we introduce our new theory in hopes to improve the accuracy of the model. There are two things to note here. We are only going to pass on some portion of the gradient to the previous layers. As seen above, we have some variable called ‘decay proportion rate’ and we are going to use to decrease the amount of gradient it can pass on to the previous layers as time goes on. As seen in the Green Box, since we multiply the gradients from future layers with decay proportion rate, as training continuous the amount of Dilated Gradient flow decreases. inverse time decay rate 2. The Dilated Gradient Flow skips every 2 Layers. As seen above in the Red Box, the gradient at time stamp 5, ONLY goes to the gradient at time stamp 3. However this architecture can be further explored to make the gradient flow much more denser. Case 4: Dilated Back Propagation + Google Brain Gradient Noise → Standard Direction of Gradient Flow → Dilated Back Propagation where we pass on some portion of the gradient to the previous layers, which are not directly connected. → Added Gradient Noise Purple Arrow Black Arrows Yellow Arrow Here we are not only adding Gradient Noise to each weight update, but also making the Gradient Flow better. Training and Results (Google Colab, Local Setting) Above are results when running the code on Google Colab. The accuracy bar represents model’s correct guesses for 100 test images. Unfortunately I forgot to print out the exact accuracy rate but we can see that Case 2 (Google Brain Gradient Noise) had the highest accuracy. Also, cases with non standard back propagation performed better than standard back propagation. In the cost over time function, we can see that standard back propagation had the highest cost rate. Above are results when running the code on my local laptop. The accuracy bar represents model’s correct guesses for 100 test images. Here it was interesting to see Case 3(Dilated Back Propagation) under performing when compared to standard back propagation. However combination of Dilated Back propagation and Google Brain’s Gradient noise have out performed every model. Interactive Code I moved to Google Colab for Interactive codes! So you would need a google account to view the codes, also you can’t run read only scripts in Google Colab so make a copy on your play ground. Finally, I will never ask for permission to access your files on Google Drive, just FYI. Happy Coding! Please click here to access the interactive code. Final Words I love frameworks such as Tensorflow, and Keras. However, I strongly believe we need to explore more different ways to perform back propagation. If any errors are found, please email me at jae.duk.seo@gmail.com, if you wish to see the list of all of my writing please . view my website here Meanwhile follow me on my twitter , and visit , or my for more content. I also did comparison of Decoupled Neural Network are interested. here my website Youtube channel here if you Reference Chang, S., Zhang, Y., Han, W., Yu, M., Guo, X., Tan, W., … & Huang, T. S. (2017). Dilated recurrent neural networks. In (pp. 76–86). Advances in Neural Information Processing Systems Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., & Martens, J. (2015). Adding gradient noise improves learning for very deep networks. . arXiv preprint arXiv:1511.06807 Seo, J. D. (2018, February 14). Only Numpy: NIPS 2017 — Implementing Dilated Recurrent Neural Networks with Interactive Code. Retrieved February 15, 2018, from https://towardsdatascience.com/only-numpy-nips-2017-implementing-dilated-recurrent-neural-networks-with-interactive-code-e83abe8c9b27 Index. (n.d.). Retrieved February 15, 2018, from https://docs.scipy.org/doc/numpy/genindex.html [1]”tf.train.inverse_time_decay | TensorFlow”, , 2018. [Online]. Available: [Accessed: 16- Feb- 2018]. TensorFlow https://www.tensorflow.org/api_docs/python/tf/train/inverse_time_decay.

Only Numpy: Dilated Back Propagation and Google Brain’s Gradient Noise with Interactive Code

Quickstart Guide: TensorFlow Core APIs

How to Train a Linear Regression Model in TensorFlow

How to Define, Save, and Restore Models in TensorFlow

Як вызначыць, захаваць і аднаўліць мадэлі ў TensorFlow

Com definir, guardar i restaurar models en TensorFlow

Cómo definir, guardar y restaurar modelos en TensorFlow

Ինչպե՞ս ստանալ, պահել եւ վերադառնալ մոդելներ TensorFlow- ում

TensorFlowでモデルを定義、保存、および復元する方法

როგორ დააყენოთ, გადარჩენა და გადარჩენა მოდელები TensorFlow

Осылайша Франклин өз беделін қайтадан қайтарып, саяси аренаға қайта

उनले हाइड्रोसिल, हर्निया, एपेक्टोमी लगायतको सेवा समेत सुरु गरे ।

How to Define, Save, and Restore Models in TensorFlow இல் மாதிரிகளை அமைப்பது எப்படி

Stories, News & Mentions about TANSO

Categories

Trending Topics

Stories, News & Mentions about TANSO

Tanso’s Stories on HackerNoon

Categories

Trending Topics