

๐จโ๐ป H2Oai ๐ CTDS.Show & CTDS.News ๐จโ๐ fast.ai ๐ฒ Kaggle 3x Expert
Index to โInterviews with MLย Heroesโ
Today, weโre talking to a very special โSoftware Guy, currently digging deep into GANsโโโโThe author of DeOldify: Jason Antic.
Jason is a CS Major and has been working as a Software Engineer for over 12 years. You might be surprised to know that he is still very new to DL, having taken it up seriously about nine months ago.
Weโll be talking about Jasonโs experience and the coolest GAN project thatโs live on a website: colorize.cc
About the Series:
I have very recently started making some progress with my Self-Taught Machine Learning Journey. But to be honest, it wouldnโt be possible at all without the amazing community online and the great people that have helped me.
In this Series of Blog Posts, I talk with people that have really inspired me and whom I look up to as my role-models.
The motivation behind doing this is that you might see some patterns and hopefully youโd be able to learn from the amazing people that I have had the chance to learn from.
Sanyam Bhutani:โ Hello Jason, Thank you for taking the time to do this.
Jason Antic: My pleasure!
Sanyam Bhutani:โโ You have created one of the most amazing โDeep Learningโ Projects, youโve been working as a Software Engineer for over a decade now.
Can you tell us how did you get interested in Deep Learning at first?
Jason Antic: Well really Iโve been interested in neural networks long before they were cool. They just seemed intuitively to have a lot more potential than other methods. But yeah, they were seriously uncool when you go back 7+ years ago. In fact, I took two artificial intelligence courses in college and neither of them covered neural networks. Years later, AlexNet swept the competition in the 2012 ImageNet challenge. That caught my attention because I knew computer vision basically just didnโt work before then. Comparatively speaking, Deep Learning was magic. I started devouring information on the subject afterwards but it wasnโt until years later that I actually started developing my own models.
Sanyam Bhutani:โ Youโve had a few failed starts to get started with DL at first, and finally took up fast.ai through till the end-what appealed to you the most about fast.ai?
Jason Antic: They simply have a much better method of teaching than other courses (I rage-quit quite a few popular MOOCs!). They start you with a huge amount of momentum right awayโโโyouโre creating an awesome dogs/cats classifier on the first day. THEN you drill down into the details. This is so motivating and more effective in terms of being able to connect the dots of what youโre learning. This is a method of teaching spelled out in a great book called โMaking Learning Whole Again.โ
Also, whatโs being taught literally is cutting edge. Uniquely so. In fact, in V3 part 1, I got to collaborate with Jeremy and Sylvain on lesson 7, and I committed the final notebook used for the GAN super-resolution bit a mere few hours before the course started. It was literally something we invented over the previous two weeksโโโGAN pretraining. I asked Jeremy if this was normal for him in preparing courses and he confirmed. Itโs mind-boggling that it actually works out but the end result is amazing!
Sanyam Bhutani:โ For the readers with a bit of less-tech background, could you share an ELI5 about what your project is and how it works?
Jason Antic: Sure! I basically created a deep learning model to colorize old black and white photos. While this isnโt the first deep learning model to colorize photos, it does a few new things that make it significantly better than previous efforts. Namely, the output is much more colorful and convincing, and this comes from setting up training of the colorizing model to involve a second modelโโโthe โcriticโโโโthat basically is there to โcriticizeโ the colorizations and teach the โgeneratorโ to produce better images. This design is called a GANโโโGenerative Adversarial Network.
Because the โcriticโ model is also a neural network, it can pick up on a lot of the nuances of what makes something look โrealisticโ that simpler methods just canโt. The key here also is that I as a programmer simply cannot comprehend how to explicitly code something to evaluate โrealismโโโโI just donโt know what all that entails. So thatโs what a neural network is here to learn for me!
Sanyam Bhutani: I think youโre the Thomas Edison of GAN(s) (or at least photo colorization using DL)-the idea didnโt work for quite a few weeks and you had quite a bit of unsuccessful experiments (over a 1000).
What made you stick to the project and not give up? How do you think can a software engineer stay motivated and not give in to the โimposter syndromeโ?
Jason Antic: Well that comparison to Edison is rather flattering! So, what made me stick to the project and not give up is this somewhat unreasonably optimistic view of mine that thereโs a solution to any reasonable problem and that itโs just a matter of effectively navigating the search space to find the answer. Effectively to me, that means doing a lot of experiments and being methodical, and to constantly question your assumptions and memory because thatโs typically where problem-solving goes wrong.
That being said, despite my undeniable successes I still to this day fall into that dark mental state of self-doubt, wanting to give up, and โimposter syndromeโ. Even earlier this week it started creeping up on me again when I was running into difficulties, and the intrusive thoughts started pouring in again. โYouโre deluded, and you were just lucky with DeOldify.โ Believe it or not that still happens.
Then I pushed through it and figured it out, and I am very excited about what will be released in the next month as of this writingย :)
How do I push through it? The belief that a solution is there and that Iโm capable of finding it simply because Iโm a normally functioning human being that can problem solve is a big one. Thatโs the key point hereโโโitโs not so much a matter of intelligence as it is of the method (and thatโs learnable!). Another motivating factor is the realization that there is in my mind no better way to spend my time then to try to solve big/cool problems, and that itโs worth the blood, sweat, and tears. Purpose and autonomy are huge motivators.
Sanyam Bhutani: There is a flipside to it as well, how does someone know when to quit a project that might just be too ambitious for the given technology?
Jason Antic: Yes, you definitely have to know when to quit, and thatโs quite the art. I say โNoโ to, and/or quit, a lot of things actually. Why? Because for everything you say โYesโ to, youโre saying โNoโ to many other things. Time (and sanity) is precious. As a result, especially lately, Iโve said โNoโ to quite a few opportunities that others find crazy to say โNoโ to.
So quitting for me is decided first on whether or not the path falls squarely in my realm of values, interests, and maintaining sanity. If youโre talking about an ambitious technological project, then you have to go one step further and evaluate whether or not itโs actually feasible. Often you simply canโt answer that right away, especially if youโre doing an ambitious project! So thatโs why you experiment. If you discover a sound reason (and not just a rage-quit rationalization) as to why something wonโt work, then quit, learn from it, and move on! But be careful on this pointโโโa lot of problems are solved handily simply by shifting perspective a bit. For this reason, Iโll stick with a problem a bit longer than seems reasonable, because often my perspective is simply wrong. Often the solution comes when I walk away (shower thoughts, basically).
Sanyam Bhutani:โ Itโs very interesting to note that Jason doesnโt have โexperienceโ with photo colorization. Yet heโs done a great job at it, even pushed a โChromatic optimizationโ update to the repository allowing DeOldify to run on GPU(s) with smaller memory.
What are your thoughts about โNon-Domainโ experts using DL to make progress in general?
Jason Antic: Itโs great that you brought up the โChromatic optimizationโ because that actually didnโt come from me. That was from somebody on Hacker News the day that DeOldify went viral that had at least some domain knowledge and suggested it as a way to get much better results much more efficiently. So I think this is an important pointโโโdomain expertise still counts. And good old-fashioned engineering and design still count. Not everything is going to be solved by your deep learning modelโโโat least as of now.
That being said, Iโve been able to get really far on DeOldify with zero domain expertise, and that idea generally excites me! I think weโve barely scratched the surface of the implications of thisโโโthat we can have a model discover things that not even experts know about yet. The challenge, of course, is figuring out what the models are figuring out and not just treat it as a black box. But I really think that weโre going to see some big breakthroughs in science in the next 10โ20 years because of this (and itโs already starting to happen to an extent). Additionally, not requiring domain expertise to be effective means many more minds can be put to the task of solving many of our worldโs problems. This is very cool.
Sanyam Bhutani:โ Your project is built on top of the fast.ai library (v 0.7), can you tell us how it has helped in the development of this project?
Jason Antic: The Fast.ai library is brilliant! It encodes a lot of best practices (such as learning rate finder) which make your life much easier as a deep learning developer. Itโs painful and silly to constantly have to operate at a low level and reinvent things (poorly) with things like Tensorflow or PyTorch. Fast.aiโs library does a lot of this for you, which means you can spend more time doing productive things. This is how progress in software generally happens, and I feel Fast.ai is leading the way. Do note too: Iโm about to push a Fast.ai V1 upgrade, and the benefits of doing this were hugeโโโspeed, memory usage, code simplification, and model quality all benefited from this largely out of the box.
Sanyam Bhutani:โ Iโve tried running DeOldify and it really really blew me away. Were there any images or scenarios that even made you go WOW?
Jason Antic: It might sound silly but the cup in the image below was my first wow moment. This was one of the first images I rendered after I finally experimented with a self-attention GAN based implementation, after failing to get this stuff to work with a Wasserstein GAN. Up to this point, I didnโt get anywhere close to this kind of detail, or interesting and seemingly appropriate colorfulness in my renders. I do acknowledge the flaws in this image (the zombie arm, for example). But I knew I was on to something after seeing this.
Sanyam Bhutani:โ DeOldify works and works really well, whatโs next for the project?
Youโve mentioned in your repository and twitter that you want to make a lot of under the hood improvements. Why do you think those are a priority over building something that will be potentially even cooler than this?
Jason Antic: I have no doubt that thereโs going to be a model in the next year or two thatโs going to blow my model away in terms of sophistication. Thatโs just the nature of progress! But what Iโm really interested in is making this stuff practically useful.
When I first released DeOldify, I was able to create some truly impressive images, but at a costโโโI had to search for quite a while just to find an image that didnโt have some sort of awful glitch or discoloration. GAN training is a double-edged sword currently in this senseโโโyou get really great results, but stability is really difficult to achieve. This is the sort of thing Iโm wrapping up on addressing now. This will make the difference between having a cool tech demo and actually being able to use the tech in the real world in a truly automated way. Itโll also enable something else thatโs way cool, but I canโt talk about that yet.ย ;)
The issue that DeOldify is currently a supreme memory hog is another thing Iโm attacking. Quite successfully Iโd add! That also will enable practical usability.
Once the items above are addressed and announced (very soon!), Iโll be looking to finally try making some money on this stuff. As you can imagine, countless friends and relatives have been asking โare you monetizing this?โ Iโll be able to finally allay their concerns about giving everything away for free LOL. And who knows what all that will involve. Just no VC money. Definitely not that.
Sanyam Bhutani:โ For the readers wondering about Jasonโs experience in Deep Learning, he had picked this project right after completing fast.ai Part 1 and 2. What pointers do you have for the enthusiasts that would like to build something as cool as DeOldify?
Jason Antic: Well, to be honest, DeOldifyโs origin story was that I had a bit of a shower thought while taking a long walk (a walk thought, I guess), where I was like โOhhhโฆGANs learn the loss function for you!โ Thatโs not actually an original thought but you donโt hear it much and I certainly hadnโt heard it at that point. Then I just paired this up with a list of projects I had compiled over time (again โwalk thoughtsโ) and figured: โLetโs try this with colorization because I have no clue what the loss function should actually be. So perhaps the GAN can learn it for me!โ
That was the intuition and I was unreasonably sure it was right. That gave me the stupid energy to spend the next six weeks doing probably upwards of 1000+ experiments (failure after failure) until I eventually stumbled upon self-attention GANs and it finally worked. And I have to emphasize that pointโโโit was a lot of failures, and a lot of unknowns, and a lot of not giving up even though I was taking a bit of a psychological beating after a while. You knowโโโself-doubt and all that jazz.
Hence my advice is this: Find something youโre interested in enough to pursue it in a manic way and guide your efforts by at least somewhat rigorous experimentation. And stay the course until you have the actual reason (evidence) to believe that what youโre pursuing is impossible as opposed to just unknown. I think this is where most people shoot themselves in the footโโโthey give up way too easily!
Sanyam Bhutani:โ Are there any upcoming updates in DeOldify that we should be really excited about?
Jason Antic: Thereโs a fast.ai v1 update coming very soon, and along with that comes the many benefits of the said upgrade! The model is going to be much faster, much smaller, higher quality, zero artifacts, and nearly zero โartistic selectionโ needed, and a large part of this comes simply from taking advantage of what is available in the new fast.ai code.
Thereโs other stuff you guys should be excited about that I just canโt talk about yet. Youโll hear about it soon or soonish. Iโm such a teaseย :P
Sanyam Bhutani:โ How can someone best contribute to DeOldify?
Are there any long-term goals that youโd like to work on with the project?
Jason Antic: Iโve been getting awesome key contributions from people in so many forms. For example, there are those who simply want something that I havenโt had time to produce yet. The Colab notebook is a great example of this. I loved that one because it made DeOldify way more accessible to a lot of people.
Another interesting key contribution was somebody on Hacker News telling me about the โChromatic optimizationโ idea they had which turned out to be the enabling factor in doing unlimited resolution image colorization. It wasnโt code, but it was so important and made DeOldify way better.
Then there are people doing awesome renders with them, sharing them, and even giving me feedback on the problems they have and the wishlist of updates theyโd like. Thatโs a great contribution too!
Contributions can come in many forms. I welcome them wholeheartedly. They make this โjobโ of mine so enjoyable and meaningful.
Sanyam Bhutani:โ Given the explosive growth rate of ML, How do you stay updated with the recent developments?
Jason Antic: Generally speaking, reading everything and knowing everything is simply not an option today. Thereโs no such thing as the โperson who knows everythingโ in the world anymore. Thomas Young apparently was the last person who had this title, and he lived two hundred years ago.
Even within just the field of machine learning, there are so many papers coming out and new information that you canโt possibly keep up. So what do you do? You filter intelligently. You choose good resources that do the hard work of distilling whatโs actually important and presenting it in a much more useful manner.
Fast.ai is an excellent example of this, and in fact, Jeremy told me that they like to think of their work as โrefactoring Deep Learningโ. We need that desperately, as there is a lot of noise to be separated from the actual signal at this point.
Itโs also much more efficient to let others in the community figure out for you whatโs great and then taking a look yourself. So follow some of your ML heroes on Twitter and see what they say! Theyโre tweeting about great papers and new developments all the time.
Sanyam Bhutani:โ What are your thoughts about Machine Learning as a field, do you think its overhyped?
Jason Antic: Speaking specifically to deep learning, I think Jeremy Howard said it best: โDeep Learning is Overhypedโฆis Overhyped.โ Simply put, Iโm strongly of the opinion that even if you just look at the capabilities of whatโs already available today, that weโve far underutilized the potential uses and problems solved with deep learning. And I think thatโs chiefly because thereโs probably not enough engineers and domain experts running with the tech yet to make this stuff work in the real world. I think people who say itโs overhyped either just like to be contrarian or lack imagination.
That being said, there are some seriously awful startups popping up here and there that make outrageous claims yet get still convinced somebody to fork over money. Itโs reminiscent of the dotcom boom in a way. But thatโs to be expectedโโโthereโs always going to be swindlers and theyโre always looking for new ways to take your money. AI and blockchain just happen to be the latest sexy things to achieve that. Keep in mindโโโthough there was a dotcom bust, the internet still really revolutionized the world. The same will happen with deep learningโโโthat much Iโm convinced of.
Sanyam Bhutani:โ Before we conclude, any tips for the beginners who aspire to go out and build amazing projects using DL but feel completely overwhelmed to even start?
Jason Antic: Go directly to fast.ai, and do the work! Great results take time and effort, and thereโs no good substitute for that.
Sanyam Bhutani:โ Thank you so much for doing this interview.
If you found this interesting and would like to be a part of My Learning Path, you can find me on Twitter here.
If youโre interested in reading about Deep Learning and Computer Vision news, you can checkout my newsletter here.
Create your free account to unlock your custom reading experience.