"Python is eating the software development world right now"

Introduction Michael Kennedy is a successful entrepreneur and software development professional. He created and hosts the weekly podcast, Talk Python To Me, which focuses on Python and related software development topics. He also founded and is the chief author for Talk Python Training, an online Python training program. We had the opportunity to interview Michael about his programming experiences, and we have included the entire transcript below. Hope you enjoy it! The Interview We at Evrone have done custom development for many years, and we try to organize professional communities in Russia, like a community for Ruby developers, for Python developers. Your podcasts are famous in Russia. So, first of all, from the Russian Python community, thank you for your hard work. Evrone: And our first question is: If Python didn't exist, which language would you choose to work with instead? It's a good question. Python has been around for a really long time, 30+ years, which I think is surprising in itself. I came from a C++ background, that's where I did my first programming. I still have love for those C-style languages, and I've thought about what language I would choose if Python weren't there. I think it probably would be one of the C languages. Probably C#. I actually like C# a lot. I think it's a really pretty language. I like Swift as a language, but I don't find the ecosystem to be as nice. I don't find the base class libraries to be nearly as supportive and clean and so on. I've done a lot of JavaScript. JavaScript is not what I would want to do more of. I don't completely hate JavaScript, but it's not my favorite thing. So I'm going to go with C#. Especially now that it's cross-platform. But I'm going to stick with Python while I can. I don't think it's going anywhere. Michael: Yeah, Python is eating the software world right now. Evrone: It really is. Especially the data science side. Michael: If you take a look into the possible future, will there be programming approaches that do not require a programmer to know a specific language syntax and write text in that syntax? What do you think? Evrone: Every 10 years or so people say, “Oh, we're not going to have programmers anymore. We're not going to need to work in text editors anymore.” Sometimes, that's because of the no-code, “drag and drop” workflow types of solutions. Sometimes, it's because of outsourcing. And there was a big concern in the US that a lot of jobs would be outsourced to places like India. I don't think that came anywhere close to being a problem. We have more programming jobs all around the world than ever before. I don't think that there's going to be a big revolution around that kind of stuff. There are much better tools that allow you to build low code/no-code solutions. But you know, what I think actually is more interesting to think of these days is artificial intelligence. Michael: I'm sure you've seen GitHub Copilot, which lets you write a slight bit of documentation and then say “make that a program” and use Codex as the underlying ML engine. It goes to all the GitHub source code and it writes that code. I don't believe that is a current solution to these problems, but we've seen machine learning go so fast. If you look 10, 20 years out, you could potentially ask an editor with artificial intelligence to write a program, and it might actually do it. That said, I don't think that means there will be no programmers. I think there will be somebody who is going to have to verify that. Somebody is going to have to maintain it and evolve it. And I don't know if artificial intelligence will be so good that you can say, “I would like to now add this feature. Please rewrite my program.” Maybe, but probably not. I actually think these GPT-3 based artificial intelligence-type things have a much more interesting impact on the future, than, you know, a “drag and drop” workflow that has no code. Yeah, we as human beings like to communicate with each other, and we are pretty evolved to do so. You have your own Python podcast scraping script in your repository that allows everyone to copy all the audio recordings of the podcast. How do you feel about people copying your work? Evrone: Yeah. Anyone can download it. It's like five gigabytes. It'll take a little while, but not very long. I've been thinking about this a lot, actually. You’ve got to think about it in terms of the source code and project you put on GitHub and the content you create that you might put on a podcast or you might put on YouTube. And for me, it comes back to thinking about what my actual job is. What am I actually trying to accomplish? With the podcast, what I'm trying to accomplish is to get the message in front of as many people as possible. I want to share the details about Flask 2.0 or how Python is being used in astronomy or whatever the topic is. I want to get that out to as many places and people as possible. And to do that, I make the podcast available for free. It's ad-supported some of the time, but, you know, it's free to distribute to everyone. There's no cost. And I encourage people to download and share it. And that's fine for me, because I also have Python courses. Some are free, but most of them cost money. And that's fair, right? I want to make something that's available for free to the world and really try to support the community and encourage that, but then also create something that people would want to support more or learn more or get more from. They can also do that. Michael: Thank you. It's really nice that you are trying to help developers. Apple is one of the companies that tries to help developers, and they revolutionized the world with their new M1 chip. They are placing the Apple Neural Engine inside the chip and offering it to developers to use. Have you had an opportunity to try it out? What do you think about it? Evrone: I'm fascinated with Apple silicon and M1. And the fact that it comes with the Neural Engine, I think will open up a lot of very interesting possibilities. We think of using GPUs and machine learning as something that uses very large, expensive hardware, these huge machines that make a lot of noise. Michael: And if we can have them on the M1, or we can have them on our phones, it's going to make some really interesting things possible, especially edge computing. The artificial intelligence and the voice assistants and all those things joining locally. So I think it's a very positive move, and it's cool that you can do so much with machine learning in Python to, you know, take advantage of these things. That said, I've only used it as a consumer, not as a developer, but I have one of my favorite editing applications for doing all sorts of artwork, like logos and whatnot. It’s this thing called Pixelmator Pro. And it does a lot of its work using ML systems. Like, if you resize the image, instead of just resizing it with some math formula, you can use an ML engine and the underlying neural stuff to do that. I love it. Yeah, we will surely see some applications like Photoshop or Affinity Design, where image, audio, and all that data will be processed with this engine. Evrone: We can say that developers really enjoy, not only hearing success stories, but also hearing about not-so-success stories. So can you tell us about your career experiences when things went awry and how you dealt with it? I'm bouncing around some ideas on which story to share here. I guess the one I'll share has to do with a company that I worked for. We tried to build an online education platform based on an in-person training platform. We had thousands of students who would come and take courses, and then we said, well, let's build an online version of this. And it seemed really natural. We have thousands of customers. We're going to build some courses, and we are going to put them online. But really, it didn't go very well. We spent a lot, you know, six months, multiple people working on this project, and the uptake was small. Michael: And one of the things I really learned, what I took away from that, is that it's not just about software development. It's about, “How do you get the message out to the world?” It's so easy to dream about building the next best thing, right? You see, well, Instagram does this, but we could do so much better. Airbnb does that. But you know what? If we applied that idea to trucks, it would be amazing. Right? Most of us could build that software. But if nobody knows about it, nobody comes to experience it. It's really hard. And so I would say the biggest challenges I have experienced multiple times were not necessarily technical problems. It’s when our technical dreams meet the real world. And then you've got to go from there. So I've also had some interesting technical failures that I've had to deal with. But the big ones that stick out are these challenges of spending a lot of time and building something beautiful but not being able to get the word out about it. Yes, writing software is one thing. And letting the world know about it is a connected thing, because the software needs to help the world. It needs users. Evrone: Lots of junior developers are using advanced ideas like the Jetbrains PyCharm or Visual Studio code with Microsoft Python extension. And for new developers, how can we avoid turning the fun of programming into constant warnings and lots of our code being highlighted as incorrect? Yeah, that's the problem, right? There are a lot of challenges for beginners, and I think some of the challenges have to do with choosing the right size of the problem to work on. If you choose a problem that's too simple, it doesn't inspire you. If you choose a problem that's too hard, even though it might seem attainable, it's only going to be frustrating. So you have to find the right level, which I think will help because you won't be working with code that's too complex. But that is very, very vague. It's not very specific or helpful. What I would give as more helpful advice is: When you see one of those errors, don't just say, “Oh, it still runs, so I'll ignore it.” But take a moment and figure out what the error means and what it's trying to teach you. So, for example, a common error that PyCharm might show you is: This local variable shadows the global variable, and your code will still work with it, but it's going to show up as a warning. And you're like, well, I don't know what that means, I'll just keep going. But if you could stop and just say, “Oh, I realize if I just chose a different name and was thoughtful about it, I would not have this confusion in my code.” Michael: It needs a little bit of “I need to slow down.” Figure out what this means and do not try to just get to the answer, but try to constantly use these cycles of learning. The same thing happens if you apply linters to your program after you've been building it for six months. There'll be pages and pages of problems that you didn't know existed. If you see those from the beginning and just address one or two a day, it's fine. But if they build up to a thousand over six months, you don't want to stop for the next week and just fix those. That's not productive. So you'll be a better programmer, and you'll avoid the warnings, if you just take a moment to figure out what it's trying to tell you. Fix it and go on. And the benefit is that, especially with PyCharm, but also somewhat with the VSCode, it will not just tell you the problems. Sometimes it will offer to fix them. And if we talk about inspiration, finding interesting and non-standard solutions is why we developers love to program. What is your advice for developers? Where can they find the strength to overcome the routine and find interest, inspiration, and happiness in software development? Evrone: We're always chasing the new shiny library, the new shiny thing we can use to solve a problem. But, a lot of times, that's not what's needed. We just need to solve a problem. It's kind of the same, but it's different enough that we need to create something new for it. I think one of the things that we can do to help overcome these challenges, or these dull, repetitive things, is to use our creativity and our programming skills to figure out how to solve the problem once and for all. Michael: So, for example, every day you get a CSV file uploaded through FTP or something dreadful like that, and then you download it and you run it through the script and it gets uploaded there. Then it goes into the database and someone else does some sort of report on it. You can, you know, find a way to learn how to write a daemon that watches a directory for a file to be dropped and then automatically does that. And you can just think of these sort of “automate the boring stuff” types of things. While originally it might have been boring, if you can find a way to make it completely automatic, or almost entirely automatic, all of a sudden, every time it runs, you can just smile and go, “Yep. That used to be no fun, but look at it now.” And those problems will also help you grow as a software developer. Maybe you know how to load a CSV and save it to Postgres or whatever. But you don't know how to create a system daemon on Linux that runs continuously in Python. Well, that would be a cool thing to learn. So there's the new shiny thing you get to play with. But also, then you never, ever have to do that again. Yeah. As a developer, how often do you have to think about the exact service or serverless platform your application will run on? And is it reflected somehow in your code? Evrone: I will say not very often do I have to think about it. And to a small degree, yes, it is reflected in the code. Here's my philosophy on these cloud platforms. There's a really interesting tradeoff you have to make. If you go all in on what they call cloud-native, and you build apps that use every single service, you can build really amazing things. You know, if you use this hosted database and that hosted queuing service and that hosted email service, and you just go down the line of all these services, you can build some really interesting apps with two restrictions. Michael: One, you're forever stuck in that particular cloud. If you build against all the Amazon APIs and services, the ability to move to, let's say, Azure or Linode or something like that, it's almost like a complete rewrite. And so you don't want to do that necessarily if you think you might want to move around. The other drawback that I don't really love is: you can’t work if you have poor Internet or if you're offline. If you're on a plane, or maybe you're stuck at home from Covid and you want to go to a park and work at a park bench because you can't take being at home anymore, you won't be able to work on your app. So for me, what I do is I just run our services on a set of Linux virtual machines. And those can run in Digital Ocean, on Linode, in AWS. And then, separate from those, I use a couple of services that are very unique. For example, I have one cloud service for creating transcripts from audio. And it's completely integrated with those APIs. But that doesn't affect where the rest of my application runs. And I have one for basically video distribution around the world, and I integrate with that service. But that is not the entire story of my applications and my infrastructure. So that's my viewpoint. I sort of tend to have simpler types of things that can be portable to different places. It's worked out really well for me. You know, maybe I could do stuff on AWS Lambda or Azure functions and life would be easier, but I don't think so. Many of us think about what the large companies are doing, the incredible scale they are doing, and think that advice applies to us. There's a really good article called “You Are Not Google.” You're not Facebook; you're not LinkedIn. And it talks about, if you see what those companies are doing in cloud computing and other amazing things, they're doing that because they have hundreds of millions of users and they can have zero downtime. You might have two thousand users, and you can afford five seconds of downtime once a month. That doesn't sound like a big difference, but it makes a huge difference for the complexity of what you have to build. There is a popular opinion that Python is slow. Have you encountered such insufficient performance? And did you use some tweaks to speed it up? Evrone: I have experienced it, but I have not seen that it's a problem for me. I'm not doing data science or large-scale mathematical computations. I'm building APIs. I'm building web applications, things like that, or little automations. And Python has actually been really, really good for those situations. It's good at interacting with these other systems. And these other systems are the network and the database and third-party APIs, like Stripe or email services, and so on. In that world, it's quite fast. I would say the average response time from the network in to network out is something like 30 milliseconds. No one's going to perceive a difference between 30 milliseconds and 20 milliseconds. You know, maybe you could say you could have fewer servers if you could put less computation in there, but most of that time is actually from waiting on databases and other sorts of things, so you can't make it go that much faster. On the scientific side, I think it gets very interesting. If you do everything in pure Python, I agree with you, it probably would be quite slow. But so much of what happens on the data science side is: “I'm going to get this library. I'm going to get Tensorflow or something like that.” And those are really just very thin shells around C-code, and the C-code is very, very fast, so you might get the data in the C, and then you just poke at it from Python. In which case that's also fast. Michael: So the speed story of Python is very interesting. I do agree with it in something like Numba or Cython. I have used Cython for a couple of things. I actually heard of Django views in Cython, which sounded like a crazy idea to me. But they were doing a lot of calculations in their views, and doing it in Cython made it faster. That's the technical side. And then the other thing that you need to think about when people talk about slow versus fast is: “What is it that you're trying to make faster?” If I could write a program in C++ that will run and give me an answer to a problem in 10 seconds, and I can write a program in Python that’ll give me an answer in five minutes, well, the C++ is faster. But if it takes me a week to write that C++ code, and it takes me half a day to write the Python code, I've just saved a ton of time. So I think it's about what you are optimizing. Are you optimizing developer speed, product speed, computation speed? There are times when C++, or something like that, like Go or whatever, makes perfect sense, but it's not as often as people would imagine it would be. Are there any peculiar things around Python that you find to be weird or awful? What would you like to change about Python to make it better? Evrone: I am quite happy with the way Python is. I don’t have any really big things. I’ll give you a couple of things that I think are very important, and then one or two that are just weird language things. Michael: The “else” statement doesn’t make any sense to me. All other languages are perfectly capable of doing loop without having “else”, doing exception handling without an “else”. It’s not used very frequently. I think there are some interesting things around type annotations that we could probably do more with. Especially, if there are type annotations, we might be able to make math faster. For example, the interpreter could potentially look into the fact that there is a type annotation for a bunch of numbers, and we are going to employ C-level numbers, instead of much slower infinite-level numbers. Things like that would be nice. But you know it’s not a huge deal for me. Also, I would like to be able to push a button in PyCharm or VScode, or run a command line thing, and very easily get an executable file I can give to somebody. You don’t have to install libraries, so you can just take it and run. I know there is Pyinstaller and Py2exe, but these things are very fragile and they are not super easy to use. And if you could, just as easily as you press “run” in PyCharm or VScode, if you could just press “build” and out comes the thing, that would be great. Alongside that, I would love support for UIs for both desktop and mobile apps. I think those two things, the distribution and the user interface, if you could make those really good, think how popular Python would be. It’s already really popular without those, but you can’t build mobile apps with it. If you could, it would go crazy. So I’d love to see that happen. Thank you a lot, Michael! It was a pleasure to have this conversation with you, and I hope we will meet each other in person at some offline conference. Thank you and have a nice day! Evrone: The conclusion We appreciate the opportunity to chat with Michael and learn from his Python expertise and experiences. Gaining valuable insight from industry professionals into the programming languages, tools, and solutions that we use every day helps us to better serve our clients and meet their software needs. We also want to thank our colleague, Nikolai Rubanov from Selectel, for assisting with the questions for this interview. Previously published here. The author of the story is the Chief Editor at Evrone .