In today’s Quality Sense episode, Federico has a conversation all about DevOps and testing with Sumit Agarwal, the Lead Cloud Architect for a global fin-tech leader with over $4.5 billion in revenues that helps clients get ahead of today’s challenges. Listen to today’s episode where they touch upon the origin of DevOps, testing, dealing with legacy code, and making the necessary culture shifts to successfully implement modern software delivery practices.
Episode Highlights:
- The origins of DevOps and how testing relates to it.
- Challenges of implementing Scrum in larger enterprises and testing legacy systems.
- How testing can help with the mindset shift and cultural changes that need to take place for DevOps to thrive.
- How to generate team spirit and proper collaboration not only with developers but also with colleagues in other areas while working remotely.
Listen Here:
S3E6 – Sumit Agarwal – DevOps and testing
Listen on Soundcloud
Listen on Spotify
Listen on Apple Podcasts
Episode Transcript:
Federico:
Hello, Sumit, how are you doing today?
Sumit:
I’m very good. Hi, Federico. Nice to be here.
Federico:
Thank you so much for joining. I saw that you are also an organizer of different conferences around the topic of DevOps. Can you tell me a little bit about that?
Sumit:
Sure. So I’m involved with All Day DevOps from a culture transformation track perspective. Derek Weeks, who is the co-founder for All Day DevOps, actually credits the track to me. We were out for dinner while he was visiting London and we got talking and I presented at the first, All Day DevOps conference, and I just mentioned to him that I believe what was missing was a cultural transformation track given DevOps has such a heavy culture component. And he said to me, if I wanted to run it, I’ve got it, so I then hosted from the next year on the culture track. I think this year was the first time that I actually moderated other tracks instead of the culture track. But it’s 24 and it started long before we were hit by COVID and everything went virtual.
So it was the first 24-hour live virtual conference to get DevOps conferences, to people who didn’t have travel budgets or were in remote parts that didn’t have DevOps conferences. And then when I moved to New York after attending my first devopsdays New York, I actually got in touch with the organizing people and started working with that committee. So that was 2019, 2020 and COVID hit us right as we ended devopsdays New York, 2020. March 5th was when the conference ended and everything pretty much locked down. If it had been towards the end of that week, we would have probably had to cancel the conference.
Federico:
Oh, just in time.
Sumit:
Just in time, yeah. So, yeah, my love for the DevOps community started with me attending devopsdays in London. And then Andi Mann, who’s with Splunk, we got talking and he introduced me to All Day DevOps. So yeah, the community overall is fantastic at sharing and promoting good ideas.
Federico:
Yeah. One of the few positive aspects of this pandemic is that we have the possibility of attending conferences all over the world without extra budget for that, right?
Sumit:
Yeah, indeed.
Federico:
Yeah. And talking about DevOps and the culture around DevOps, what’s your view on how testing is related to DevOps?
Sumit:
So, I think the name just happened, DevOps the name just came about because of the history of it with a Birds of a Feather session, and then Patrick Debois setting up DevOps days Kent, and the conversation from there going into, how do we… And I think agile was already a thing. And so scrum, agile, were already things and so when they use the word dev, they kind of assumed we were talking about dev teams, which included testing and QA as a function within dev. And the conversation was about how we get agility to infrastructure and the operations aspect.
“So the name, DevOps, didn’t exclude testing, we’ve heard DevQAOps, DevSecOps, but that wasn’t the idea of it all. It assumed that QA was already part of that dev wording.“
And so, yeah. And then there’s the continuous delivery aspect that comes into it and I think you can’t do continuous delivery without that staying incorporated into it. The whole idea of DevOps as Gene Kim has talked about and everywhere else. Pipelines or automation on pipelines is an integral part of the DevOps culture, the DevOps mindset.
The whole idea is increasing feedback loops and shortening those feedback loops, the cycle time. And by the speed of delivery, you actually want to make it less risky and risk equals testing in some respect. So I believe in all of this, testing and risk reduction are at the heart of DevOps.
We want to have more confidence in what we’re deploying into any environment. So we want to move left on everything. So whether that’s unit testing or any other aspect, it should be incorporated into.
Federico:
I really like talking about the culture around DevOps, because I have seen that many people confuse the term DevOps with continuous delivery or automated pipelines or things like that, but what’s happened with the so-called manual testing or exploratory testing. How is that connected with DevOps?
Sumit:
The thing is, and as I was talking about being more comfortable with software releasing or releasing software, and we then start talking about, and Martin Fowler spoke about the testing pyramid, there’s so much that automated testing can do that there is still space for manual and exploratory testing. So what DevOps and automation don’t mean that you can replace that human aspect of it when something that automation can only cover so much, if what needs to be in a certain place, you could make it transparent on a screen and still have the automation pass.
You could probably check the colors, but how many permutations and combinations and then there’s the whole accessibility aspect of what if I’m color blind? And does everything match up? And yes, you could automate all of that, but the amount of time you would spend to automate that versus actually having a human test that manually or exploratory best thing, it doesn’t take away from that fact. And you still need somebody who has a testing mindset to come up with what needs to be tested and what should be tested automatically every time we’re putting it through the system versus, or through the pipeline versus what should be done once. Like I built it, I’m not going to be changing some of that, and that’s more of an exploratory thing. Yeah, I mean, again, UX and design, etcetera, you can’t automate tests. That is something usable that has to be very much a manual thing.
Federico:
Yeah. The mindset, I think this is the key here, is like having the mindset, and then with this testing mindset, you can decide which things you should automate in order to be more efficient and less error-prone or improving your processes. But also there are many things and aspects that you should continue doing enough in a manual way.
Sumit:
I mean, with the agile delivery, what are you changing more frequently and what is likely to break, you automate that piece, but what you don’t change and what isn’t adding value. If you end up spending 50 days writing an automated test for something that never changes you’d really be not adding value to your testing process. So there’s the space for manual testing on that front.
Federico:
You have a background working in a big financial company, I think when talking about agile, many people tend to think of small scrum teams or maybe start-ups. How is it different in a big corporation?
Sumit:
So I think with a big company there are multiple challenges. One, you’ve got a huge amount of scale and there’s the whole legacy aspect as well. We have systems that were written in the ’80s and ’90s, well before TDD was a thing or test automation was a thing or waterfall ways of delivery of those systems, and they continue to make a lot of money for the organization. So there’s been partial rewrites, there are components that are added with more modern technology, ways of delivery have changed on those things from waterfall to agile, but still the legacy aspect as, and there are various different definitions of legacy, some people call it legacy equals money-making, legacy equals non-tested code, so from that perspective, I’d call it a combination of all of those, that there may be bits and pieces that are tested from a regression testing perspective.
So we’ve got regression tests that have been written out, and those have been offshore to continue doing manually there are even things around trying to automate that in an end-to-end fashion. So there are aspects and we were talking about agility. So there are aspects of agility around, yes, I need to be able to deliver in two, three, four weeks sprints, but testing and the whole aspect of a dev equals dev and test… that might not be true in some big organizations because of the legacy aspect of the code.
Yeah, it just takes far too long, the code base is so big that it wouldn’t be possible to automate fully. And so there’s obviously lots of different initiatives in a big organization and with so many systems that govern how to get better at covering that regression. And I think the only way to do that is with an iterative approach as you write new codes and I think, one of the great books on that is Working Effectively with Legacy Code, which talks about how to introduce more automated testing into code that is non-tested.
Federico:
Yeah. And maybe by doing so you can adopt more agile practices.
Sumit:
Yeah. And another thing is sometimes with compliance and how your audit controls are written, et cetera, that you have it written in your compliance that you will have segregation of roles and duties, that you need that QC separate unit that certifies that something has been tested. And then you start getting those silos, which then introduce some of that repetition. So yes, I’ve got things covered with TDD or unit tests, but you still need to cover the whole thing because in your controls you’ve got it written that the whole working software is going to be tested by the QA team, because they can now trust what’s been done in code already with TDD. So, coming back to the culture aspect and what some of the greats have talked about in the DevOps field is how do you change these processes to make use of the automation?
So how do we talk to audit and rewrite those controls to say, we trust our pipeline and the dev team wouldn’t be to change or disable tests, because that’s what you’re trying to guarantee with some of these controls is yes, there’s a body that knows how to test. We’re going to take that, build that into the pipeline now. And as a result, we can now say, okay, it isn’t the function of a separate silo to verify the software because the pipeline itself ensures that once a test is introduced, it can’t be taken off. And then the pipeline’s insuring it and so you start building in that level of trust with the pipeline and all of that. So yeah, again, some of this comes from the added knowledge that all of the spaces add to DevOps, I think.
Federico:
So, as I understood, you’re proposing to automate some of the scenarios that maybe an audit could include these automated scenarios in that pipeline, so you can run them more frequently and have early feedback, and also working there in the communication between the different silos. So you can maybe avoid duplications in that course to verify these types of things.
Sumit:
Absolutely. And when I talked about the test pyramid that has been spoken about previously, you want to have lots of very quick running unit tests so that you get a lot of confidence in the code as it’s coming out. Early feedback.
So the developer checks in code, it runs tons of small little tests that run in memory without saying, for example, using a relational database. And then as you move up the stack, the tests start becoming slower and probably more fragile. So you want to do less of those, and as a result, you don’t want that duplication, if something’s already been verified at the unit test level, then why do them at the integration test. You don’t need to be doing all the different scenarios at the higher layers, you want to do a very minimum of those. And that’s all kind of cultural as well as how do you change that in a big enterprise where you’ve got these controls? So those controls need to be rewritten with automation and everything in play.
Federico:
Yeah. And many people involved, I think that this makes the challenge even harder because there are many people involved in all the processes. I find it fascinating that you can try to modify the culture of a company through technology or through the technical aspects of how you’re working with automation.
Sumit:
Yeah. And there have been some debates on Twitter that I’ve followed, which is all about, is DevOps about tools? Or is it about culture? And we don’t need a tool, we need a culture or the other way around. And I think a couple of those conversations, people were just debating for the sake of debating, they never thought it was one or the other. And people taking stands on the side of those just to make a point and that’s why it has to be the trifecta of people, process, and technology. So technology is the tool.
So you have to change mindsets in big enterprises, and when I say code living on for 30, 40 years, there are different mindsets. There are people who’ve grown up doing it a certain way and you are asking, and they’re probably on the verge of retirement at some time and you’re asking them to now do things very, very different from what they know and they’ve grown old doing that. So you need to change mindsets, you need to change processes because they’re written in, you need to comply with what you’ve set. That’s an audit requirement, the government comes after you, the clients come after you, the authorities come after you if you don’t follow what you put down, so there’s a lot of work on that side. And then yes, the technology has moved on to help make those changes possible, like we couldn’t have done that before, but we can now because the technology makes it possible.
Federico:
We have to work in both aspects at the same time, in the technology and the mindset of the people involved in all the process.
Sumit:
Yeah. And so, one of the things that we talk about DevOps engineers or automation engineers, and that’s one thing. But one of the definitions, I think I picked it up from Puppet, was:
“A DevOps engineer is somebody who has the ability to actually affect change.”
And one of the first DevOps meetups that I went to, I think it was a Java meetup, but the talk was about DevOps. And when the presenter was showing all the books that he recommends, I think there was a whole slide dedicated to change management-related books, because DevOps from that perspective is also about changed, and testing is about change.
So it is all about, we want to be constantly bringing change into our software, into our organization, and one of the other things there is, how do you continuously bring about change? Because that’s what we’re trying to do with software delivery, it’s continuous delivery of new features, bug fixes, etcetera. There’s continuous testing to make that possible, but we also then need to keep looking at our processes to keep adjusting and improving from that point.
Federico:
Yeah, you know that at Abstracta we offer software testing services. But, we’ve been working a lot of times with companies, trying to help them adopt an agile culture or a DevOps culture from our activities in software testing. Because I really believe that from this perspective, you can have some influence in different areas to help with the feedback loops or so many different aspects that can contribute to this culture and mindset aspect.
Sumit:
Yeah. And while we’re talking about that, for example, without the cloud, it would be very difficult to spin up test environments at will. Can you do that on a private cloud? Do you have the capacity to just be able to spin up? Let’s say 20 environments, 50 environments to just very, very quickly run your tests. But then adopting the cloud can be a very process-oriented change for an organization because they’re not ok with putting their data onto the cloud. And then things like, can I do copy backs from production into my lower environments? Do I have to do masking? Again, we could have technology help with that because again, could help spin that masking workload very, very quickly versus previously you just didn’t have the ability to one, bring up so many environments or be able to have that compute available to mask because if you’ve got really large databases, you can’t.
So I think it’s all of that putting it together, but you also then have training effort because how many of your people truly, if they’ve been working on legacy stuff and in the data centers, how do you get them on up to speed? So that’s one of the very big challenges. I mean, even training people in the testing mindset. So if I’ve always been a manual tester that basically executes these scripts that have been written out manually on those systems, now asking them to do automated regression tests or become an SDET, software developer in testing.
There’s a big learning curve because you’re completely changing the skillset. How to test a system interactively is very different from testing a system programmatically, and you have to start making architectural changes to the software to allow for that testing to happen as well.
Federico:
Yeah. Many changes in the tools that you use, in the activities that you do, but also in the way you interact with your peers and areas of your organization, right?
Sumit:
Yeah. I mean, we spoke for business before, and one of the challenges is I’ve been approached by enough companies that are offering testing as a service. And what I liked about getting in touch with you was that you were actually affecting change in how to think about testing from a modern agile perspective.
So if I just want manual testers in an offshore location, there’s plenty of body shops out there that’ll do that work, but really what’s needed to advance software delivery with a DevOps mindset has to do with adopting the change and the culture around. How do I improve the feedback loops?
And I think in the State of DevOps report, Nicole Forsgren and others called out that large scale functional outsourcing causes problems, or is correlated with lower delivery speeds. So, should you be wholesale outsourcing QA, or should you be bringing in more from a consulting perspective, bringing in people to help improve your ways of doing that?
So, it’s interesting. Large scale, again, from a large company’s perspective, you’ve got plenty of things to think about from how much of this is going to get modernized, or do I need to start saving costs and kind of more expensive locations by shifting network across the globe, the world’s flat. So how do you?
Federico:
So for me, this is a very interesting topic because mainly nowadays with the pandemic we are all working and collaborating from different parts of the world, or we don’t need to be seated in the same room to foster collaboration. So the thing here is how to generate that team spirit and proper collaboration, not only with developers, also with testers, with people working in support, with people working in different areas of the organization, this is a great challenge, and solving this, I think we can make a huge impact.
Sumit:
Yeah, that’s another topic that’s very close to my heart is around organizations and the social-technical systems that make up an organization and collaboration for global teams and getting to know people at a personal level, because in a small team, I mean, are we using the chat and collaboration systems like Slack and teams… just for work, but are you really bringing your whole self to work? And then if we’re going to be part of a team that is remote and virtual, for example, how do you make that personal connection? Because there’s so much more to a person than the working life. Like if they’re having a challenge at home, and they’re not fully present, as a team, it’s impacting the team’s productivity. And that’s another side of it that we got to think about is how to bring the team closer together while still being virtual.
There’s some great stuff from, I was actually recently revisiting Gitlabs’ remote working culture documents there that published them openly across. They actually talk about making informal communication formal. So the whole idea is that if you’re in an office and you go grab a coffee, you’ll bump into somebody, the conversation’s just going to happen. You don’t need to be very formal about setting up that informal communication because you’ll bump into people, you’ll meet people, you’ll meet new people within an organization.
But, when you’re fully virtual, those opportunities don’t automatically come up. So you have to be very intentional about creating that space for that informal communication. Like do you set up meetings for a happy hour, or coffee time and organizing a lunch so that you can have informal communication, but making it more intentional. For example, we spent months trying to coordinate this, but we intentionally did, so yes.
Federico:
So we already have the topic for our next conversation! But I agree that this is a very interesting topic and we could discuss different related aspects. And it’s very connected with the DevOps culture because collaboration is key and motivation is key. Trying to wrap up this conversation today, I have a couple of questions.
One, you already mentioned some books and some interesting material that I will add to the episode notes, but do you have any other book to recommend to our listeners?
Sumit:
So my latest favorite that I’ve done one very quick read of, but I need to read over and over again, is Sooner, Safer, Happier, and we’ve talked about it today, a lot about agility. And the book tries to focus on how to be nimble and agile without using agile as a big, A, Agile, and a big T, Transformation. So it summarizes a lot of DevOps ideas into the book around agility. So Sooner Safer Happier.
Federico:
Perfect. Great. Thank you. What about habits? Do you have any habits that you will suggest to form in order to improve productivity or your happiness at work?
Sumit:
So I think it’s a journey for me, but one of the things for me within the DevOps space has been reading. I end up reading and listening to podcasts, etcetera. So when I used to drive to work, because I didn’t get a chance to read as much, so I would listen to stuff more. And with the pandemic, all of it has ground to a halt a little bit, but I still try to read as much as possible.
Federico:
Yeah, there are many things to learn from other people, always.
Sumit:
Yeah. The thing is I do Twitter a bit and there are people I follow from the DevOps community, there’s always new content and new thoughts. Also, one of the things I’d recommend is if you are in testing, for example, or an agile scrum master type person, try to broaden the reach there because I feel… When I look at conferences, DevOps, so people assume automation engineers, and Ops, but even from an organizing perspective is what I don’t see is as many people coming from the testing community or looking at the agile community, and it’s very important to widen that scope to look at other communities and what’s happening in those. So bringing in testing people, or looking at content from those other fields is quite important.
Federico:
Yeah, totally. And do you want to invite our listeners to do something, reach out or whatever?
Sumit:
You can reach out to me on Twitter it’s @aga_sumit, on Twitter and that’s probably the most, from a social media perspective where I’d like to talk. LinkedIn becomes too formal.
Federico:
Okay, perfect. Sumit, thank you so much for all the conversation and all the knowledge and ideas that you share. I really appreciate it.
Sumit:
Thank you very much Federico. It’s been a pleasure.
Federico:
See you. Bye-bye.
Sumit:
Take care.
Also published here.
