Quality Sense Podcast: Andrey Momot – The “Holy Trinity” of Software Quality

Today’s guest is an inspiration for any test engineer who wants to drive their organization’s quality engineering and shift-left testing practices, working in a way that devs and testers can best collaborate. In this episode, get to know Andrey Momot, a QA Lead at , the company that pioneered the Digital Adoption Platform to simplify user experiences by combining insights, engagement, guidance, and automation capabilities. Quality Sense WalkMe Listen or read the transcript below to discover Andrey’s keys to success while working in an environment where developers execute manual testing and he owns the overall test and automation strategy. Episode Highlights: Collaborating with developers when you are the team’s sole QA How to sell the idea to developers that they manually test their own code The test automation solution that WalkMe has success with What Andrey calls the “holy trinity of quality” and how teams can achieve it Relevant Links: Connect with Andrey on Linkedin Listen Here: Quality Sense, a Software Testing Podcast · S3E7 – Andrey Momot – The “holy trinity” of software quality Listen on Soundcloud Listen on Spotify Listen on Apple Podcasts Episode Transcript: Federico: Hello, Andrey. How are you doing today? Andrey: Hi, Fede. I am great. Thank you so much for having me. I’m super excited. Federico: Thank you so much for accepting the invitation. It’s great to have you here on the show. So Andrey, can you tell me how you ended up working in software testing? Andrey: Sure. So before I started to work in the tech industry, I worked in retail. So I got to manage stores and interact with customers, very different than what I’m doing right now. And after several years, I was looking for a change. So I started my degree. I finished a degree in industrial engineering and at the end of my first year, I was hired for a support role at a small financial software company. After two years or so, I then got an amazing opportunity to work as the sole QA for a cybersecurity startup, which was a great experience for me and I was able to learn a lot and I never looked back, go quality! Federico: Let’s just start talking and understanding, what do you do at WalkMe? Andrey: Yeah, sure. So currently, I’m the QA domain lead of WalkMe’s all-in-one analytics platform, which is called Insights. So I’m in charge of all the quality processes and activities related to my product as the main offering, which is the Digital Adoption Platform, in short “DAP.” Basically our product helps users to use the technology faster and more efficiently, by shifting the burden of learning every new feature process and platform to our Digital Adoption Platform, to our product. And the product that I’m working on, basically we’re providing general user behavior and core analytics for WalkMe’s content. Federico: Amazing. I remember you told me that you work as the only tester in your team, right? Andrey: Yeah. So I’m the sole QA domain lead for Insights. Yeah. So I’m in charge of all the quality activities. Federico: How does it feel to be the only tester in the group? Andrey: Yeah. So a lot of responsibility, I will say that for sure. You definitely get to know every area of your product really well, so you need to be on point of everything at all times, but it’s of course challenging, but in a good way, I think, and of course I’m not reliant on other QA engineers. So I get to do things differently. Federico: This is one of the main topics I’d like to address today because I’m really interested in how to involve developers in testing activities. Andrey: Definitely. Federico: So how are developers in your team involved in testing activities today? Andrey: Yeah, sure. So in addition to writing unit tests, which I think all developers are doing and in charge of, our developers are also responsible for writing test plans and then manually executing them. I know, manual testing, developers, that never happens, but this is the approach that we took. Federico: Great. So can you tell us more or less the process that you follow for the different stages? When do they test? How do they prepare those test plans or test cases? I don’t know, what’s the approach and the process? Andrey: Definitely. Yeah. So each feature basically enters the design phase. Okay. So where crucial areas we’ll be taking into account, all the needed parts and changes will be documented of course, by the developer, in our knowledge base. So every feature basically has its own article. And then when we have new developers that are joining, they can basically read the architecture of all of our features, sort of the onboarding experience. So after that, the developer will write the test plan for the feature that he will be working on, which will be also included in the same article. After that, I will be reviewing it with the developer to make sure that all the necessary test cases and edge cases have been added. And so before the design is finalized, the developer sets a meeting with the team to present the design and the test plan and the whole store. And so if something will come up in the meeting, we can end that as well. And basically then he starts to develop the feature. After that, the feature enters the testing phase, which basically starts usually at the local environment in which the developer is working. Then he will continue further to our alpha and beta environments. So these environments are mimicking our production. Testing new features there will provide us with results that are pretty much similar if I was testing it on production, for example. So after that, after the testing phase is finished, we begin the acceptance testing, which is done by me and sometimes by our product managers as well. And if everything is okay, we’re deploying the PR to production. So each new PR is reviewed and approved by at least two developers. Then the code goes through our CI/CD, the unit tests, of course, in our automation suites. I would also like to mention that we are now deploying new PRs to production after 2PM, except for Sundays. So that we’ll be able to minimize the risk of potential regressions that we’ll have enough time to revert or fix the issues so that our customers are not affected. Testim.io Federico: Yeah, it’s a healthy practice. That’s great. So I understood that you are involved in a very early stage of the definition of features. Andrey: Definitely. Federico: So that’s great. And how is that interaction between you and the developers? So you say that you review the documentation and the test plan they prepare and what things come up in this type of conversation? What’s the typical feedback that you could provide? You have the testing activities even before starting development. Andrey: Sure. I think that the typical feedback that is always coming up are —a lot of times there are some UI tests that the developers forget about them as well as, and maybe data testing. So it depends on the feature. So we can have a feature that will integrate with a lot of other areas in the product and sometimes developers are not aware of that. So I put my architect hat on, and then I guide them through those missing areas and then we add the needed tests. Yeah. So basically it depends on the feature. Sometimes it can be nearly perfect and then I almost don’t have anything to say. Sometimes there will be some tests that are involving more UI. Federico: Yeah. I guess that the testing mindset is the way that you typically contribute in a discussion with a developer. And I really think this is very beneficial for both parties because probably this developer is going to understand and think in another way in the future when preparing new test plans. So that’s amazing. Andrey: Definitely. I do think that really sitting with a developer, not just reviewing the test plan, but also explaining the whole architecture of the product or the feature he’s working on and then looking for those missing areas can really put this whole picture together for him and for me as well, if there are things that I’m not aware of as well. Federico: Yeah, totally. And probably this helps you to do a better job when you are doing the acceptance testing at the end of the process? Andrey: Definitely. It’s a both way partnership. Federico: Yeah. Yeah, totally. Okay. And you also mentioned that there are two developers approving the PR, so do they do some peer review in the code? Andrey: Yeah. They’re reviewing the code. Yeah. They’re not testing the feature. They’re just doing a review to make sure that… We usually prefer that those two developers that are reviewing the code, the PR, will also be familiar with the area that was changed so that they can really truly give a good review for that situation. Federico: Okay. Andrey: Yeah. The main thought behind the acceptance testing is that I’m going over the new features like a user would do. Federico: Yeah. Andrey: Any user, which just now sees the feature for the first time. Although I’m definitely seeing that feature a lot of times before that, but yeah, that is definitely the main approach. Federico: Perfect. You also mentioned test automation. Can you delve into it a little bit? Andrey: Sure. I can talk about it for hours! Federico: Okay. Andrey: I think that for us definitely test automation is crucial. More than six months ago, we were using the Nightwatch.js as our automation framework. And our product of course still continues to grow, but at that point, our product was already big. And we started to encounter a lot of regressions and difficulties with maintaining fast releases. So eventually we’ve come to the conclusion that our safety net had holes in it. So we couldn’t really rely on Nightwatch. So then as the name states, Nightwatch, although I didn’t feel like someone was watching after me! Federico: Okay. Andrey: Because we had a lot of issues with regressions and slow deploys. So we immediately did a POC with Cypress, which went really well. And while we were doing that, our CEO, Dan introduced us to Testim.io and asked us to do a POC with that as well. So at first I was like Testim? I barely even know her, but really in a matter of a week, we had a robust suite for our production, which really showed us the tremendous potential we still have. As Katya mentioned in , they really take the whole aspect of the test. And you don’t even have to deal with that at all. her episode DevOps Federico: Yep. Andrey: It’s codeless for the listeners that are not familiar with Testim. So it’s codeless. It helps you to create a new test in a blink of an eye. And yeah, as I already mentioned, it’s really easy to use and it’s really fast. So for example, currently we are covering all of our environments with Testim. We have more than 60 active tests. We have a very high success rate and every run is completed in a matter of minutes. So for example, with Nightwatch, we had to wait at least half an hour for our deploy to be complete, not the whole deploy, just the part with Nightwatch, 30 minutes. Federico: Okay. Andrey: So it’s a crucial time that we could have used for something else. And I’m not even talking about the whole retries and failures. So definitely Testim.io for us. Automation as a whole is crucial and Testim really delivers the dream. I don’t want to sound like a commercial, but it’s true. It’s true. I could also add to that, that in the last six months, more than the 40 regressions were caught by Testim, which is incredible. Federico: It’s really nice to have these kinds of metrics in order to also, again, imagine to convince your management that this investment made sense, right? Andrey: When they saw that we could cover new features and a lot of areas in the product fast and with results, they were immediately convinced even before we had the stats about regressions and what I just mentioned. Federico: Great. And who is the person in charge of preparing the test scripts? Is each developer, or is it you, or how do you do it? Andrey: Right. So currently I’m the only one who is in charge of creating new tests for Testim. I’m the only one who is supporting it and changing it. And so the developers are not involved in that yet. We are thinking about helping them and guiding them with using it in their local environments so that they will be able to use Testim on local environments, because this is something that we’re still not doing yet. Although, as I mentioned, we are using Testim.io for all of our environments except for the devs local environment. Federico: Cool. Another question I have is, which was the original motivation for involving developers in these types of activities in testing? Andrey: Yeah. So the original background story for that is that our current Director of R&D, prior to WalkMe, he worked at AT&T where they involve their developers in manual testing. So when he joined us, he basically came with the same agenda. Prior to his arrival, I was complaining a lot about the quality of features that I had to test. So basically all the features that were arriving at my table were at a very low quality, most of them, not all of them. And so that was also one of the reasons. We said, okay, besides having developers manually test their features, they also need to care about the quality of their features. It’s not just about testing. It’s not just about saving costs. It’s also, as we talked before, helping them understand the whole picture. And basically, as I said, care about the quality because sometimes you can develop something and then let’s say you don’t care that much or you’re under pressure, so you don’t have time for that. So once you have a specific time for testing as well, the whole mindset changes. Federico: I’ve talked with some developers that have the idea, this mindset, that someone else is going to test and if it’s something broken there, they will have a bug report to fix it. But of course, this is not the most efficient way to collaborate, right?! Andrey: Definitely. As you just mentioned, in our industry, developers really rely on QA and they’re almost never testing anything they’re doing as a QA would do. So it’s sort of a given. You as a developer, you’re developing something and then you know that eventually it moves forward and if there are any issues, someone else will find them. So changing to this approach, definitely gives this, in my opinion, a better mindset regarding quality of features and the development process as a whole. Federico: So you mentioned that before taking this approach, you were complaining a lot, I guess now you’re very happy with the new approach. What about the developers? How did they feel about it when this new manager proposed this idea? Andrey: Yes. We can have a separate episode just for that question! Federico: I can imagine. Andrey: Yeah. So at first, not everyone was on board with this change. And it took us some time to really start this new process. I think that besides the upper management support, you also need to involve some other methods and you need to explain and convince and sell the idea to developers that besides them doing more work, it is beneficial for the team, for the product, for the company. So, yeah, I think that besides hitting the table and then saying, “Listen, you need to manually test.” You need to do more things on top of that. And as I mentioned before, it’s a partnership. 100%. You need them and developers need you. You know, when I’m saying them, I mean, QAs. And you definitely need to have a sort of personal approach. You need to guide them, you need to do it… At first, I did basically everything for them. I did the test plans and I scheduled the meetings and in some cases, I had to chase down people, to really sit with them and tell them, “Listen, this is the test plan. This is when we need to do X, Y, Z.” And of course behind every test case, there supposed to be a reason at first, at least for the beginning. So sometimes developers can look at a test plan and they can say, “Why should I test all the filter types? It’s very repetitive.” And then you need to tell them, “Listen, there might be a risk for regression, even though you didn’t touch the other filters.” Federico: Yeah. Andrey: In this example. So you need to definitely convince people. You need to guide them. It must be very personal. You cannot just write them on Slack, “Hey, this is a test plan,” or, “I need the test plan and send it back to me for review.” And then, “Bye.” No, it has to be as a partnership, as I already mentioned several times. Federico: Yeah. And also training and continuous learning for both. Andrey: Of course. Definitely for both sides. So me too, I will also learn new information from the developers, for my work process and for further things after this particular feature. Federico: I wonder, do you have something in mind that is the most important thing that you have learned from these interactions with developers? When collaborating with them, thinking about how to test something? Because I really believe that probably from our perspective, there are things that a developer can contribute, to open our eyes from something that we couldn’t see without their contribution, right? Andrey: Definitely. I agree. I think I can tell that from several occasions where I was reviewing a test plan, the developers were showing me their code. And while I was seeing the code and they were explaining things, I could definitely think about use cases that I forgot while I was reviewing, or test cases they forgot and they didn’t even think about. So definitely, as you said, it opens your mind for more possibilities when you’re seeing what’s underneath the hood of the car. Federico: Yeah. Andrey: So you press on gas, but you don’t know what is going on underneath. It definitely helps in the process. Federico: Yeah. Totally. Andrey: And I will also like to mention, for the whole convincing developers to manually test- “ Catching bugs in the early stages eliminates risks of potential support tickets. And we both know how developers love support tickets.” So, yeah. If we’re catching the bug early, great, immediately, we can say, we’ll not get a support ticket for that. So it’s another way of thinking about it when you’re trying to convince or sell the idea to- Federico: Persuade. Right. Andrey: Persuade. Yeah. Right. Definitely. When you’re trying to persuade your developers. Federico: Yeah. Totally. Talking about the benefits that they are going to receive by doing this, yeah. That’s a great, great- Andrey: 100%. Federico: Amazing. The question that I think everyone has after listening to all of what you’ve shared is, does it work? Would you recommend this approach to a friend? Andrey: I think that I will definitely recommend it to a friend, to a colleague, to anyone. I really think that involving developers and letting them be responsible for manual testing is the future. This is where we are moving forward, where we are going forward in the tech industry. I think that involving them in manual testing also eliminates the back and forth between QA and developers, which saves a lot of time. And in my opinion, the developing process is more efficient when new features are ready for acceptance, the quality is already there. And it is much better to release the product. You’re feeling much more comfortable than before. “ “ I truly think that with a strong automation infrastructure as a safety net, plus manual testing effort from developers and the guidance of a QA lead, teams can really achieve the ‘Holy Trinity’ of quality. Federico: Great. Andrey: It is really possible, I think. And regarding automation tools, you can have as many automated tools as possible. We have currently a lot of options and most of them are great, except for one thing. “You can buy automation, but you cannot buy mindset. So you can have all the tools in the world, but if you don’t have the right mindset, you probably will not reach the quality you desire for your product.” Federico: Cool. Andrey, I have a couple of final questions for you. One is, if you have any book to recommend our listeners to read? Andrey: Yeah. Sure. Okay. So I’m a big fan of Harlan Coben. I love his books. I recently read several books from his recent releases. So for our listeners who are looking for some drama and some thriller books, definitely I recommend . In my opinion, I think it’s an amazing thriller book. Netflix also has a series based on the book. So for those who maybe- The Stranger Federico: Prefer the film. Andrey: … Prefer to see the series and don’t like reading books, they definitely can watch the series. Federico: Okay, great. What about habits? Do you have any habits to recommend people to follow that could improve their productivity, for example? Andrey: Yeah, sure. I think that at this period of time, especially with the pandemic, mindfulness is very important. . We need to exercise our mind. So definitely meditations and if you’re having a stressful day, taking those 15 minutes or 30 minutes just to meditate and relax and to connect with your body and mind, I think it’s definitely crucial And yeah, I definitely believe in that. For me, from my own experience, I think it sometimes really relaxes me and then I can continue with my day, like nothing happened. Federico: Yeah. I think this is one of the silver linings of the pandemic that many people got into to meditation or to even yoga or things like this. And talking also for myself, I never tried before and this year was the first time and I felt that I really needed it and it made a difference. So thanks for sharing. Andrey: Yeah. Especially that most of us are in our homes, so we can basically do a lot of things while we are working. Federico: Yeah. Totally. Do you want to invite our listeners to do anything? Andrey: Yes. Sure. I would invite listeners to connect with me on . If you have any questions regarding automation or involving developers, with manual testing or testing as a whole, feel free to connect with me. I’ll be happy to hear stories and help more people in similar positions. LinkedIn Federico: You just gave us like a small view of all the things that are important in order to involve and collaborate in another way with developers. So I’m sure you have many other stories to share about it. So thank you so much Andrey! Andrey: Definitely. Federico: Have a nice day. Andrey: Thank you so much Fede. Federico: And see you soon. Bye-bye. Andrey: Bye-bye.