How to Pick a Test Management Solution: 5 Mistakes to Avoid

There are many test management solutions, and they share essential functionality. They all offer test cases & scenarios, reporting, and integrations with other QA and/or software development tools. How do you choose one and not regret it later? Simple: go over less obvious features. For years, we have been seeking a perfect solution for our enterprise clients. The one that would unify testing in a huge team. Our clients needed to plan better, reuse test cases & resources, and automate testing. The critical requirement was the need to simplify preparation for regulatory audits because many of our clients are BFSI companies. At one point, we understood that it would be easier to create our own all-in-one test management solution taking into account the clients’ needs instead of juggling those tools that were on the market but couldn’t fulfill the needs of enterprise clients. The few available options at the time had too many drawbacks, so we had to walk this road ourselves. Our own test management tool was launched in 2013. Almost a decade later, we still talk to in-house quality assurance specialists, clients, and clients’ software testers to see what they miss in modern test management solutions. Below are just some of their musings. On-Premise vs Cloud Versions The model may not be your primary criterion, but it can become a deal-breaker. In most cases, you will be just fine running a test management solution . In fact, it is preferable for the following reasons: deployment on the cloud (and associated man-hour costs) lie with the solution provider. Upkeep and can be scaled depending on the project. Prices start low enables work-from-home, both as a policy and emergency Covid containment measure. Online availability Unfortunately, it’s the online availability that can be a problem. FinTech companies and banks, let alone government agencies can’t risk personal data getting leaked online. Sensitive industries have to use , which entails: on-premise software access (office-only or corporate VPN). Restricted for an enterprise license. Higher start fee Various maintenance options that . inevitably involve your IT resources to address your company’s unique needs. Tailored solution Large companies can often use cloud software for some projects yet be forced to on-premise products for others. Even if you’re not a large company (yet!), the safest option is to pick a test management solution that runs both on cloud and on-premise. Surprisingly, Jira + a test management plugin for it is not the answer: they are retiring the on-premise offering. From my experience, on-premise is a very much in-demand model. This year, we have had one On-Premise client for every two Cloud clients. Don’t take it as a dig on the competition: the distribution is a natural reflection of our client portfolio. For an evangelistic tool like Jira with a lot of SaaS companies as clients, maintaining the on-premise offering might indeed be unfeasible. Desktop Client vs Web The choice is about both preference and performance. Some people prefer to have everything they use at work as separate apps. After all, it makes sense to split tools where you actually work, talk about work, and show your progress. Desktop apps also have better offline functionality, although some web applications handle internet outages well. client vs web The desktop convenience, however, is a double-edged sword. Some test management solutions may , such as QTest lacking a Linux version. Desktop-only products can also frustrate management: you can’t simply use your phone to glance at unresolved issues or stability dashboards. lack versions for individual operating systems , both desktop and web apps can get things done at scale. Modern software architecture offloads as many calculations from the user as possible. When either cloud or on-premise server does the hard labor, web apps run virtually as well as desktop apps. Performance-wise On the other hand, (no matter how you load them) due to architectural compromises. Here’s a review of from a seasoned user. some solutions can slower become Polarion The test management tool of my company ( ) has both a desktop and web version and most clients prefer the web one. Desktop aqua adepts are power users of the few features that we have not moved to the web yet. The company’s workspace synchronizes changes done in both versions, so every employee is free to use the one they like better. aqua ALM Neglecting User Permissions Modern test case management solutions target more than just testers. You have the project manager who checks if things are going well. Sometimes, a product owner might get curious if any bugs would delay a new feature. Finally, you have software developers that need to know what bugs to fix first. The , and things only get harder when you run multiple projects. use scenarios go beyond basic “read/write permissions” Luckily, dedicated test case management solutions handle user permissions much better than general-purpose task boards. Advanced products come with more than just several presets of permissions (e.g. Developer, Tester, Manager). When the functionality is available at all, you can usually or even change them for each individual user. Here are just some scenarios: create roles with more granular permissions can see/comment on bug reports, but not edit them. Junior Project Manager can mark a bug as done, but only Testers can verify that it is fixed and archive the ticket. Senior Developer can access issues and tests from their projects only. Members of the Agile team If you would appreciate such control, run possible scenarios in your head before picking a tool. Here’s an abstract from a review on from a user who didn’t enjoy this particular aspect: Tuleap Apart from granularity, the . Some cloud solutions allow role-based access only if you buy a more expensive package ( charges at least 50% extra for that). Individual user permissions might be restricted to enterprise (see ) and/or on-premise licenses only. offers custom user permissions on all cloud and on-premise plans. difference in user permission often comes down to price Qase’s TestRail Aqua Underdeveloped Filters Now that you’ve decided what users can access, let’s see how they can access it. Even for a QA specialist, test management solutions give more information than one can handle or even need at the moment. This is an even bigger issue for managers. The common answer is functionality to and/or having a certain deadline. Such filters, however, usually lack flexibility and require too much clicking. Even when you usually cycle through similar sets of filters, you have to set them up every time. see only items assigned to somebody So, it would be great to have all items in one place, but also have a neat and fast way to slice through them. . Our solution was Views, which are advanced filters and presets of filters You can choose a number of filters that you would like to apply at the same time, save them as a View, and switch to it whenever you like. They save a few minutes every day, which adds up to hours and workdays. Views are also a powerful management tool to assess an individual’s workload and mid-sprint progress. Here are some examples: can see which Software Tester has the least tickets to assign them a reproduction of a recently reported critical bug. QA Lead can verify whether the Software Tester re-assigned items before leaving for vacation. QA Lead can see if there are any unresolved defects that caused a recent spike in the abandonment rate. Product Owner Last but not least, . QA Leads can set them up for testers to help them see their area of responsibility. Testers can share their views to speed up sprint planning and daily standups. Views can be a great collaborative tool You save time by creating views and increase transparency by sharing them — looks perfect to me! Subpar Logging In industries with mandatory external audits, software development and testing should be traceable. who deployed a new version of the software and how well it was tested beforehand. Regulatory compliances require that you know This functionality is often what separates test management solutions from regular task boards. On the surface, logging is a non-issue. Practically every cloud tool shows you the history of changes to individual items/tickets. This information helps even if you are not legally required to store it. But what happens if the cloud solution shuts down? How do you access the changes if a card was permanently deleted? Simple: you . In my case, aqua logs all actions of users. I have information on how all present and past employees interacted with my workspace, projects in it, and even individual items. decouple public change history from regulatory-compliant logging The record is unaffected by items and even entire projects removed from the workspace. , so it always displays a genuine timeline of changes. The log cannot be edited Now that you know the extra criteria for picking the right test management solution, it’s time to scout them. Good luck! First published and reposted with permission. here