How To Solve the Problem With Key Metrics In a B2B Product

To understand your business and where it’s heading, it’s crucial to choose the right metrics and to be able to see them all in context. By tracking and taking action in improving essential metrics you can add value to the product and position it for better performance. To learn how B2B companies solve the problem with key metrics in a product, I caught up with Yuri Brankovsky.

Yuri Brankovsky is Chief Product Officer at GetOutfit, a fast-growing online clothing selection service. Yuri has experience in different spheres: HRTech, GameDev, and FashionTech at such companies as Wargaming, VCV.ai, as well as participation in various acceleration programs.

Yuri, welcome! Can you tell us about your experience in B2B, how did you end up in this field?

Y.B.: First of all, I was interested in new technological products which do not just automate some processes, but change the whole market or at least a part of it. And a vendor I know told me that VCV.ai was looking for new people to invest in and expand internationally. VCV is a video interview service that allows candidates to record video responses to questions prepared in advance by the recruiter. This reduces recruitment time, saves manager's time interviewing irrelevant candidates, and allows an initial screening, including hard and soft skills.

And all of that is done online?

Y.B.: Yes, exactly. The candidate doesn't have to spend time going to the office for an interview, and the recruiter doesn't have to spend an hour interviewing a candidate who isn't really a good fit based on the first answers to the manager's questions.

At the same time, since you collect a lot of data about the candidate, can you create whole profiles of specialists?

Y.B.: Basically, this is one way of developing the service, but it wasn't the main strategy at the time.

Okay, what is the fundamental difference between B2B and B2C in terms of just determining the effectiveness of the product? I mean in both cases revenue is the main indicator, right?

Y.B.: Certainly, the key metric in a product will be money. But when we talk about the efficiency of product development, it is also important to understand how fast the product grows, why it grows, and which parts of it have more of an impact on growth and which have less of an impact. In B2C this happens quite unambiguously: if the product solves the user's problems, he pays for it. If he's satisfied with the quality, he extends his subscription and invites his friends.

Isn't it like that in B2B?

Y.B.: No. For example, recruiters and HR use video interviews, but the decision to buy the product is made by the HR director, who hasn't seen this product with his own eyes. It turns out that some people use it, and others buy it. And here it is not quite clear how to connect one with the other in terms of metrics. For example, a company may use the service rarely, but at the end of the year, it still has to renegotiate the contract. And, conversely, it can use it often during the trial period, but end up refusing to use it for a fee.

And you, as a vendor, can't influence that?

Y.B.: I would put it another way: this influence is not always unambiguous. After all, a company may end up renewing a contract because of the charisma of a manager or a sales director. Or because the company is simply uncomfortable with switching to another service or re-training the staff. Therefore, the analysis of product metrics becomes more complicated. In general, in some cases, it is not possible to determine what affects particular things. As a result, the product more often acts as an account manager who collects feedback from the client or a researcher who analyses the market to understand in what direction to develop the product.

Why is it inconvenient?

Y.B.: Because there aren't many of the product's hypothesis-testing tools that produce statistical results. This means that there is a higher probability that the hypothesis will not be claimed because the evaluation of its potential is subjective. For example, there may be 100 companies with a total of 200 people using the product. At the same time, the companies are from different areas, which does not allow for a uniform sample and a split test.

In addition, the service is not used regularly, in different ways. How does one know if a product is becoming more valuable or not? When the number of users is estimated in thousands, then we can talk about MAU, DAU, and Retention. And these figures correlate with financial metrics. In B2B Retention may not be high within one company, but in the end, the customer will pay. So it's hard to establish a correlation if there aren't many users.

So what did you do then?

Y.B.: We needed to find some kind of a metric and amplify it so that its changes would be more visible. That metric could be, for example, the number of sessions. But how do you understand the scale of changes if there are not many users in general? We opted for the time between sessions. That is, the more often the user uses the product as a whole or a separate feature, the better. And to make changes more noticeable, the metric can be presented on a logarithmic scale.

And what does that give you?

Y.B.: The point is that the linear scale shows the value of the value "as is". And if the changes are not so intensive (in our case due to a small number of users) and uneven (because the distribution of sessions is uneven since the use of the service by a small number of customers is more "impulsive"), and the rate of change in the metric relative to its minimum value is difficult to determine. At the same time, on a logarithmic scale, the graph is plotted against the increase in value.

And what does this mean from an analytical perspective?

Y.B.: Imagine a product is first used by a small number of people, but after a while the number of customers grows. On a linear scale, you would see the change in the metric as a sharp spike, as if the product wasn't growing before, and suddenly everyone "ran to use it". If you compare it to the beginning of the graph, it actually looks like that. Whereas on a logarithmic scale, this spike will be reduced because it accounts for the rate of growth of the index. In simple terms, the product used to grow too, even if there were fewer customers. As a result, instead of deciding that it "popped" after this release, you can draw more realistic conclusions. That is, the logarithmic scale kind of focuses on the rate of change of the function, not the increment.

That is, as you said, it allows you to amplify exactly the changes in the metric, so that the effect of this or that event is more obvious.

Y.B.: Exactly. And it is from this data that you can conclude whether a product is effective or not. The difficulty, however, is that in this case it is important to understand the essence of the logarithm, and for this, you have to readjust your thinking a bit, especially if you don't have a mathematical background. I didn't have one, and it took me a while to get used to this metric.

How did you come to it in the first place?

Y.B.: Thanks to networking. We didn't have our own analyst, and conventional metrics didn't work for us. So we turned to expert analysts for help. We told them about our goals and asked them to help us define the indicator. And as a result, we talked to analysts at a product conference and it turned out that this metric was the most suitable.

And what advice would you give to those who are now looking for metrics for their B2B product?

Y.B.: Well, first of all, it's important to understand that in B2B there are specifics: if a product solves an employee's problem in any way, they can continue to use it, even if the interface is very inconvenient. In B2C, the user would have already given up and switched to another product. Particularly because the cost of switching is lower. So regardless of which metric you choose, it's important to communicate with customers. Understand how they use the product, combine similar work scenarios, hypothesis improvements in those scenarios, then conduct UX tests and implement new functionality in the product. But use metrics more as an indicator of the value of the product. Compare how often it is used, why the time or frequency of use differs from company to company.

At the same time, do not forget that in addition to functionality that solves a particular problem, there are selling features that sound and look impressive when sold, but may be used only once a month. For example, the reports for the manager can be prepared once a month, or even once a quarter. So, analytics is rarely used, but if you don't have analytics for the executive, the product may not be purchased at all, that's the thing.

Yuri, thank you very much for the story. It was extremely interesting to hear about the experience of someone who has worked in different areas and segments, especially since the topic of analytics and data product management is very relevant today.

Y.B.: Glad to share my experience, I hope it will help someone developing a product.