Jev AI Will Make You Money – But Exactly How Much?

Predictively Investing In Customers

Using a model is the only way it realizes value. Operationalize it, deploy it, launch it or put it to production. All different ways to say the same thing. You need to determine what the system will do when it receives a customer complaint such as:

“I am writing to report a significant problem with the centralized account management portal, which currently appears to be offline. This outage is blocking access to account settings, leading to substantial inconvenience. I have attempted to log in multiple times using different browsers and devices, but the issue persists.”

If this is for B2B, such as a payroll service, you could respond with a retention offer of, say, $250 off their subscription. That might pay off well, averaging $700 of net future earnings from customers correctly targeted and convinced to stick around.

Of course, the reason you’re using AI in the first place is that detecting (predicting) a customer’s intent to leave based on such a message is fuzzy. It’s a judgement call. It’s probabilistic. It’s a job for predictive AI.

Accordingly, judgement models such as Jev – and machine learning models in general – respond with a probability. That’s potentially useful – if you can determine how to best threshold on its probabilities. Do you invest the $250 discount when the probability of customer cancelation is above 80%? Or above 60%? Or what? Each wrongly targeted discount costs you $250 – the customer wasn’t going to leave and you gave them $250 off anyway.

Simply picking a threshold that “feels right” is a grave error. Why do all this data-driven AI work only to top it all off with a subjective “gut check”? It’s true that some thresholds might feel good – 90% seems pretty darn confident – but the business realities are situational and only by playing them out arithmetically can you drive the decision with true objectivity. Yet even interactive Jev explainers have you focusing on setting the confidence threshold without considering the resulting business value of the operations that would be driven on that basis.

Visualizing The Value Of Predictive AI

There’s a fundamental, straightforward view that’s missing from most Jev projects – and, in fact, from most predictive AI projects in general: A profit curve.

The potential earnings from retaining at-risk customers depends on how many you invest in – represented by the X-coordinate. Think of the most risky customers on the left, and least risky on the right. If you draw the line at 10%, that corresponds with offering a retention discount to the most risk 10% – according to Jev’s probabilities. The estimated net earnings at that position is about $4.5 million. This is calculated over 10,000 customer messages from a public synthetic testbed, scaled to an assumed 500,000 messages, taking into account that each correctly targeted offer provides a $700 average win, and each incorrectly targeted offer costs $250 in lost revenue.

More pain, more gain. If you and your colleagues are willing to be more aggressive, you can maximize the earnings by extending the offer to 17%, in which case you’ll profit an estimated $5.2 million. By delving further down the list, this means you’re providing the offer to less risky customers, which means more frequent incorrect targeting. If the bottom-line profit is the only consideration, that tradeoff is worth it. Indeed, the profit curve (more generally, value curve) of a predictive AI projects usually looks something like this. There’s a sweet-spot Goldilocks zone: Don’t treat too few and don’t treat too many.

In some circumstances, you may want to be even more aggressive. Say your near-term priority is growing the customer population. Retaining customers is often a much more cost-effective way to do so than acquiring brand new customers “off the street.” So, if you go all the way to 46%, the estimated profit is around zero. This would be the best approach for retaining the most customers at net zero cost.

A business metric – such as the financial win – must be the project’s driving force. Not accuracy. Predictive AI projects have, for decades now, measured predictive performance almost exclusively in terms of technical metrics that are at best poor proxies for business values, such as accuracy, precision, recall and a slew of others familiar to all data scientists. But if you aren’t measuring business value, your project can’t be pursuing business value – and, perhaps most importantly, your project can’t be selling the potential value to business stakeholders that must understand the value in order to authorize deployment. For that reason, most predictive AI projects fail to launch.

Uncertainty Can Be So Hard To Cope With

Why are business metrics so rare in predictive AI? After all, they’re obviously the originating motivator for most any project. The reason is, a business metric such as profit depends on business factors that are subject to change or uncertainty. In the simple example covered so far, each case’s cost/benefit is set at $250 and $700, respectively. But those are at best only solid estimates. A slightly more sophisticated approach that would better estimate the upside of retaining any given customer would vary each upside based on that customer’s history – that is, based on the expected value for retaining them specifically. Some customers are more valuable to retain than others.

Uncertainty brings up trust issues. Since the value calculation hinges on uncertain factors, it can be hard to trust. Some of your colleagues will inevitably express skepticism, and the potential value of operationalizing your predictive AI scheme may begin to feel obscured by uncertainty.

But we can’t let uncertainty derail these projects. Predictive AI is the very act of playing the odds better. It is the data-driven streamlining of organizational micro-decisions. If we hid from opening a can of “uncertainty” worms, we’d be stuck with the clearly sub-optimal practice of driving decisions by the gut.

The antidote to this uncertainty hurdle is to interactively explore the effects of business-factor “wiggle room.” For example, the figure below shows how things look if we assume the average win for rightly targeting a retention offer is $800 rather than $700.

At the previously-optimal point of targeting the 17% most risky, the profit jumps from $5.2 to $8 million. And yet, that would still miss some opportunity, since now the peak estimated profit would be $9.3 million, by targeting almost 30%. Within this “wiggle room” range, there’s uncertainty – but whichever setting you think is more precise, we’re compelled to deploy. We’re seeing viable business value rather than only presuming or hoping for it.

The Business Console For Predictive AI Projects

The profit curve – more generally, “value curve,” since profit isn’t always the main KPI – is only one piece of the business console needed for running a predictive AI project. Zooming out a bit, here’s some of the bigger picture:

There aren’t a million things to consider, but there are more than a couple. Now we’re comparing three different models – in this case, all variations on Jev usage:

• Normal Jev: Simply ask it, “Is this customer expressing an intent to cancel, request a refund or discontinue service?” – along with the customer messagre itself. This is the use of Jev covered so far in this article.
• Aggressive Jev: When asking Jev, also instruct it to predict “as if false negatives are 5 times as costly as false positives.” That gives it an incentive to correctly identify at-risk customers, even at the expense of more often falsely identifying no-risk customers. This is why there’s a bit of a hump at the far left for “Aggressive Jev.”
• Sensitive Jev: This was a failed experiment in which we asked Jev to classify in a way that favored the customer’s perspective by identifying interactions where, for example, the customers gave up on solving their problem out of fatigue.

In the zoomed-out console shown, some other options and considerations have also come into view. The business factors and their “wiggle room” are shown at the very bottom, where sliders allow you to change (“wiggle”) them. Changing their setting affects changes to the shapes of the three overlapping profit curves.

Finally, the top shows the value of three other business metrics besides profit: The number of customers given the offer, the number of times offers were rightly done and the number done wrongly. Each such value depends on where the decision boundary is set, namely, the horizontal position within the profit curve. As shown, there is a decision boundary, a vertical line on the profit curve – moving it right or left alters the values in the table accordingly.

You can access this console setup immediately here and try it out interactively, moving the sliders and decision boundary. Just find the “Jev” example on that webpage and click.

Taking predictive AI’s business value seriously means making the value explicit, and interacting with potential deployment options and their ramifications. Let’s get concrete with value and drive the use of Jev – and of all predictive models – according to value. With that, we’ll welcome a long-overdue industry make-over that predictive AI demands. Then we’ll have finally achieved the maturity and professionalization that this tech deserves.