Using Champion x Challenger in decision strategies

Published on: 2024-08-10 18:36:47

When you start using a new decision model for an existing decision strategy, the goal is to verify whether the new model or strategy performs better than the previous one.

A sound practice is to use a Champion x Challenger validation approach. The goal is to compare the performance of the Champion, which is the current decision model, against the Challenger, which is the new decision model.

Decisimo decision engine

Try our decision engine.

This approach is especially useful when the new decision model is not fully developed yet, and there is still uncertainty about its performance.

In high-impact decision strategies such as credit underwriting, Champion x Challenger should be standard practice. Testing and gradual rollout help keep portfolio performance stable.

Using a Champion x Challenger validation approach gives quick feedback on how the new model performs, and helps identify where it needs improvement.

How it can be done?

Here is a simple example of how the Champion x Challenger validation approach can be used.

Suppose we have a decision model that determines which offers to present to customers. The model has been in use for some time, and we now want to develop a new decision model that uses a different approach to offer selection.

As we develop the new decision model, we keep the historic baseline model as the Champion, which continues to evaluate most cases. The new decision model acts as the Challenger and is used to evaluate a holdout.

By holdout, we mean a sample, usually random, that is evaluated by the Challenger.

If we find that the new model outperforms the existing one, we can replace the existing model with the new one. The replacement can be gradual or done in one step. If the new model does not outperform the existing one, we can improve it further or continue using the existing model.

Advantage of Champion x Challenger

The main advantage of the Champion x Challenger validation approach is that it lets us compare the performance of the new model against the existing model or baseline without waiting for the new model to be fully developed.

Another advantage is that it shows the real-world impact of using the model for decision-making, because in-vitro simulations are often very different from live decisions.

Practical implementation in a decision engine

Most Champion x Challenger implementations use one random number or several random numbers. The random number is part of the data object inside the message used to execute a decision flow.

When only one part of the decision flow is being challenged, one random number may be enough.

When there are multiple challenger areas in the decision flow, multiple random numbers should be used. Each challenger area gets its own random number and runs with that number. This prevents possible cross-correlations.

At the end of execution, all decision results are stored together with the random number that was used.

Any model within a decision flow can be used for Challenger x Champion. This can be a new decision table segmentation, a single decision table row, one rule, or a whole rule set. You can also challenge the performance of different third-party data sources against each other.

When to especially use Champion x Challenger?

Champion x Challenger is especially useful when the outcome of decisions takes longer to unfold.

For example, in consumer lending, credit risk performance and default vintage shape can take half a year to show fully. In that case, switching blindly to a new model would make it hard to see whether vintage performance changed because of the model or because of external conditions.

By keeping Champion and Challenger vintages side by side, performance evaluation becomes easier. If sample sizes are statistically significant, there is less doubt about other factors.

Another example is a marketing campaign, where outcomes such as customer lifetime value can take longer to develop. In those cases, the decision may be designed well, but outlier performance and the propensity to take up additional services may still be unclear.

Conclusion

Champion x Challenger is a sound validation practice when developing and using new decision models. It allows us to compare the performance of the new model against the existing model or baseline quickly.

When using Champion x Challenger, we need to make sure sample sizes are large enough for statistical significance, and we need to account for possible selection bias.

Where possible, Champion x Challenger should be used in cases where decision outcomes take longer to unfold.

Decisimo decision engine

Try our decision engine.

Related Articles

  • Attribute and Model Management: How to Track Stability Without Weakening Your Decision Strategy

    Adding more attributes to a model does not always improve decision quality. If predictors are poorly grouped, weakly represented, or unstable over time, they can degrade model performance and create risk for the wider decision strategy. This article explains how to manage attributes and models with stability in mind, and why binning and categorizing predictors remains a practical way to keep automated decisions explainable, traceable, and reliable.

  • Building Anti-Fraud Competency Without Losing Control of Your Data

    Anti-fraud is not a product you install once and forget. It is an operating capability built from data, decision logic, traceability, and continuous tuning across the customer journey. Companies can buy parts of that stack, but if they outsource the core logic and enriched data, they often create a lock-in problem that is hard to unwind later.

  • The True Nature of Fraud and How to Build Anti-Fraud That Works

    Most anti-fraud programs focus on single bad applications, stolen identities, or suspicious transactions. That matters, but it misses the main point. The real fraud problem is not isolated opportunists. It is organized actors who treat fraud like a business, calculate return on investment, and target the weakest operators first.

  • How to integrate Maxmind minFraud Insights into a decision engine - Decisimo

    This guide shows how to integrate Maxmind minFraud Insights into your decision engine. It covers account setup, license key generation, HTTP authentication in the Decisimo portal, attribute mapping, and decision flow deployment. Follow these steps to call minFraud in parallel with other data sources, so your decision logic can evaluate IP data and risk scores quickly at runtime.

  • 3 minutes guide - Decisimo - Decision Intelligence Services

    This 3-minute guide shows how to connect Decisimo, configure decision logic, and test a release before it goes live. It walks through the setup in 3 steps, from API integration to production deployment, so you can start making decisions with a clear process.