How to Configure FM Scoring

Funnel Metrics supports a wide variety of scoring scenarios, which can result in different application behaviors and different consumption of CPU time.

Background

When a response is processed, it goes through a number of steps. For the purposes of this article, we’ll focus on three stages:

• Response Handling: This is where CampaignMembers are examined (typically during a before-trigger) to see if they are responses. Repeat responses are processed here as well. At the end of the stage, every response that is to be processed is set to a ‘Pending - Scoring’ status.
• Scoring: This is where the CampaignMember is evaluated to determine if it is a qualified response (typically during an after-trigger). At the end of this stage the CampaignMember is given a status of ‘Pending - TQR’
• Resolution: This is where qualified responses are disposed - they are given a response status and the lead or contact is updated. Sync occurs, notifications, assignments, etc.

There are three scoring modes. Synchronous, Asynchronous and Manual. In Synchronous scoring, all three stages occur at once. As a result, you will never actually see the CampaignMember have a ‘Pending - Scoring’ or ‘Pending - TQR’ status - the statuses are changed internally in the application but are never even written to the database. Because all three stages are being processed in a single execution context - the CPU load can be significant, especially with batch operations.

In the asynchronous scoring mode, the current execution ends after response handling - leaving the CampaignMember with the ‘Pending - Scoring’ status. An asynchronous request is placed and the scoring begins soon after in it’s own execution context. Because the scoring and resolution are in a separate execution context, the chances of CPU timeouts is reduced considerably, as not only is there no response handling (which uses CPU time), but async execution contexts have higher limits that those launched by triggers.

The manual scoring mode is almost identical to the asynchronous scoring mode. The only difference is that scoring is not started until some other logic outside of RM (such as customer defined Apex) sets the ‘Do Pending Scoring’ field on the Lead or Contact to true (it’s a checkbox), to tell RM that the response is ready to be scored.

Initially, the three scoring modes were intended to support the following scenarios:

• Synchronous scoring for cases where all of the scoring information is available during creation of the CampaignMember
• Asynchronous scoring for cases where customer logic would be able to set the scoring information during the creation of the CampaignMember - so the scoring information would be available by the time the async operation started.
• Manual scoring for cases where an external scoring engine or logic would perform scoring after CampaignMember creation, and be able to notify RM when the scores were ready. Most of the scenarios that were anticipated for manual scoring were rendered obsolete when score reactivation was implemented - so this mode if effectively never used.

Since then we have found that asynchronous scoring has value in addressing other issues - particularly around CPU time usage.

Note: Response prompt processing is always synchronous. This includes responses that are requested by the API to be treated as response prompts.

Configuring When RM Chooses Asynchronous Scoring

There are two types of scoring configuration. The default scoring mode, and scenarios where RM will dynamically choose asynchronous scoring.

Default Scoring Mode

The default scoring mode is configured on the RM scoring settings page.

If “Evaluate score immediately” is set, scoring will by default take place immediately.

If “Evaluate score asynchronously” is set and “Evaluate score immediately” is not set, scoring will by default take place asynchronously.

If neither checkbox is set, scoring will be manual.

User Based Async Scoring

If an external service like Marketo sends in a very large number of individual responses very quickly, and those responses are all on the same Campaign, there is a high likelihood of record locking errors occurring. This is because each of the individual responses is locking the related campaign, and if the requests come in faster than they can be processed, some of the requests will fail with a record lock error.

To address this we have implemented “User based async scoring” which is set in advanced configuration. This section allows you to define one or more users for special handling. For these users, if responses arrive in small batches (typically individual responses, but you can configured the threshold that defines the maximum number of records in a batch for it to be considered a small batch), those responses will be processed asynchronously after a set delay.

The reason for the delay is to allow RM to combine these asynchronous requests into larger batches. This dramatically reduces the chances of locking errors - as we can score batches of responses much more efficiently than individual responses. Also, we process those batches one at a time, which also reduces the chances of lock errors.

Batch Size Based Scoring

We’ve seen many case where RM works fine until a customer tries to upload a large batch of responses at once. This results in a large effort both on our part and the customer’s part to eliminate those CPU errors - usually by analyzing debug logs and identifying automation (workflows, processes) and other packages, that are consuming CPU time.

It (finally) occurred to us that it would be much easier to simply switch to asynchronous scoring for larger batches.

On the scoring configuration page there is a section called “Asynchronous Processing”

The scoring batch size defines the size of batches to be used during the scoring phase of asynchronous scoring. The evaluation batch size defines the size of batches to be used during the resolution phase of asynchronous scoring.

By default, both batch sizes will be 200.

Historically, these have only been used when already in asynchronous scoring mode. However, effective RM release 1.37.6, RM will automatically switch to asynchronous scoring when the size of a batch exceeds this number.

Instructions - Evaluating Scoring Configuration

The following approach is recommended when thinking about configuring scoring:

1. Choose the default scoring mode based on the business need.
2. If the customer is using an external service (still looking at you Marketo) that is configured to insert large numbers of CampaignMembers that are tied to the same campaign, or if they determine that lock errors are occurring in that scenario, configure the integration user for that service for user based async scoring. By default use a threshold of 1. Use the default timeout (5 minutes?) and adjust to a larger number if you still see lock errors.
3. If the customer is running into CPU errors when uploading CampaignMembers in batches, reduce the size of the scoring batch (make sure they are upgraded to at least 1.37.6).

For #2 and #3, it is strongly recommended that you test asynchronous scoring on a sandbox to ensure there are no unexpected side effects. To do this, on the sandbox change the default scoring mode to asynchronous. Test CampaignMember insertion - but not response prompts (since response prompts are always synchronous).

Remember, that in async scoring, the scoring will not occur right away. So you will likely need to refresh the record page (of the CampaignMember and lead or contact) to see the results.