Scalability and Large Data Volumes

Overview

Customers sometimes ask about Full Circle's scalability to support large Salesforce orgs with over 10 million records. Full Circle is highly optimized to scale. This article details what that scalability looks like in practice.

Large Data Volume Best Practices

Salesforce publishes a document on best practices for implementation of large data volumes. It can currently be found here.

It is especially important that Full Circle customers with large data sets adopt the best practices in the above-mentioned Salesforce document. Otherwise, problems that occur in their org may be falsely attributed to Full Circle applications. For example, if Full Circle updates a record and a resulting flow or trigger has a scalability issue, the problem may be attributed to our database update rather than the flawed code or automation. If this occurs, a debug log will need to be captured and analyzed to identify the issue.

Scalability Issues

With large data volumes, scalability is a set of related issues. Let's examine each one individually.

Query Selectivity

Once a Salesforce table exceeds 100,000 records, query selectivity rules come into play. These are discussed in the Salesforce documentation for Apex and for large volumes.

All queries in Full Circle applications are designed with selectivity requirements in mind.

Data Skew

The Salesforce database is a relational database. This means it is possible for a large number of records to reference a single record, e.g. Contacts to Accounts. In larger orgs, it is more likely for a very large number of records to reference a single record. This results in data skew.

Excessive data skew is problematic in two ways:

First, it can result in a large number of record locks and thus database lock errors. Full Circle applications are generally tolerant of these errors – detecting them and in most cases retrying the failed operation. Thus, when these problems occur it is highly unlikely that our applications are the root cause of the issue.

Second, there are certain common Apex design patterns that fail when excessive data skew occurs. These can take the form of heap errors or CPU timeouts. For example: code that attempts to update every contact on an account when the account is modified might start failing when there is skew on that account. All Full Circle applications are designed to avoid these types of design patterns, and where possible, to recover from errors resulting from data skew.

Throughput

Larger orgs are more likely to have larger amounts of data being processed at any given time. It is essential that these orgs process data in bulk (per Salesforce best practices). Integrations that edit or insert records one at a time can consume excessive resources (such as async requests) if they occur at a high rate. Our applications are optimized for bulk operations. We also have mechanisms to combine multiple async requests on orgs that have ill-behaved integrations. However, we cannot completely compensate for these scenarios.

We recommend any triggers and automation be modified to ignore DML updates performed by our applications. This is discussed here.

Bulk Updates on Large Volumes

Unsurprisingly, performing any type of data operation on every record in a table will take much longer on large tables than on small ones.

Outside of the initial launch period where certain applications perform bulk operations during or immediately after installation, our applications do not generally perform bulk updates or run batches on large tables.

Campaign attribution runs batches on accounts and opportunities – and its rebuild time is impacted by table size. We recommend avoiding rebuilds of closed opportunities on these orgs.

Matchmaker runs batches on accounts on request.

Digital Source Tracker runs batches on Campaign Members that occur on a single day.

While these batches will take more time in larger orgs, they are designed with Salesforce best practices in mind, and will not fail on these larger tables. They just take longer to run.

Summary

At Full Circle, we have years of experience working with customers with very large databases (> 10 million leads or contacts). As a result, we have been able to optimize our code not only based on best-practices theory, but based on real-world environments.

We are very confident in the ability of our applications to handle these larger orgs.