Change Data Capture: Changing the Game

The new Salesforce feature Change Data Capture does not require a feature license and offers a low code resilient approach to keeping data in sync between two or more systems.

 

New in the Spring 19 release

The new “Change Data Capture” was in preview mode on the previous Salesforce release, but is now an official feature!  What is it? When should you/can you use it? Basically, it’s appropriate when you require integration with an external system but don’t have a middleware tool such as MuleSoft in place. But let’s take a deeper look:

 

Resiliency: It’s not enough to get the data from point A to point B

What do I mean by that? When you set out it might be tempting to code an Apex Trigger to respond to data events and call out to an external system and keep that system current with data when Salesforce is the system of record.  Don’t! The first roadblock here is that you cannot make callouts from the context of a trigger, but you can work around this by queuing up the work in a future method.  This approach might work but when the system is under pressure and limits are exceeded due to mass updates, your external data turns into an unknown state of ‘mostly correct’.  In order to make the integration resilient, two important features are required. First, the message or event must be durable because when limits are exceeded the events still need to fire.  Second, in the event of system maintenance or down time, the event must be persisted and replayed.  Change Data Capture retains these events for 3 days!  This is done in order to handle heavy pressure on the system Overflow events when thresholds are exceeded. 

There is an additional type of event called a Gap Event.  There may be non-transacted database events such as archiving of records or data cleanup jobs that do not trigger a normal Change Event, Gap Events alert you to record changes from these types of operations.

 

Salesforce takes care of the hard parts

When working with integrations, a common approach is to establish data synchronizations per data object or table.  The downside with this approach is that it is difficult to keep the related objects consistent.  A more code heavy approach is to traverse the data structure or object graph and find any related records that were also changed as part of the same transaction.  Reporting back this entire data structure can be quite large when only a few fields were modified.  Additionally, reporting back fields that are part of the data structure but didn’t actually change can lead to executing process logic when it’s not necessary.  For example, your external system may have a database trigger for when a book title changes. However this change within Salesforce didn’t actually update the title, it just updated that the product is active status.  Firing off logic like this when it is not necessary may create audit records that make it difficult to track what’s actually occurring as well as adding pressure to the system which may make things either run slower or time out.  Ideally the system only returns fields that have changes along with the relevant Id’s and timestamps along with the related records that have also been modified in the same transaction.  This is exactly how the Salesforce Change Data Capture feature works!  This is one of the more difficult areas to code correctly as well as thoroughly test, using this feature should reduce the complexity and difficulty of your next integration.

 

Middleware: That’s on the roadmap, but not in place today

Something that should be noted is that point to point integrations become unmanageable as the number of integrated systems and number of integrations grow.  Middleware such as MuleSoft and publishing / subscribing to Platform Events from Salesforce are often the recommended approach.  If your project involves standing up a Salesforce Org or extending one with Service Cloud and your enterprise does not already have a middleware tool in place, it may be cost prohibitive to include one in your project plan.  Standing up a middleware solution is typically a fair amount of work and not something you should try to squeeze into an existing project plan.  If you do have a Middleware platform in place you can still take advantage of the Change Data Capture feature to gain a transactional view of related records and only the fields that have actually changed.

 

Connectors: Connecting the other side

If this is a point to point integration, we can’t use middleware connectors.  Instead the client will need to subscribe to the channel that the Salesforce events are published on.  The Salesforce Streaming API is the gateway and comes with a code example written in Java using the Enterprise Messaging Platform (EMP) Connector: https://developer.salesforce.com/docs/atlas.en-us.api_streaming.meta/api_streaming/code_sample_java_client_intro.htm

Whether you are integrating with an Oracle E-Business Suite ERP system or another cloud based app this example should get you going in the right direction.

 

Getting Started: Salesforce Blog and Trailhead

I recommend going through the following Trailhead on Change Data Capture:

https://trailhead.salesforce.com/content/learn/modules/change-data-capture

And reading the following blog by Katia Hage from Salesforce:

  https://developer.salesforce.com/blogs/2018/08/what-is-change-data-capture.html

 

 

 

 

About Steve

Steve MunLeeuw is is Technical Architect at Statera.