Salesforce CDC (Change Data Capture): The Ultimate Guide

In today's era, the domain of data reigns supreme. With data being generated every second, its ever-changing nature is a key factor.

In a time-sensitive environment, it becomes crucial for businesses to capture every bit of available information. Advanced technologies and software are already commanding the attention of organizations to meet the escalating demands of data-driven work.

Since data can potentially make or break a business, it is of monumental importance to obtain real-time, nearly accurate data at your fingertips. Fortunately, there is a fast-paced solution - Change Data Capture.

What Is Change Data Capture?

Change Data Capture (CDC) is a software solution that identifies and captures the minuscule changes occurring in a database. Each and every change is tracked as well as processed. This can occur in batches, with new information being added to the repository during the defined period of capturing data changes.

This continuous process allows for updated data to be integrated into the database system of the organization as and when the shifts in data happen. Change Data Capture is hailed as an efficient process for data warehouses that greatly reduces downtime while using real time data ensuring that information present in databases is not obsolete.

Adopting CDC has become pivotal in today's business landscape to acquire accurate data almost instantaneously accessing change data capture events.

Salesforce Overview

Customer Relationship Management (CRM) is an invaluable function of every business. However, each customer interaction can lead to vast amounts of customer data often being generated. To streamline the process of correctly managing customer interactions and data, businesses use CRM software.

Salesforce is one of the leading cloud-based CRM software globally. With the advent of Salesforce, you are able to engage meaningfully with your customers. You can consolidate all your data from various sources and use Salesforce's comprehensive tools to produce innovative solutions constantly.

Salesforce is highly scalable, helping you grow your business by leaps and bounds. Here, you are provided a unified view of every customer's journey. This can be put to use to extend the lifetime value of your customers, identify ways to constantly improve your offerings, and reduce business costs without compromising on your growth strategies. 

Why Implement CDC in Salesforce?

Cloud CRM systems, such as Salesforce, are increasingly favored by organizations to address business challenges effectively. However, integrating these cloud CRMs with internal data management systems poses a common obstacle.

Challenges of Cloud CRM Integration:

• Many organizations encounter difficulties when integrating cloud CRMs like Salesforce with internal data management systems.
• The reporting module necessitates synchronization between the CRM and data warehouse for real-time insights.

Benefits of Change Data Capture (CDC):

• Implementing change data capture (CDC) with Salesforce enhances data integrity and facilitates insights.
• Amid business growth, maintaining Salesforce and capturing new insights can be challenging, but CDC ensures long-term data integrity.

Key Stages of CDC Salesforce Data Replication:

• Initial Dataset Replication: The entire dataset is replicated in an external system as the first batch.
• Ongoing Synchronization: Newly added or updated data is continuously synchronized with the system, including new records and modifications.
• Duplicate Record Management: Managing and rectifying record duplication in the database is crucial. Utilizing a robust data integration system like Airbyte ensures corresponding updates are executed efficiently.

How to set up Salesforce With Airbyte?

Now that you have understood how beneficial it is to set up change data capture in your Salesforce, you must also know what is the best way to go about it. With every new thing, there comes a set procedure that you must follow. 

It may be daunting to perform all the steps in salesforce events, especially when you have limited experience in customer relationship management software systems. To ease your transition with Salesforce CDC, Airbyte brings to you a fairly easy guide that you can follow for setting up platform events in your Salesforce account.

Prerequisites: Before diving into the setup process, ensure you meet the following prerequisites:

• Have a Salesforce account with Enterprise access or a purchased API quota.
• Consider setting up a read-only user specifically for Airbyte to ensure data integrity.

Creating a Read-Only User for Airbyte:

If you are wondering how to create a steadfast read-only user for Airbyte, take a look at the simple steps laid out below:

• To commence operations, it is important to be logged in first. Start by logging into your Salesforce with an admin account.
• You will notice a gear icon located on the top right of your screen. There, you need to click on the Setup option.
• Your current screen will have a Navigation bar on the left side. Choose the Administration tab, where you have to click on Users, then Profiles.
• A Profile page will appear. If you are a new user, simply click on New Profile.
• Under the Users Tab, click on New Users. You will be required to fill out certain fields.
• Ensure that you select Salesforce for the License field and Airbyte Read Only User for the Profile.

If you happen to be an existing user, you must follow the former three steps laid out above. Post that, take a look at the steps laid out below:

• Select the option of Read Only. Next, enter Read Only Airbyte User and save this for your Salesforce.
• On the Profile page, you must click on the Edit option.
• You will have to scroll down until you find Standard Objects Permission and Custom Objects Permission. This is a crucial step, for you have to click on the Read checkbox. This gives Airbyte the permission to make a copy of all data objects that require updation.
• Simply checking the box is not enough. Do not forget to go to the top and save everything.

It should be good for you to know that the Salesforce connector in Airbyte allows you to read both Standard Objects as well as Custom Objects. With Airbyte, you can import all previously available objects, and custom object which are then read and treated individually.

Step 1: Configure the Source

Once you have completed the above steps, you need to log in to your Airbyte Cloud account. Under the Sources tab, you must select and configure the Salesforce connector.

Step 2: Configure the Destination

Now, you may choose the destination as per your requirements. Once done, you can establish the destination connector for the Salesforce source. 

Step 3: Configure the Connection

Create a connection between your data source and destination and configure the replication frequencies as you desire. This will sync your data with Salesforce CDC.

How to set up and use Salesforce CDC manually?

Salesforce, one of the most popular cloud-based CRM systems, is utilized by various businesses worldwide. However, a challenge with this system lies in its setup and integration with a company's internal systems and reporting modules. Change Data Capture (CDC) serves as the implementation architecture facilitating real-time data synchronization between two such systems. Salesforce's streaming events API enables support for Change Data Capture.

Step 1: Enable CDC in Salesforce 

Navigate to the "Change Data Capture" section within the setup tab to configure Change Data Capture on an object. Within this section, you can find the desired object for which Change Data Capture needs to be enabled. This includes both standard and custom objects. For this example, let's work with a custom object named "product_item," which comprises two fields: "product_name" and "product_description." The developer selects the relevant table from the list and adds it to the entities chosen for Change Data Capture.

Step 2: Subscribe to the event channel

To complete the process, we'll need to subscribe to the event channel. Salesforce provides an intuitive Java tool called the EMP Connector for this purpose. It's capable of printing out event messages and subscribing to events. You can obtain the tool by cloning the GitHub repository from the following link: git clone https://github.com/forcedotcom/EMP-Connector.git.

Step 3: Use the below command to build the jar

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/Employee__ChangeEvent

Step 4: Access Salesforce Events

You can access Salesforce events by subscribing to a particular URL. Developers can use the /data/ChangeEvents URL to access events for all objects. The URL format for specific objects will be as follows by default.

‍

/data/<Object_Name>ChangeEvent

For accessing change data events on custom objects the below format is to be used.

/data/<Object_Name>__ChangeEvent

In our case, since our object product_item is a custom object, we will use the below URL

‍

/data/product_item__ChangeEvent

Step 5:  Subscribe to Events

To subscribe to events in the provided URL, open a shell and use the following command along with the JAR file.

‍

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/product_item__ChangeEvent

‍

The user credentials for this sign-in are the username and password. An additional authentication option is to use an OAuth-based token. Please keep the command window open for the above shell to take additional actions.

Step 6: Create and Explore Object

After subscribing to the event, let's establish a new "product_item" object and explore the nature of the event. Navigate to "product_item" from the App Launcher. Fill in the "Product Name" and "Product Description" fields by clicking "New," then press "Save."

Step 7: Monitor Event Messages

‍

Go to the terminal where the employee connector is running. You will find messages in the following format:

‍

{ "schema": "-pszPCNGM45tUPU1ftkjxEA", "payload": { "LastModifiedDate": "2020-01-25T20:36:12.000Z", "OwnerId": "005RM000001vI54mYAE", "CreatedById": "005RM000001vI54mYAE", "ChangeEventHeader": { "commitNumber": 65842604581, "commitUser": "005RM000001vI54mYAE", "sequenceNumber": 1, "entityName": "product_item__c", "changeType": "CREATE", "changedFields": [], "changeOrigin": "com/salesforce/api/soap/47.0;client=SfdcInternalAPI/", "transactionKey": "00051c2e-a75a-3f97-03fc-cdf4e16d9d3c", "commitTimestamp": 1569443783000, "recordIds": [ "a00RM0000114ICTYA2" ] }, "CreatedDate": "2020-01-25T20:36:12.000Z", "product_name": "Laptop", "product_description": "High-performance laptop with SSD", "LastModifiedById": "005RM000001vI54mYAE", }, "event": { "replayId": 15053 } }

‍

The changeType field in the format above is significant as it indicates the type of change that occurred. It could contain one of the following four values: CREATE, UPDATE, DELETE, or UNDELETE. The transaction key and sequence number indicate the transaction containing the specific commit and its committed order. Replaying the same event for up to three days is possible with the replayed. You can configure Salesforce Change Data Capture in this manner.

‍

Benefits of Utilizing a CDC Solution

Change Data Capture (CDC) solutions offer numerous advantages for businesses, enabling them to streamline operations, enhance decision-making, and stay competitive in today's dynamic business environment. When combined with platforms like Salesforce and Airbyte, CDC becomes even more powerful, providing businesses with scalable and efficient change data and management solutions.

1. Streamlined Operations with Salesforce CDC:

• Salesforce offers top-of-the-line services for businesses, and when integrated with CDC, it becomes even more streamlined and scalable.
• Example: Consider a business that interacts with multiple purchase vendors daily. With CDC, transaction data is continuously updated, providing relevant and up-to-date information for faster decision-making.

2. Enhanced Data Integration with Airbyte:

• Airbyte, an open-source data integration engine, complements Salesforce by enabling integration with existing databases and keeping data in sync.
• Businesses can build and maintain custom Salesforce connectors, authenticate accounts, and ensure seamless data flow.

3. Detailed Data Change Notifications:

• Salesforce CDC provides detailed notifications for record creation, updates, modifications, and deletions, ensuring businesses stay informed about all data changes.
• This granular level of data capture helps businesses distinguish between changes originating from external systems and those captured through CDC.

4. Customized Data Replication:

• When establishing a Salesforce connector with Airbyte, businesses can choose specific data sets to replicate.
• The Filter Salesforce Object feature allows businesses to select and filter data according to their preferences, ensuring only relevant data is replicated.
• Airbyte also provides the option to replicate the entire Salesforce database for comprehensive data management.

By leveraging CDC solutions like Salesforce and Airbyte, businesses can optimize their data management processes, gain valuable insights, and drive better business outcomes. With streamlined operations, enhanced data integration, detailed change notifications, and customized data replication options, businesses can stay agile and competitive in today's digital landscape.

Limitations of the Custom-Code Approach for Salesforce CDC

Salesforce Data Capture is a powerful solution that utilizes change data capture to enhance data usage for businesses. Implementing Salesforce Change Data Capture (CDC) through custom code presents several challenges and constraints. Here's a concise overview:

1. Complex Implementation Process:

• Requires deep understanding of Salesforce and downstream databases.
• Custom programming needed for data warehouse synchronization adds complexity.

2. Concurrent Client Limit:

• Increase in CometD or Pub/Sub API users may hit Salesforce CDC's concurrent client limit.
• Salesforce Enterprise access has a maximum limit of 1000 CometD subscribers.

3. Daily Rate Limits by Salesforce:

• Salesforce imposes daily rate limits, affecting connector functionality when exceeded.
• Airbyte's Incremental Sync mode mitigates this by syncing only new changes.

By recognizing and addressing these limitations, businesses can effectively navigate the challenges associated with custom-code approaches to Salesforce CDC implementation.

Salesforce CDC Vs. Platform events

The same streaming API technology, which allows for high-volume, low-latency, asynchronous, and dependable messaging between systems, is used by both CDC and Platform Events. When deciding which one to use for your integration scenarios, you should take into account a few of their differences. Here are a few of the primary variations - 

• Platform Events record changes in custom event objects you define, whereas CDC records changes in standard and custom objects. 

• When data changes, CDC automatically publishes events, but Platform Events publishes events manually when you use the Apex or REST API publish method.

• Platform events contain the fields you define for the custom event object, while CDC events contain the record ID, change type, change origin, transaction key, entity name, and modified fields of the data change.

• Platform events are unordered and non-replayable, which means you can only receive the events that are currently on the streaming channel. In contrast, CDC events are ordered and replayable, allowing you to retrieve previous events based on their sequence number or time window.
• Platform events are retained for 72 hours, whereas CDC events are retained for 24 hours.

Real-world Use Cases of Salesforce Change Data Capture (CDC)

Real-world examples highlight the practical applications of Salesforce CDC across industries, showcasing its versatility and effectiveness in driving business outcomes. Here's how CDC benefits organizations:

1. Sales Pipeline Tracking:

• CDC enables sales teams to track opportunities, monitor stage transitions, and identify sales trends in real-time.
• Empowers sales managers to make informed decisions, allocate resources effectively, and prioritize high-value opportunities.

2. Lead Management:

• Captures updates to lead records, facilitating prompt follow-up and effective lead nurturing.
• Optimizes lead conversion rates and improves sales performance.

3. Customer Service Ticketing:

• Tracks changes to case records, enabling prompt responses to inquiries and prioritization of critical issues.
• Enhances customer satisfaction and loyalty by delivering exceptional service experiences.

4. Inventory Management:

• Monitors changes to product records, optimizing inventory levels and anticipating demand fluctuations.
• Ensures efficient inventory management and minimizes stockouts or overstock situations.

5. Marketing Campaign Monitoring:

• Captures updates to campaign records, allowing assessment of campaign effectiveness and optimization of marketing spend allocation.
• Maximizes return on investment (ROI) for marketing campaigns.

Salesforce CDC enhances decision-making, improves operational efficiency, and drives business success across various industries and functions. Its real-time data capture capabilities enable organizations to stay competitive in today's dynamic business environment.

Conclusion

With this comprehensive guide, you can now navigate your way through Salesforce with Airbyte. Log in to your Airbyte Cloud account and get started right away!

Future trends in Salesforce Change Data Capture (CDC) are poised to revolutionize data management and analytics. Key developments include enhanced integration, AI-driven insights, advanced security, complex data model support, real-time reporting, blockchain integration enable change data capture, cross-cloud compatibility, and improved user experience. Stay ahead of the curve with these upcoming advancements in Salesforce CDC.

FAQs

What is a CDC event?

Change data capture (CDC) is a technique to detect changes made to a database or external system and then provide these changes to an downstream system.

What are Apex triggers?

Apex triggers are essential components in Salesforce, executing actions in response to specific events on records. Written in Apex, Salesforce's proprietary language, triggers automate processes, enforce data validation rules, and trigger additional logic pre or post record manipulation. They are vital for customizing Salesforce's behavior to meet business needs efficiently and effectively.

What is the difference between platform event and CDC in Salesforce?

The CDC captures the changes in custom and standard objects whereas the platform event captures the changes in custom event objects created. CDC publishes data changes automatically, and Platform events publish events manually using publish methods from Apex or REST APIs. 

Why Implement CDC in Salesforce?

Salesforce provides CDC with numerous advantages for users. Salesforce enables companies to improve customer interactions with customers and make informed business decisions. CDC provides a seamless way for organizations to track data and improve decision-making by automating the processing of the data.

What is the limitation of CDC in Salesforce?

Salesforce will publish events up to 1M events. If your business contains more custom fields than a single long field, the limit is possible. If so, changes are not delivered to salesforce events but instead event messages are replaced by gaps in event messages.