ICYMI—SecurityScorecard Talks Streaming Data Pipelines @ Data In Motion Tour 2023

When you want to learn about successful data streaming use cases and Confluent Cloud from experienced practitioners, you go to the #DataInMotionTour.

DIMT 2023 is well underway in North America, and attendees are experiencing a full-day of technical breakout sessions, guided demos, and workshops. At our recent stop in Jersey City, Brandon Brown , a Senior Staff Software Engineer from SecurityScorecard , shared how data streaming has helped uplevel one of the company’s newest products: Automatic Vendor Detection.

Businesses that need to evaluate their data systems for security risks or threats turn to SecurityScorecard – which continuously monitors and scores over 12 million rated companies worldwide. Offering Automatic Vendor Detection has meant giving its customers a more complete, up-to-date view of third- and fourth-party connections and the products they integrate with their data systems.

When first developed, this product originally relied on batch-based data pipelines. But as SecurityScorecard continued to ingest and process more data, delivering timely cybersecurity insights to customers using batch processing became even more challenging.?

As part of a larger effort to make streaming data accessible across the organization, Brandon and other engineers on SecurityScorecard’s pipelines team began modernizing the company’s data pipeline architecture. By the time Automatic Vendor Detection went live over a year ago, it was primarily event-based, which allowed SecurityScorecard to onboard new data feeds much faster.

During his breakout session at DIMT2023, Brandon explained how migrating from batch-based pipelines to streaming data pipelines with Confluent has allowed his team to collaborate effectively with the company’s threat intelligence team and significantly improve Automatic Vendor Detection by:?

→Onboarding additional sources of data for vendor detection

→Creating an end-to-end integration test suite for faster debugging

→Putting in place data governance to standardize data schemas across the company

Brandon also shared how using Confluent Cloud has allowed the pipelines team to focus on empowering engineers and sharing data streams across teams while saving on infrastructure costs.?

You can learn more about how industry leaders like SecurityScorecard are using data streaming to build better products and deliver a better customer experience at DIMT2023. We still have nine stops left in North America – with our Washington, D.C., Bay Area, and Montreal stops up next.

Sign up to attend #DIMT2023 in a city near you – or check out the recap from our recent stop in Boston to learn more about the event.