We joined the startup after several data initiatives had failed, so the engagement had to address both the platform and the business's confidence in data. The existing eventing foundation was not robust enough to support growth across marketing, analytics and leadership use cases.
The first step was a complete redesign of the domain events. The goal was a streaming-first data infrastructure in which different teams could build on a shared representation of operations instead of maintaining separate, inconsistent interpretations.
The data stack included Segment, BigQuery, Mixpanel, Iterable, Impact and other operational tools. Because these products served different consumers, simply connecting each source to each destination would not have provided enough control. We introduced a custom control plane to coordinate the system:
flowchart LR
sources[Product and business events] --> segment[Segment]
segment --> bigquery[BigQuery]
segment --> mixpanel[Mixpanel]
segment --> iterable[Iterable]
segment --> impact[Impact]
contracts[Data contracts] --> control[Custom control plane]
control --> segment
control --> validation[Pipeline and data validation]
validation --> alerts[Alerts and notifications]
The control plane orchestrated pipelines, enforced validations defined by data contracts and governed how data moved from sources to destinations. This created one place to manage delivery expectations across a toolchain that otherwise had separate configuration and monitoring surfaces.
We redesigned the event model and aligned it with the shared operational concepts required by marketing, analytics and leadership. We then made the surrounding systems work together by coordinating pipeline execution, data validation and destination delivery through the control plane.
Governance and observability were included in the architecture from the beginning. When a pipeline or validation failed, the platform generated prompt alerts and notifications so the team could investigate before downstream consumers relied on incomplete or incorrect data.
- established a streaming-first event foundation shared across marketing, analytics and leadership use cases
- introduced robust observability to verify that data pipelines operated as planned
- used data profiling to help the team detect and react quickly to accidental product regressions
- enforced data contracts and governed data delivery across Segment, BigQuery, Mixpanel, Iterable, Impact and connected tools
