Tracking design & implementation

Tracking implementation for analytics does not need to be more complicated than it sounds, but it does need an architecture that keeps event definitions, deployments and data quality signals under control. I have been doing tracking implementations for the last 4+ years myself, and I recently finished reading Timo Dechau’s book — The Analytics Implementation Workbook. I wish this was the kind of resource I had access to earlier in my data career. This post highlights some commonalities between the experience of Timo Dechau and mine. I absolutely recommend reading the book, whether you’re a a data consultant just starting out, an IT team planning to develop tracking internally or a seasoned data expert willing to get some idea sparks. It’s consumable, pleasant to read and I found myself getting to my notebook every couple of pages to note down ideas for improving the work I do for my consulting clients.

I cannot reinforce the importance of having automation across the data infrastructure — an opinion shared by Timo and covered across several chapters of the book. In a tracking implementation that I designed and implemented recently, I approached things gradually:

manage schemas in JSONSchema in a centralized dedicated repo; this is my source of truth and provides proper version control (no tool-specific implementation or formats, especially if proprietary)
simplify/restructure the schemas, taking benefit of the degree of control introduced with the above; in my case, this meant centralizing object definitions as reusable artifacts and referencing them across events — a customer and an order are now the same consistent structure wherever they are used in events; using JSONSchema standard referencing mechanisms or a lightweight builder are the way to go here
build scripting for deployments; don’t think rockets yet — just a way to make schemas from your repo make it into your key systems; in my case, this meant creating/updating Tracking Plans in Segment automatically, in the desired environment (nonprod or prod)
build CICD to automate such deployment scripting; in other words, don’t run deployment scripts manually, but rather adopt a mechanism in which schemas changes are pushed to the PR, they go though a review process, and upon PR merge they are automatically deployed with proper traceability
leverage active metadata to ship schemas, descriptions and data quality checks (expectations) from the repo to other systems as well — think MixPanel, Iterable, Attentive etc and enabling your users to get immediate access to event and field descriptions on the spot, without having to go to another system

The architecture I prefer is intentionally boring:

flowchart LR
    repo[Tracking schema repo] --> checks[PR validation]
    checks --> deploy[CI deployment job]
    deploy --> segment[Segment tracking plans]
    deploy --> metadata[Active metadata sync]
    segment --> warehouse[BigQuery event tables]
    warehouse --> monitoring[Quality and freshness monitoring]
    monitoring --> alerts[Slack alerts and dashboards]

The important design decision is to make the schema repository the contract, not a documentation afterthought. Product analytics, engineering and data teams should review the same event names, property types, required fields and descriptions before the event reaches production. In practice, I split the work into four implementation layers:

Event contract design. Define naming conventions, required context fields, reusable objects and ownership for each event family.
Schema validation. Use JSONSchema to catch type, enum and required-field issues before changes are merged.
Deployment automation. Promote approved schemas into Segment tracking plans through a CI job, with different targets for non-production and production.
Operational monitoring. Profile landed event data in BigQuery and alert on volume, freshness, null-rate and distribution changes.

That sequence avoids the common failure mode where the tracking plan looks good in a spreadsheet but production events drift away from it within a few sprints. Observability is another key consideration that often gets overlooked. Lack of observability leads to the inability react to issues & failures, which leads to data problems not being fixed and, ultimately, loss of trust in data. In data, loss of trust is literally the worst-case scenario.

When I have ownership of a setup, I create a guardrail system that regularly checks the state of the setup and tells me what I need to work on. […] Is it a requirement before you build an event data setup? No, but I recommend setting it up in a lean initial version. Trust me, it will save you time and, more importantly, give you confidence in the data setup. This always comes in handy when someone asks in a meeting: ‘Can I trust these numbers?’.

In my engagements as a data consultant, I realized that observability rarely comes out-of-the-box. Logging is part of observability and you should not reinvent logging, that’s for sure. But when it comes to tracking observability, this oftentimes goes to a business level where you need insights and interpretations, not just technicalities. What I implemented once was a technical metrics system on top of events data landed by Segment in BigQuery. For events I looked at daily counts and freshness, for fields I looked at not-null counts, distinct counts, string min/avg/max lengths, numeric min/avg/max values, and for low-cardinality fields I also looked at distribution profiling. All of these went into a system in which thousands of technical metrics were collected automatically, with information further fed into a Slack notification system and data quality dashboards. The resulting monitoring table had one row per metric observation, with dimensions such as event name, field name, metric type, environment and collection date. Keeping the model generic made it easy to add new checks without creating a new table for every event or tool. That choice also made the dashboards easier to maintain: the same chart could show freshness for one critical checkout event or null-rate trends for a whole family of product events. The people/team aspect of tracking implementations is another core aspect which doesn’t get the attention it deserves. Communication is paramount in data work, whether it’s working with product & development teams on the design & implementation or working with stakeholders to communicate or to align. You need to identify the proper stakeholders, keep them engaged and be permanently in touch — otherwise you cannot make data deliver value in an organization. The book articulates such a simple yet powerful concept — collect and socialize wins in data. Asking your customers “What win, value or interesting fact did you get recently when working with the event data? What impact had this insight have on your work and the business in general?“. Do this as part of periodic, targeted interviews with your key stakeholders — Timo covers quite extensively the topic of how to do this, and the chapter covering the Customer dimension of an event data setup is the one which got me the most ideas to implement with my consulting customers as well. Timo also brings in sight the importance of how data teams are set to work with engineering teams. There needs to be no divide. We must strive to remove any walls between engineering and data, in order to build on the vision of making tracking first-class citizen — and that means having engineering as a trusted partner, since the very early design phase of tracking setups and all the way through implementation and delivering outcomes. My main takeaway is that tracking quality is not produced by one tool. It comes from a clear contract, a deployment path that engineers trust, and monitoring that tells the data team when reality diverges from the design.Also published on Medium