Differences-in-Differences

Differences-in-Differences#

A B2B software company rolls out an AI sales assistant to its representatives. The rollout was not simultaneous: some regions received it in Q1, others in Q3, and some never adopted it at all. Management wants to know whether the tool improved productivity. How do we estimate the effect when treatment assignment was not random?

This section introduces the differences-in-differences (DiD) estimator, the most widely used quasi-experimental method in applied economics and in data analysis at technology firms. We begin with visual intuition, formalize the estimator, discuss the assumption that sustains it, and conclude with the modern version for staggered designs.

1. The Logic of DiD: The Control Group as Counterfactual#

After the AI assistant launched, representatives in the regions that received it sell, on average, 25 % more than those in regions that never adopted it. Can we attribute this difference to the assistant?

Not directly. Before the launch, regions that would receive the assistant already sold around 10 % more than the others — a pre-existing gap driven by market size, team tenure, and other hard-to-measure factors. Comparing the two groups after the launch conflates the program’s effect with that pre-existing advantage.

What if we instead looked at the change in sales within the treated regions — comparing each region to itself before and after? That sidesteps the level-gap problem, but creates a new one: sales grow every quarter due to general market expansion, seasonal factors, and team growth. A before-after estimate for the treated group alone would pick up those time trends on top of the program effect, inflating the estimate.

We need a counterfactual: what would have happened to sales in the treated regions if they had not received the assistant? Regions that never adopted it give us exactly that benchmark — as long as we are willing to assume that, absent treatment, both groups would have followed the same trend. The difference in the changes in sales (change in treated minus change in controls) cancels both the pre-existing level gap and the common time trend, leaving only the treatment effect.

That is DiD in one sentence: the time variation of the control group is the counterfactual for the time variation of the treated group.

Exploration: The DiD estimator in action#

We now use simulated data to illustrate the DiD strategy. The simulation generates data for two groups of regions (treated and control) across two periods (before and after the launch). The key parameter is τ — the true effect of the assistant on sales. The background parameters (baseline group gap α = 10, common trend λ = 5) are fixed.

Explore the simulation in “2×2” mode:

Look at the line chart. Identify the treated and control groups. Notice that the two groups start at different levels — there is a pre-existing gap. Where does the DiD strategy use the control group’s trajectory?
Look at the 2×2 table below the chart. The bottom-right cell (“DiD”) shows the difference of changes. Compare it with the Treat×Post coefficient in the regression table.
Change τ to different values (for example, 5, 10, 30). Observe how the DiD entry in the table and the regression coefficient track τ exactly — regardless of the baseline gap or the common trend.

What do we observe?#

The DiD estimator recovers τ precisely, regardless of how large the baseline gap α is between the groups or how steep the common time trend λ. Both cancel in the double difference: the change in the control group (Δλ) is subtracted from the change in the treated group (Δλ + τ), and what remains is exactly τ. The 2×2 table and the regression coefficient report the same number — and that number equals the true effect we set.

2. The DiD Estimator: Double Difference and Regression#

When DiD works: an intuitive case#

We will state the key assumption formally in the next section, but let’s first understand the intuitive case where DiD recovers the effect exactly. Suppose that, in the absence of treatment, the outcome for unit \(i\) in period \(t\) follows:

\[Y_{it}(0) = \alpha_i + \lambda_t + \varepsilon_{it}\]

where \(\alpha_i\) is a time-invariant unit-specific constant — the baseline sales level for each region, driven by its market size, team quality, and other stable characteristics — and \(\lambda_t\) is a time effect common to all regions. Both treated and control groups share the same \(\lambda_t\): neither group grows faster than the other on its own. With treatment, the potential outcome is:

\[Y_{it}(1) = Y_{it}(0) + \tau\]

so the observed outcome for a treated unit after the launch is \(\alpha_i + \lambda_t + \tau + \varepsilon_{it}\).

The 2×2 table and the double difference#

Taking group averages, let \(\bar{\alpha}_C\) and \(\bar{\alpha}_T\) be the average unit effects for control and treated groups, and \(\Delta\lambda = \lambda_{\text{post}} - \lambda_{\text{pre}}\) be the common time trend:

	Before	After	Change
Control	\(\bar{\alpha}_C + \lambda_{\text{pre}}\)	\(\bar{\alpha}_C + \lambda_{\text{post}}\)	\(\Delta\lambda\)
Treated	\(\bar{\alpha}_T + \lambda_{\text{pre}}\)	\(\bar{\alpha}_T + \lambda_{\text{post}} + \tau\)	\(\Delta\lambda + \tau\)
DiD			\(\boldsymbol{\tau}\)

The level gap \((\bar{\alpha}_T - \bar{\alpha}_C)\) cancels because it appears in both periods. The common trend \(\Delta\lambda\) cancels because both groups share it. What remains is exactly \(\tau\).

The differences-in-differences estimator is the difference of changes:

\[\widehat{\tau}_{DiD} = \bigl(\bar{Y}_{T,\text{post}} - \bar{Y}_{T,\text{pre}}\bigr) - \bigl(\bar{Y}_{C,\text{post}} - \bar{Y}_{C,\text{pre}}\bigr)\]

The algebraic derivation for the general case is in the appendix. The key result: when both groups share the same time trend, the level gap and the common trend cancel, and the estimator recovers exactly \(\tau\).

Regression equivalence#

The same estimate is obtained by running OLS on:

\[Y_{it} = \beta_0 + \beta_1 G_i + \beta_2 \text{Post}_t + \hat{\tau}\,(G_i \times \text{Post}_t) + \varepsilon_{it}\]

The coefficient \(\hat{\tau}\) on the interaction \(G_i \times \text{Post}_t\) is algebraically identical to the double difference of means. The formal proof is in the appendix. The regression form is convenient because it produces standard errors directly, allows adding additional controls, and generalizes to multiple periods and multiple groups. The OLS table in the simulation panel shows exactly that coefficient — watch it track τ as you vary the effect size.

3. The Parallel Trends Assumption#

When DiD fails: differential trends#

It is easy to construct a case where the 2×2 DiD breaks down. Return to the simulation and switch to “Pre-trends Inspection” mode. This reveals a new parameter, δ, which captures whether the treated regions were already growing at a different rate than controls before the launch.

Set δ = 0. The DiD estimate matches τ exactly — the two groups run parallel before the launch.
Now raise δ to 10 or lower it to −10. Observe that the DiD estimate diverges from τ. The treated group’s trend was already different from the control group’s trend before treatment, and that pre-existing divergence is absorbed into the estimator as a spurious effect.

To see why, we extend the model. Let \(G_i \in \{0,1\}\) indicate whether region \(i\) belongs to the treated group, and \(\text{Post}_t \in \{0,1\}\) whether period \(t\) is after the launch. Suppose the untreated potential outcome is:

\[Y_{it}(0) = \mu + \alpha G_i + \lambda \,\text{Post}_t + \delta\, G_i \cdot \text{Post}_t + \varepsilon_{it}\]

This extends the model from Section 2 by adding \(\delta\): a differential trend for the treated group that would exist even without treatment. The 2×2 table now becomes:

	Before (\(\text{Post}=0\))	After (\(\text{Post}=1\))	Change
Control (\(G=0\))	\(\mu\)	\(\mu + \lambda\)	\(\lambda\)
Treated (\(G=1\))	\(\mu + \alpha\)	\(\mu + \alpha + \lambda + \delta + \tau\)	\(\lambda + \delta + \tau\)
DiD			\(\tau + \delta\)

When δ ≠ 0, the estimator returns \(\tau + \delta\) rather than \(\tau\). The bias equals δ — visible in the simulation as the gap between the true τ and the Treat×Post coefficient. The algebraic derivation is in the appendix.

Parallel trends assumption: For \(\widehat{\tau}_{DiD} = \tau\) we need δ = 0, i.e.:

\[\mathbb{E}[Y_{it}(0) \mid G=1, \text{Post}=1] - \mathbb{E}[Y_{it}(0) \mid G=1, \text{Post}=0] = \mathbb{E}[Y_{it}(0) \mid G=0, \text{Post}=1] - \mathbb{E}[Y_{it}(0) \mid G=0, \text{Post}=0]\]

In words: the counterfactual change in the treated regions (what would have happened without the assistant) must equal the observed change in the control regions.

Examining trends with the simulation#

In “Pre-trends Inspection” mode, the simulation generates three periods (pre, before, after), plotting the full trajectory of both groups including a period before any treatment. This is the key diagnostic tool for evaluating the parallel trends assumption in practice.

With δ = 0, the two lines are parallel in the pre-period — the slope is the same in both groups before the launch. The pre-treatment coefficient in the event-study chart is close to zero.
With δ ≠ 0, the lines diverge before treatment. In real data, this pattern would be evidence against the parallel trends assumption — and a warning sign for the analysis.

This type of plot — verifying that there are no differential pre-treatment trends — is standard practice in any DiD analysis. If the treated regions were already growing faster than controls before the launch, the post-treatment comparison conflates the assistant’s effect with that pre-existing dynamic.

4. Staggered DiD: Cohorts and the Goodman-Bacon Decomposition#

The staggered setting#

Our SaaS scenario has a detail that complicates the analysis: the assistant did not reach all treated regions at the same time. Some received it in Q4 (early cohort), others in Q7 (late cohort), and a group never received it. This staggered design is the rule rather than the exception in corporate implementations — support resources, training logistics, and technical team availability make sequential rollouts the norm. The simulation below captures exactly this setting: 10 quarters, two treated cohorts, and a never-treated group.

Exploration: three staggered scenarios#

Use the dashboard to compare what happens under three different assumptions about how the treatment effect behaves:

Homogeneous effects — every region gains the same τ = 20 units of sales regardless of when it adopted the assistant. Watch Panel A: both cohorts jump by the same amount at their respective launch quarters. Panel D shows TWFE and Callaway & Sant’Anna (C&S) tracking each other closely.
Heterogeneous effects — the early cohort gains more (τ = 30) than the late cohort (τ = 15). Panel D starts to show a gap between TWFE and C&S. Why? We will explain this next.
Dynamic effects — the treatment effect grows with time-since-adoption: τ(e) = min(5(e+1), 20). Panel A now shows a ramp-up after each cohort’s launch. The TWFE coefficient in Panel D diverges the most in this scenario. Panel C (Goodman-Bacon decomposition) will reveal why.

TWFE and the Goodman-Bacon Decomposition#

The natural extension of the 2×2 DiD to a panel with multiple periods and multiple cohorts is the two-way fixed effects (TWFE) estimator:

\[Y_{it} = \mu_i + \lambda_t + \hat{\tau}\, D_{it} + \varepsilon_{it}\]

where \(\mu_i\) absorbs region-level differences, \(\lambda_t\) absorbs the common time trend, and \(D_{it} = 1\) when region \(i\) is under treatment in period \(t\). The appeal is obvious: it looks like a direct generalization of the regression equivalence we established in Section 2.

The difficulty is that it is not immediately clear what this regression is actually comparing. The Goodman-Bacon theorem (Goodman-Bacon, 2021) provides the answer: the TWFE coefficient is a variance-weighted average of every 2×2 DiD that can be constructed from the data. In our two-cohort design, there are four implicit comparisons:

Early cohort vs. never-treated — early-adopter regions compared to those that never adopted, using periods before Q4 as the pre-period. Clean comparison.
Late cohort vs. never-treated — late-adopter regions compared to never-treated, using periods before Q7 as the pre-period. Clean comparison.
Early cohort uses late cohort as control (middle window) — early-adopter regions are the “treated” group; late-adopter regions serve as the control during a window that falls after early adoption but before late adoption begins. Clean comparison: the late cohort has not yet been treated.
Late cohort uses early cohort as control (post window) — late-adopter regions are the “treated” group, but the control is the early-cohort regions, which have already been treated for several quarters. This is the problematic comparison.

Comparison 4 is the risky one. If the treatment effect is dynamic — growing over time — then the early-cohort regions used as “control” are not experiencing zero treatment effect: their effect has been accumulating since Q4. Using them as a counterfactual for the late cohort subtracts a large positive effect from the estimate, biasing \(\hat{\tau}_{TWFE}\) downward. In extreme cases this can flip the sign of the coefficient entirely.

Explore this in the dashboard: switch to dynamic effects and examine Panel C (Goodman-Bacon decomposition). The red dots mark comparison 4 — the late-vs.-early post-window. Notice how the weight on those red comparisons, combined with the inflated early-cohort effect, drives \(\hat{\tau}_{TWFE}\) away from the true ATT. With homogeneous effects, all four comparisons are valid, the red dots carry zero contamination, and TWFE is unbiased.

The Callaway & Sant’Anna Estimator#

The Callaway & Sant’Anna (C&S) estimator starts from a simple principle: never use an already-treated group as the control for another group’s 2×2 comparison. The building block is \(ATT(g,t)\) — the average treatment effect on the treated for cohort \(g\) in period \(t\):

\[\widehat{ATT}(g,t) = \bigl[\bar{Y}_{g,t} - \bar{Y}_{g,\,g-1}\bigr] - \bigl[\bar{Y}_{C,t} - \bar{Y}_{C,\,g-1}\bigr]\]

where \(C\) is a clean comparison group — either never-treated or not-yet-treated in period \(t\) — and \(g-1\) is the period immediately before cohort \(g\)’s launch. Each \(ATT(g,t)\) is a standalone 2×2 DiD that never involves a contaminated control. This is precisely what prevents the forbidden comparison in case 4 above.

Aggregation. Once all \(ATT(g,t)\) estimates are in hand, C&S aggregates them. The dynamic aggregation used in Panel D averages \(ATT(g,t)\) values by event time (\(e = t - g\)), producing one coefficient per period relative to treatment:

\[\widehat{ATT} = \sum_{g} \sum_{t \geq g} \omega_{g,t} \cdot \widehat{ATT}(g,t)\]

Inspect the C&S ATT(g,t) heatmap in Panel B of the dashboard. Each cell \((g, t)\) is one 2×2 estimate. Pre-treatment cells (grey) are set to zero by construction. With homogeneous effects, all post-treatment cells look alike. With heterogeneous or dynamic effects, the heatmap reveals the pattern: early-cohort cells differ from late-cohort cells, and cells further from the launch date reflect accumulated effect dynamics. This granularity is what TWFE collapses into a single number.

Switch between “never-treated” and “not-yet-treated” comparison groups: the ATT(g,t) estimates shift slightly. The never-treated group is the cleanest control; the not-yet-treated group offers more data but introduces a small risk of contamination if effects are anticipated before the formal adoption date.

5. Event-Study Plots#

An event-study plot presents DiD estimates for each period relative to the launch — from several periods before adoption to several after. The horizontal axis shows event time (\(e = t - g\), where \(e = 0\) is the launch period); the vertical axis shows the estimated coefficient. Panel D of the simulation is the canonical example.

Post-treatment estimates: treatment effect dynamics#

Each post-treatment coefficient (\(e \geq 0\)) targets \(ATT(g, t)\) — the average effect on treated cohort \(g\) in calendar period \(t = g + e\). The collection of these estimates traces out the dynamics of the treatment effect. A flat profile after adoption suggests a constant effect. An upward-sloping profile (as in the dynamic scenario) suggests the effect accumulates as regions become more skilled with the assistant. Event-study plots in the post-treatment window answer the question: does the effect arrive immediately, or does it grow?

In Panel D, the C&S coefficients (by cohort, then aggregated by event time) directly target these \(ATT(g,t)\)’s. The TWFE event-study line is obtained by replacing the single treatment dummy \(D_{it}\) with a full set of event-time interactions and omitting \(e = -1\) as the baseline:

\[Y_{it} = \theta_t + \eta_i + \sum_{e \neq -1} \beta_e \cdot \mathbf{1}\{t - G_i = e\} + \varepsilon_{it}\]

This produces identical estimates to C&S when effects are homogeneous, but diverges when effects differ across cohorts — for the same reason as the scalar TWFE: some post-treatment interactions implicitly compare cohorts with different effect levels.

Pre-treatment estimates: falsification tests#

Each pre-treatment coefficient (\(e < 0\)) estimates:

\[\tau_{-k} = \mathbb{E}[Y_{t=g-k}(0) - Y_{t=g-1}(0) \mid D=1] - \mathbb{E}[Y_{t=g-k} - Y_{t=g-1} \mid D=0]\]

This is a differential-trend or pre-trend estimate: it measures whether the treated and control groups were following parallel trajectories in the pre-treatment window. Under no-anticipation and parallel trends, all \(\tau_{-k}\) equal zero.

Three practical lessons from the recent literature (Baker et al. [2026]; Roth [2022]; Rambachan and Roth [2023]):

Pre-trends test a related but different assumption. The parallel trends assumption requires that trends be equal in every post-treatment period. Pre-trend estimates measure differences in pre-treatment periods — the “wrong” window. Small pre-trends provide some reassurance but do not constitute a direct test of the assumption that governs post-treatment inference (Baker et al. [2026], §5.1.2).
Tests for pre-trends are low-powered. With few pre-treatment periods or small cohorts, statistical tests of \(\tau_{-k} = 0\) have low power to detect real violations. Failing to reject flat pre-trends says little about the magnitude of violations that would materially bias the post-treatment estimates.
Consider the size of plausible violations explicitly. Rather than asking only “are pre-period coefficients close to zero?”, a more informative question is “how large would a parallel-trends violation need to be before it seriously biases the ATT estimates?” Rambachan and Roth [2023] provide tools for bounding ATT under violations of specified magnitude — a disciplined alternative to the informal eye-test.

In the simulation, explore the event-study panel under each of the three effect types. With homogeneous effects, pre-trends are flat and TWFE matches C&S throughout. With dynamic effects, the post-treatment TWFE line dips below the C&S line — reflecting the contamination from the Goodman-Bacon forbidden comparison — while the pre-treatment window stays flat. The lesson: flat pre-trends are necessary but not sufficient for a clean TWFE estimate in staggered designs.

6. Summary and Decision Tree#

Differences-in-differences identifies causal effects by comparing the time variation between treated and control groups. Its key assumption — parallel trends — is weaker than the conditional independence assumption: it only asks that trends would be the same in the absence of treatment, not that levels would be. When the design is staggered, TWFE can fail if effects are heterogeneous across cohorts; the Callaway & Sant’Anna estimator solves this by restricting each 2×2 comparison to uncontaminated control groups.

Decision tree for DiD analysis:

Is treatment simultaneous for all treated units? Yes → standard 2×2 DiD or TWFE with two periods. No → continue.
Are effects homogeneous across cohorts and constant over time? Yes → TWFE may suffice. No (or unknown) → use Callaway & Sant’Anna.
Is there a “never-treated” group? Yes → use it as the primary comparison group. No → the “not-yet-treated” group is the alternative, with greater contamination risk.
Do you have enough pre-treatment periods? Yes → plot the event study and evaluate pre-trends. No → the parallel trends assumption is harder to justify and must be supported with institutional arguments.
Are you reporting only TWFE? If the design is staggered and effects may be heterogeneous, report C&S alongside TWFE and discuss any discrepancies.

Appendix: Formal Derivations#

A.1 Algebra of the double-difference estimator#

Setup. Given the DGP:

\[Y_{it}(0) = \mu + \alpha G_i + \lambda \,\text{Post}_t + \delta\, G_i \cdot \text{Post}_t + \varepsilon_{it}, \qquad \mathbb{E}[\varepsilon_{it}] = 0\]

\[Y_{it} = Y_{it}(0) + \tau\, G_i \cdot \text{Post}_t\]

The four cells of the 2×2 table in expectation:

\[\mathbb{E}[Y \mid G=0, \text{Post}=0] = \mu\]

\[\mathbb{E}[Y \mid G=0, \text{Post}=1] = \mu + \lambda\]

\[\mathbb{E}[Y \mid G=1, \text{Post}=0] = \mu + \alpha\]

\[\mathbb{E}[Y \mid G=1, \text{Post}=1] = \mu + \alpha + \lambda + \delta + \tau\]

Step 1 — Time change in the control group:

\[\Delta_C \equiv \mathbb{E}[Y \mid G=0, \text{Post}=1] - \mathbb{E}[Y \mid G=0, \text{Post}=0] = \lambda\]

Step 2 — Time change in the treated group:

\[\Delta_T \equiv \mathbb{E}[Y \mid G=1, \text{Post}=1] - \mathbb{E}[Y \mid G=1, \text{Post}=0] = \lambda + \delta + \tau\]

Step 3 — Double difference:

\[\widehat{\tau}_{DiD} = \Delta_T - \Delta_C = (\lambda + \delta + \tau) - \lambda = \tau + \delta\]

Conclusion. When \(\delta = 0\) (parallel trends), \(\widehat{\tau}_{DiD} = \tau\). The estimator cancels \(\alpha\) (level gap) and \(\lambda\) (common trend). When \(\delta \neq 0\), the bias is exactly \(\delta\).

A.2 Equivalence with OLS regression#

Setup. Consider the regression:

\[Y_{it} = \beta_0 + \beta_1 G_i + \beta_2 \text{Post}_t + \beta_3\, (G_i \times \text{Post}_t) + \varepsilon_{it}\]

The four cells of the 2×2 table in terms of the coefficients:

\[\mathbb{E}[Y \mid G=0, \text{Post}=0] = \beta_0\]

\[\mathbb{E}[Y \mid G=0, \text{Post}=1] = \beta_0 + \beta_2\]

\[\mathbb{E}[Y \mid G=1, \text{Post}=0] = \beta_0 + \beta_1\]

\[\mathbb{E}[Y \mid G=1, \text{Post}=1] = \beta_0 + \beta_1 + \beta_2 + \beta_3\]

Step 1 — Matching with the DGP:

\[\beta_0 = \mu, \qquad \beta_1 = \alpha, \qquad \beta_2 = \lambda, \qquad \beta_3 = \delta + \tau\]

Step 2 — Applying OLS exactly (with the four cell means as sufficient statistics), \(\hat{\beta}_3\) satisfies:

\[\hat{\beta}_3 = (\bar{Y}_{G=1,\,\text{Post}=1} - \bar{Y}_{G=1,\,\text{Post}=0}) - (\bar{Y}_{G=0,\,\text{Post}=1} - \bar{Y}_{G=0,\,\text{Post}=0}) = \widehat{\tau}_{DiD}\]

Conclusion. The OLS coefficient on the interaction \(G_i \times \text{Post}_t\) is algebraically identical to the double-difference estimator. The regression formulation adds automatic standard error computation and allows extending the model with additional controls \(X_{it}\) without altering the interpretation of \(\hat{\beta}_3\) as the DiD estimator.

References#

The interactive dashboards in this section also draw on Angrist and Pischke [2009], Card and Krueger [1994], Roth et al. [2023], Callaway and Sant'Anna [2021], Goodman-Bacon [2021], and Callaway et al. [2024].

[AP09]

Joshua D Angrist and Jörn-Steffen Pischke. Mostly harmless econometrics: An empiricist's companion. Princeton university press, 2009.

[BCC+26] (1,2)

Andrew Baker, Brantly Callaway, Scott Cunningham, Andrew Goodman-Bacon, and Pedro H C Sant'Anna. Difference-in-differences designs: a practitioner's guide. Journal of Economic Literature, 64(2):498–557, 2026.

[CSantAnna21]

Brantly Callaway and Pedro H C Sant'Anna. Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2):200–230, 2021.

[CSantAnnaQG24]

Brantly Callaway, Pedro H C Sant'Anna, Alexander Quispe, and Carlos Guevara. Csdid: difference-in-differences with multiple time periods in python. 2024. Available at: d2cml-ai/csdid.

[CK94]

David Card and Alan B Krueger. Minimum wages and employment: a case study of the fast-food industry in new jersey and pennsylvania. American Economic Review, 84(4):772–793, 1994.

[GB21]

Andrew Goodman-Bacon. Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2):254–277, 2021.

[RR23] (1,2)

Ashesh Rambachan and Jonathan Roth. A more credible approach to parallel trends. The Review of Economic Studies, 90(5):2555–2591, 2023.

[Rot22]

Jonathan Roth. Pretest with caution: event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3):305–322, 2022.

[RSantAnnaBP23]

Jonathan Roth, Pedro H C Sant'Anna, Alyssa Bilinski, and John Poe. What's trending in difference-in-differences? a synthesis of the recent econometrics literature. Journal of Econometrics, 235(2):2218–2244, 2023.