Bases de donnéesConnectez n'importe quelle base de données et analysez vos données instantanément·FichiersImportez des fichiers CSV ou Excel et explorez-les avec l'IA·ChatPosez vos questions en langage naturel — dialoguez avec vos données·Tableaux de bordCréez des dashboards interactifs à partir de vos requêtes en quelques secondes·IALaissez l'IA écrire le SQL à votre place·GraphiquesVisualisez les tendances avec des graphiques générés automatiquement·No-codeAucune connaissance SQL requise — demandez simplement en français·PartagePartagez vos dashboards en direct avec votre équipe en un clic·InsightsDétectez automatiquement les tendances et anomalies cachées dans vos données·ExportsTéléchargez vos résultats en CSV, Excel ou PNG instantanément·Bases de donnéesConnectez n'importe quelle base de données et analysez vos données instantanément·FichiersImportez des fichiers CSV ou Excel et explorez-les avec l'IA·ChatPosez vos questions en langage naturel — dialoguez avec vos données·Tableaux de bordCréez des dashboards interactifs à partir de vos requêtes en quelques secondes·IALaissez l'IA écrire le SQL à votre place·GraphiquesVisualisez les tendances avec des graphiques générés automatiquement·No-codeAucune connaissance SQL requise — demandez simplement en français·PartagePartagez vos dashboards en direct avec votre équipe en un clic·InsightsDétectez automatiquement les tendances et anomalies cachées dans vos données·ExportsTéléchargez vos résultats en CSV, Excel ou PNG instantanément·
AnalityQa
FonctionnalitésPar métierTarifsOutilsBlog
ConnexionCommencer
Blog›Product

A/B Test Results That Tell You What to Ship

Declaring a test winner based on a raw conversion rate difference without checking sample size, segment effects, or guardrail metrics is how teams ship regressions they celebrate as wins.

Try AnalityQa AI AI free →See live examples
Product team reviewing analytics

The problem

  • →Most teams look at the headline metric lift and declare a winner without calculating whether the result is statistically significant at their actual sample size.
  • →Segment-level effects — where a variant wins for one user group and loses for another — are invisible in aggregate results and only surface when someone slices the data manually.
  • →Guardrail metrics that should not move — page load time, support ticket volume, error rates — are rarely checked in the same analysis pass as the primary metric, so regressions go undetected until they show up in customer complaints.
  • →Experiment data from feature flag tools like LaunchDarkly or Optimizely lives separately from product event data and business outcome data, making a complete analysis require joins across three systems.

Why the usual approach breaks down

Significance testing is easy to get wrong and hard to explain

Choosing between a t-test, a z-test, a chi-squared test, or a Mann-Whitney test depends on the metric type and distribution. Using the wrong test produces a p-value that looks meaningful but is not. Explaining the result to a non-technical stakeholder adds another layer of difficulty that often leads teams to skip the explanation entirely.

Segment-level analysis multiplies the number of comparisons and the chance of false positives

Running significance tests on ten segments without a multiple-comparison correction inflates the chance of a false positive substantially. Most ad-hoc segment analyses in spreadsheets skip this correction, leading to confident but wrong conclusions about which segments the variant helps.

Experiment data is siloed from the metrics it is supposed to move

LaunchDarkly or your feature flag system knows who was in which variant. Your product database knows what those users did. Your data warehouse or billing system knows the revenue impact. Joining these three sources to get a complete picture requires either a pre-built data pipeline or a data team engagement.

Guardrail metric checks are skipped because they require extra queries

Checking whether the variant degraded page load time, error rates, or churn alongside the primary metric means running additional queries on additional tables. Under time pressure, this step is routinely skipped — until a shipped variant is discovered to have caused a silent regression.

How AnalityQa AI AI solves it

Upload your data — or connect it live — and ask in plain English.

01

Upload your experiment data and get a statistically rigorous result in one query

Upload a CSV with variant assignment and metric outcomes, or connect your product database. AnalityQa AI AI selects the appropriate test for your metric type, calculates the lift with confidence intervals, and states plainly whether the result is significant at your chosen threshold.

02

Segment-level effect heatmap with multiple-comparison correction

Ask for a segment breakdown by plan tier, acquisition channel, or device type and AnalityQa AI AI applies a Bonferroni or Benjamini-Hochberg correction automatically, so the segments flagged as significant are actually worth acting on.

03

Guardrail metric check in the same pass as the primary metric

Specify your guardrail metrics — latency, error rate, unsubscribe rate — and AnalityQa AI AI evaluates them alongside the primary metric in one session. Any guardrail that moves significantly in the wrong direction is flagged before you ship.

04

Auto-join experiment assignments to product and business outcome data

Upload your variant assignment file and your event or revenue data separately. AnalityQa AI AI joins them on user ID and makes the merged dataset available for all queries — primary metric, guardrails, and segment analysis — without any manual data preparation.

05

Plain-English results summary for non-technical stakeholders

After the statistical analysis, ask for a summary you can paste into a product review. AnalityQa AI AI writes a concise, jargon-free recommendation stating the lift, the confidence level, the segment findings, and the guardrail status — no translation required.

You askedGenerated in 4.2s

"Calculate the lift in 7-day conversion rate for variant B vs. control, with 95% confidence intervals."

MRR

€328k+4.1%

Net retention

112%+3pp

Churn

2.4%−0.6pp

Summary table: conversion rate by variant, absolute lift, relative lift, 95% CI, p-value

Last 12 mo

Heatmap: variant lift by plan tier x device type, with significance markers

Table: guardrail metric check — direction, magnitude, significance, ship/hold flag

A dashboard built in AnalityQa AI — from question to chart, no SQL.

Real examples

Paste your data. Ask. Ship.

You

Calculate the lift in 7-day conversion rate for variant B vs. control, with 95% confidence intervals.

AI

AnalityQa AI AI runs a two-proportion z-test, calculates the absolute and relative lift, computes the 95% CI, and states whether the result clears the significance threshold at your sample size.

Summary table: conversion rate by variant, absolute lift, relative lift, 95% CI, p-value
You

Show me a segment-level effect heatmap for plan tier and device type, with multiple-comparison correction.

AI

It computes the variant effect for each segment combination, applies a Benjamini-Hochberg correction, and renders a heatmap where cells are shaded by effect size and marked significant or not.

Heatmap: variant lift by plan tier x device type, with significance markers
You

Check our guardrail metrics — page error rate, session length, and 30-day churn — for any regressions from the variant.

AI

AnalityQa AI AI runs significance tests on each guardrail metric and produces a status table showing the direction of change, the magnitude, and whether it is statistically significant.

Table: guardrail metric check — direction, magnitude, significance, ship/hold flag
You

How long would I need to run the test to detect a 5% relative lift in activation rate with 80% power?

AI

It computes the required sample size based on your baseline activation rate, desired lift, and power threshold, then estimates the days to reach that sample at your current traffic volume.

Sample size and runtime estimate: days to 80% power for 5% relative lift
You

Write a product review summary of the test results — lift, confidence, segment findings, guardrail status.

AI

AnalityQa AI AI produces a concise paragraph summarising the headline result, the most notable segment effect, and the guardrail check outcome, written for a non-technical audience.

Text: product review summary paragraph, ready to paste

What teams get out of it

✓Teams catch false positives from underpowered tests before shipping decisions are made.
✓Segment-level heatmaps with multiple-comparison correction surface actionable heterogeneous effects that aggregate results hide.
✓Guardrail metric checks in every analysis pass prevent silent regressions from reaching production.
✓Plain-English summaries reduce the time from test completion to stakeholder decision from days to the same afternoon.

Frequently asked questions

Which statistical tests does AnalityQa AI AI use for A/B test analysis?+

The test is selected based on your metric type. Conversion rates and proportions use a two-proportion z-test or chi-squared test. Continuous metrics like revenue per user or session length use a t-test or Mann-Whitney test depending on the distribution. You can override the selection if your team has a preferred method.

How does it handle multiple testing when I analyse several segments?+

By default, AnalityQa AI AI applies a Benjamini-Hochberg false discovery rate correction when you request a segment breakdown with more than three segments. You can also choose Bonferroni correction or no correction if you prefer, with a note in the output about the implications.

Can it detect network effects or interference between variants in experiments with social features?+

AnalityQa AI AI does not model network interference automatically, as this requires cluster-randomised experiment designs that depend on your specific product graph. You can upload cluster-level data and it will run appropriate cluster-level tests, but detecting interference from individual-level data is not supported.

How is experiment data containing user IDs handled?+

AnalityQa AI AI does not use uploaded data for model training, and supports pseudonymised or hashed user IDs if you prefer not to upload raw identifiers.

Can it analyse experiments where the randomisation unit is not the user — for example, page views or sessions?+

Yes. Specify the randomisation unit when you upload the data and AnalityQa AI AI adjusts the variance calculation accordingly. Analysing at a different unit than the randomisation unit — for example, aggregating page-view-randomised data to the user level — inflates false positives, and AnalityQa AI AI will flag this if it detects the mismatch.

How do I connect experiment assignment data from LaunchDarkly or Optimizely to my product event data?+

Export your variant assignment log from your feature flag tool and your event or outcome data as separate CSV files, or connect the database tables directly. AnalityQa AI AI joins on user ID and makes the merged dataset available for the full analysis — primary metric, segments, and guardrails — in one session.

What plan do I need to run guardrail metric checks alongside primary metric analysis?+

Multi-metric analysis — primary metric plus guardrails in the same session — is available on all plans including the free tier. Scheduled experiment monitoring with automated alerts is available on Pro and Business plans.

Related guides

Product

Know If Your Feature Launch Is Actually Working

SaaS / Customer Success

Understand How Customers Actually Use Your Product

Your data has answers. Start asking.

Upload a file or connect your database. Your first dashboard, in under 5 minutes.

Try AnalityQa AI AI free →

No credit card required

AnalityQa

Le workspace tout-en-un pour les data analysts et engineers.

Produit

  • Fonctionnalités
  • Tarifs
  • Blog
  • Outils gratuits

Société

  • À propos
  • Contact

Légal

  • Confidentialité
  • CGU

© 2026 AnalityQa AI. Tous droits réservés.

Tous les systèmes opérationnels