A loyalty team runs an offer test, and the offer group spends more than the holdout. Statistical significance answers the first question that follows. Is the gap bigger than chance alone would produce? It says nothing about whether the gap pays for the offer.
What does statistically significant mean?
A result is statistically significant when the data would be unlikely if there were no real effect. The test starts from a null hypothesis, the claim that the groups do not differ. It then asks how often chance alone would yield a gap this large. NIST defines the p-value as the chance of a result at least as extreme as the one seen, if the null hypothesis is true (NIST).
The analyst sets a cutoff before the test, called the significance level or alpha. If the p-value falls below it, the result is called significant. At 0.05, the test will reject a true null hypothesis 5% of the time. NIST calls the choice “somewhat arbitrary” and lists 0.1, 0.05 and 0.01 as common values (NIST).
How do you calculate statistical significance?
Every test follows the same steps. State the null hypothesis. Choose alpha and the sample size before any data comes in. Run the test and compute a test statistic, such as a z or t score. Turn that score into a p-value and compare it with alpha.
For two conversion rates, the z score is the gap between the rates divided by its standard error. The standard error shrinks as the groups grow. So the same one-point gap can fail in a small test and pass in a large one. In NIST’s sample size formula, the members you need grow with the square of the spread over the shift you want to catch (NIST).
| Term | What it means | In a loyalty offer test |
|---|---|---|
| Null hypothesis | The claim that there is no real difference | The offer does not change member behavior |
| p-value | The chance of a result at least this extreme if the null is true | How often a gap this big would show up with no offer effect |
| Significance level (alpha) | The cutoff set before the test, often 0.05 | The false alarm rate the team accepts |
| Type I error | Calling a chance result real | Rolling out an offer that does nothing |
| Type II error (beta) | Missing a real effect | Dropping an offer that works |
| Power (1 minus beta) | The chance the test catches a real effect of a given size | Set by the holdout size and how much spend varies |
| Minimum detectable effect | The smallest effect the test is sized to catch | The smallest lift worth paying for |
Why is 0.05 the threshold for significance?
The habit goes back to Ronald Fisher. In 1925 he wrote that the value for which P = .05, or 1 in 20, is 1.96 or nearly 2. He called it “convenient to take this point as a limit” for judging a result significant (Fisher, 1925). The line began as a working rule of thumb.
Some scientists want a stricter line. In 2018, a large group of researchers proposed moving the default from 0.05 to 0.005 for claims of new discoveries (Benjamin et al., 2018). Business tests seldom need that bar. The right alpha depends on what a false alarm costs. A wrong rollout to millions of members costs more than a wrong subject line.
What is the difference between statistical and practical significance?
Statistical significance says a gap is unlikely to be chance. Practical significance says the gap is big enough to matter. The American Statistical Association drew this line in a 2016 statement. It says a p-value does not measure the size of an effect. Nor does it measure how much a result matters. Business decisions, it adds, should not rest only on a p-value passing a line (ASA, 2016).
Scale makes the two easy to confuse. Kohavi and Thomke describe a Bing headline test that raised revenue 12%, worth more than $100 million a year in the US (HBR, 2017). In a program with millions of members, even a few cents per member can clear 0.05. The real test is whether the lift covers the reward that bought it. Report the size of the effect and its range next to the p-value.
What mistakes break significance tests in marketing?
The first is peeking. A team checks the dashboard each day and stops when the result turns significant. The math assumes the sample size was fixed in advance. Evan Miller shows that peeking ten times turns a reported 1% level into a true 5% (Miller, 2010). Fix the sample size first and read the result once.
The second is testing many things at once. Each comparison at 0.05 carries its own 5% risk of a false alarm. Test 20 segments where the offer does nothing, and the chance that at least one looks significant is about 64%. The Bonferroni method guards against this by testing each comparison at alpha divided by the number of comparisons (NIST).
The third is the wrong unit. If members are assigned to groups, analyze the result by member. One member’s purchases are tied to each other. Counting each transaction as its own data point makes the test look more certain than it is.
How do loyalty programs use statistical significance?
Loyalty teams use it to read holdout tests. A random group of members gets no offer, and the rest get it. Our entry on control group testing covers the design. Significance then tells the team whether the gap in spend is beyond chance. The loyalty program ROI guide turns that gap into profit.
Spend is the hard metric to test. Members range from people who buy once a year to people who buy every week, so spend varies far more than a yes or no outcome. That spread widens the standard error, and the holdout has to grow to match. A rate, such as the share of members who shop, reaches a clear answer with fewer members.
Real loyalty field tests often land close to the line. In a café reward program, researchers gave some members a 12-stamp card with two bonus stamps. Others got a plain 10-stamp card, so both needed ten coffees. The bonus group finished in 12.7 days against 15.6, about 20% faster, with t = 2.0 and p < .05 (Kivetz, Urminsky and Zheng, 2006). That t value sits right at Fisher’s “nearly 2.”
For illustration, a grocery program sends a double points offer to 20,000 members and holds out 20,000 more. Within 30 days, 25.0% of the offer group shop, against 24.0% of the holdout. A z-test on that one-point gap gives z = 2.33 and a p-value near 0.02. With 2,000 members a side, the same rates give a p-value near 0.46.
Now price the significant result. One point of 20,000 members is 200 extra shoppers. At $30 of margin each, that is $6,000. The bonus cost $1.50 for each of the 5,000 shoppers in the offer group, or $7,500. The test passed at 0.05, and the offer lost $1,500.
How GRAVTY handles statistical significance
A significance test is only as good as its groups and its records. GRAVTY keeps every earn, burn, tier change and offer response on one member record through Member 360. Agentic AI Compass lets a team compare a treated segment against a held-out one in plain language. On lji.io, AgenticTest runs live A/B simulation before launch and watches lift while a campaign runs.
Frequently asked questions
What is a p-value in simple terms?
A p-value is the chance of a result at least as extreme as yours if there were no real effect. A p-value of 0.03 means a gap that big would show up about 3 times in 100 by chance. It is not the chance that your hypothesis is true. The ASA lists that misreading among its six principles (ASA, 2016).
What does 95% confidence mean in an A/B test?
It means the test used a significance level of 0.05. When the two versions truly perform the same, the test will still flag a winner 5% of the time (NIST). It does not mean a 95% chance that the winner is better.
How large should a loyalty holdout group be?
Large enough to catch the smallest lift worth paying for. The needed size grows with how much member behavior varies and shrinks as the lift you need gets bigger (NIST). For illustration, catching a move from 24% to 25% in a shopping rate takes about 29,000 members per group. That assumes alpha of 0.05 and 80% power.
Can a statistically significant result be wrong?
Yes. At a 0.05 level, about 1 test in 20 of an offer with no effect will still come out significant. Peeking raises that rate. Evan Miller shows that ten peeks turn a reported 1% level into a true 5% (Miller, 2010). Repeat a surprising win before you scale it.
Does statistical significance prove that an offer caused the lift?
Only when members were assigned to groups at random. Random assignment makes the groups alike, so a significant gap points to the offer. In a member against non-member comparison, a significant gap only says the groups differ. Members chose to join, so part of the gap was there before the program.
