Measuring the impact of AWS Textract as a fallback for users whose ID verification failed at Margins (primary provider) due to mistyped ID numbers
When a Ghana KYC user's ID verification fails at Margins (the primary provider) with a FAILED_VERIFICATION reason, the treatment group gets a second chance via AWS Textract. Textract uses OCR to read the ID number directly from the uploaded card image, and if it differs from what the user manually typed (i.e., the user made a typo), a new KYC record is created with the corrected ID and Margins verification is retried automatically.
Why it works: Users often mistype their ID numbers during KYC submission. Textract reads the correct ID from the card image itself, fixing the mismatch that caused the initial failure.
The experiment has been running since Feb 25, 2026 with a 50/50 split:
Note: This fallback only triggers for FAILED_VERIFICATION — not for selfie failures, face comparison mismatches, or other rejection reasons. Textract is the 1st fallback tier, positioned before the existing Chenosis fallback in the chain.
KPI: Did the user ultimately pass KYC (KYC_VERIFIED = true) after being assigned to the experiment?
Confidence intervals do NOT overlap — Control upper bound (65.20%) < Treatment lower bound (66.28%)
| Week | Control Users | Control Pass Rate | Treatment Users | Treatment Pass Rate | Difference |
|---|---|---|---|---|---|
| Feb 23 – Mar 1 | 985 | 64.67% | 1,003 | 70.69% | +6.02 pp |
| Mar 2 – Mar 8 | 1,493 | 65.51% | 1,478 | 67.86% | +2.35 pp |
| Mar 9 – Mar 15 | 655 | 57.40% | 680 | 63.82% | +6.42 pp |
| Mar 16 – 17 | 18 | 61.11% | 16 | 75.00% | +13.89 pp |
Treatment outperforms control in ALL weeks consistently. The effect is stable and directionally consistent.
Note: Week of Mar 9 shows lower overall rates for both groups (users assigned more recently have had less time to complete KYC). Mar 16-17 has very few assignments and is excluded from conclusions.
The Textract ID fallback increases overall KYC verification success by 4.39 percentage points. This is highly significant (p = 0.000234) with strong power (95.7%).
Treatment group has 156 more KYC-verified users than control (2,158 vs 2,002), despite nearly identical group sizes. These are users who would have otherwise failed KYC.
Unlike the Mar 9 analysis which showed inconsistent weekly trends, the current data shows treatment winning every single week. The effect is real and stable.
Tables: banking_service.experiment_assignments + banking_service.user
Join: experiment identifier (USER_ID) = user.id
KPI: User's current KYC_VERIFIED boolean status
Test: Two-proportion Z-test (two-tailed, 95% CI, Wilson score intervals)
Effect size: Cohen's h = 0.0925 (small but practically meaningful)
| Aspect | Mar 9 Analysis | Mar 18 Analysis (Current) |
|---|---|---|
| KPI | Post-assignment recovery rate (from notification events) | Final KYC_VERIFIED status (from user table) |
| Data source | BACKEND_NOTIFICATIONS events | banking_service.user |
| Sample size | 5,217 (2,610 + 2,607) | 6,328 (3,151 + 3,177) |
| Period | Feb 22 – Mar 9 | Feb 25 – Mar 17 |
| Result | Not significant (p ~ 0.26, -1.54 pp) | Significant (p = 0.000234, +4.39 pp) |
The KYC_VERIFIED approach captures the user's final state regardless of when/how they completed verification. This includes users who may have verified days after assignment. The event-based approach in the previous analysis only captured immediate post-assignment recoveries and may have missed delayed successes.
The results are conclusive: treatment significantly improves KYC pass rate by 4.39 pp with high statistical confidence (p = 0.000234) and strong power (95.7%). The effect is consistent across all weeks.
After shipping to 100%, continue monitoring: