Incident: November 13, 2025
Report and dashboard numbers inflated for certain data providers
What Happened?
Issue: On November 13, 2025, customers reported that certain dashboards and reports were showing highly inflated numbers.
We sincerely apologize for the disruption this caused to your workflows and reporting schedules.
Impact Summary
What was experienced:
- Reports and dashboards were showing inflated numbers for specific data providers.
Timeline
Nov 12, 2025 - 14:55 MT (21:55 UTC)
Performance optimization deployed to production
Nov 13, 2025 - 09:12 MT (16:12 UTC)
First customer report received from CSM team
09:45 MT (16:45 UTC)
Engineering identifies root cause in previous day's deployment
09:59 MT (16:59 UTC)
Incident formally created in Better Stack; Critical Incident Response Team alerted
10:00 MT (17:00 UTC)
Incident acknowledged by Jason Collins; team begins investigation
10:08 MT (17:08 UTC)
Fix developed and tested against reported customer accounts
10:34 MT (17:34 UTC)
Production build pipeline started for release/25.11.6.0
10:39 MT (17:39 UTC)
Fix deployed to production as release/25.11.6.0
10:43 MT (17:43 UTC)
Fix verified; dashboards and reports showing correct numbers
10:53 MT (17:53 UTC)
Incident resolved and closed
Root Cause
On November 12, 2025, at 2:55pm MT, we deployed a performance optimization designed to reduce database query load for certain data integration types. This optimization inadvertently changed how the system filters certain data by advertiser within an agency.
Our Response
Our engineering team quickly identified the root cause and implemented a fix that restored proper advertiser-level filtering. The fix ensured that the performance optimization works correctly while maintaining proper advertiser-level data filtering.
Immediate Actions Taken:
- Rapid diagnosis: Identified the root cause within 10 minutes of it being reported
- Immediate Patch Deployed: A patch was developed and deployed immediately following diagnosis
What We're Doing to Prevent This
We've identified several improvements to prevent similar incidents and improve our response capabilities:
- 🔄 Data Anomaly Detection: We're always seeking ways to improve how we test and verify data. This includes looking for new and creative ways to monitor and detect data anomalies, which would help us prevent situations like this. Status: Continuously ongoing
Our Commitment to You
We understand that accurate reporting is critical to your business, and we fell short of our commitment to data accuracy during this short window.
We're grateful for your patience and continued trust in NinjaCat.
Questions or Concerns?
If you have any questions about this incident or continue to experience any issues with report processing, please don't hesitate to reach out to our support team. We're here to help.
Contact Support:
- Email: [email protected]
- In-app chat support
Updated 2 days ago