Incident: November 3-5, 2025

📘

What Happened?

Between Monday, November 3rd and Wednesday, November 5th, 2025, NinjaCat customers experienced significant delays in report processing. Reports that normally complete in seconds were taking multiple hours, with some reports appearing stuck in a "Getting Data" state.

We sincerely apologize for the disruption this caused to your workflows and reporting schedules.

Impact Summary

What was experienced:

• Reports stuck in "Getting Data" state for extended periods (multiple hours)
• Significant delays in dashboard updates and scheduled reports
• Slower than normal report generation across all report types

Timeline

Sunday, November 2 (Evening)

Our data warehouse provider (Snowflake) deployed an internal optimization that unexpectedly degraded our query performance. This change was not communicated to us in advance.

Monday, November 3 (8:00 AM MT)

Issue detected as customers began reporting delays. Our team immediately began investigating and identified the root cause within 90 minutes.

Monday, November 3 (10:05 AM MT)

Emergency infrastructure scaling deployed - we increased our processing capacity by 400% to maintain service levels while working with Snowflake on a permanent fix.

Tuesday, November 4 (Throughout Day)

Processed 35,000 reports while implementing code optimizations to improve query resilience.

Tuesday, November 4 (8:03 PM PT)

Snowflake reverted their internal change, restoring normal query performance.

Wednesday, November 5 (10:30 AM MT)

All report backlogs cleared and system performance returned to normal. Incident fully resolved.

Root Cause

The incident was caused by an undocumented internal optimization deployed by our data warehouse provider, Snowflake. This change altered how certain database queries are executed, causing our report queries to slow from approximately 5 seconds to over 4 hours.

This issue was compounded by the timing - it occurred during one of our highest-volume reporting periods of the year, when the first of the month and first Monday of the month fell close together, resulting in nearly double our normal report volume.

Important note: While the root cause was outside our direct control, we identified several improvements to our systems and processes to better handle similar situations in the future.

Our Response

Our team took immediate action to minimize customer impact:

Immediate Actions (Monday Morning)

• Rapid diagnosis: Identified the root cause within 90 minutes of detection
• Engaged vendor support: Opened critical support ticket with Snowflake
• Emergency scaling: Increased processing capacity by 400%
  • Scaled from 35 to 75 data warehouses
  • Increased report workers from 80 to 320
  • Added additional application servers
  
  

Ongoing Improvements (Tuesday-Thursday)

• Code optimizations: Implemented query improvements to make our system more resilient to vendor changes
• Backlog processing: Maintained elevated capacity to clear the accumulated report backlog
• Continuous monitoring: Enhanced monitoring to track recovery progress

What We're Doing to Prevent This

We've identified several improvements to prevent similar incidents and improve our response capabilities:

• ✅ Query Optimization: We've restructured our database queries to be more resilient to vendor infrastructure changes, reducing our vulnerability to similar performance degradations.
• 🔄 Incident Response Training: We're developing comprehensive runbooks and conducting regular training exercises for emergency scaling like we did here for high-severity incident response. Status: Continuously ongoing
• 🔄 Vendor Relationship Management: We're working with Snowflake to establish better communication channels for infrastructure changes to prevent similar incidents.

Our Commitment to You

We understand how critical timely reporting is to your business. This incident has reinforced our commitment to building a more resilient platform and improving our ability to respond quickly when issues arise.

We're grateful for your patience and continued trust in NinjaCat.

Questions or Concerns?

If you have any questions about this incident or continue to experience any issues with report processing, please don't hesitate to reach out to our support team. We're here to help.

Contact Support:

• Email: [email protected]
• In-app chat support