There were NO outages this weekend.  We found at least one area of code causing a slow memory leak that has been fixed. We are still analyzing memory dumps from Sunday to make sure there are no further memory leaks. 

Site Outage Sunday, Nov 23: (12:10–12:35, 2:32 - 2:48pm PM MST)

Similar to last week’s 6 PM interruption, the good news is that these outages were not related to the database-storm issue from earlier in the season. Those metrics continue to look solid.

This interruption was caused by the memory-pressure issue we’ve been actively working on. We recently rolled out caching improvements to shrink large in-memory objects (especially standings data, which naturally grows each week). Those changes helped, but they didn’t fully prevent today’s spike. The downtime was also extended because, at the same time due to sheer bad luck, AWS console access was misbehaving on our end, which delayed us manually issuing a clean restart. We have additional fixes scheduled for mid-week, and we’ll also be upgrading the server to double its CPU and memory to provide more headroom going forward.

Further investigation of this consistently high-traffic window revealed additional opportunities for code optimization. This outage stemmed from the same underlying memory-pressure issue, compounded by increased CPU demand from a surge of /home page traffic. We have additional improvements planned specifically for the Home page—these will benefit performance across the entire site—and the upcoming server upgrade will greatly improve stability during peak-traffic bursts.

Site outage Monday Nov 17  4:55am - 8:15am

This was not our issue, its Cloudflare and affecting websites across the globe. We are switching from Cloudflare to AWS Route 53, which is considered the absolute best.

Site outage Sunday Nov 16  2:35pm (~10 min), 5:25pm (5 min), 6:20pm (35 min) MST.

The outage at ~6:20pm, was not related to heavy traffic, but instead due to a memory leak over time that hit hard during this window but still allowed just enough activity to not trigger a restart of the app. Both developers were heading home form the awesome Broncos victory over the Chiefs and therefore were not able to do a timely manually restart. 

Site outage Sunday Nov 9, 11:45am, 2:15pm to 2:35pm MST.

We experienced two short outages today at approximately 11:45am MST and again from 2:15pm to 2:35pm MST. The issue was traced to high memory usage, primarily triggered by the GameDay page’s caching behavior under load. This was different from the typical database traffic spikes we see on Sunday afternoon.

We have already made code changes to reduce memory usage and prevent this specific condition from recurring. The changes will be uploaded to the site late this evening, so its possible a brief outage could happen before then.