Quadric - Notice history

While deploying some application and configuration updates a cache restart was required. When restarting the cache, loading from a corrupted saved snapshot caused startup failures.

After removing the corrupted cache snapshot, we were able to successfully restart our cache service from a differently slightly older snapshot.

We will investigate solutions to mitigate this problem if it occurs again.

We've identified the issue with the cache and are working on a fix.

New features were deployed requiring a reconfiguration of our docs search index. The reconfiguration required re-indexing all of the documentation.

The improved search functionality required more CPU resources compared to the previous implementation. When the re-indexing was triggered, high CPU usage occurred causing a backlog of index writes and ultimately connection timeouts to the search service. Automatic retries of index record insertions further compounded the issue.

After cancelling all the search index tasks, to avoid the automatic retry, we were then able to increase our search service capacity. With the capacity increased, we were able to attempt the indexing operation again.

We shall review the incident and make changes to reduce the risk of resource exhaustion if we need to rebuild our search indexes.

We implemented a fix and are currently monitoring the index rebuild.

We are rebuilding our search index. This may take several minutes.

A recent code change requiring docs search index rebuild is experiencing connection timeouts. We are evaluating possible solutions.