How actually a glitch at Google caused a global technical meltdown?

OVERVIEW

On 14th Dec 2020, there was a big outage of Google services. It started around 3:45 AM PST and users complained YouTube is not working, Gmail throwing errors, Meet also down, Drive not opening. Later Google Workspace Status Dashboard displayed the status of all its services where it was clearly visible that the whole bunch of services were down. The outage lasted for a duration of 47 minutes. This incident is tagged as “Google Cloud Infrastructure Components Incident #20013” on Google Cloud Status Dashboard.

What was the glitch?

The company reveals the glitch and the root cause behind it. Google OAuth, the customer-facing Google service was unavailable along with Google’s central identity-management system, blocking users from login into any of Google services.

Google says the root problem was an issue with the reduced capacity in its storage quota management system. As part of a migration, the User ID Service was moved to a new quota system in October but some parts of the old quota system were left in place which incorrectly reported the usage for the User ID Service as zero.

<<<Also read: Memory Management: Android vs iOS>>>

Google’s User ID Service handles unique identifiers for user accounts that authenticate OAuth tokens and cookies. Google OAuth is used for authentication and authorization from Google’s central identity management system.

The issue was detected by Google’s automated alert system and end-users also reported the same over the internet. The services were immediately recovered when the new quota for the User ID service took effect.

The global technical meltdown

Here is the Google Workspace Status Dashboard displaying all the Google services for 14th Dec which were 100% impacted for 47 minutes.

Incident Start: 2020-12-14 03:45 PST

Incident End: 2020-12-14 04:35 PST

Console login to Google cloud was also not working although the cloud service accounts in the GCP were functioning normally without any issue.

This outage not only impacted Google itself but all the Google customers who rely on Google services and the Google cloud platform. Here is the list of Google cloud services that had a significant impact –

● Cloud Console

● Google BigQuery

● Google Cloud Storage

● Google Cloud Networking

● Google Kubernetes Engine

● Google Workspace

● Cloud Support

The global reaction

The global reaction was big and open. Many global leaders, CEOs expressed their opinion on the public cloud. Google spokesperson said, “Services like YouTube were working in incognito mode but a login requiring service was unavailable and experienced errors during the outage.” Bob Venero, CEO of Holbrook said, “The cloud isn’t going to run always, there will be outages and shutdowns. The companies totally dependent on the public cloud are risking their credibility and stock price. Dell Technologies CEO Michael Dell also commented on the public cloud that it can be more expensive and risky than the on-premise solutions if outages become more and more frequent.

Google apologized for this huge inconvenience to its users and thanked them for their patience and support.

References:

How actually a glitch at Google caused global technical meltdown?

How actually a glitch at Google caused a global technical meltdown?

OVERVIEW

What was the glitch?