Case Study: How did Google store the entire internet on cheap, dying hard drives?

Reading Time: 8 mins

How did Google store the entire internet on cheap, dying hard drives?

Goal: Understand how sharding, replication, and leader election address the capacity and reliability limits of single machines.

Problem 1: A Mountain of Trash Hardware

In the early 2000s, Google faced the challenge of storing petabytes of web crawl data. Buying high-end supercomputers from legacy vendors was financially prohibitive. Instead, they opted for a different strategy: linking thousands of cheap, consumer-grade hard drives. Back then, these were spinning disks, not the SSDs of today.

The Catch: Consumer HDDs are unreliable. With a Mean Time To Failure (MTTF) that was laughably low, having 10,000 spinning disks meant daily failures were a certainty. Google needed software resilient enough to handle such unreliable hardware.

Problem 2: The Capacity Problem (Sharding)

The Problem: The web crawl reached 100 Terabytes, while the largest hard drive was just 1 Terabyte. A single machine couldn't handle it.

The Solution (Chunks / Sharding): Google's engineers split the massive file into 64-Megabyte pieces, known as "Chunks."

The Two-Tier Architecture: Google developed a two-tier system with a Leader and numerous Chunk Servers.

  • Chunk Servers: These are the cheap machines storing the 64MB data chunks.

  • Leader: This central node acts as a directory. It uses a Hash Function on file names to locate which Chunk Servers hold specific chunks. This is Hash Partitioning. By hashing file names, storage is evenly distributed across the cluster.

Problem 3: The Reliability Problem (Replication)

The Problem: With 1,000 chunks across 1,000 Chunk Servers and low MTTF, server failures were inevitable. Losing even one chunk could corrupt the entire 100TB file.

The Solution (Replicas): Paranoia-driven design. Each 64MB chunk is saved as three identical copies (Replicas) on different Chunk Servers, ideally on separate power racks.

The Result is Self-Healing: When a Chunk Server fails, the Leader redirects requests to a surviving replica and commands the creation of a new replica on a healthy server.

Summary: The Impact

By integrating Sharding for capacity, Replication for reliability, and Leader Election for availability, the Google File System (GFS) demonstrated that a highly scalable and reliable system could be built from unreliable hardware. This architecture laid the groundwork for Hadoop (HDFS) and the modern cloud.

Read the Original Paper

Explore the original 2003 paper on the Google File System, a seminal work in computer science that initiated the "Big Data" era. Focus on the overarching architecture.

If the embed doesn't load in your browser, open the paper directly: The Google File System (research.google).

Where This Leads

GFS was the starting gun. Each of the limits it ran into got its own line of research, and each of those is its own page in this module:

  • Master as a single point of failure → Consensus & Leader Election replaces one Master with a Paxos/Raft-coordinated council.

  • Resharding pain when machines join or leave → Distributed Sort & Hash introduces consistent hashing (the ring), used by Dynamo and Cassandra.

  • 3× replication's storage overhead → Distributed File Systems traces the move to erasure coding in Colossus (~50% overhead instead of 200%).

  • Master metadata RAM ceiling → Distributed File Systems covers the move to sharded metadata (Colossus, HDFS Federation).

The next page picks up the thread from here, generalizing GFS into the four-generation pattern (GFS → HDFS → S3 → Colossus).