mini-GFS

A minimal, educational implementation of the Google File System (GFS) architecture in Go, using gRPC for internal communication. Files are split into chunks, replicated across chunkservers, and encrypted end-to-end before they ever leave the client.

The system has three components, each a separate binary:

• Master (master/) — the single coordinator. Keeps track of which files exist, which chunks make up each file, which chunkservers hold each chunk, and which chunkservers are alive (via heartbeats). Persists this metadata to master/metadata.json so it survives restarts.
• Chunkserver (chunkserver/) — dumb storage nodes. Each one stores chunk blobs on local disk (in a chunks_<port>/ directory) and exposes WriteChunk / ReadChunk / ReplicateChunk RPCs. You run as many of these as you want; each sends periodic heartbeats to the master so it's counted as alive.
• Client (client/) — a gateway that also serves the web frontend. It talks to the master and chunkservers over gRPC on behalf of whoever uploads/downloads a file through the browser.

All inter-service communication (client ↔ master, client ↔ chunkserver, master ↔ chunkserver for replication) uses gRPC, defined in proto/master.proto and proto/chunkserver.proto.

1. Browser POSTs the file to the client's /upload endpoint.
2. The client generates a random AES-256 key and encrypts the file with AES-GCM.
3. The AES key itself is encrypted with the client's RSA public key (RSA-OAEP).
4. The client calls AssignChunk on the master, which picks ReplicationFactor (currently 2) live chunkservers and returns their addresses plus a new chunk handle (UUID).
5. The client writes the encrypted chunk to every assigned chunkserver.
6. The master persists the mapping of filename → chunk handle → locations → encrypted AES key to metadata.json.

1. Browser hits /download?file=<name>.
2. The client asks the master (GetChunkLocations) for the file's chunks, their storage locations, and their wrapped AES keys.
3. For each chunk, the client reads the encrypted bytes from one of its replica locations, decrypts the AES key with its RSA private key, then decrypts the chunk data.
4. Decrypted chunks are concatenated in order and streamed back to the browser.

Chunkservers send a heartbeat to the master every 5 seconds. A background loop on the master checks every 10 seconds for servers that haven't been heard from in over 15 seconds and marks them dead. If a chunk had a replica on a dead server, the master picks a live server that doesn't already have a copy and asks it to pull the chunk directly from a surviving replica (ReplicateChunk), then updates the chunk's location list.

• Chunk data is encrypted client-side with AES-256-GCM before it's sent anywhere — chunkservers only ever see ciphertext.
• The AES key is itself encrypted with RSA-OAEP using a keypair generated on first run and stored in client/keys/ (private.pem / public.pem).
• The master stores the RSA-wrapped AES key per chunk, but never sees the plaintext key or data.

You need 4 processes: 1 master, at least ReplicationFactor (2) chunkservers, and 1 client.

# terminal 1 — master (must be run from master/ so metadata.json lands there)
cd master && go run .

# terminal 2 — chunkserver on port 50052
cd chunkserver && go run . -port 50052

# terminal 3 — chunkserver on port 50054
cd chunkserver && go run . -port 50054

# terminal 4 — client + web frontend
cd client && go run .

Then open http://localhost:8080 in a browser. Drag a file in (or click to choose one) and hit Upload — you'll get back the chunk handle it was stored under. Enter a filename in the download box to fetch it back.

master/       — coordinator: chunk assignment, metadata, liveness tracking
chunkserver/  — chunk storage node (data + gRPC handlers)
client/       — gRPC gateway + HTTP/web frontend (upload & download)
proto/        — protobuf service definitions and generated Go code

If you edit proto/*.proto, regenerate the Go bindings with:

protoc --go_out=. --go_opt=paths=source_relative \
       --go-grpc_out=. --go-grpc_opt=paths=source_relative \
       proto/master.proto proto/chunkserver.proto

This is a learning project, not production software:

• Single master — no failover if it goes down, and it's a single point of metadata truth.
• No authentication/authorization between services — anyone who can reach the ports can read/write chunks.
• No chunk checksums beyond AES-GCM's built-in tamper detection.
• Replication factor is fixed at 2 and hardcoded.