Welcome to Part 1 of the Go Distributed Systems Lab series! Over the course of 20 hands-on projects, we are building core distributed systems primitives from the ground up using Go 1.22+ and the standard library (net, sync, context, log/slog, encoding/binary).
Before jumping into raw socket framing, gossip protocols, or Raft consensus, we need to master the foundational concurrency building blocks inside a single process: Goroutines, Channels, and Communicating Sequential Processes (CSP).
💡 The Philosophy: Share Memory by Communicating
In traditional concurrent programming (like C++ or Java), thread synchronization often relies on shared memory protected by mutexes, lock-free queues, or read-write locks.
Go flips this model with a core design principle:
"Do not communicate by sharing memory; instead, share memory by communicating."
By passing ownership of data structures through Go channels, each pipeline stage operates on isolated memory. This eliminates data races by design without requiring explicit lock management (sync.Mutex).
🏗️ Architecture & Component Design
In this first module (01-message-passing), we construct a 3-stage data processing pipeline:
+------------------+ Job Channel +------------------+ Result Channel +-------------------+
| Producer | -------------------------> | Worker | ------------------------> | Collector |
| (Generates Jobs) | (Buffered, cap=10) | (Isolated State) | (Buffered, cap=10) | (Aggregates Data) |
+------------------+ +------------------+ +-------------------+
1. Ingestion Stage (Producer)
Generates typed Job values and pushes them into a direction-constrained buffered channel (chan<- Job). When generation finishes, it closes the channel to broadcast an end-of-stream signal.
2. Processing Stage (Worker)
Consumes from <-chan Job using Go's for job := range in construct. The worker maintains internal execution metrics (e.g., processedCount) entirely within its local stack scope—no locks required.
3. Collector Stage
Receives output items from <-chan Result and processes them asynchronously until the channel is closed.
💻 Full Implementation
Here is the complete standard-library implementation using Go 1.22+:
package main
import (
"fmt"
"log/slog"
"os"
"sync"
"time"
)
// Job represents a unit of work passed between isolated pipeline stages.
type Job struct {
ID int
Payload string
Processed bool
}
// Result represents the output produced by the processor stage.
type Result struct {
JobID int
Output string
Timestamp time.Time
}
func main() {
// Initialize standard structured logger (log/slog)
logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
slog.SetDefault(logger)
slog.Info("Starting message passing pipeline", "stage", "initialization")
// Channels acting as bounded queues
jobs := make(chan Job, 10)
results := make(chan Result, 10)
var wg sync.WaitGroup
// Stage 1: Ingestion Producer
wg.Add(1)
go producer(jobs, 5, &wg)
// Stage 2: Processing Worker (Isolated State Pipeline)
wg.Add(1)
go worker(1, jobs, results, &wg)
// Stage 3: Aggregator / Collector
done := make(chan struct{})
go collector(results, done)
// Synchronize pipeline termination
wg.Wait()
close(results) // Signal collector to exit once upstream workers complete
<-done // Block until collector drains remaining results
slog.Info("Pipeline processing complete", "status", "success")
}
// producer generates work items and sends them through the buffered channel.
func producer(out chan<- Job, count int, wg *sync.WaitGroup) {
defer wg.Done()
defer close(out) // Closing channel signals EOF to downstream consumers
for i := 1; i <= count; i++ {
job := Job{
ID: i,
Payload: fmt.Sprintf("task-payload-%d", i),
}
slog.Info("Producing job", "job_id", job.ID)
out <- job
time.Sleep(50 * time.Millisecond)
}
}
// worker encapsulates isolated processing state without shared memory locks.
func worker(id int, in <-chan Job, out chan<- Result, wg *sync.WaitGroup) {
defer wg.Done()
// Isolated state local to this goroutine - thread-safe without mutexes!
processedCount := 0
for job := range in {
processedCount++
slog.Info("Worker processing job",
"worker_id", id,
"job_id", job.ID,
"local_total", processedCount,
)
// Simulate computation / processing overhead
time.Sleep(100 * time.Millisecond)
out <- Result{
JobID: job.ID,
Output: fmt.Sprintf("PROCESSED[%s]", job.Payload),
Timestamp: time.Now(),
}
}
}
// collector receives final stage outputs until the stream is exhausted.
func collector(in <-chan Result, done chan<- struct{}) {
for res := range in {
slog.Info("Result collected",
"job_id", res.JobID,
"output", res.Output,
"time", res.Timestamp.Format(time.RFC3339Nano),
)
}
close(done)
}
🔍 Key Go Concurrency Patterns Explained
1. Directional Channel Types
Notice the parameter signatures in our pipeline functions:
-
out chan<- Job: Send-only channel. Writing into this is allowed; reading or closing from inside the caller scope (if not intended) produces a compile-time error. -
in <-chan Job: Receive-only channel. The worker can only consume from it.
Restricting channel direction at API boundaries prevents accidental closed-channel writes or unauthorized channel closures.
2. Clean Channel Drain Mechanics
When producer() executes close(out), it doesn't delete existing values inside the buffer. Instead, it marks the channel as closed.
Downstream in worker():
for job := range in { ... }
The for range loop continuously extracts items until the channel is empty AND closed, at which point the loop cleanly exits.
3. Structured Logging with log/slog
Since Go 1.21, log/slog provides structured key-value logging natively. In distributed systems, plain text strings become impossible to query. Using slog.Info("...", "job_id", job.ID) ensures consistent log parsing across asynchronous routines.
⚠️ Architectural Limitations of In-Process Message Passing
While in-process channels provide clean abstractions, they have clear boundaries when designing real-world distributed systems:
-
Process Boundary Restrictions: Channels are strictly in-memory data structures managed by the Go runtime scheduler (
m:ngoroutine multiplexing). They cannot cross host, network, or process boundaries. - Volatile Memory: If the application panics or crashes, all messages currently sitting inside buffered channels are lost permanently.
- Coordinated Backpressure: Channels handle backpressure synchronously (blocking on send when full). However, if an upstream system floods the channel faster than workers can consume, memory usage grows up to channel capacity before blocking cascades backward.
- No Error Return Channels: This basic pattern lacks a mechanism for workers to report execution errors or request task retries.
🚀 What's Next?
In Part 2, we will address the asynchronous communication problem by building a Correlated Request/Reply mechanism. We will implement unique correlation IDs, response channels, and request multiplexing to turn one-way pipelines into interactive distributed calls!
What concurrency pattern do you use most in your Go services? Drop a comment below! 👇
Source code of the same can be found at : https://github.com/pckrishnadas88/go-distributed-systems-lab/tree/main/01-message-passing