Design a Load Balancer

Problem Statement

Design a software-level Load Balancer. The load balancer must support routing client requests to a pool of backend server instances using pluggable algorithms (like Round Robin, Weighted Round Robin, or Least Connections). Additionally, it must support dynamic server registration and run background health checks to automatically isolate unhealthy nodes and restore them once they recover.

Asked In Companies

Google Uber Amazon Apple

---

Functional Requirements

• Support pluggable load balancing routing algorithms: Round Robin, Weighted Round Robin, and Least Connections.
• Allow dynamic server registration and deregistration in the server pool.
• Execute periodic background health checks to ping servers, marking them offline if pings fail.
• Only route requests to backend instances that are currently healthy.
• Ensure thread safety when accessing the shared server pool and incrementing request loads concurrently.

---

Objects Required

• ServerInstance (Data container representing backend instances)
• LoadBalancingStrategy (Interface defining routing contracts)
• RoundRobinStrategy (Concrete algorithm looping sequentially)
• WeightedRoundRobinStrategy (Concrete algorithm distributing by weight ratios)
• LeastConnectionsStrategy (Concrete algorithm targeting idle nodes)
• LoadBalancer (Main controller orchestrating routing and health check runs)

---

ServerInstance Class

The ServerInstance class holds properties for an active server node, including target parameters like IP, traffic weight, active connection counter, and current status flags.

public class ServerInstance {
    private final String ip;
    private final int weight;
    private int activeConnections;
    private boolean isHealthy;

    public ServerInstance(String ip, int weight) {
        this.ip = ip;
        this.weight = weight;
        this.activeConnections = 0;
        this.isHealthy = true;
    }

    public String getIp() { return ip; }
    public int getWeight() { return weight; }
    
    public synchronized int getActiveConnections() { return activeConnections; }
    public synchronized void incrementConnections() { activeConnections++; }
    public synchronized void decrementConnections() { 
        if (activeConnections > 0) activeConnections--; 
    }

    public synchronized boolean isHealthy() { return isHealthy; }
    public synchronized void setHealthy(boolean healthy) { isHealthy = healthy; }
}

The constructor sets the initial metrics and marks the node as healthy. The methods modifying connection state and health markers are synchronized to ensure thread safety when accessed by multiple threads.

---

LoadBalancingStrategy Interface & Implementations

Applying the **Strategy Design Pattern** allows us to swap the load balancing algorithms at runtime without modifying the caller class.

import java.util.List;

public interface LoadBalancingStrategy {
    ServerInstance selectServer(List<ServerInstance> servers);
}

The selectServer() method evaluates a list of healthy servers and returns a single instance based on the chosen strategy. Here are the three concrete implementations:

public class RoundRobinStrategy implements LoadBalancingStrategy {
    private int index = 0;

    @Override
    public synchronized ServerInstance selectServer(List<ServerInstance> servers) {
        if (servers.isEmpty()) return null;
        ServerInstance server = servers.get(index % servers.size());
        index = (index + 1) % servers.size();
        return server;
    }
}

The RoundRobinStrategy uses a synchronized index counter to return server instances sequentially, ensuring even distribution.

public class WeightedRoundRobinStrategy implements LoadBalancingStrategy {
    private int index = 0;

    @Override
    public synchronized ServerInstance selectServer(List<ServerInstance> servers) {
        if (servers.isEmpty()) return null;

        // Sum weights of all healthy servers
        int totalWeight = 0;
        for (ServerInstance server : servers) {
            totalWeight += server.getWeight();
        }

        // Select server based on index modulo total weight
        int targetWeight = index % totalWeight;
        index = (index + 1) % totalWeight;

        for (ServerInstance server : servers) {
            targetWeight -= server.getWeight();
            if (targetWeight < 0) {
                return server;
            }
        }
        return servers.get(0);
    }
}

The WeightedRoundRobinStrategy calculates total group weights and allocates traffic proportionally. Servers with higher weights receive a larger share of requests.

import java.util.Comparator;

public class LeastConnectionsStrategy implements LoadBalancingStrategy {
    @Override
    public ServerInstance selectServer(List<ServerInstance> servers) {
        if (servers.isEmpty()) return null;
        // Find the healthy server instance with the lowest active connection count
        return servers.stream()
                .min(Comparator.comparingInt(ServerInstance::getActiveConnections))
                .orElse(null);
    }
}

The LeastConnectionsStrategy uses streams to select the server with the lowest connection count, which is ideal for routing intensive I/O operations to idle resources.

---

LoadBalancer Class

The LoadBalancer manages the server instances, executes routing decisions, and runs background health checks.

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;

public class LoadBalancer {
    private final List<ServerInstance> servers;
    private LoadBalancingStrategy strategy;
    private final ScheduledExecutorService healthCheckerScheduler;

    public LoadBalancer(LoadBalancingStrategy strategy) {
        this.servers = new CopyOnWriteArrayList<>(); // Thread-safe concurrent list
        this.strategy = strategy;
        this.healthCheckerScheduler = Executors.newSingleThreadScheduledExecutor(runnable -> {
            Thread t = new Thread(runnable);
            t.setDaemon(true);
            t.setName("LB-Health-Checker");
            return t;
        });
    }

    public void registerServer(ServerInstance server) {
        servers.add(server);
        System.out.println("Registered server: " + server.getIp());
    }

    public void unregisterServer(String ip) {
        servers.removeIf(server -> server.getIp().equals(ip));
        System.out.println("Unregistered server: " + ip);
    }

    public synchronized void setStrategy(LoadBalancingStrategy strategy) {
        this.strategy = strategy;
        System.out.println("Load balancer strategy changed to: " + strategy.getClass().getSimpleName());
    }

    public ServerInstance routeRequest() {
        // Filter out unhealthy nodes
        List<ServerInstance> healthyServers = servers.stream()
                .filter(ServerInstance::isHealthy)
                .collect(Collectors.toList());

        if (healthyServers.isEmpty()) {
            throw new RuntimeException("503 Service Unavailable: No healthy servers found.");
        }

        ServerInstance selected = strategy.selectServer(healthyServers);
        if (selected != null) {
            selected.incrementConnections();
        }
        return selected;
    }

    public void startHealthCheck(long intervalMs) {
        healthCheckerScheduler.scheduleAtFixedRate(this::checkServersHealth, intervalMs, intervalMs, TimeUnit.MILLISECONDS);
    }

    private void checkServersHealth() {
        for (ServerInstance server : servers) {
            // Mock ping test. Assume all servers return true unless simulated offline.
            boolean isReachable = mockPing(server.getIp());
            server.setHealthy(isReachable);
        }
    }

    private boolean mockPing(String ip) {
        // Simulated failure rule: Server with IP "192.168.1.3" is simulated as down
        return !ip.equals("192.168.1.3");
    }
}

Here is an explanation of the core operations in the LoadBalancer class:

• The constructor configures the routing strategy and sets up the scheduler thread pool using thread-safe collections (CopyOnWriteArrayList).
• registerServer() and unregisterServer() handle dynamic membership updates in the pool.
• routeRequest() filters the list to include only healthy servers, delegates selection to the strategy, increments the active connections on success, and returns the selected server.
• checkServersHealth() runs in the background to evaluate server health and update status flags dynamically.

---

Main Driver Class

This class tests our Load Balancer, showing request routing, strategy updates, health checks, and connection count tracking.

public class Main {
    public static void main(String[] args) throws InterruptedException {
        LoadBalancer lb = new LoadBalancer(new RoundRobinStrategy());

        ServerInstance s1 = new ServerInstance("192.168.1.1", 3); // weight 3
        ServerInstance s2 = new ServerInstance("192.168.1.2", 1); // weight 1
        ServerInstance s3 = new ServerInstance("192.168.1.3", 1); // weight 1 (unstable)

        lb.registerServer(s1);
        lb.registerServer(s2);
        lb.registerServer(s3);

        System.out.println("\n--- Testing Round Robin Strategy ---");
        for (int i = 0; i < 5; i++) {
            ServerInstance server = lb.routeRequest();
            System.out.println("Routed request #" + i + " to: " + server.getIp());
            server.decrementConnections(); // complete execution
        }

        System.out.println("\n--- Testing Weighted Round Robin Strategy ---");
        lb.setStrategy(new WeightedRoundRobinStrategy());
        for (int i = 0; i < 5; i++) {
            ServerInstance server = lb.routeRequest();
            System.out.println("Routed request #" + i + " to: " + server.getIp());
            server.decrementConnections();
        }

        // Simulate health checker running
        System.out.println("\n--- Triggering Health Check (s3 will fail) ---");
        lb.startHealthCheck(500);
        Thread.sleep(600); // Wait for health check run

        System.out.println("\n--- Routing Requests post health check (s3 should be skipped) ---");
        lb.setStrategy(new RoundRobinStrategy());
        for (int i = 0; i < 4; i++) {
            ServerInstance server = lb.routeRequest();
            System.out.println("Routed request #" + i + " to: " + server.getIp());
            server.decrementConnections();
        }
    }
}

The main() driver registers server instances, executes routing using different strategies, runs a mock health check to detect and isolate an unhealthy server, and verifies that the system continues running normally.

---

Class Diagram

---

Also See

• Most Commonly Asked Low Level Design Interview Questions
• Top 10 Low Level Design Interview Questions