In this guide, we’ll walk through a framework for scaling systems from 0 to 1M users. As we dive into the details, we’ll reference several key topics that I’ve previously covered in depth. Feel free to check these out:

1. A crash course on Caching—Part 1, Part 2 & Part 3

2. A crash course on Rate Limiters - Part 1 & Part 2

3. What are Load Balancers?

4. What is Redis?

5. What is Kafka?

Starting with a Single Server Setup

In the early stages of a web application, a single server setup often provides all the necessary resources. In this configuration, a single physical or virtual server typically hosts the web server, database, and cache. The simplicity of this architecture allows for quick deployment, minimal configuration, and low costs, making it ideal for small projects or MVPs (Minimum Viable Products).

However, as user demand grows, this design quickly encounters limitations. A single server, no matter how powerful, has finite resources in terms of CPU, memory, storage, and network bandwidth. Running all application components (such as web server, database, and cache) on one server can lead to resource contention, where processes compete for CPU time and memory, leading to slower response times and degraded performance.

While starting with a single server is simple, transitioning to a distributed architecture becomes essential as traffic increases. This evolution ensures that no single component becomes a bottleneck, allowing each part of the application to scale as needed.

Request Flow: Understanding User Requests and Responses

In a typical web application request flow, understanding each step from user request to server response is crucial. Here’s a breakdown of this flow:

1. DNS Resolution: The first step is the Domain Name System (DNS) lookup. When a user enters a URL, DNS translates the human-readable domain name into an IP address that the browser can use to connect to the server.

2. Web Server Processing: Once the request reaches the server, the web server (e.g., Apache, Nginx) processes the incoming request. This server can route requests to the appropriate part of the application, whether it’s static HTML content or dynamic data processing.

3. Application Logic and Database Query: For dynamic requests, the application server may need to retrieve data from the database. This step could involve complex queries or multiple database operations to gather the necessary data.

4. Response Generation: The application logic processes the data, formats it (often as HTML, JSON, or XML), and sends it back to the client.

5. Rendering the Response: Finally, the client (usually a browser) renders the response for the user, displaying the content or updating the page based on the application’s logic.

In this setup, each step is crucial for the application’s responsiveness and reliability. A delay or failure at any point affects the overall user experience.

Challenges with single server setup: Single Point of Failure and Scaling Issues

One of the most significant challenges of a single-server architecture is that it represents a single point of failure. If the server goes down, whether due to hardware failure, software issues, or unexpected traffic surges, the entire application becomes inaccessible. Downtime is a major concern for businesses, as it can result in lost revenue, reduced user trust, and potential reputational damage.

Moreover, a single server limits scalability. Adding more traffic means increasing CPU, memory, and storage demands on this single machine, which can only be scaled up to a certain point. Eventually, performance will degrade, leading to slower response times, increased latency, and a poor user experience.

Database Separation and Scaling

As traffic and data requirements increase, the database becomes one of the first components to be decoupled from the web server. By moving the database to a separate server or cluster, we can scale the application layer and data layer independently.

• Improved Performance: Separating the database reduces the workload on the web server, which no longer has to manage storage-heavy tasks. Instead, the web server can focus solely on handling requests, while the database server handles data queries and transactions.

• Enhanced Scalability: With a dedicated database server, scaling the application becomes easier. The application layer can be scaled horizontally (by adding more web servers) while keeping the database layer stable.

• Increased Reliability and Security: By isolating the database, we can implement separate backup strategies and security measures, reducing the risk of data loss and improving data access controls.

Decoupling is an important step for any application with growing data needs, as it allows each layer to grow at its own pace, making the architecture more modular and easier to manage.

Relational Databases (RDBMS) vs. Non-Relational (NoSQL) Databases

Choosing the right database type is essential for scaling the application effectively. Here’s a comparison of RDBMS and NoSQL databases:

• Relational Databases (RDBMS): These databases, such as MySQL and PostgreSQL, store data in structured tables with predefined schemas. They are ideal for applications where data integrity, relationships, and ACID (Atomicity, Consistency, Isolation, Durability) compliance are important. RDBMS is highly suitable for applications that rely on complex relationships, such as e-commerce platforms with inventory, customer, and order relationships.

• Non-Relational Databases (NoSQL): NoSQL databases, such as MongoDB and DynamoDB, are designed for handling large volumes of unstructured or semi-structured data. They often prioritize availability and scalability over consistency, which makes them suitable for applications with high read/write requirements, low-latency needs, or applications with dynamic data structures (e.g., social media platforms). NoSQL databases are usually schema-less, allowing for flexibility as data requirements evolve.

In practice, many applications employ a polyglot persistence strategy, using a combination of RDBMS and NoSQL databases to leverage the strengths of each type.

Scaling Approaches

To handle increased demand, there are two primary approaches to scaling infrastructure: Vertical Scaling and Horizontal Scaling.

Vertical Scaling: Adding More Power to a Single Server

Vertical scaling, also known as scaling up, involves increasing the resources (CPU, RAM, or storage) of an existing server. For example, upgrading a server from 16 GB to 32 GB of RAM or from 4 CPU cores to 8. This approach is straightforward and doesn’t require changes to the application architecture.

• Pros: Simplicity and low configuration changes. Vertical scaling is quick and can often be done with minimal disruption.

• Cons: Limited scalability. Physical hardware has constraints, and each upgrade becomes increasingly costly. There’s a point at which further vertical scaling becomes impractical or even impossible.

For small applications, vertical scaling may suffice. However, it becomes unsustainable as traffic and storage demands grow, especially for applications with unpredictable workloads.

Horizontal Scaling: Distributing Load Across Multiple Servers

Horizontal scaling, or scaling out, involves adding more servers to distribute the load across multiple instances. This approach often involves load balancing, where requests are spread across multiple servers to avoid overloading any single machine. Horizontal scaling is more complex than vertical scaling but offers nearly unlimited scalability if implemented properly.

• Pros: Greater flexibility and reliability. By distributing load, horizontal scaling can prevent bottlenecks, reduce downtime, and support rapid increases in traffic.

• Cons: Complexity in setup and maintenance. Horizontal scaling often requires architectural changes, such as implementing distributed databases, caching layers, and synchronization across servers.

Load Balancing for High Availability and Performance

A load balancer distributes incoming traffic across multiple servers, ensuring no single server handles too much load. This setup improves performance and maintains uptime by allowing the system to continue functioning even if one server fails.

• Failover: If a server goes down, the load balancer automatically routes traffic to other healthy servers, maintaining uptime.

• Scalability: With a load balancer, you can add more servers as traffic grows, smoothly scaling your application without disrupting users.

• Security: By positioning servers behind the load balancer, you limit direct access to servers, adding an extra layer of security.

Load balancing allows applications to handle large traffic volumes while ensuring that each server performs optimally.

Optimizing Response Time with Caching

Caching reduces load on the database by storing frequently accessed data in memory. This speeds up response times significantly, especially in applications with a lot of read requests.

• Cache Layers: Use caching solutions like Redis or Memcached as an additional layer to store commonly requested data.

• Read-Through Cache: The application checks the cache for data before querying the database. If the data isn’t in the cache, it’s fetched from the database, stored in the cache, and returned to the user. This minimizes database calls.

• Cache Expiration: Use time-to-live (TTL) values to automatically remove data from the cache after a set time, helping balance memory usage and data freshness.

• Eviction Policies: When cache memory fills up, eviction policies like LRU (Least Recently Used) or LFU (Least Frequently Used) manage which items to remove, optimizing memory for frequently accessed data.

Caching can dramatically improve response times and reduce load, especially in high-traffic, read-heavy applications.Content Delivery Network (CDN) for Global Performance

A CDN helps reduce latency for delivering static content by caching files on edge servers close to users. This offloads traffic from your main servers and speeds up content delivery.

Content Delivery Network (CDN) for Global Performance

A Content Delivery Network (CDN) caches static files (like images, stylesheets, and scripts) on servers around the world, reducing load on your main servers and improving response times for users in different regions.

• How CDNs Work: When a user requests content, the request is routed to the nearest CDN server. If the file is already cached, it’s served immediately. If not, the CDN fetches it from the origin server, caches it, and serves it to the user.

• Advantages of CDNs:

  • Reduced Latency: By serving files from a server close to the user, CDNs reduce the time it takes to load static content.

  • Cost-Effective Scaling: Offloading traffic to CDN servers reduces demand on your primary servers, lowering infrastructure costs.

CDNs are essential for applications with users in multiple regions, providing faster load times and reducing the load on backend servers.

Stateless Web Tier for Horizontal Scaling

In a stateless design, each request is handled independently, meaning that no server relies on previous requests to process the current one. This setup allows requests to be routed to any available server, simplifying load distribution and disaster recovery.

• Session Management: In a stateless architecture, session data is not stored on individual servers but in a centralized data store like Redis or an RDBMS. This approach allows users to be routed to any server without losing session data, supporting seamless scaling and failover. Centralized session storage also ensures that user data remains consistent across requests.

• Autoscaling: With stateless design, you can automatically scale up or down based on traffic without disrupting user sessions. As demand increases, additional web servers can be added, distributing requests evenly and supporting higher throughput.

Database Replication for Fault Tolerance

Database replication creates one or more copies of the database on different servers, which provides redundancy and high availability. This approach is especially valuable in production environments where database downtime can affect user experience.

• Master-Slave Replication: In this setup, the master server is responsible for handling all write operations, while slave servers are designated for read operations. This division of roles helps increase read throughput by allowing reads to be distributed across multiple slave servers. If the master server fails, the system retains the latest read-only data on the slaves, minimizing potential data loss.

• Automatic Failover: If the master node fails, a slave node can be promoted to master to take over write operations. This process typically happens quickly and with minimal downtime, helping to keep the system operational even during outages.

Sharding for Large-Scale Database Distribution

Sharding divides a large database into smaller parts, called shards, that are spread across multiple servers. This division distributes the load and improves overall database performance, making it an effective strategy for applications with rapidly growing data.

• Sharding Key: Selecting an effective sharding key is crucial. This key, such as a user ID in a social media app, determines how data is distributed across shards. An even distribution reduces the risk of any single shard becoming a bottleneck, helping maintain consistent performance as data grows.

• Challenges:

  • Hotspots: Certain entities, like high-profile users, may generate excessive traffic and overload their shard. A common solution is to use dedicated shards for popular entities or to split traffic using a balanced sharding strategy.

  • Re-sharding: As the data volume increases, you may need to modify the sharding strategy to distribute data more effectively. Techniques like consistent hashing can help re-balance the data across shards without extensive downtime, though this process requires careful planning.

Multi-Data Center Architecture for Global Availability

Deploying applications across multiple data centers improves performance, increases resilience, and allows applications to serve users worldwide with reduced latency. This setup supports both disaster recovery and user experience.

• GeoDNS: By using GeoDNS, you can route users to the closest data center, which reduces latency and speeds up response times. For example, users in Europe can be directed to a European data center, while users in Asia are directed to an Asian data center. This geographic routing also improves regional performance.

• Data Synchronization: Data is synchronized across all data centers, ensuring that users get a consistent experience no matter where they are located. Replicating data in this way allows each data center to serve as a backup for others, enhancing disaster recovery.

Asynchronous Communication with Message Queues

Message queues facilitate asynchronous communication between services, allowing them to operate independently. This decoupling enhances scalability and resilience by enabling each service to function and scale at its own pace.

• Decoupling Services: With a message queue, one service can produce messages, and another can consume them without having to be available at the same time. This setup improves reliability, as services can recover from downtime and pick up where they left off without losing data.

• Use Case Example: For example, in a media platform, when a user uploads a video, the request can trigger background processes for encoding and thumbnail generation. These processes run independently, using a message queue to coordinate tasks without slowing down the user’s upload experience.

Monitoring, Logging, and Automation for Operational Excellence

As systems grow, effective monitoring, logging, and automation are crucial to maintaining reliability, performance, and security.

• Metrics Collection: Monitoring involves collecting key metrics, such as CPU and memory usage (host-level metrics), database and cache performance (application-level metrics), and user engagement (business metrics). By tracking these metrics, teams can identify and address performance bottlenecks before they affect users.

• Centralized Logging: Centralized logging aggregates logs from all services in a single location. This setup simplifies troubleshooting and enables teams to spot patterns, detect anomalies, and identify potential security risks.

• CI/CD Pipelines: Continuous integration and continuous deployment (CI/CD) pipelines automate the testing and deployment process, ensuring that updates are rolled out consistently and safely. This automation reduces human error and speeds up the release of new features and bug fixes.

Conclusion: Scaling to Millions of Users

Scaling from a single-server setup to a multi-million-user system requires a methodical approach, incorporating strategies like database sharding, caching, stateless architectures, and load balancing. By focusing on modular components and adopting distributed data management practices, engineers can ensure that systems are resilient, high-performing, and able to meet the demands of large-scale, geographically distributed users.

Mastering these principles is essential for any engineer building and maintaining scalable systems in today’s fast-evolving tech landscape.

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

If you are a recent subscriber, here’s what you might have missed:

1. How to crack any system design interview?

2. How to use Redis in system design interviews?

3. A crash course on Caching—Part 1, Part 2 & Part 3

4. How to design a payments system?

Kafka deep dive concepts

How to achieve Scalability in Kafka?

In a Kafka environment, achieving scale relies on optimizing message handling, resource distribution, and throughput. On strong hardware, a single broker can manage around 1TB of data and process roughly 10,000 messages per second (this depends on message size and hardware specs, but serves as a rough guideline). If your requirements are within these limits, scaling may not be necessary. Here’s a breakdown of key strategies for scaling Kafka effectively.

Add more brokers for horizontal scaling

Adding brokers to the Kafka cluster is the most straightforward way to scale. This spreads the load and improves fault tolerance. However, to fully benefit from additional brokers, ensure that your topics have enough partitions to allow for parallelism. Without adequate partitions, new brokers won’t effectively balance the load.

Keep the message size small

Kafka performs best with smaller messages, which enable faster processing and save storage space. This can be achieved this by batching messages and configuring compression on producers using settings like compression.type and batch.size. Sample configs:

message.max.bytes=1048576 # Maximum message size should be 1MB
batch.size=16384 # Maximum size of a batch in bytes
linger.ms=1 # Time to wait before sending the batch
compression.type=snappy # Compression codec

Add more partitions to high-traffic topics in Kafka

For high-traffic topics, increase partition count and replication factor for better load distribution and fault tolerance. Here are ways to manage hot partitions effectively:

1. Random Partitioning (No Key): Without a key, Kafka randomly assigns partitions, ensuring even distribution but no message ordering. Suitable if order doesn’t matter.

2. Salting Keys: Add a random number or timestamp to keys (e.g., ad IDs) to spread the load evenly across partitions, though this may complicate later aggregation.

3. Compound Keys: Combine the primary key (like ad ID) with another attribute, like region or user segment, to balance traffic across partitions.

4. Back Pressure: Slow down the producer if partition lag is high. Managed Kafka services may offer this by default, or you can manually check and adjust the producer rate.

How to achieve Fault Tolerance and Reliability with Kafka

Consumer rebalancing in Kafka

When consumers fail or new ones join a group, Kafka redistributes partitions to keep messages flowing. This rebalancing keeps partitions actively consumed but can add latency if it happens too frequently. You can control rebalancing speed with session.timeout.ms and max.poll.interval.ms settings to reduce unnecessary delays.

Replication strategies

Kafka's replication factor sets the number of copies for each message across brokers, adding fault tolerance and data protection. A higher replication factor minimizes data loss risk during broker failures but requires more storage and bandwidth. Use KAFKA_REPLICATION_FACTOR when creating topics to match your system’s availability needs.

Processing guarantees in Kafka

Each partition acts as an append-only log, with unique offsets for every message. Consumers commit these offsets after processing, allowing them to pick up exactly where they left off after restarts, ensuring no missed or duplicated messages. Figure 1 shows how Kafka consumers manage offsets

Kafka offers three offset commit methods, each providing distinct processing guarantees:

At-most once processing

In this mode, each message may be processed only once or skipped if there’s a failure, as offsets are automatically committed after polling. With enable.auto.commit set to true, any processing failure means Kafka won’t reprocess the message, which can lead to data loss. This option works best for non-critical data. Figure 2 shows the relevant code snippet.

At-least once processing

This mode ensures each message is processed at least once but may result in duplicates if retries occur after a failure. Here, enable.auto.commit is set to false, and offsets are committed only after processing completes successfully. It’s ideal when data loss is unacceptable, such as in logging applications, even if it means handling duplicate messages. Figure 3 shows the relevant code snippet.

Exactly once processing

EOS guarantees each message is processed exactly once using transactions and idempotent producers (enable.idempotence=true). Consumer offsets are committed only after a successful transaction, preventing both duplicates and data loss. This level of reliability is essential for sensitive applications like financial transactions, where precision is critical. Figure 4 shows the relevant code snippet.

Handling Errors and Retries in Kafka

Kafka retries help prevent message loss during temporary issues like network disruptions. Retries can be set up for both producers and consumers.

Producer retry strategies

Producer retries are configured using the retries (max attempts) and retry.backoff.ms (interval between retries) settings. When a producer faces a temporary issue, such as network instability or a broker being briefly unavailable, these settings allow Kafka to retry sending the message instead of discarding it. See Figure 5 for code example

Consumer Retry strategies

Consumers will attempt to process messages a set number of times before moving them to a Dead Letter Queue (DLQ). Figure 6 provides a detailed breakdown of how the retry process operates on the consumer side, including the transition to DLQ.

Here is a step-by-step breakdown:

• Main consumer

  • A producer sends a message to the Main Topic, and if the consumer fails to process it, the error context is captured and sent to a dedicated Retry Topic, isolating them from the main processing flow.

  • The original message ID and timestamps help track failures and aid debugging, while error details and service version information help identify recurring issues.

• Persistence in MySQL

  • Failed messages are logged in MySQL with the current retry count and timestamps, helping to analyze failure patterns.

  • Error messages and service versions provide context for failures, supporting effective troubleshooting.

• Scheduled Retries via Airflow

  • Airflow regularly picks up failed messages to ensure all retries are processed promptly.

  • Exponential backoff in retries helps avoid system overload during high failure rates.

• Dead Letter Queue

  • If the message still fails processing even after N retry attempts from the retry topic, it is sent to a Dead Letter Queue (DLQ).

  • DLQ is a separate Kafka topic that stores failed messages indefinitely, allowing developers to analyze them later, determine the failure cause, and fix them. The DLQ ensures no message is lost, providing a fallback for unresolvable errors.

Performance optimizations in Kafka

Use Schema Registry for data consistency

In messaging or event streaming systems, data is often serialized using a schema format before being published to topics or queues. However, as data producers and consumers evolve, they might have different expectations of data formats, leading to issues such as:

• Schema Mismatch: Producers may publish data in formats that consumers cannot read if they are using outdated or incompatible schemas.

• Maintenance Complexity: Manually updating schemas across multiple applications can be error-prone and challenging to coordinate.

A schema registry is a centralized service that manages schemas for systems such as Kafka. It ensures that both producers and consumers use compatible schemas.

Benefits of using a Schema Registry:

1. Decoupling Schemas: Producers and consumers can work independently, allowing easier upgrades and maintenance.

2. Central Schema Tracking: All schemas used in production are stored in a central location.

3. Versioning Benefits: Facilitates schema changes by managing different versions, ensuring backward compatibility, and minimizing errors during updates.
   

What is Apicurio in Kafka?

Apicurio Schema Registry is a runtime server that stores and manages schemas for data serialization. Key advantages of Apicurio:

• Open Source: Supported by an active open-source community.

• Multi-Format Support: Supports Avro, JSON Schema, and Protobuf.

• Schema Evolution and Versioning: Provides schema versioning, supporting backward and forward compatibility.

• Schema Validation: Validates schemas against predefined rules to prevent deployment of invalid schemas.

How does Apicurio enhance Kafka's performance?

Figure 7 shows the steps involved when using a schema registry:

1. The producer registers the schema in the Apicurio Schema Registry before producing messages.

2. Before sending the message to Kafka, the producer retrieves the schema from the registry to serialize data.

3. The producer sends the serialized message to Kafka with a reference to the schema.

4. The consumer retrieves the schema from the registry to understand the data format.

5. The consumer uses the schema to deserialize the message and consume it.

Consumer Lag in Kafka

Kafka consumer lag refers to the difference between the last produced message offset and the last consumed message offset for a particular consumer group (Figure 8). It indicates how far behind a consumer is in processing messages from a Kafka topic. Monitoring consumer lag is crucial because high lag can lead to delays in message processing, impacting system performance and user experience.

What causes consumer lag in Kafka?

1. Slow Processing: If the consumer application takes longer to process messages, it can lead to lag. For example, a consumer that performs complex data transformations may struggle to keep up with incoming messages.

2. High Throughput: A sudden increase in message production can overwhelm consumers that are not properly scaled. For instance, during a flash sale, a retail application may receive more order messages than its consumers can handle.

3. Network Latency: Poor network conditions between Kafka brokers and consumers can delay message delivery. For example, a consumer located far from the broker may face lag due to increased latency.

4. Consumer Configuration: Misconfigurations, such as having too few consumer threads or setting low session timeouts, can also cause lag. For instance, a low value for max.poll.records might limit the number of messages a consumer can process at once.

How to mitigate consumer lag?

1. Scaling Consumers: Increase the number of consumer instances or partitions to handle higher message throughput effectively.

2. Optimizing Processing Logic: Streamline the consumer application logic to reduce processing time. For instance, break down complex operations or offload some tasks to background workers.

3. Tuning Kafka Configurations: Adjust configurations like fetch.min.bytes, fetch.max.bytes, and max.poll.interval.ms to optimize message retrieval and consumption speed.

4. Batch Processing: Implement batch processing to reduce the frequency of consumer polls and improve throughput.

When to use Kafka in system design interviews?

Kafka can be used either as a message broker or a stream processor. As a message broker, it decouples services, ensuring asynchronous communication. As a stream processor, Kafka provides a framework for real-time data transformation, aggregation, and analytics.

Kafka as a message queue

Ticket Booking Waiting Queue

In systems like BookMyShow, Kafka queues ticket requests by user ID and timestamp to process them in the correct order. Each consumer handles requests one at a time to avoid race conditions and ensure fair processing. Failed requests are requeued for retries, so users have a fair chance to secure tickets.

Payment Processing with Kafka: Example from Stripe

Payment services (P2P, Card, B2B) send transaction events to Kafka, which then streams these to different services. For example, Kafka sends transaction data (such as transaction ID, location, and IP address) to the Fraud Detection Service for compliance checks. At the same time, successful transaction events are sent to the Wallet Service for balance updates and to the Ledger Service for accounting. Check out Accounting 101 in Payment Systems for more on accounting system design.

Using Kafka for YouTube video transcoding

For YouTube, Kafka facilitates video transcoding by publishing upload details to a topic for real-time processing. Its replication feature protects against data loss, while multiple consumers handle tasks concurrently, ensuring timely video delivery in various formats. This setup enhances user experience by reducing buffering and enabling support for diverse devices.

Kafka for message streaming

Kafka should be used for message streaming when:

• You require continuous and immediate processing of incoming data, treating it as a real-time flow. For example, designing an Ad click aggregator

• Messages need to be processed by multiple consumers simultaneously. For example, while designing Facebook live comments, we can use Kafka as a pub/sub system to send comments to multiple consumers.

Congratulations on making it through! With these two guides, you’ll be well-prepared for interviews that require an in-depth understanding of Kafka.

In a recent poll, I asked what topics you’d like to see next, and most of you voted for Kubernetes and Elasticsearch. Upcoming articles will be about these topics. Looking forward to exploring them together!

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

Building high-performance systems requires a deep understanding of reliable messaging solutions like Apache Kafka. In this series, we’ll explore Kafka’s foundational elements and advanced features such as schema registry, commit strategies, and retry mechanisms, helping you design notification systems and event-driven architectures that can handle heavy data traffic.

As an early engineer at Barq, I’ve worked on scaling our systems to support 2 million users and have witnessed Kafka’s scaling challenges firsthand. I tried to look for a practical, interview-focused Kafka guide but couldn’t find any resource that fully addresses the real-world issues engineers face.

Cracking system design interviews covers how the "Design Deep Dive" part often decides interview outcomes, especially for senior roles. While How to use Redis in system design interviews covers Redis in detail, this two-part series will focus on Kafka—a component that often plays a central role in scaling discussions.

If you are a recent subscriber, here’s what you might have missed:

1. How to use Kafka in system design interviews?

2. A crash course on Caching—Part 1, Part 2 & Part 3

3. A crash course on Rate Limiters - Part 1 & Part 2

4. How to design a payments system?

List of topics covered in this 2-part series:

Before diving in, I want to share that I focus on making each piece of content as relevant and valuable as possible for system design interviews. That’s why I take the time to produce only in-depth, long-form content.

With that in mind, it would mean a lot if you could take a moment to answer a quick poll. Your input will help me choose the topics that matter most to you👇.

Basics of Kafka

Apache Kafka is a distributed event-streaming platform created by LinkedIn and later open sourced by the Apache Software Foundation. It’s built to handle large volumes of data in real time, enabling multiple applications to share and process data streams efficiently.

Unlike tools that process data in batches at set times, Kafka allows data to be processed continuously as it arrives, which is crucial in areas like e-commerce, IoT, social media, and finance.

Let’s look at a relatable example to understand how Kafka is used.

A real-world example of Kafka

Consider a ride-hailing platform like Uber, where each ride generates a stream of updates: the driver’s location, trip status (e.g., started, ongoing, completed), and the estimated time of arrival (ETA).

These updates are placed in a queue by a process known as the producer. On the other side, another process called the consumer reads from this queue and performs its business logic.

Let’s look at the problems with this simple setup (Figure 2).

Problem 1: Scaling to Millions of Rides

Uber handled nearly a million rides per day in 2023. For each ride, many events are generated at once, creating a huge volume of data. A single queue can’t handle this volume.

Solution: To manage this, Kafka allows data to be grouped into topics. Different types of updates are sent to separate topics, as shown in Figure 3:

• driver-locations: For live location updates of drivers.

• trip-updates: For ride status updates (start, ongoing, or completed).

Problem 2: Consumers Overwhelmed by Location Data

With so many location updates coming in, a single consumer quickly becomes overloaded, slowing down processing and causing delays.

Solution: Kafka allows multiple consumers to work together in a consumer group, where each event is handled by only one consumer in the group. For example, Figure 3 shows the driver-location-group which has two consumers, each handling a part of the data, speeding up processing and avoiding duplication.

Problem 3: High-Volume Data and Processing Delays

A global platform like Uber has thousands of rides happening at once. If all updates go to the same topic, it can create bottlenecks, slowing down tasks like ETA prediction.

Solution: Kafka lets us split topics into partitions based on specific keys (Figure 4). For example, the driver-locations topic can be divided by driver ID, so each driver’s updates go to a separate partition. This balances the workload, reducing delays and making sure each update is processed promptly.

Why choose Kafka? Key benefits in distributed systems

1. Scalability: Kafka can handle millions of messages per second by spreading data across multiple servers, making it suitable for high-traffic applications.

2. Reliability: Kafka’s replication ensures that data remains available even if some servers fail, providing strong reliability for critical applications.

3. Independent Communication: Kafka’s design allows services to send and receive messages independently, enabling multiple services to access the same data without direct connections.

4. Data Persistence: Kafka can store data for a set time, letting services replay past data when needed, which is useful for reprocessing after failures or for accessing historical data.

Kafka components explained

In the ride-hailing example above, we highlighted several components through a problem-solution approach. Now, let’s explore each of these components in greater detail.

Figure 5 shows a high-level architecture with all the components in a Kafka setup.

Kafka topics

• A topic is a stream of data where a specific type of data is stored. For instance, in the ride-hailing example we discussed above, different topics were created for specific use cases (eg: “driver-locations” or “trip-updates”)

• Kafka has configurable retention policies for topics. For example, a topic can be set to keep data only for a few days, after which the older messages are automatically deleted.

Kafka partitions: a key to parallel processing

• Each topic can be split into multiple partitions, allowing messages to be processed independently. This makes Kafka scalable because messages are spread across multiple servers instead of all being stored in one place.

• Partitions are replicated across different servers (or “brokers”) in a Kafka cluster. This replication ensures that if one server fails, other servers have copies of the data, making Kafka resilient against data loss and system failures.

Kafka brokers: distributing and managing data streams

• Brokers are the servers in a Kafka cluster that handle data storage, processing, and retrieval. Each broker stores one or more partitions of a topic and handles requests from clients (producers) and servers (consumers).

• In Kafka, each partition has one broker assigned as the leader, which manages the read and write operations. Other brokers have replica copies of the data in the partition but don’t process requests unless the leader fails.

Kafka producers: creating and sending messages

• Producers are services or applications that send messages to Kafka topics. They decide which topic and partition to send data to, often based on a unique key (eg: driver ID). This ensures that messages related to the same entity (like a driver) are sent to the same partition, keeping their order.

• Producers can be set up with SSL/TLS for secure data transfer. Additionally, they can be configured to prevent duplicate messages, ensuring that each message is sent only once, even if there are retries.

Kafka consumers: reading and processing data streams

• Consumers are services or applications that read messages from Kafka topics. They consume messages at their own pace, which allows them to process data independently and resume from where they left off if interrupted.

• Each consumer tracks its “offset” which represents the last message it successfully read. By tracking offsets, Kafka enables consumers to resume from the correct position in the event of a failure.

• Consumers can be grouped into “consumer groups” allowing Kafka to distribute messages across multiple consumers. For example, if you have a group with three consumers, Kafka will assign each partition of a topic to a different consumer in the group. This setup ensures each message is processed only once within the group.

Zookeeper: ensuring fault tolerance and scalability

• ZooKeeper is a separate service used to coordinate the Kafka brokers. It stores metadata about the cluster, including which broker is responsible for each partition.

• When a broker fails, ZooKeeper helps reassign leadership of its partitions to another broker, ensuring Kafka continues to operate smoothly.

How does Kafka work?

Step 1: How are messages produced in Kafka?

When an event occurs, a Kafka producer formats a message (often called a record) and sends it to a Kafka topic. Each message includes a required field (value), which contains the main data, and three optional fields:

• Headers: These are optional key-value pairs that store metadata about the message, such as HTTP headers.

• Key: Used to specify which partition the message should go to, ensuring that all messages for a use case go to the same partition.

• Timestamp: Marks the time the event occurred, helping Kafka to maintain chronological order within each partition.

For example, if a producer is set up to send messages on driver locations, it might send a message with the driver’s ID as the key, the current GPS coordinates as the value, and optional metadata (eg: device type) in the headers.

Step 2: How does Kafka partition messages?

Kafka decides the partition for each message depending on whether a key is present or not:

• If the key is specified, Kafka applies a hashing algorithm to the key, ensuring that all messages with the same key end up in the same partition.

• If no key is provided, Kafka can use a round-robin approach to distribute messages randomly across the available partitions or a different partitioning logic configured by the producer.

Step 3: Broker assignment in Kafka

After Kafka decides on a partition, it identifies the broker that holds that partition through the metadata it maintains about the cluster. The producer uses this information to send messages directly to the broker responsible for the intended partition.

Step 4: Kafka’s message storage

Each partition in Kafka functions as an append-only log (Figure 8), where messages are added sequentially at the end of the log file. This log structure is central to Kafka’s architecture for several reasons:

• Immutability: Once written, messages cannot be changed or deleted, avoiding inconsistencies.

• Efficiency: Writing messages to the end of a log file minimizes disk seek times, making it a quick operation.

• Scalability: The simplicity of the append-only log allows Kafka to scale horizontally by adding more partitions across multiple brokers.

Figure 9 shows sample producer code using kafkajs, a popular Node.js client for Kafka

Step 5: Managing message offsets in kafka

Each message in a Kafka partition is assigned a unique sequential offset, which helps consumers track their position as they read messages from the topic (Figure 10). By keeping track of offsets, consumers can resume reading messages from the last known position if there is an interruption or failure, ensuring that no messages are lost or skipped.

Consumers can have different types of offset commit strategies, which I will cover in the next article.

Step 6: Kafka replication for fault tolerance

Kafka uses a leader-follower model for replication to ensure data durability and availability.

• Leader Replica: Each partition has a leader replica located on one broker, which handles all read and write requests.

• Follower Replicas: Each partition also has follower replicas on different brokers that are ready to take over if the leader fails.

• Synchronization: Followers continuously sync with the leader to ensure they hold the latest data. If the leader replica for a partition fails, one of the in-sync followers is promoted to become the new leader, minimizing downtime.

Zookeeper is responsible for managing this replication process, monitoring broker health, and reassigning leadership when a broker goes down.

Step 7: Consuming messages in Kafka

In most setups, consumers poll Kafka to fetch new messages, allowing them to control the pace of consumption. Consumers can be grouped into consumer groups to handle high message volumes where each partition is read by only one consumer.

For example, if multiple replicas of a service need to process driver location updates from a driver-location-updates topic, each replica can be in the same consumer group, with each service reading from separate partitions.

Figure 11 shows a sample consumer code using kafkajs

Kafka messaging strategies: acks=0, acks=1, acks=all

In Kafka, message delivery mechanisms are controlled by the acks parameter in the producer configuration. This parameter sets the number of acknowledgments required from the Kafka cluster before a message is considered successfully delivered. The value of acks can be 0, 1, or all. Let’s break down each one of these:

acks=0: Unacknowledged message strategy in Kafka

• The producer sends the message without waiting for any acknowledgment from the broker.

• Pro: Since the producer doesn’t wait for a response, message delivery is fast.

• Con: There’s no message delivery guarantee since there is no confirmation the message reached the broker.

• Best for non-critical data like logging or metrics where losing some messages is acceptable.

acks=1: Single acknowledgment messaging strategy in Kafka

• The producer waits for an acknowledgment from the leader broker once it has written the message to its local log.

• Pro: Since the leader broker acknowledges message receipt, this setting reduces the chance of data loss compared to acks=0.

• Con: Waiting for an acknowledgment from the leader introduces a slight delay.

acks=all: Ensuring full replication in Kafka messaging

• The producer waits for acknowledgments from all in-sync replicas (ISRs), ensuring that the message is fully replicated across the cluster.

• Pro: Guarantees the strongest delivery guarantee by waiting for all in-sync replicas to acknowledge the message.

• Con: Waiting for all replicas to confirm receipt can introduce significant delay, as network and processing time increases.

• Necessary for strict use cases such as financial transactions.

Practical use cases of Kafka

Using Kafka for Logging, Monitoring & Alerting

Log/metrics collector agents such as OpenTelemetry push logs and metrics from application services to ElasticSearch (for indexing) via Kafka. Kibana helps in visualizing logs, while Apache Flink helps in aggregating and visualizing metrics.

Kafka for data streaming in recommendations

Kafka processes real-time user clickstreams to enhance recommendation systems. User activity data is ingested into Kafka, which routes it to Flink for processing, enabling real-time aggregations and calculations. The processed data is sent to a Data Lake from where it’s fed to machine learning models to generate recommendations based on user interactions.

Change Data Capture (CDC) with Kafka: Keeping databases in sync

Kafka can be used to sync source and destination data stores in real-time. In a CDC scenario, Kafka captures real-time changes from source databases via transaction logs. These logs are routed to various sinks such as ElasticSearch, Redis, and replica DBs via respective connectors.

Kafka for system migration

Disaster recovery is essential for large-scale applications where high availability is critical. Kafka supports this by replicating entire production workloads across data centers. With Kafka Connect, large volumes of events can be easily ingested, making it an ideal tool for replicating production data.

Conclusion

This post was more about learning the basics of Kafka along with its most common use cases. In the next post, I will try to cover as many scaling challenges as possible based on my practical experiences with Kafka in a large-scale fintech system. Stay tuned 🤘🏻.

That’s all for today. Thanks for reading!

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

Why use probabilistic data structures?

Analyzing massive Terabyte-sized datasets is increasingly common in web analytics and online advertising. Traditional methods, like Hadoop and MapReduce, can lead to slow, high-latency processes, making them unsuitable for real-time applications.

While simple metrics, such as total page views or average conversion prices, can be efficiently summarized using daily data or in-stream counters, more complex metrics—like unique visitor counts or the most frequent items—are resource-intensive to compute directly.

The rest of the article will explore three probabilistic data structures that can efficiently handle such complex queries. But first, let’s walk through an example to understand the challenges involved.

Real-world examples of Probabilistic Data Structures

Given a dataset with 10 million integers, where no more than 1 million are distinct (resulting in a 40MB raw dataset size at 4 bytes per integer), here are some key questions we need to address:

• Does the dataset contain a specific element (Membership Query)?

• What are the most frequent elements (heavy hitters or top-k elements)?

• What are the frequencies of those elements?

• How many elements fall within a specific range (Range Query, like SELECT count(v) WHERE v >= c1 AND v < c2 in SQL)?

For the first question, using a sorted list or hash table would need around 4MB space (1M * 4 bytes). This works for smaller datasets, but for datasets that grow into terabytes, it’s not scalable. A probabilistic data structure like LogLogCounter, however, can handle this in just 2KB space!

Sounds interesting, right? Let’s dive into the data structures that help answer each of these questions efficiently.

Read on till the end for a concise comparison between probabilistic and non-probabilistic data structures.

What are probabilistic data structures?

Probabilistic data structures are designed to efficiently answer the questions mentioned above while optimizing for time and space but with an acceptable margin of error.

In simple terms, non-probabilistic data structures provide exact answers—such as "How many items are there? Exactly 1,600,000"—probabilistic data structures might respond with "How many items are there? Approximately 1,523,425, with a 99% probability."

Types of Probabilistic Data Structures

Bloom Filters

What are bloom filters and why use them?

Bloom filters can estimate if a dataset contains a specific element (Membership Query) in a time-efficient manner. They can return 2 responses:

• If a Bloom filter returns False, the item is not in the set.

• If it returns True, the item might be in the set, but there’s a chance of a false positive (we’ll have a look at why this happens shortly).

How do bloom filters work?

Membership Query using bloom filters

A Bloom filter has two key components:

1. Data structure: A bit array of length n. For example, a bit array of length 10 (with indices 0-9) [Figure 1]

2. Hash functions: There are k hash functions that output an index between 0 and n-1, setting the corresponding bit in the array.

Let's see how items are added and looked up in a Bloom filter.

Adding an item

To add an item to a Bloom filter, the following steps are performed:

1. The item is hashed using k hash functions.

2. The modulo operation with the bit array length (n) is applied to the hash outputs to find the positions (buckets) in the array.

3. The bits at these k positions are set to 1.

In Figure 2, the items X and Y are added as follows:

For X:

• hash1(X) mod 10 = 1

• hash2(X) mod 10 = 2

• hash3(X) mod 10 = 4

For Y:

• hash1(Y) mod 10 = 4

• hash2(Y) mod 10 = 5

• hash3(Y) mod 10 = 7

The bucket at index 4 is set to one both the items X and Y.

Looking up an item

To check if an item is present in a Bloom filter, the following steps are taken:

1. The item is hashed using the same k hash functions as during the insert operation.

2. The modulo operation with the bit array length (n) is applied to each hash output to find the corresponding array positions (buckets).

3. The bits at these positions are checked.

   1. If any of these bits are 0, the item is not in the Bloom filter.

   2. If all bits are 1, the item might be in the Bloom filter due to potential hash collisions or bits set by other items.

Examples (Figure 3 & Figure 4):

For Y (existing item)

• hash1(Y) mod 10 = 4

• hash2(Y) mod 10 = 5

• hash3(Y) mod 10 = 7

Since all the bits at these positions are 1, Y might be in the filter.

For Z (item that doesn’t exist)

• hash1(Z) mod 10 = 3

• hash2(Z) mod 10 = 5

• hash3(Z) mod 10 = 8

Since the bits at positions 3 and 8 are 0, Z is not in the Bloom filter.

False Positives

In Figure 5, the Bloom filter is queried to check if F is a member. The following calculations are made:

• hash1(F) mod 10 = 2

• hash2(F) mod 10 = 4

• hash3(F) mod 10 = 7

Bits at positions 2, 4, and 7 are set, indicating that item F might be present. However, this is a false positive because X and Y have set those bits. Therefore, F is not a member of the Bloom filter, illustrating the inherent uncertainty and possibility of false positives in Bloom Filters.

Time & Space Complexity of bloom filters

Bloom filters are more efficient than hash sets for large data lookups due to their constant time and space complexity.

• Time Complexity: O(k) to add and lookup an element, where k is the number of hash functions.

• Space Complexity: Constant space usage, as Bloom filters utilize a fixed number of bits. Each item contributes only a few bits, and bits are reused. Notably, Bloom filters do not store actual values.

Practical use cases of Bloom Filters

The bloom filters are useful in high-ingestion systems to prevent expensive disk seeks. Few applications of bloom filters

• Reducing disk lookups for the non-existing keys in a database

• Determining whether a user ID is already taken

• Filtering out previously shown posts on recommendation engines

• Checking words for misspellings and profanity with a spellchecker

• identifying malicious URLs, blocked IPs, and fraudulent transactions

• Log-structured merge tree (LSM) storage engine used in databases such as Cassandra uses a bloom filter to check if the key exists in the SSTable

• MapReduce uses the bloom filter for the efficient processing of large datasets

Bloom filter calculator

hur.st bloom filter calculator can be used to choose the optimal size for the bloom filter and explore how the different parameters interact. The accuracy of the bloom filter depends on the following:

• number of hash functions (k)

• quality of hash functions

• length of the bit array (n)

• number of items stored in the bloom filter

Count-Min Sketch: Efficient frequency counting

What are bloom filters and why use them?

A Count-Min Sketch (CMS) is a data structure specifically optimized for two scenarios without the need to store the actual elements:

1. Estimating the frequency of elements in a stream

2. Identifying heavy hitters

CMS utilizes a 2-D array of size w x d (width × depth) [Figure 6], where w represents the number of counters in each row and d denotes the number of rows. Each row is associated with its hash function.

How does a count-min sketch work?

A. Frequency Estimation using count-min sketch

Adding an item

1. The item is hashed using d independent hash functions. Each hash function corresponds to one row in the 2D array and maps the element to one of the w counters in that row.

2. For each hash function, increment the counter at the corresponding position in the array.

Figure 7 shows the counters of each position after the items X and Y are added to the CMS:

For X:

• hash1(X) mod 4 = 2

• hash2(X) mod 4 = 3

• hash3(X) mod 4 = 1

For Y:

• hash1(Y) mod 4 = 3

• hash2(Y) mod 4 = 3

• hash3(Y) mod 4 = 2

The counters at the above positions in the corresponding rows are incremented by 1.

Querying an item’s frequency

1. Apply the same d hash functions to the query item.

2. Find the counter at the corresponding position in its row for each hash function. The estimated frequency of the element is the minimum value of these counters.

Figure 8 shows how the frequency of an item X is determined:

• hash1(Y) mod 4 = 2 (counter = 1)

• hash2(Y) mod 4 = 3 (counter = 2)

• hash3(Y) mod 4 = 1 (counter = 1)

The estimated frequency of Y is min(1, 2, 1) = 1.

Time & Space Complexity of count-min sketch

• Time Complexity: O(d), where d is the number of hash functions (or rows).

• Space Complexity: O(w × d).

B. Heavy Hitters using count-min sketch

CMS combined with a heap can efficiently identify heavy hitters/elements with frequencies greater than k% of the total dataset size (N). The approach is simple:

1. Initialize a CMS to track the frequency of elements and an empty heap to store the top elements. Set a counter N to track the total number of processed elements.

2. For each new element, update the CMS and estimate its frequency. If the frequency exceeds k x N, add the element to the heap.

3. Periodically prune the heap by removing elements that no longer meet the k x N frequency threshold.

C. Range queries using count-min sketch

An array of Count-Min Sketches can also handle Range Queries (eg: SQL query "SELECT count(v) WHERE v >= c1 AND v < c2), though I won't go into detail here as it's outside the focus of this article.

This is mainly relevant when designing systems like Databases (eg: Clickhouse), Analytics tools (eg: Mixpanel), or Metrics platforms (eg: Grafana). If you're interested in learning more, I've included references at the end for further reading.

Practical use cases of Count-Min Sketch

• Network Traffic Monitoring: Estimate the frequency of IP addresses in a traffic stream.

• Data Streams: Track word counts in massive text streams (e.g., search query trends).

• Approximate Counting in Distributed Systems: Manage resource usage and limit counters in large-scale systems where exact counting is expensive.

Skip List

What are skip lists & why use them?

A Skip List is a probabilistic data structure that provides an alternative to balanced trees (self-balancing BST and AVL trees). It uses multiple linked lists to represent different "levels" of the dataset, allowing for fast lookups, insertions, and deletions with logarithmic time complexity.

Figure 9 shows the representation of a SkipList.

There are multiple levels, with each level containing a subset of elements from the previous one, also in sorted order. While it may look like nodes appear multiple times, like Node 5 being in all three levels, it's the same node connected across different levels. For example, Node 5 connects to Nodes 24, 15, and 7 at various levels.

How do skip lists work?

Let’s have a look at various CRUD scenarios in a Skip List

Adding an item

The gif in Figure 9 explains this scenario:

1. Start at the topmost level and move forward through the list until you find a node larger than the element to be inserted.

2. Drop down to the next level and continue searching from that position, repeating this process until you reach the base level.

3. Insert the element at the correct position in the base level.

4. Flip a coin to decide if the element should be promoted to the next higher level.

5. If the coin returns heads, insert the element at the same position in the higher level, repeating the promotion and insertion process.

6. Stop promoting the element when the coin returns tails, finishing the insertion process.

Querying an item

The gif in Figure 11 explains how an item is queried in a SkipList:

1. Start at the topmost level and move forward, comparing each node’s value with the target.

2. If a larger node is found without matching the target, drop down to the next level and resume the search.

3. Continue this process at each level, moving down and checking nodes until you reach the base level.

4. Stop if the target is found at any level, otherwise the search concludes at the base level. If the target is still not found, it is not in the skip list.

Deleting an item

Figure 12 illustrates deletion of an item from a SkipList:

1. Traverse the list at the topmost level to find the node just before the element to be deleted.

2. Drop to lower levels from the current node, continuing to search for the node before the target element.

3. At each level where the element exists, update pointers to bypass and unlink the target node.

4. Once the element is removed from all levels, the deletion is complete.

Time and Space Complexity of a skip list

• Time Complexity:

  • Average Case: The level of a node is determined by coin flips and fewer nodes appear on higher levels. This results in the average time complexity of O(log n) for most operations, as only a few levels need to be traversed.

  • Worst Case: Although rare, coin flips can create an unbalanced skip list where most nodes are concentrated in just 1 or 2 levels, leading to a worst-case time complexity of O(n).

• Space Complexity: O(n), where n is the number of elements.

Use cases of skip lists

Skip lists are useful for fast lookups in large datasets. Some practical use cases include:

• In-Memory Databases: Used for indexing and maintaining sorted data with fast insertions and lookups.

• Priority Queues: Efficiently manage dynamically changing priorities.

• Redis Sorted Sets: Skip lists are the underlying data structure in sorted sets.

References for reading on probabilistic data structures

• https://highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/

Conclusion

The table in Figure 13 quickly compares commonly used probabilistic data structures, highlighting their use cases, error rates, and memory consumption.

It's worth noting that LogLog Counter and Stream-Summary are two additional probabilistic data structures, which we will explore in detail later.

Thanks for reading!

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

1. Versatility: Redis isn't just a cache; it handles diverse use cases, such as rate limiting, session management, leaderboards, etc.

2. Simplicity and Speed: Redis excels at executing simple operations quickly, making it perfect for high-performance systems.

Redis can do much more than just caching and can be a major technology in the Design Deep Dive interview phase. In this article, we’ll dive into the key systems you can design using Redis.

If you are a recent subscriber, here’s what you might have missed:

1. How to crack any system design interview?

2. A crash course on Caching—Part 1, Part 2 & Part 3

3. What are Load Balancers?

4. A crash course on Rate Limiters - Part 1 & Part 2

Redis Basics: Why and how Redis is essential for system design

Redis is an open-source, in-memory data store renowned for its speed and efficiency. It supports various data structures like strings, hashes, lists, and sets, making it ideal for core use cases such as caching, rate limiting, pub/sub messaging, and even designing social media feeds.

Why Is Redis Fast?

• In-Memory Storage: Redis stores data in RAM, making it significantly faster than disk-based storage.

• Single-threaded Architecture: Redis processes all requests on a single thread, avoiding the overhead of context switching and locks.

• Non-blocking I/O: An event loop and asynchronous request handling ensure low latency and high throughput.

• Optimized Data Structures: Redis uses efficient data structures for fast read/write operations.

Understanding Redis data structures

Strings are commonly used for:

1. Storing session data for users in web applications

2. Caching frequently accessed API responses.

Lists are commonly used for:

1. Implementing a message queue for asynchronous tasks.

2. Storing a timeline in social media applications.

Sets are commonly used for:

1. Tracking unique visitors to a website.

2. Maintaining a list of unique hashtags used in posts.

Sorted Sets: This data structure combines a hash table and a sorted tree. Each entry has a string "member" (the key) and a double "score" (the sorted value). This allows you to access entries directly by member or score, and retrieve members in order based on their score.

Use cases of Sorted Sets:

1. Creating a leaderboard for an online game.

2. Ranking search results based on the number of likes for a keyword.

Hashmaps: Useful for storing structured data like objects. Commonly used for:

1. Storing user profiles with fields like name, email, and age.

2. Caching database query results in a structured format.

Essential Redis commands for system design

SET foo 1  
GET foo     # Returns 1
INCR foo    # Returns 2
ZADD leaderboard 100 user1    # Adds a user to a sorted set (leaderboard) with score 100
ZRANGE leaderboard 0 0 WITHSCORES    # Returns the top user and score in the leaderboard
SADD users user1     # Adds user1 to a set of users
SMEMBERS users     # Retrieves all members of the set
HSET user:1 name "John Doe" age 30    # Stores structured data in a hash
HGETALL user:1    # Retrieves all fields and values from the hash
XADD mystream * name Sara surname OConnor    # Adds an item to a stream

Top Redis use cases in System Design interviews

In this section, we will explore each system that can be built on top of redis in detail along with the data structures that make this possible.

Redis as a cache

The most common use case of Redis is as a “cache”. It is deployed in a multi-node setup where Consistent Hashing is used to distribute keys across nodes to ensure high availability. Cache Aside is the most commonly used caching strategy where Time To Live (TTL) is often employed on each key for expiry. Redis guarantees you'll never read the value of a key after the TTL has expired and the TTL is used to decide which items to evict from the server - keeping the cache size manageable even in the case where you're trying to cache more items than memory allows.

Redis as a distributed lock: Ensuring data consistency in System Design

Another use case of Redis is as a distributed lock. Occasionally we have data in our system and we need to maintain consistency during updates, or we need to make sure multiple people aren't acting at the same time.

Tip💡: Most databases (including Redis) will offer some consistency guarantees. If your core database can provide consistency, don't rely on a distributed lock which may introduce extra complexity and issues. Your interviewer will likely ask you to think through the edge cases in order to make sure you really understand the concept.

Figure 2 shows how a simple distributed lock works. When we want to try to acquire the lock, we run INCR with a TTL. If the response is 1 (ie: we own the lock), we proceed. If the response is > 1 (ie: someone else has the lock), we wait and retry again later. When we're done with the lock, we can DEL the key so that other proceesses can make use of it.

Redis as Leaderboard in Gaming applications

Redis' sorted sets maintain ordered data which can be queried in log time which make them appropriate for leaderboard applications. The high write throughput and low read latency make this especially useful for scaled applications where something like a SQL DB will start to struggle.

Figure 3 shows how this happens:

• Game Service adds players to the Redis sorted set (leaderboard) using ZADD, associating each player with a score and unique ID.

• When a player's score changes, the application updates the sorted set with the new score using ZADD, which automatically updates the existing score.

• The application fetches the top 10 players from the leaderboard using ZRANGE leaderboard 0 9 WITHSCORES, ordered by score.

• The rank of a specific player is found using ZREVRANK leaderboard player_123.

Redis as a Rate Limiter: Best Practices for High Traffic Systems

As a data structure server, implementing a wide variety of rate-limiting algorithms is possible. A common algorithm is a fixed-window rate limiter where we guarantee that the number of requests does not exceed N over some fixed window of time W. A crash course on Rate Limiting talks about different rate-limiting algorithms and their pros & cons.

Figure 4 shows a simple implementation of a Rate Limiter in Redis. When a request comes in, we increment (INCR) the key for our rate limiter and check the response. If the response is greater than N, we wait. If it's less than N, we can proceed. We call EXPIRE on our key so that after time period W, the value is reset.

Locating nearby drivers with Redis Geohashes in ride-hailing apps

Redis can help locate nearby drivers in ride-hailing services like Uber. Sorted Set is the data structure that facilitates this and the algorithm involved is called Geohashing

• Geohash: A 52-bit integer created by interleaving the bits of a driver's latitude and longitude coordinates.

• Sorted Sets: It stores the driver ID with the corresponding geohash as the score, allowing the set to be sorted based on location (see the sorted set table in Figure 5).

Figure 5 shows a step-by-step breakdown of how this happens:

1. Driver Location Service:

   • Updates Redis with driver locations using GEOADD drivers:current_location 13.3 38.1 "driver_123" (sorted set name, latitude, longitude, driver ID).

   • A geohash is calculated from lat/long pair and is stored as a score with the driver_id in a sorted set (drivers:current_location)

2. Ride Request Service:

   • Queries Redis for nearby rides using GEOSEARCH drivers:current_location FROMLONLAT 15.0 37.0 BYRADIUS 2km (sorted set name, user’s lat/long, radius).

   • Computes a geohash/score from the user’s coordinates to find the closest drivers in the drivers:current_location sorted set.

   • Retrieves driver IDs and additional details (availability, ETA) from Redis or another data store.

Tips 💡:

1. There are additional complexities to consider in “Finding nearby drivers in Uber”, such as the status of drivers who may be available or currently completing another ride.

2. Depending on search traffic, it may be necessary to maintain different sorted sets for various locations.

3. Redis is best utilized for scenarios where latitude and longitude updates occur frequently (eg: finding nearby drivers in Uber). For use cases with infrequent location changes, like storing the locations of all outlets for a store, static data storage is more appropriate.

Redis for async communication: Message Queues

Redis provides a List data structure that can be utilized as a queue to facilitate asynchronous communication between services. Below is a step-by-step overview of how to implement Redis for an image processing use case:

1. The user uploads an image and the image service pushes a job to the Redis list image_queue using LPUSH.

2. The background worker retrieves the job from the queue using BRPOP, waiting if the queue is empty.

3. The worker processes the image (in this case, resizing it).

4. Once the image processing is complete, the worker pushes a completion notification into another Redis list, processed_images.

5. The notification service reads this message to inform the user about the completion.

Downsides of using Redis as a message queue

While Redis can handle this use case, it's not ideally suited for large-scale systems where solutions like Kafka are more appropriate due to the following limitations of Redis:

1. Lack of Message Durability: As an in-memory store, Redis can lose messages if it crashes or restarts. Although it supports persistence through AOF (Append Only Files) and RDB (Redis Database), there are chances of messages getting lost between snapshots.

2. No Message Re-delivery Guarantee: Unlike Kafka, Redis lacks a message commit mechanism, meaning there's no assurance that lost messages will be retried.

3. Not Designed for Scale: Being single-threaded, Redis can become a bottleneck in high-throughput scenarios, whereas distributed message brokers like Kafka or RabbitMQ are built for scalability.

4. No Message Ordering Guarantees: Redis does not guarantee the order of messages when multiple consumers are involved.

Storing and analyzing time series data in Redis

Time series data is a sequence of data points collected or recorded at successive time intervals, typically in chronological order. Each data point is associated with a timestamp, which represents when the data was captured. Common examples of time-series data:

1. Stock Prices: Daily closing prices of a stock over time

2. Weather Data: Temperature readings recorded every hour or day

3. Server Metrics: CPU utilization of a server over time

4. IoT Sensor Data: Sensor readings from devices over time

5. Financial Transactions: Sequence of financial transactions like payments made over time, categorized by timestamp.

Redis provides a NoSQL data structure called RedisTimeSeries which allows storing and querying time series data.

Figure 7 shows the steps involved:

1. The client sends data to the Analytics Service with a timestamp and the number of visits (can also be aggregated by the client and sent at once).

2. The Analytics Service adds this information to Redis using the TS.ADD command, which stores the number of visits with the given timestamp in Redis time series DB

3. TS.RANGE command can be used to retrieve time series data between two timestamps/in the last hour.

4. TS.MRANGE command can be used with an aggregation option (AGGREGATION count 60) to answer queries such as “how many visits occurred per minute”.

Managing user sessions with Redis

Another popular use case for Redis is managing user sessions. Its built-in support for time-to-live (TTL) makes it easy to automatically expire sessions after a designated interval.

Figure 8 shows the steps involved:

1. When a user logs in, the Login Service stores the session data for each user (456_session_token: {user_id: 456, logged_in: true}) in Redis using the HSET command.

2. A TTL (eg: 1 hr) is set for every token EXPIRE 456_session_token 3600.

3. During each API request, the login service retrieves the session data by using the HGETALL command on the session token (456_session_token).

4. Session expires after 1hr and the user is prompted to login again.

Redis for top posts in Social Media: Real-Time ranking and engagement optimization

Designing social media feeds or displaying the top posts related to a specific topic is another prevalent use case for Redis. By utilizing sorted sets (ZSET), Redis effectively stores posts as members and uses the number of likes as scores to rank them. As posts receive likes, their scores are updated in real time, maintaining accurate rankings.

Figure 9 shows the steps involved:

1. Whenever a new post is created, relevant keywords are extracted and stored in a Redis sorted set. For instance, a post like “Google launched Gemini…” would use the command ZADD keyword:google 100 post_id_12345 to store its likes.

2. The score of a post can be based on likes, impressions, or other metrics.

3. When a user searches for a keyword (e.g., "google"), the top posts are retrieved using ZREVRANGE keyword:google 0 9 WITHSCORES, which returns the top 10 posts sorted by likes.

4. When a post receives a like, its score is updated to reshuffle rankings with ZINCRBY keyword:google 1 post_id_12345, increasing the likes for that post to 101

5. To manage space, less popular posts are periodically removed. For example, ZREMRANGEBYRANK keyword:google 100 -1 removes all posts ranked below the 100th position, and ZREMRANGEBYSCORE keyword:tiger -inf 9 deletes posts with scores below 10.

Other use cases of Redis:

In addition to the use cases already covered, Redis can also be utilized to develop the following systems:

1. Real-time messaging using Pub-Sub (Figure 10)

2. Approximate the cardinality of a large set using HyperLogLog

3. Full-text search using the RedisSearch module

4. Storing nested JSON without re-serialization costs using RedisJSON

Conclusion

Redis is a powerful and versatile tool for system design interviews. Its straightforward data structures make it easy to reason about scaling implications, enabling you to engage in in-depth discussions with your interviewer without requiring extensive knowledge of Redis internals.

Apart from Redis, understanding other key technologies such as Kafka, Elasticsearch, Cassandra will give you an edge during deep dives in a system design interview. Will write separate posts on them soon 😉.

Thanks for reading!

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

Before we begin, here’s a quick recap of what we covered in the previous parts:

• Caching Part 1: What is caching, How it happens at the hardware level, Where should caching be implemented?

• Caching Part 2: Distributed Caches, Types of Caching Strategies, Types of Cache Eviction Strategies, Comparison between popular Distributed caches

If you are a recent subscriber, here’s what you might have missed:

1. How to use Redis in system design interviews?

2. How to crack any system design interview?

3. What is Kafka?

4. How to use Kafka in system design interviews?

5. How to design a payments system?

Thundering Herd Problem in Distributed Caching

Understanding the Thundering Herd problem

In a high-traffic environment, it’s common for one or more resources—like databases, cache nodes, or application servers—to become overwhelmed by an influx of requests. When this happens, traffic may unintentionally be redirected to a single instance, leading to cascading failures and resulting in service unavailability. This scenario, characterized by a sudden spike in incoming traffic, is known as the Thundering Herd Problem.

The Thundering Herd Problem can be seen in caching in two ways:

Cache Stampede Issue

Cache stampede is a situation that occurs when a popular cached key expires, causing multiple clients to request the same data simultaneously. This surge in requests overwhelms the underlying data store, as each client attempts to fetch or recompute the missing data at once. As a result, the sudden influx of traffic can lead to significant performance degradation and service unavailability.

Figure 2 shows the situation which leads to a cache stampede situation.

Example: E-commerce platforms during flash sales face cache stampede when the product availability cache expires, leading to a flood of DB queries as users check out at the same time.

Cache Avalance in Distributed Systems

Cache avalanche occurs when multiple cached keys in a single node expire simultaneously, or when a cache node goes down, leading to an overwhelming number of requests directed to the underlying data store or other cache nodes.

Unlike a cache stampede, which is triggered by the expiration of a single key, a cache avalanche involves the mass expiration of many keys at once. This scenario can severely strain backend DBs and result in performance issues, as they struggle to handle the sudden surge in requests.

Figure 3 shows one of the situations which leads to a cache avalanche situation.

Solutions to Thundering Herd

There are several strategies to address the thundering herd problem. Let’s explore each one of these in detail

Using Consistent Hashing

As discussed in the previous article on Consistent Hashing, distributing keys across multiple nodes, combined with replication, is crucial for maintaining high data availability in the event of node failures. This approach helps mitigate the effects of cache avalanches and cache stampedes by balancing the load among remaining servers, thereby reducing the risk of a single point of failure.

Implementing Key Locking/Leasing

This approach is similar to mutexes in operating systems, where only one process can access a shared resource at a time. In caching, when multiple clients try to access an expired key, the first request locks the cache to update the key, while subsequent requests wait for the cache to refresh. This helps minimize the number of requests to the backend data source.

Figure 5 shows how this works:

1. Key Expiration: When a popular key (k1) expires, the first application server (A1) querying it experiences a cache miss.

2. Lock Acquisition: A1 acquires a lock on the cache for k1, preventing other servers from accessing it until the update is complete.

3. Request Queuing: Other servers trying to access k1 while it's locked must wait in a queue.

4. Data Retrieval: A1 retrieves the latest data for k1 from the DB and updates the cache.

5. Release of Lock: After updating, A1 releases the lock, allowing queued servers to access the updated value.

Applying Random Expiration Policies

To prevent cache avalanche, implement staggered expiration by assigning a random time-to-live (TTL) to each cached key.

This technique effectively spreads out the expiration times of cached entries, minimizing the risk of multiple items expiring simultaneously. By introducing randomness in the TTL, you reduce the chances of overwhelming the backend system with a flood of requests when a cache reset occurs.

Using Circuit-Breaking Techniques

Circuit breaking stops further requests to the database until it is back online and able to handle traffic again.

Request Rate Limiting and Load Shedding

Request rate limiting and load shedding help prevent cache stampede by managing how many requests are processed. These techniques can be applied per user, client, or system-wide to maintain stability during peak loads.

Cache Penetration in Distributed Systems

Overview of the Cache Penetration Problem

Cache penetration occurs when clients, often with malicious intent, send requests for keys that do not exist in both the cache and the underlying database. Figure 7 shows an example scenario:

1. Requesting Non-existent Keys: Clients attempt to access a non-existent key("k1") which is absent in the cache.

2. Cache Miss and Database Query: Since the cache does not contain the key, the system queries the underlying DB for that key.

3. Null Response from Database: The database also lacks this key, resulting in a null response.

4. Overwhelming the Database: The malicious clients repeatedly send requests for the non-existent key, leading to a flood of queries that overwhelm the database and can degrade its performance.

Example: A news website might face this when many bots query outdated or non-existent articles, resulting in high backend load since the cache never stores these responses.

Solutions for Cache Penetration

There are two ways to mitigate cache penetration. Let’s look at each one of these.

Storing “null“ in cache to reduce load

Store a placeholder value like "null" in the cache for non-existent keys (Figure 8). This prevents unnecessary hits to the database by serving future requests directly from the cache. Set a TTL (Time to Live) for these entries to free up cache space after a reasonable period.

Using Bloom Filters to Prevent Misses

Bloom filters are probabilistic data structures used to quickly check whether a key is absent before querying the cache or database. If the filter suggests that a key isn’t present, it avoids hitting the cache or database, reducing unnecessary load, all while operating in constant time.

That's it for our overview of Bloom Filters for now. I'll explore the topic in more detail in a future deep dive.

Now, let’s return to the challenges associated with distributed caches. 👇

Hot Key Problem in Caching

What is the hot issue?

In a distributed cache system, keys are distributed across different nodes based on the sharding strategy. However, frequently accessed keys may cluster on the same node, leading to congestion. This phenomenon is known as the hot key problem. For instance, keys "k5" and "k6" located in node 2, can potentially overwhelm that node due to high traffic.

Example: If a specific video goes viral on YouTube, the cache for the video metadata might become overloaded as millions of users access the same data, causing delays in responses.

Solutions for Hot Key Problem

Replication for better Load Management

To effectively manage hot keys, it’s important to increase their replication factor beyond the standard approach used in a distributed cache cluster.

For instance in Figure 16, if k5 and k6 are identified as hot keys, create N replicas (k5-1, k5-2, …, k5-N) and distribute these replicas across various nodes. This way, when a read request for one of these keys is made, it can be served from any of the replicas at random, allowing for even traffic distribution.

This strategy helps prevent any single node from becoming overloaded, ensuring smoother performance for high-demand keys.

Applying Rate Limiting for high-traffic keys

To manage excessive requests for a key, you can implement rate limiting by placing additional requests in a wait queue. These queued requests will be processed only after the current ones have been completed successfully.

For a more in-depth understanding of rate limiting, I’ve written separate articles, Rate Limiting Part 1 and Part 2, which include detailed explanations and cheat sheets. I encourage you to check them out and share your feedback 😉!

Large key problem in Caching

What is the Large Key problem?

The large key problem occurs when a cache is used to store oversized data, such as large JSON responses, images, or big data blobs. Figure 17 shows a large key like "k2" with a 2MB payload can cause several issues:

• Network Bandwidth: Frequent access to large keys consumes more bandwidth, slowing down lookup times.

• Update Overhead: Partial updates to large keys force the entire value to be rewritten, impacting performance.

• Cache Eviction: If evicted or invalidated, reloading large keys from storage is slow, further degrading query performance.

Solutions for Large Keys

Using Compression to Optimize Storage

One effective approach to managing large values in caching is compressing them before storage. Compression reduces the memory used by cached data, making it easier to store larger values without consuming excessive cache space.

When a value exceeds a specified size, the system compresses it (using algorithms like GZIP, LZ4, or Snappy) before caching it. On retrieval, the data is decompressed for use.

However, compression adds latency to both read and write operations and can slow down debugging, as values must be decompressed before inspection.

Chunking Large Keys for Better Management

For handling large data structures like JSON, lists, or hashmaps in the cache, one effective strategy is to split the data into smaller pieces and store them under separate keys.

Figure 19 shows how user video metadata (like video views and resume timestamp) is cached.

Instead of storing everything in one large object (eg: user_id -> video_metadata), the key can be split into smaller parts. For instance, user_1_v1_views: 10 and user_1_v1_resume_at: 10:20 to store views and resume timestamps for video v1 in separate keys, reducing the size and complexity of individual entries.

Hybrid Caching Approaches for Large Keys

In some cases, using a hybrid caching strategy with multiple cache layers can be efficient. Smaller, frequently accessed items can be stored in fast in-memory caches like Redis or Memcached, while larger, less frequently accessed data can be kept in disk-based caches, databases, or object storage systems.

For example in Figure 20, instead of storing large objects directly in the cache, the cache holds a reference (like an Amazon S3 URL) pointing to the data stored elsewhere. When large data is needed, the application retrieves it from the external storage using the cached reference.

Using Proxy Clusters for Scalability

A dedicated multi-node storage cluster can sit between app servers and the distributed cache cluster to handle large keys. While slower than in-memory caches, this layer, using NVMe or SSD drives, offers a faster alternative to disk-based storage, maintaining the benefits of caching for large keys.

Due to the added complexity of managing another storage layer, this approach should be considered an outlier. It is worth exploring only when there are many large keys and simpler solutions, like compression or chunking, aren’t effective.

Memory Fragmentation in Caching

What is the memory fragmentation problem?

Fragmentation occurs when free memory becomes scattered in small, non-contiguous blocks throughout the system. This happens over time as data is repeatedly allocated and deallocated in the cache, making it inefficient since these fragmented blocks may be too small to hold new larger cache entries.

This issue is more common in in-memory caches like Redis or Memcached, especially when key-value sizes vary widely or memory reclamation isn’t consistent. As memory becomes fragmented, it can lead to reduced cache performance and difficulty in handling larger items.

Example (Figure 22):

Imagine a Redis cache system with a memory capacity of 1 GB. Initially, small keys of size < 5 KB each are stored in the cache. When a large key (eg: 10KB) is added, it won’t fit in smaller spaces and will start creating small gaps in the system. Even though the cache might have 500 MB of free memory available, it might only be in small chunks like 2 KB, 4 KB, and so on. These fragmented blocks cannot be used to store the larger entries, and eventually, the system runs into out-of-memory errors even though, theoretically, there is still enough memory available.

Solutions to Mitigate Memory Fragmentation

Most of the popular distributed caches already have inbuilt mechanisms to deal with memory fragmentation issues. Let’s look at how Redis and Memcached solve this:

Applying Defragmentation techniques

The Defragmentation/Garbage Collection process compacts memory blocks by relocating data to create contiguous blocks of free space. This can reduce fragmentation and make better use of available memory. Redis, for example, triggers memory defragmentation when it detects high levels of fragmentation, which can be configured through Redis commands like MEMORY PURGE

Using Slab Allocation for Efficient Memory Management

Instead of dynamically allocating memory for each new cache entry, Memcached uses predefined memory chunks (slabs) of fixed sizes to store objects. Objects are grouped by size, and each size group is allocated memory from a corresponding slab. For example, objects of size 16 KB might all be stored in 16 KB slabs, while larger objects are stored in their appropriate size slabs. This prevents small objects from leaving gaps between larger ones, thus reducing fragmentation.

This brings us to the end of this series on caching! This was the longest and the most important part because it shares a lot of practical challenges and mitigation strategies while dealing with caching systems at scale.

Congrats if you made it here 🎉! Just like previous posts, sharing a consolidated cheat sheet on caching problems & solutions 🧷.

PS: Cheatsheets of previous parts of the series — Part 1 & Part 2.

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

Get in Touch

You can find me on LinkedIn or Twitter.

Today, we will focus on Distributed Caching, exploring how large-scale systems manage caching effectively. The key topics we'll cover include:

1. What is Distributed Caching and its advantages?

2. How does distributed caching work?

3. What is Consistent Hashing?

4. Types of caching strategies

5. Types of cache eviction strategies

6. Popular examples of distributed caches

If you are a recent subscriber, here’s what you might have missed:

1. A crash course on caching—Part 1 & Part 3

2. How to crack any system design interview?

3. A crash course on Rate Limiters - Part 1 & Part 2

4. What is Kafka?

5. How to use Redis in system design interviews?

Before we dive deep, here's a distributed caching cheat sheet 🧷.

What is distributed caching?

A naive approach to caching data is to store it in a single-node system, but this method faces several challenges:

• Scalability: How to handle the vast volume of data that needs to be cached?

• Fault Tolerance: How can we ensure high availability of cached data if one node fails?

• Load Balancing: How to handle a high volume of concurrent read requests, such as a Twitter feed accessed by millions of users?

Distributed caching addresses these challenges as systems scale to accommodate millions of users and large datasets.

In a distributed caching system, multiple nodes store replicas of each dataset. This architecture enhances fault tolerance and ensures high availability while allowing load balancing of requests across nodes, enabling the system to efficiently handle millions of reads.

How Distributed Caching works?

Figure 2 shows the components of a distributed cache system and how it works:

1. Cache Client - When caching data, the client library utilizes a hashing or sharding algorithm to determine the appropriate node or partition for data storage.

2. Cache Cluster

   1. A dedicated multi-node cluster operates with a master-slave configuration to manage all read and write operations.

   2. Data is replicated across multiple nodes to enhance fault tolerance and ensure high availability.

   3. All write operations are directed to the master node, which determines how to distribute the data to the slave nodes.

   4. Coordination mechanisms, such as distributed locks or consensus protocols, keep cache nodes synchronized, especially during simultaneous data modifications by multiple nodes.

3. Sharding - Data is divided into shards, each stored on a different cache node. This approach helps distribute data evenly and enables horizontal scaling of the cache.

4. Caching Strategies - Policies for read and write operations must be established to determine how data is updated between the database and the cache or vice versa.

5. Eviction Policies - Caches implement eviction strategies, such as LRU (Least Recently Used), LFU (Least Frequently Used), or TTL (Time to Live), to remove outdated or less frequently accessed data and create space for new entries.

Types of sharding algorithms in distributed caching

Sharding algorithms are essential for distributing data across multiple nodes in a distributed caching system. They determine which node to read from or write to. There are 3 types of sharding algorithms:

Modulus Sharding

Modulus sharding involves assigning a key to a shard based on the hash value of the key modulo the total number of shards.

Pros & cons

✅ Uniform Distribution: The modulus function evenly distributes data across shards, avoiding hot spots since the hash output is random.

✅ Simplicity: Easy to implement, as it relies on a simple hash function to determine the shard based on a key.

❌ Scaling Difficulty: Modulus-based sharding complicates adding or removing shards because keys have to be redistributed, causing significant data migration.

❌ Poor Range Queries: Since the sharding is based on the hash of keys, executing range queries (eg: getting keys between 1000 and 2000) becomes inefficient and requires querying multiple shards.

Range-based Sharding

Range-based sharding assigns keys to specific shards based on predefined key ranges. With this approach, the system can divide the key space into specific ranges and then map each range to a particular shard.

Pros & cons

✅ Efficient Range Queries: Range-based sharding optimizes range queries by storing data in specific shards, reducing the need to access multiple shards.

✅ Logical Data Grouping: Related data is stored together, enhancing retrieval efficiency.

❌ Load Imbalance Risk: Uneven data distribution can overload certain shards if many values fall within a specific range.

Consistent Hashing in Distributed caching

Modulus and range-based sharding are easy to manage but struggle with scaling. When a node goes down or a new node is added (a common scenario in distributed systems), keys must be completely redistributed, leading to downtime and complexity.

Consistent hashing (Figure 5) solves this problem by mapping both keys and nodes onto a fixed-size ring using a hash function. This method minimizes key redistribution when nodes change, improving scalability and stability.

How does consistent hashing work?

1. When a key is requested, the system applies a hash function to map the key to a specific position on the ring.

2. The system traverses the ring in a clockwise direction from the key's position until it encounters the first node.

3. This node is responsible for storing the data associated with the key.

Handling failed nodes in Consistent Hashing

When a node in the ring goes down, all keys assigned to that node are reassigned to the next available node in the ring, while other keys remain unaffected.

Figure 6 explains this scenario. When node 4 fails, the key k6 which was previously assigned to node 4, is reassigned to the next node in the ring i.e. node 1.

Handling new nodes in Consistent Hashing

When a new node is added to the ring, all keys that lie between this new node and the previous node are remapped to the new node, while keys assigned to other nodes remain unaffected.

Figure 7 explains this scenario. When a new node (node 5), is added to the cluster, the key k6, previously assigned to node 4, is now reassigned to the new node 5.

Types of Caching Strategies for Distributed Systems

There are five types of caching strategies, categorized into two subtypes based on their focus: read strategies and write strategies.

1. Read strategies: Cache Aside, Read Through

2. Write strategies: Write Around, Write Back, Write Through

In the following sections, we will explore each strategy in detail and discuss their pros & cons.

Cache Aside

1. The application requests data for a key from the cache.

2. If the key is present in the cache (cache hit), it is returned to the application.

3. If key is not present in the cache (cache miss), the application:

   1. Fetches data from the main DB.

   2. Updates the cache with latest data for future reads.

Pros & Cons of cache aside

✅ The system can tolerate cache failures, as it can still read from the storage.

❌ The application must manage both cache and storage, complicating the code.

Read Through cache

1. The application requests data for a key from the cache.

2. If the key is present in the cache (cache hit), it is returned to the application.

3. If key is not present in the cache (cache miss), the cache:

   1. Fetches data from the main DB.

   2. Updates itself with the latest data and returns it to the application.

Pros & Cons of read-through cache

✅ The implementation is simple, as the application only needs to manage cache (not storage).

❌ The system cannot tolerate cache failures as it doesn’t rely on DB.

❌ The cache populates only after the first request, leading to slower initial access for new data— an issue for high-throughput systems that need pre-warmed caches.

Write Around cache

1. All the write requests go directly to the DB and read requests land on the cache.

2. If the key is present in the cache (cache hit), it is returned to the application.

3. If key is not present in the cache (cache miss), the cache:

   1. Fetches data from the main DB

   2. Updates itself with the latest data and returns it to the application

Pros & Cons of write around cache

✅ Cache contains the data that it really needs (frequently read).

❌ Frequent cache misses for recently written data can cause inconsistent read performance, as data may need to be fetched from the database.

Write Back cache

1. All the data is written directly to the cache.

2. The actual write to the DB happens later, asynchronously, or when the cache data is evicted.

Pros & Cons of write-back cache

✅ Writes are faster since they are only performed in the cache, deferring the actual database write.

✅ Multiple writes to the same data can be combined into a single database operation, reducing the number of I/O operations.

❌ The cache holds the most recent version of the data, so if other services access the database directly, they might see stale or outdated data.

Write Through cache

1. All write requests from the application land on the cache.

2. The actual write to the DB also happens in sync before returning response to the application.

Pros & Cons of write-through cache

✅ Since writes are performed in both the cache and the database simultaneously, the cache is always in sync with the DB, ensuring data consistency.

✅ By writing data to the cache during writes, future reads benefit from cached data, enhancing read performance.

❌ Every write operation incurs the overhead of writing to both the cache and the database, which can slow down the write performance.

Cache eviction strategies in Distributed Caches

Cache eviction is crucial for maintaining fresh and relevant data. There are four main types of cache eviction policies:

Least Recently Used (LRU)

LRU evicts the cache entry that has not been used for the longest time. It assumes that data that hasn't been accessed in a while is less likely to be needed soon.

Pros & Cons

✅ Adapts well to scenarios with temporal locality (data accessed recently is likely to be accessed again soon).

❌ Requires tracking the usage order of all items, which can be computationally expensive for large caches.

Example: Web browsers often use LRU to manage their cache of visited web pages. The most recently visited pages are kept in the cache, while the least recently accessed pages are evicted to make space for new ones.

Least Frequently Used (LFU)

LFU evicts the items that are accessed the least number of times. It tracks the frequency of access to each cached item and removes those that have been used the least over time.

Pros & Cons

✅ Works well in scenarios where frequently accessed data is important (product details on an e-commerce website).

❌ Harder to implement as it requires tracking access counts.

Example: Search engines use LFU in their query caching mechanisms to keep the most frequently searched queries cached, allowing fast retrieval for popular searches.

First In First Out (FIFO)

FIFO evicts the oldest item in the cache, based on when it was added, regardless of how often it has been accessed. The first item added is the first item removed.

Pros & Cons

✅ Simple to implement since it doesn't require tracking access times or counts.

❌ Doesn’t account for how often or how recently an item is used, which can lead to suboptimal cache efficiency.

Example: Message brokers like RabbitMQ may use FIFO in their in-memory message cache, where older messages are evicted to ensure space for newer ones without considering usage patterns.

Time To Live (TTL)

TTL evicts items automatically from the cache after a predefined time period, regardless of how frequently or recently the item was accessed. It’s a time-based expiration method.

Pros & Cons

✅ Useful when data is time-sensitive, such as temporary data or cached results that may become invalid after a certain period.

❌ Items may still be relevant when evicted if they haven’t been accessed frequently during the TTL window.

Example: User session tokens can be cached with a predefined TTL (eg: 5 mins). Once the TTL expires, the token is evicted from the cache, and a new session token must be generated.

Popular distributed caching solutions

Redis and Memcached are two most popular open-sources distributed caches. A quick comparison matrix between them:

Additionally, Amazon ElastiCache is a fully managed in-memory caching service from AWS that supports both Redis and Memcached, giving developers flexibility in choosing the right engine for their use case.

It offers multi-AZ (Availability Zone) deployments with automatic failover, ensuring high availability even during an AZ outage.

ElastiCache can also auto-scale the number of nodes in the cluster based on traffic demand, enabling seamless handling of fluctuating workloads without manual scaling efforts.

That’s all for today. Thanks for reading 🙏🏻!

In the next issue, we will explore key challenges with caches & how to solve them, how to implement caching (with code snippets), & key gotchas about how NOT to do caching.

Creating each article, along with the accompanying diagrams, takes about 12-14 hours of research and hard work. If you enjoy my content, please ❤️ and subscribe to encourage me to keep creating more such valuable content! 🙏🏻.

Since caching is a broad topic, I will explore its various facets in a comprehensive 3-part series. In this first issue, we will start with the basics and cover the following topics:

1. What is caching & why is it necessary?

2. How does caching occur at the hardware layer (CPU/OS Level)?

3. Where to implement caching?

4. Real-world examples from large-scale systems

If you are a recent subscriber, here’s what you might have missed:

1. A crash course on caching - Part 2 & Part 3

2. A crash course on Rate Limiters - Part 1 & Part 2

3. What is Kafka?

4. How to use Kafka in system design interviews?

5. How to design a payments system?

Before we dive deep, here's a quick caching cheat sheet 🧷.

What is caching and why is it necessary?

What is caching?

Caching is a technique used in computing to store frequently accessed data in a fast storage area, usually in memory. Instead of using a database which can be slow due to disk reads, caching lets the system quickly access the same data. This speeds things up and makes the system run more efficiently.

Example

Figure 2 illustrates a practical example of how caching works. In this scenario, data is first retrieved from the cache (stored in memory). If the data is not found in the cache, it is then fetched from the database. This contrasts with a system without caching, where data is always retrieved directly from the database, leading to longer response times.

Key terms in caching

• Cache Hit: When data is found in the cache, it’s called a cache hit, allowing quick access without going to the DB.

• Cache Miss: A cache miss happens when the data isn’t in the cache, so the system has to fetch it from the DB.

• Cache Hit Ratio: The cache hit ratio measures how effective the cache is. It’s the number of cache hits divided by total requests—higher ratios mean better performance.

Pros & Cons of caching

Pros:

1. Faster data access: Caching speeds up data retrieval by avoiding slow database or disk access.

2. Reduces backend load: By serving frequent requests from the cache, it eases the load on databases, APIs, and backend systems.

3. Better user experience: Faster responses lead to smoother interactions. This improves user satisfaction.

Cons:

• Limited capacity: Caches have less space than disks. So, they can't store much data or provide long-term persistence.

• Stale data: Cached data can become outdated if not refreshed, causing inconsistencies with the DB.

• Higher costs: Storing data in memory is more expensive than on disk, and memory is more limited.

• Cold cache delays: When the cache is empty (a "cold cache"), initial data requests may still experience slower response times. Note: We will discuss problems with caching in detail in a later issue.

Caching at the hardware level: CPU and OS caching explained

Hardware caching is built into systems to make data access faster. CPUs and operating systems have caches that help retrieve frequently used data quickly, reducing the need to access slower memory or disks. In this section, we’ll cover how CPU and OS-level caches work to improve system performance.

Figure 3 shows different mechanisms used by the CPU and OS for caching.

CPU Caching

CPUs use multi-level caches to store frequently accessed instructions for faster processing. There are four key types of caches found in the CPU:

1. L1 Cache - The fastest and smallest cache, typically under 64KB, with each CPU core having its own dedicated L1 cache.

2. L2 Cache - The second level of caching, ranging from 512KB to 1MB, and also dedicated per core.

3. L3 Cache - The largest but slowest cache, usually over 3MB, shared between all CPU cores.

4. Translation Lookaside Buffer (TLB) - Stores recently used address mappings to speed up virtual-to-physical memory translation. It resides in the Memory Management Unit (MMU) and helps optimize memory access.

OS-level caching

1. Page Cache - Located in main memory, it stores frequently accessed data pages, reducing the need for slower disk access.

2. Inode Cache - An in-memory cache that stores file metadata (inodes). It speeds up access to file properties like size and permissions. This improves file system performance.

Where should caching be implemented?

Caching can be applied at different points in a system's infrastructure to enhance performance and reduce latency. Key areas include browsers, CDNs, API gateways, databases, and distributed caches. Let’s examine each one to see how caching optimizes system performance.

Client-side caching: Improve user experience and reduce latency

Also known as browser caching, this involves storing static assets like HTML, JavaScript, and images on the client side. This helps speed up page loads by avoiding repeated downloads from the server.

Figure 5 shows how HTML is cached in the browser with an expiration time. Future requests can then be served from the cache, instead of reloading from the server.

Example: When a user visits an e-commerce website, the browser caches images, stylesheets, and scripts. On future visits, these assets load from local storage. This speeds up the experience by avoiding unnecessary downloads.

Types of client-side caching:

• Local Storage Cache: Stores static assets such as images and HTML in the browser’s local storage.

• Service Worker Cache: Allows applications to work offline by caching responses. It enables features like Progressive Web Apps (PWAs) to function without an internet connection.

CDN caching: Enhancing web performance through Content Delivery Network

A Content Delivery Network (CDN) caches static content such as images, videos, stylesheets, and scripts on servers located in various geographic areas. This helps reduce latency by delivering content from a server closest to the user.

Example: In video-streaming services like Netflix or YouTube, CDNs store video content in different locations. When a user in India watches a movie, the CDN delivers it from a nearby edge server instead of the original source.

Types of CDN caching:

• Static Content Caching: Stores static files like images, videos, and scripts for quick access.

• Dynamic Content Caching: Some CDNs can also cache dynamic content using methods like edge-side includes or smart caching algorithms.

Caching at API Gateway/Load Balancer

API gateways cache responses for frequently accessed API endpoints to reduce the load on backend services. This is ideal for read-heavy APIs with relatively static data.

Example: A currency conversion API that updates its rates every 10 minutes can cache responses at the gateway, reducing the need to hit backend services for every request.

Types:

• Per-Route Caching: Caches responses for specific API routes.

• Header-Based Caching: Cache can vary depending on headers like user-agent, locale, etc.

Distributed Caching

Distributed caching stores data across multiple nodes. It ensures quick access and high availability in large-scale systems. Highly scalable systems use it to improve access speed across multiple regions or services.

Example: A social media platform may store user session data in a distributed cache (eg: Redis Cluster), ensuring that users can access their sessions quickly regardless of which region they’re in.

Redis and Memcached are two widely used distributed caching systems.

Figure 8 illustrates a typical setup — 90% of read requests from clients are directed to a distributed cache cluster with multiple nodes. This design ensures high availability—if one node goes down, data can still be served from another node that holds a replica. Only 10% of requests go to the database due to cache misses.

Note: We will discuss distributed caching in detail in the next article.

Caching in Relational databases

Relational databases, such as MySQL, use in-memory caching to store query results or frequently accessed data. This approach speeds up subsequent access by reducing the need for costly disk I/O operations or executing complex queries each time.

Example: An e-commerce site could cache the results of a popular query like "top-selling products," which doesn't change often, in a Redis cache to avoid re-executing the same query repeatedly.

Types of caches in relational DBs:

1. Transaction log: It records all atomic changes made to a database (eg: credit/debit in banking applications). It is used primarily for recovery purposes, ensuring that all operations can be rolled back in case of failure.

2. Write-Ahead Logging (WAL): WAL is a specific logging technique where changes are written to a log before being applied to the database. This ensures that data can be recovered in case of a crash.

3. Replication log: They record changes to the primary database. Then, they send these to replicas to ensure consistency in distributed systems.

4. Bufferpool: An area in memory where frequently accessed data pages from the database are cached. Example: In e-commerce, product information that is accessed frequently is cached in the buffer pool. This allows for quick retrieval at peak shopping times.

5. Materialized views: It’s a precomputed result set that is stored like a table. It can be refreshed periodically to reflect changes in the underlying data. Example: In a reporting application, a materialized view aggregates sales data for quick access during analysis. This allows users to retrieve summary reports faster than querying the raw data directly.

Caching in message queues for high-traffic applications

Message queues store messages on disk until they are processed by consumers. You can set custom retention policies to keep messages for a specific period. This allows for retries if consumers are unavailable or in case of failures, such as a Kafka outage.

Example: In an e-commerce application, when a user places an order, the order details are sent to a message queue (eg: RabbitMQ or Kafka). The message is stored on disk until a fulfillment service processes it. A retention policy keeps unprocessed messages for a set duration eg: 7 days) to enable retries in case of failures.

Caching in search systems

Search systems, such as Elasticsearch or Solr, index documents to facilitate quick retrieval based on queries. They use different indexing techniques, including forward and inverted indexing, to optimize search performance.

Types of caching in search systems:

• Forward Index

  • The forward index maps documents to their terms, allowing quick access to the content of each document.

  • Example: Google Docs maintains a forward index that links document IDs to headings, enabling it to quickly generate the table of contents for billions of documents in real time.

• Inverted Index

  • An inverted index (Figure 11) maps terms to their corresponding documents, allowing for rapid lookups. It is the backbone of most modern search engines.

  • Example: In a search engine for a news website, the inverted index lists terms like "election," "president," and "vote," with pointers to the articles that contain those terms. If a user searches for "president," the system quickly retrieves all relevant articles without scanning every document.

How Netflix, Facebook, and Twitter leverage caching for efficiency?

Let’s look at some real-world examples of caching in large-scale systems:

Netflix

1. Content Delivery Network (CDN) : Netflix uses its own CDN, called Open Connect, to cache video content closer to the users. By deploying servers at ISPs (Internet Service Providers), Netflix reduces latency and bandwidth costs while ensuring high-quality video streaming.

2. Personalization and Recommendations : Netflix caches personalized recommendations and metadata about shows and movies. This allows the recommendation engine to quickly provide relevant suggestions without repeatedly querying the backend systems.

Facebook

1. Memcached Deployment: Facebook is known for its large-scale deployment of Memcached to cache data retrieved from its databases. This caching layer helps reduce the load on databases, allowing them to scale horizontally.

2. TAO (The Associations and Objects): Facebook developed TAO, a geographically distributed data store that caches and manages the social graph (relationships and interactions between users). TAO ensures that frequently accessed data, such as friend lists and likes, are served quickly, improving the overall user experience [Resource]

Twitter

1. Timeline Caching: Twitter caches timelines (feeds of tweets) to ensure that users see updates quickly. By caching these timelines, Twitter reduces the need to query the database for every user request, significantly improving response times.

2. Redis for In-Memory Caching: Twitter uses Redis for various caching purposes, including caching user sessions, trending topics, and other frequently accessed data.

Thanks for reading 🙏🏻!

This concludes the first part of our 3-part series on caching. In the next issue, we will explore key concepts such as distributed caching, types of caching strategies, and cache invalidation/eviction methods.

Creating each article, along with the accompanying diagrams, takes about 12-14 hours of research and hard work. If you enjoy my content, please ❤️ and subscribe to encourage me to keep writing! 🙏🏻

Why does this matter?

• It can determine whether you’re seen as a fit for junior or senior roles.

• It also has a direct impact on your compensation.

This article will explore a proven framework. It will help you ask the right questions and focus on what matters in a system design interview. ⚡️

If you’ve recently subscribed, these articles will help you get up to speed on system design:

1. How to use Redis in system design interviews?

2. How to use Kafka in system design interviews?

3. A crash course on Caching—Part 1, Part 2 & Part 3

4. A crash course on Rate Limiters - Part 1 & Part 2

5. How to design a payments system?

Before we dive deep, here's a quick system design cheat sheet 🧷

Step 1: Clarifying System Design Requirements

The first step in any system design interview is to grasp the problem. It's vital to ask good questions. They ensure a deep understanding of the system's purpose, users, and limits.

1.a Types of requirements in system design

A key part of this step is distinguishing between the two broad types of requirements.

Functional requirements

• Core features: What is the primary function of the system?

• Feature prioritization: Are there any features that are more critical or time-sensitive?

• Users: Who are the main users (e.g., customers, internal teams like developers, DevOps, QA)?

• User interaction: How will users interact with the system—via web, mobile, or API?

• External integrations: Does the system need to connect with third-party services or APIs?

• Business logic: Must the system follow any specific rules or constraints?

• Notifications: Should there be real-time or delayed notifications (eg: emails, push notifications)?

• Error handling: What should happen in the case of a system error or failure?

Non-functional requirements

High-Priority Considerations:

1. Scalability: How many users and requests should the system handle? Plan for future growth.

2. Availability: Must the system be highly available, or can it have some downtime?

3. Latency: What are the acceptable latency and throughput for efficient system functioning?

Low Priority Considerations:

1. Consistency: Can the system tolerate eventual consistency? Or does it need strong consistency?

2. Security: How critical is securing the APIs and protecting data? Are there specific security requirements or compliance needs?

3. Accuracy: How important is absolute accuracy in the system? Can it tolerate minor discrepancies, or must it be 100% accurate?

Tip💡: For senior roles, emphasize non-functional requirements like scalability, availability, and performance. For junior roles, focus more on the core functional requirements of the system.

1.b Capacity Estimation

The second sub-step estimates the needed system capacity. It is based on insights from the previous step. This estimate is vital for sound design choices in the next stages of the interview. Capacity estimation helps guide choices around infrastructure, scaling strategies, and performance optimization.

We can break down capacity estimation into four key aspects:

• Users

  • Estimate the number of monthly and daily active users.

  • Account for peak activity periods by estimating the number of users.

• Storage

  • Estimate volume through activity patterns (eg: 2 daily posts for a Twitter-like system).

  • Identify the type of data (eg: text, images, videos) that is being stored and the expected duration of storage.

  • Calculate the total storage capacity required based on the volume and type of data.

• Requests

  • Estimate the system's Queries Per Second (QPS) based on the number of users and their activity.

  • Provide a peak QPS estimate to ensure the system can handle high traffic.

  • Divide QPS into read and write operations to inform design decisions, such as database architecture and caching.

• Memory

  • Estimate the memory required to store frequently accessed data for rapid retrieval.

Figure 3 below shows a back-of-the-envelope calculation for a Twitter-like system based on the above aspects (These numbers are illustrative, not actual Twitter data).

Step 2: Propose a high-level system design

With the requirements and capacity estimation phase complete, the next step is designing the system. This process has three key parts (Figure 4), each of which will be discussed in detail.

2.a High level design

Create a simple block diagram. It should show the key system components. It must outline the high-level flow of data and requests. This starts from the client, goes through the backend, and returns the response.

Figure 5 highlights the essential components that should be emphasized in any high-level design.

The diagram should include the following components:

• Client Applications

  • Specify if the client is a mobile app, web app, or an internal dashboard for internal teams.

  • If internal teams are the users, the client may be a CLI tool within the VCN.

• Content Delivery Networks (CDN)

  • Use case: If handling media storage or streaming, CDNs improve performance by caching media close to users.

  • CDNs can also help limit rates at the edge to protect backend services.

• API Gateway

  • Entry point for all incoming requests into the VCN, handling routing to respective services.

  • Functions: Load balancing, rate limiting, and sometimes authentication (e.g., AWS API Gateway).

• Application Servers

  • Deploy microservices for specific functionalities (e.g., user onboarding and authentication services).

  • Bonus: Briefly mention Monolith vs Microservices but focus on microservices, which is the modern approach.

• Database

  • Specify database type based on the use case.

  • Examples: MySQL for relational data and MongoDB for unstructured data in document-oriented databases.

• Caching

  • Use Redis or Memcached to reduce database load by caching frequently accessed data.

  • Bonus: Mention caching strategies such as write-through or TTL for cache invalidation.

• Message Queues

  • Required for asynchronous communication, like OTP or email delivery.

  • Bonus: Compare Kafka (distributed, scalable) vs RabbitMQ (traditional messaging) if needed.

• External Services

  • Outline external dependencies like payment processors and alert services. Note any third-party integrations critical to the core functionality

  • Bonus: Discuss methods of status updates like webhooks, status APIs, or event-driven updates.

• Bonus Components (Optional): If time allows, and you understand them, add these optional components to your block diagram.

  • Ingress Gateway

    • Sits at the edge of the Kubernetes cluster, directing traffic from the API gateway to the microservices.

  • Egress Gateway

    • It handles outbound requests from microservices to external services. It sits at the edge of the Kubernetes cluster.

2.b API Design

Designing the API and choosing the right protocols are crucial. They define how system components communicate and how clients access the system's features.

Figure 6 shows the key aspects of API design.

• Types of Communication Protocols:

  • HTTPS: Standard for secure communication in most APIs (eg: REST APIs for user authentication or data fetching).

  • WebSockets: Ideal for real-time data exchange where low latency is key (eg: live chat, stock market updates).

  • gRPC: Best for fast, efficient communication between microservices (eg: internal service-to-service calls).

  • Message Protocols: Use in asynchronous scenarios (eg: Kafka or RabbitMQ for event-driven architectures).

• Types of APIs:

  • REST: It is often used for CRUD operations, like creating, reading, updating, and deleting user details.

  • GraphQL: Allows clients to request specific data fields (eg: flexibly fetching custom user profiles).

  • XML: Suitable for legacy systems or applications needing a strict schema (eg: financial data exchange systems).

• Follow Common Conventions:

  • Plural Naming: Use plurals for resource names (eg: /v1/users for multiple users).

  • Versioning: Use versioning for backward compatibility (eg: /v1/users not /users/v1).

  • Pagination: Always paginate search or listing APIs to handle large datasets (eg: GET /v1/users?page=5&pageSize=20).

• Idempotency:

  • Ensure critical APIs, like payment processing, are idempotent. This prevents duplicate charges. For example, use an idempotency key like {idempotency_key: "abcd-fdfs"} with POST /v1/payment/initiate.

• Data in API Calls:

  • Define request/response body structures for consistency and ease of use. For example, include user details in the POST /v1/users request body.

  • For use cases such as fraud detection, geolocation and IP address are often required. Pass this data via headers (eg: x-latitude for location and x-ip-address for the user's IP).

  • Mention response status codes clearly:

    • 200 OK: Request was successful and data is returned.

    • 201 CREATED: A new resource was successfully created.

    • 403 FORBIDDEN: The request is valid, but the server is refusing to fulfill it.

    • 302 FOUND: The requested resource has been temporarily moved to a different URL.

• API Segregation:

  • Public APIs:

    • These are open to external clients, often with few restrictions. For example, anyone can create an account via the public POST /v1/signup.

    • Implement rate limiting to prevent abuse. Inform clients of limits via headers (eg: x-rate-limit: 1000 for a max of 1000 requests per user).

  • Private APIs:

    • Require authentication and are accessible only to authorized users. For example, to access sensitive account details, use POST /v1/account with a Authorization: Bearer <token> header.

    • Use JWT-based tokens for managing sessions and validating users securely.

  • Internal APIs:

    • Internal services or dashboards rely on these APIs for operations not exposed to external clients. For example, GET /v1/users retrieves user data for internal admin dashboards.

    • To ensure that they are not accessible from outside the network, block these at the CDN/API gateway level.

2.c Database Design

This step (Figure 7) involves designing the DB schema and choosing a DB type. We must also address concerns about query performance through normalization, partitioning, and caching.

• Database schema

  • Table Structure - Map out the key tables and include important fields. Clearly define the data types for each attribute. Eg: user_id as INT, product_name as VARCHAR.

  • Primary & Foreign Keys - Specify primary and foreign keys to establish relationships between tables. Eg: product_id as the primary key in Products table and foreign key in Prices table.

  • Indexing: Identify columns that should be indexed for faster query performance. Eg: index product_name and category_name in an e-commerce data to improve search speed.

  • Denormalization - Create denormalized tables to avoid complex joins in read-heavy queries. Eg: In an e-commerce app, a denormalized table combining product details, reviews, and prices reduces the need for multiple table joins.

  • Partitioning - Partition large tables by date or range. Eg: Partition orders by month in an online store to improve query performance for recent transactions.

  • Caching Frequently Accessed Data - Implement caching for commonly queried data like product prices or user profiles to reduce load on the database and improve performance.

• Types of Databases

  • Relational (SQL) - Store data in structured tables with rows and columns, useful in case of relations between data. Eg: MySQL, Postgres, CockroachDB (distributed SQL)

  • Non-Relational (NoSQL) - Flexible schema design; can store various data types. Types of NoSQL DBs:

    • Key-value - Simple database where data is stored as key-value pairs. Eg: Redis → used for caching and real-time analytics.

    • Column store - Store data by columns rather than rows, optimizing read-heavy queries. Eg: Cassandra → Well-suited for building metadata stores in distributed systems; Clickhouse → Optimized for high-performance analytics and data warehousing, ideal for processing large volumes of data quickly.

    • Graph - Store data as nodes and edges, ideal for complex relationships. Eg: Neo4j → Used for social networks and recommendation engines.

    • Document - Store data in JSON-like documents; flexible schema. Eg: MongoDB → Popular for applications requiring dynamic data structures.

Before wrapping up this section, here are some useful tips to follow during the high-level Level Design step:

Tips💡:

1. Avoid including irrelevant components. For eg, in a payment system, there's no need to include authentication in the high-level design.

2. At this stage, avoid going into details about individual components. Allocate time to dive deeper later on in the discussion.

3. Note down key points to discuss in the next section:

   1. API scaling

   2. DB scaling

   3. Concurrency

   4. Failure scenarios

   5. Consistency

Step 3: Address Key Issues in the design

This is the final step which involves identifying and addressing the key challenges the system is likely to face (Figure 8). This phase needs a structured approach to problem-solving. Collaborate with the interviewer to find which areas to explore in depth.

Key points to keep in mind:

1. Non-Functional Requirements: Emphasize scalability, performance, reliability, and other non-functional factors based on prior discussions.

2. Collaborate on Focus Areas: Work with the interviewer to pick specific sections or problems to explore in-depth.

3. Follow a 4-step problem-solving framework to apply to any problem in this section (Figure 9):

   • Articulate the problem: Clearly define the challenge being addressed.

   • Generate multiple solutions: Present at least two potential solutions.

   • Discuss trade-offs: Compare the pros and cons of each solution. Consider cost, complexity, and scalability.

   • Pick and deep dive: Choose the most appropriate solution and analyze it in detail with the interviewer.

Potential problem areas and solution approaches:

• Scalability:

  • Application Scaling:

    • Discuss whether to scale horizontally (adding more servers) or vertically (upgrading CPU and memory of existing servers).

    • Example: In a high-traffic e-commerce platform, horizontal scaling might be preferred to handle spikes during peak sales.

  • Database Scaling:

    • Add indexes to critical columns to improve read performance.

    • Partition large tables into smaller ones to reduce query time.

    • Normalize tables to reduce data redundancy, or use denormalization to optimize read-heavy operations.

  • Caching:

    • Identify high QPS APIs where caching can reduce load (eg: frequently accessed product pages in an e-commerce system).

    • Discuss cache storage options like Redis or Memcached, outlining the pros and cons of each.

    • Present different caching strategies (eg: write-through, write-back) and select the most appropriate one for the system’s needs.

    • Define cache invalidation strategies, either manual (via code) or automatic with TTL (Time To Live).

  • Message Queues:

    • Replace asynchronous communications with message queues to handle task processing efficiently.

    • Discuss strategies to ensure exactly-once delivery, critical for systems like payments (Read Avoid double payments to learn the key strategies)

      • Example: In a payment processing system, idempotency keys ensure that users are charged only once, even if the request is retried.

    • Implement retry mechanisms for handling failed deliveries at both producer and consumer sides, ensuring reliability.

• Reliability:

  • Single Point of Failures (SPOF): Identify and eliminate SPOFs by introducing redundancy (eg: multiple load balancers, database replicas).

  • Failure Handling:

    • Implement exponential back-off strategies for retries, preventing the system from being overwhelmed.

    • Log persistent failures in a DB and retry later using cron jobs or background workers until successful completion.

    • Apply circuit breaker patterns to prevent cascading failures, protecting the system from becoming overloaded.

  • Data Replication & Disaster Recovery (DR):

    • Discuss strategies to handle large-scale failures, such as a cloud service outage (eg: AWS going down).

    • Explore disaster recovery strategies, ensuring high availability through data replication across regions or cloud providers.

• Monitoring and Alerting:

  • Implement logging and metrics collection to monitor system health and performance.

  • Use the ELK stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis.

  • Discuss how to use OpenTelemetry for collecting traces, offering visibility into application performance and failure points.

  • Define alerting mechanisms to notify teams of critical issues in real-time.

This 3-step template (with many sub-steps😄) should help. It will let you tackle any system design interview question without fear!

The goal is not to find a perfect solution. It's to show you can analyze complex problems, make design choices, and communicate your reasoning. 😄

Thanks for reading 🙏🏻!

The details in this article are based on 6+ yrs of practical experience that has come from designing real systems and seeing problems first hand. If you liked the deep dive, do give a ❤️ and subscribe to encourage me to publish more such deep dives.

In this post, we'll dive deeper into:

1. Different types of rate-limiting actions

2. Key rate-limiting algorithms with their pros & cons.

Before we begin, here's a quick rate-limiting cheat sheet 🧷.

Types of rate-limiting actions

When building a rate limiter, it’s crucial to determine whether your use case requires rejecting excess requests outright or handling them with a delayed or lower-quality response. Generally, rate limiters perform one of three actions (Figure 2) on excess requests:

What is Blocking in rate limiting?

Excess requests exceeding the limit are denied access to the resource. It is commonly expressed as an error message such as HTTP status code 429 (Too Many Requests).

Examples:

1. Social media platforms like LinkedIn and Instagram return a 429 error when detecting web scraping attempts to prevent misuse.

2. Too many login attempts on public APIs (gmail login) are rejected to prevent brute-force (DDoS) attacks.

How does Throttling happen in rate-limiting?

At times, responses to requests from certain clients need to be throttled by redirecting them to lower-quality servers. This reduces the strain on high-performance resources while still maintaining service availability.

Examples:

1. Streaming platforms like Hotstar and Netflix reduce stream quality for users who exceed their data cap to manage bandwidth.

2. Cloud storage services like Google Drive slow down upload speeds for users who surpass their bandwidth limits.

What is Shaping in rate limiting?

Shaping involves directing requests from certain clients to a low-priority queue, causing their processing to be intentionally delayed.

Examples:

1. During high traffic days, CDNs put additional users in a wait queue, whose requests are processed only after previous users are done processing

2. In online gaming events, free-tier users may face delays in match-making as paying subscribers are prioritized for faster access.

Types of rate-limiting algorithms

Token Bucket Algorithm

How does Token Bucket work?

1. Tokens are added to a bucket at a constant rate.

2. Each incoming request consumes one token from the bucket.

3. Requests are denied if the bucket is empty and no tokens are available.

Pros & Cons of Token Bucket

✅ Simple to implement and understand.
✅ Accommodates short-term request spikes up to bucket capacity.
❌ Doesn’t ensure a perfectly smooth request.

Leaky Bucket Algorithm

How does Leaky Bucket work?

1. Requests are added to a fixed-size queue and processed at a steady rate.

2. Requests are discarded if the queue is full.

Pros & Cons of Leaky Bucket

✅ Requests are processed steadily, preventing sudden bursts from overwhelming the system.
❌ Slightly more complex to implement compared to Token Bucket.

Fixed Window Counter Algorithm

How does Fixed Window Counter work?

1. Stores the number of requests in the current window and limits them (eg: 100 requests per minute).

2. When a new request arrives, it's either accepted or rejected, depending on whether the current window can accommodate it.

Pros & Cons of Fixed Window Counter

✅ Simple to understand, and clear rate limits for each time window.
❌ Poor handling of boundary bursts; can allow twice the rate of requests at edges (shown in Figure 5).

Sliding Window Log Algorithm

This is the most commonly used rate-limiting algorithm.

How does Sliding Window Log work?

1. Records the timestamp of each incoming request.

2. Upon a new request, expired timestamps from previous windows are cleared.

3. The number of requests in the current window is counted. If it’s below the allowed limit, the request is accepted and logged; otherwise, it’s denied.

Pros & Cons of Sliding Window Log

✅ Precise and smooth rate-limiting with no edge cases.
❌ Memory-intensive for high-volume APIs as it requires storing all timestamps.

Sliding Window Counter Algorithm

How does Sliding Window Counter work?

1. This approach combines Fixed Window Counter and Sliding Window Log algorithms.

2. Instead of recording every request timestamp, only the total number of requests in the previous fixed window is logged.

3. When a new request arrives, the “weighted number of requests” for the current window is calculated. For example (Figure 7):

   1. Assume Window size: 1 min, Window capacity: 4 requests/min, and a new request arrives at t=2:15

   2. The current window (1:15 - 2:15) consists of 75% of the previous window (1:15 - 2:00) and 25% of the current window (2:00 - 2:15).

   3. Instead of tracking all timestamps, only the request count in the previous fixed window is logged.

   4. The “weighted number of requests” is calculated as: requestsInCurrentWindow + (weight * requestsInPreviousWindow) = 2 + (0.75 * 3) = 4.25.

   5. Since adding the new request would bring the total to 4.25 (exceeding the limit of 4), the request is rejected.

Pros & Cons of Sliding Window Counter

✅ More accurate than Fixed Window Counter
✅ More memory-efficient than Sliding Window Log because it only tracks the total number of requests per window, rather than storing individual timestamps for each request.
✅ Smooth out edges between windows.
❌ Slightly complex to implement.

Thanks for reading! I hope this helped clarify the different types of actions and algorithms used in designing a rate limiter. However, two key factors to consider, based on your use case are:

1. Choosing the right rate-limiting algorithm will depend on the specific behavior and needs of your system.

2. The data structure used to store rate limiter data should be chosen with efficiency and scalability in mind.

See ya in the next one 👋🏻!

Rate limiting is like a traffic cop for the internet, controlling how often users can perform actions. It prevents server overloads and adds security by stopping malicious activity, ensuring smooth system performance.

Before moving ahead, here's a rate-limiting cheat sheet 🧷.

If you’ve recently subscribed, these articles will help you get up to speed on system design:

1. System design interview crash course

2. A crash course on rate limiting (Part 2)

3. A crash course on Caching—Part 1, Part 2 & Part 3

4. How to use Redis in system design interviews?

Benefits of Rate Limiting

• Prevent server overload: A rate limiter blocks excessive requests to protect servers from Distributed Denial of Service (DDoS) attacks.

• Prevent resource starvation: It ensures fair access to resources by capping the number of requests users can make, avoiding resource starvation.

• Reduce costs: By controlling traffic, rate limiting helps cut down on unnecessary usage, ultimately lowering operational costs.

Designing Rate Limiters: Key Concepts and Principles

Designing a rate limiter involves grasping these essential concepts:

1. Identifier: Distinguishes individual users or systems, typically using unique identifiers like IP addresses or user IDs.

2. Window: Specifies the period during which requests are counted (e.g., a minute or an hour).

3. Limit: Sets the maximum number of requests allowed within the defined time window.

Figure 2 illustrates this concept with relevant examples:

Ex 1: Twitter limits the number of tweets per user to 300 per 3 hours →
[Identifier: user_id, Window: 3 hours, Limit: 300 tweets]

Ex 2: Users can create 10 accounts per day per IP address →
[Identifier: user_id_ip_address, Window: 1 day, Limit: 10 accounts]

Where should rate limiters be placed? strategic points

The next key question is: where should rate limiters be implemented?

This depends on the specific use case. For example, if rate limiting is needed for database queries, it would be placed at the database level. On the other hand, for blocking excessive traffic on high-traffic days, rate limiting could be applied at the CDN or API gateway.

Broadly, there are six key points in the infrastructure stack where rate limiters can be deployed. Let’s dive into each of them:

Rate limiting at Application Servers

This is a common rate-limiting strategy because many rate-limiting needs are custom and often require integration directly into the application code.

Examples:
1. Social Media: Twitter limits the number of tweets per user to 300 per 3 hours.
2. Live Streaming: Hotstar uses custom rate limiting to prevent abuse of the free tier by restricting unlimited access.

Rate limiting at the CDN layer

💡💡 A Content Delivery Network (CDN) is a network of distributed servers that caches and delivers web content closer to users, speeding up load times and improving performance.

Eg: When you watch a video on YouTube, it uses a CDN to deliver videos from servers near you, reducing buffering and improving streaming quality.

Rate limiting at the CDN level manages traffic at the edge, blocking excessive requests before they reach your main infrastructure.

Examples:
1. E-commerce: Control new user onboarding during high-traffic days by placing extra users in a CDN wait queue.

Rate Limiting at the API Gateway

💡💡 An API gateway is a server at the front of your infrastructure that manages incoming client requests and routes them to the appropriate services within your application.

Eg: In a shopping app, the API gateway handles requests for product information, user accounts, and orders, directing them to the appropriate backend services.

Rate limiting at the API gateway level controls traffic between client and backend services.

Examples:
1. SaaS: Limiting free users to 1000 API req/day, while enterprise users get 100K req/day
2. Social Media: Instagram blocks repeated scraping attempts from same IP or location.

Rate limiting at the Ingress Gateway level

💡💡 An Ingress Gateway manages and routes external traffic into a Kubernetes cluster, handling tasks like load balancing and SSL termination.

It’s advantageous for efficiently directing traffic to different services, such as routing /user requests to one service and /payment requests to another.

Rate limiting at Ingress Gateway controls traffic between API gateway and ingress gateway OR between internal services as shown in the diagram

Examples:
1. Internal dashboards: If both client applications and internal dashboards access the User Details API, rate limiting at the ingress gateway can prioritize requests from client-facing applications while restricting access for internal dashboards.
2. E-commerce: Limits requests between Order Service and Inventory Service to prevent overload.

Managing traffic with rate limiting at the database

Rate limiting between Application servers and Database servers protects the DB from overload by limiting heavy query traffic.

Examples:
1. Limiting the no. of simultaneous DB connections to ensure fair access for all application replicas.

Rate limiting at Egress Gateway for outbound control

💡💡 An Egress Gateway manages and routes outgoing traffic from a Kubernetes cluster, handling tasks like rate limiting and connection monitoring.

It’s advantageous for controlling traffic to external services, such as enforcing security policies for outbound connections and managing bandwidth usage effectively.

Third-party services often enforce their rate limits, causing excess requests to fail. Rate limiting at the Egress Gateway helps prevent this by capping outgoing requests and managing network resources effectively.

Examples:
1. Oauth: Google caps the number of OAuth requests from a third party during high-traffic login periods.

Conclusion

Thanks for reading! In this post, we’ve covered why Rate Limiting is important, its core concepts, and where to place rate limiters in your infrastructure. It’s important to:

1. Understand the scope of the rate-limiting problem to determine the appropriate infrastructure layer for implementation.

2. Communicate rate limits clearly to API users, ideally through response headers, to enable effective retry and back-off strategies.

In the next post, we’ll explore key rate-limiting algorithms and how to choose the right one for your needs, accompanied by a cheat sheet 😉.

Liked this article? Make sure to 💙 click the like button.

Feedback or addition? Make sure to 💬 comment.

Know someone who would find this helpful? Make sure to 🔁 share this post.

Over the years, load balancers have become an essential pillar of system architecture, serving as the front-line controllers that intelligently distribute incoming network traffic across multiple servers.

Similar to an air traffic controller coordinating the safe landing of planes by assigning them to available runways, load balancers regulate data flow within a system, ensuring horizontal scalability and optimal resource utilization.

Before we dive deeper, here's a quick cheat sheet summarizing today’s article 🧷

Let’s understand each component in detail:

Functions of a load balancer

1. Distribute Traffic: Load balancers evenly distribute incoming requests across multiple servers to prevent overloading any single server.

2. Fault Tolerance: They detect server failures and redirect traffic to healthy servers, ensuring continuous service availability.

3. Rate Limiting: Load balancers can control traffic flow by limiting the number of requests sent to servers, preventing overload.

4. Session Persistence: They maintain session data by consistently routing a user’s requests to the same server for the duration of a session.

Types of Load Balancers

1. Layer 4: Operates at the transport layer, distributing traffic based on IP addresses and TCP/UDP ports. Eg: HAProxy, AWS ELB

2. Layer 7: Works at the application layer, routing traffic based on content such as HTTP headers and URLs. Eg: Nginx, Traefik, AWS ALB

3. Cloud-Based: Load balancers hosted in the cloud, offering scalable, on-demand traffic management without hardware overhead. Eg: Cloudflare, AWS Route 53

4. Hardware: Physical load balancers deployed on-premises, providing high-performance traffic distribution with dedicated infrastructure. Eg: F5, A10, Citrix ADC

Types of load balancing algorithms

1. Round Robin

How does it work?
Requests are distributed sequentially across servers, looping back to the first server after reaching the last one.

In Figure 2, requests 1, 2, 3 and 4 go to servers 1, 2, 3 and 1 respectively.

Pros & Cons
✅ Simple to implement and effective for servers with equal capacity.
❌ Doesn't account for server load or processing power.

Example: Default algorithm used by Kubernetes Load Balancer for routing traffic to service pods.

2. Sticky Round Robin

How does it work?
The first client request goes to a random server, and all future requests stick to that server unless it's unavailable.

In Figure 3, requests 1 & 2 go to server 1 while requests 3 & 4 go to server 2

Pros & Cons
✅ Ensures session persistence, ideal for applications requiring user state retention.
❌ Not suitable for applications with short-lived sessions.
❌ If a server goes down, session data can be lost unless state is stored centrally.

Example: Redis Cluster uses this to direct a client’s requests to the same Redis node, ensuring consistent access to cached data.

3. Weighted Round Robin

How does it work?
Servers are assigned weights based on their capacity, with higher-weight servers handling more requests.

In Figure 4, Out of 4 requests, 3 requests (0.8 * 4 = 3.2) go to server 1 and 1 request (0.1 * 4 = 0.4) goes to server 2.

Pros & Cons
✅ Balances load based on server capacity.
✅ More efficient for systems with servers of varying strength.
❌ Doesn’t factor in real-time load (active connections).

Example: AWS Elastic Load Balancer uses this to direct more traffic to EC2 instances with higher capacity and computing power.

4. IP Hash

How does it work?
Calculates a hash value from the client’s IP address and uses it to determine the server to route the request.

In Figure 5, requests 1 & 2 are from same IP in China, hence go to server 1 where hash(IP)=0, while requests 3 & 4 from same IP in India go to server 3 where hash(IP)=2

Pros & Cons
✅ Ensures consistent routing of requests from the same client to the same server.
✅ Good for session persistence without needing cookies.
❌ Can lead to uneven load distribution if many clients share similar IP addresses.
❌ IP changes can break persistence.

Example: E-commerce platforms like Amazon use this to ensure that the same server handles all cart requests for a user, such as adding, viewing, or removing items from the cart.

5. Least connections

How does it work?
Incoming requests are directed to the server with the fewest active connections.

In Figure 6, all requests go to server 2 because it has the least no of client connections (50)

Pros & Cons
✅ Balances real-time traffic load efficiently.
✅ Prevents servers from becoming overloaded.
❌ Doesn’t account for server capacity or request complexity.

Example: PgBouncer distributes queries across PostgreSQL instances based on least connections, preventing overload on any single instance.

6. Least Response Time

How does it work?
Incoming requests are routed to the server with the fastest response time.

In Figure 7, all requests go to server 1 because it has the least current response time (1ms)

Pros & Cons
✅ Reduces latency for time-sensitive applications.
❌ May lead to instability if servers experience temporary performance spikes.

Example: Booking platforms (eg: Expedia) use this at the API/ingress gateway level to route travel booking requests to the fastest available server.

So far so good. I will publish separate deep-dive diagrams on load balancers in upcoming posts!

Also, the next post will be about something exciting and a bit more challenging 😉! Stay tuned and don’t forget to subscribe to support my work 🙏🏻.

Let’s talk credit cards today. In every credit card transaction, the merchant is charged a Merchant Discount Fee (MDR) of 2-3%, which is divided among the issuer, acquirer, POS provider, and card networks. Figure 1 illustrates how this MDR is split among the various parties involved.

Previous issue talks about the BTS actions after you swipe your credit card at a merchant’s terminal. Today, we’ll dive into the revenue breakdown, revealing how each party involved in a credit card transaction earns from the fees.

1. Acquirer Markup

   1. Typically ranges between 0.5% to 0.75% of the transaction amount.

   2. This is split between the Acquirer and the POS provider.

   3. In cases where the POS provider is also the acquirer (eg: Paytm), the same party retains the full markup.

2. Interchange Fee

   1. Usually between 1% to 1.25% of the transaction value.

   2. Paid by the Acquirer to the Card Issuer.

   3. The Issuer earns the largest share since it assumes the risk by paying the merchant, even if the cardholder defaults.

3. Network and Assessment Fee

   1. Typically around 0.2% of the transaction amount.

   2. Both the Issuer and the Acquirer pay this fee to the Card Network (eg: VISA)

4. Interest and Late Fees

   1. Earned by the Issuer from cardholders who fail to repay their credit card bills on time.

   2. This forms a significant part of the issuer’s overall revenue from card transactions.

It’s a win-win for everyone, right? No wonder why credit card is called “the most profitable product in banks”! See you in the next one ✌🏻.

Today, let's dive into the mechanics of what happens behind the scenes when someone swipes their Credit Card at a merchant's Point of Sale (POS) terminal. The transaction process can be broken down into two key flows:

1. Authorization Flow – the initial step that determines whether the transaction is approved or declined in real-time.

2. Settlement Flow – the End of Day (EOD) process where the funds are transferred from the cardholder’s bank to the merchant’s account.

   

Authorization Flow

1. After conducting thorough due diligence, the Issuer Bank issues credit cards to customers. This is a complex process in itself, deserving a separate post!

2. Transaction is initiated when a customer swipes their card at a merchant’s POS terminal — often from fintechs like Paytm or Pinelabs.

3. The POS machine has its own OS, which securely transmits tokenized card details, the txn amount, and encrypted merchant info to the Acquiring Bank.

4. The Acquiring Bank processes card payments for businesses and forwards the transaction to the relevant card network (or card scheme). In this example, HDFC Bank issued the credit card, and VISA serves as the card network.

5. VISA performs Anti-Fraud (AF) and Anti-Money Laundering (AML) checks and forwards secure transactions to the Issuer Bank which is responsible for:

   1. Checking card rules (active/inactive, card limits, domestic/international) and performing AML/AF checks.

   2. Debiting amount from credit card and doing accounting (Checkout Accounting in payment systems) of amount/revenue/tax.

   3. Notify the customer via SMS about the transaction's success or failure and relay the response back to VISA.

6. VISA sends the response to the Acquirer, which then relays it to the POS terminal. The POS displays a success or failure message and generates a transaction receipt.

Settlement flow

1. At the end of the day, the POS machine’s software sends all transaction data to the Acquiring Bank for reconciliation. This can be done via bulk APIs or by uploading files to a file server. Check out Reconciliation system to learn about designing a recon system.

2. After reconciling transactions, the Acquiring Bank sends the batch file to VISA, which then forwards it to the Issuer Bank.

3. The Issuer Bank performs the final reconciliation and follows these steps:

   1. Successful Reconciliation: The actual funds are transferred from the Issuer Bank to the Acquiring Bank.

   2. Failed Reconciliation: The amount is refunded to the customer’s credit card.

4. Finally, the funds are settled into the merchant’s bank account after deducting all applicable fees.
   

Important points to consider

1. Customer’s Credit Card Debiting: The customer’s credit card is debited immediately at the time of the transaction.

2. Merchant’s Account Crediting: The merchant’s bank account is credited at EOD.

3. Revenue Share and Fees: Different parties involved receive various revenue shares (fixed or percentage), which are deducted at different stages of the process. (For details, see our upcoming post on revenue sharing.)

4. Refund Scenarios: The amount must be refunded to the customer’s credit card in the following cases:

   • Pending Transactions: If a transaction is pending due to network issues during authorization, any debited amount will be refunded.

   • Reconciliation Failure: If reconciliation for a transaction fails, a refund must be processed.

5. AML and Anti-Fraud Checks: AML and AF checks are specialized and must be conducted at each step: by the acquirer, card network, and issuer.

By analyzing hundreds of factors, including content quality, user behavior, and contextual signals, Google ensures that users find exactly what they're looking for, whether a simple fact or a comprehensive research topic.

An in-depth analysis of how Google's complex ranking system works was recently released after 100s of hours of research. Presenting a high-level view:

Originally deep dive by Mario Fischer from SearchEngineLand - https://searchengineland.com/how-google-search-ranking-works-445141.

I will try to break this down into important services from a system design pov and key strategies while setting up SEO for Google search in a later post.

Reconciliation in payment systems is the process of comparing internal systems' records with those in external systems at regular intervals to figure out and fix discrepancies. It is vital because:

1. It uncovers issues in accounting systems that often go unnoticed until this critical step, ensuring errors are caught and corrected.

2. Effective reconciliation identifies anomalies and potential fraud by comparing internal records with external data.

3. Successful reconciliations enable actual fund transfers between payment systems, banks, and third-party platforms.

The process broadly involves the following steps:

1. Data ingestion and transformation: Workflow orchestrators like Apache Airflow or big data processors like Apache Spark are generally used to fetch, transform, and normalize data from external sources through jobs running at regular intervals.

   1. Transaction statements from external services are received in CSV, excel or via APIs.

   2. All data is converted into a common format through a transformation job.

   3. Data is normalized by aligning timestamps, converting currencies, and eliminating duplicates for accurate comparison with internal records.

2. Reconciliation Service

   1. Data collection pipeline sends normalized data to reconciliation service either through streaming platforms like Kafka or API call.

   2. Transactions are fetched from internal accounting system (read Accounting in payment systems 👆🏻👆🏻) for comparison.

   3. The service matches external and internal transactions, identifying discrepancies like missing entries or amount mismatches (Figure 1).

3. Reporting: Results can be viewed either in recon dashboards built inhouse/using Retool or visualization platforms such as Metabase.
   

Ensuring eventual consistency in Airbnb's payment system was crucial as they transitioned to a Service Oriented Architecture (SOA). This migration presented numerous challenges, including:

1. How to handle race conditions if a user clicks “Book” twice?

2. How to ensure that refunds are processed only once?

3. What if the response was lost or the client timed out?

4. How would clients recover if it failed to consume the response?
   

Idempotency, as discussed in Avoiding double payments 🤯 is how Airbnb successfully navigated these obstacles and achieved eventual consistency in a distributed payments system. At a high level, Figure 1 illustrates a simple example scenario with duplicate requests and ideal idempotent behavior. No matter how many charge requests are made, the guest will always be charged at most once.

The Problem Statement 🧐

1. They needed a generic yet configurable idempotency solution to be used across various Payments services.

2. The solution should be able to classify errors as retryable and non-retryable and handle them accordingly.

3. The solution had to be an ultra-low latency to handle the payment scale.

The Solution ✅

Airbnb engineering teams built “Orpheus”, a general-purpose idempotency library, to use across multiple payments services. Fun fact: Orpheus is the legendary Greek mythological hero who was able to orchestrate and charm all living things 🦄.

Key design principles used in Orpheus (Figure 2):

1. Building a separate, stand-alone idempotency service wouldn’t be sufficient because they needed ultra-low latency. Choosing to build a library was the solution.

2. Idempotency design

   1. The client has to pass a unique idempotency key into the framework, representing a single idempotent request.

   2. The client should ensure that mutation of request payload between retry attempts is not allowed.

3. Low-level design

   1. Database transactions are combined in different parts of the codebase to ensure atomicity. Airbnb used Java lambdas to achieve this.

   2. At code level, every API request is separated into three distinct phases: Pre-RPC, RPC, and Post-RPC. This ensures a clear distinction between DB and network/API calls.

4. Exception Handling

   1. Orpheus was designed to classify responses as “retryable” or “non-retryable”.
      

Idempotency design

The client must own several key responsibilities when interacting with a service that uses an idempotency library like Orpheus:

1. Pass in a unique idempotency key for every new request. It’s important to reuse the same idempotency key for retries.

2. Persist these idempotency keys to the database before calling the service (to later use for retries).

3. Properly consume successful responses and subsequently unassign (or nullify) idempotency keys.

4. Ensure that mutation of the request payload between retry attempts is not allowed.

5. Carefully devise and configure auto-retry strategies based on business needs (using exponential backoff or randomized wait times (“jitter”) to avoid the thundering herd problem).
   

Low-level design

They wanted to avoid mixing network communication with database calls because it can result in bad things like rapid connection pool exhaustion and performance degradation. Hence, the following segregation was implemented:

1. Pre-RPC (Remote Procedure Call): Details of the payment request are recorded in the database.

2. RPC: Network call to external service is made and the response is received. This is the place to do one or more idempotent computations or RPCs (for eg: query service for the status of a transaction first if it’s a retry attempt).

3. Post-RPC: Details of the response from the external service are recorded in the database, including its success and whether a bad request is retryable or not.

To maintain data integrity, two simple ground rules were set:

1. NO service interaction over networks in Pre and Post-RPC phases.

2. NO database interactions in the RPC phases.

Exception Handling

All validation errors, such as invalid input and states (for eg: you can’t refund a refund) are categorized as non-retryable (4XX HTTP statuses). All subsequent retries of the same request are expected to fail in the same manner. This categorization is handled by creating a custom, generic exception class. Broadly, Figure 3 shows how error categorization happens.

Additional considerations