Clik - A Self-Deployable URL Shortener

2026-03-28T00:00:00+00:00

Random Click</figcaption>

Building a URL shortener is a classic system design interview question. I built one to find out how many of those whiteboard answers actually survive contact with real code.

The Stack</h2>
No abstractions over abstractions. Here is what it runs on:

Java 21</li>
Spring Boot 3.4.2 handles HTTP so I can focus on what matters underneath</li>
MySQL 8.0 (the database choice is load-bearing, explained below)</li>
Spring Data JPA + Hibernate with Envers for write auditing</li>
Spring Security with JWT-based auth for URL ownership</li>
Maven for builds, nothing clever</li> </ul>
Two tables. That is the entire data model.
`short_urls -> internal_id (BIGINT PK), long_url, short_code, user_id (FK nullable), click_count, created_at users -> id (PK), email, password_hash, first_name, last_name, ...</code></pre> The redirect path is a single indexed lookup on short_code</code>. That query runs on every click, so that is the one that matters.`
The ID Problem</h2> I did not use AUTO_INCREMENT</code>. I did not use UUID</code>. Here is why neither worked for me. A UUID is a 128-bit number, not a string. The mistake most people make is storing it as VARCHAR(36)</code> (the human-readable hyphenated form) instead of BINARY(16)</code>. If you store a UUID as BINARY(16)</code>, comparisons are fast and the index stays compact. But the real problem is still randomness. UUID v4 is randomly generated, so every insert lands at a random position in the B-tree, not the end. That causes node splits, which scatter data across disk pages. At millions of rows a randomly-keyed table fragments badly, and reads that miss the buffer pool pay disk seek latency. UUIDv7 is worth knowing about. It is time-ordered with a 48-bit millisecond timestamp prefix, standardized in RFC 9562, and eliminates the fragmentation problem entirely. If I were starting fresh and wanted a 128-bit ID space without building my own generator, UUIDv7 is the honest answer. MySQL and Postgres support it natively as of recent versions. I still went with Snowflake-style IDs because the shard ID is baked into every key. You can look at any internal_id</code> and immediately know which machine generated it, which is useful for debugging and operational routing. UUIDv7 does not give you that without a separate column. AUTO_INCREMENT</code> is sequential and avoids fragmentation. But it couples your ID space to a single database node. The moment you shard, you have a coordination problem. So I use a Snowflake-inspired 64-bit integer that is sequential, distributed, and carries operational metadata inside the number itself:
`+--------------------------------------------------------------+ | Timestamp (41 bits) | ShardID (10 bits) | Seq (12 bits) | +--------------------------------------------------------------+ | 63 | 22 | 12 | <- bit positions +--------------------------------------------------------------+</code></pre>`

41 bits of milliseconds since a custom epoch (2023-01-01 00:00:00 UTC). About 69.7 years of headroom before the timestamp field overflows.</li>
10 bits of shard ID. 1024 unique shard identifiers embedded in every key. Reading bits 12 through 21 tells you which machine generated the ID.</li>
12 bits of sequence. 4096 unique IDs per millisecond per shard before rollover.</li> </ul>
The actual implementation:</h2>
private synchronized long generateInternalId(int shardId) { long timestamp = System.currentTimeMillis() - epoch; int sequenceNumber = sequence.getAndIncrement() & 0xFFF; return (timestamp << 22) | (shardId << 12) | sequenceNumber; }</code></pre> Two things worth understanding here. First, timestamp << 22</code>. Shifting left by 22 puts the timestamp in the most significant bits of the 64-bit integer. Since timestamp dominates the value numerically, IDs generated later are always larger numbers than IDs generated earlier regardless of shard or sequence. A quick example: timestamp = 100 -> 100 << 22 = 419,430,400 timestamp = 101 -> 101 << 22 = 423,624,704</code></pre> The second ID is always bigger. That is why every new insert goes to the rightmost leaf of the B-tree and index fragmentation is essentially zero. Second, & 0xFFF</code>. The sequence counter keeps incrementing forever, but we only have 12 bits reserved for it. & 0xFFF</code> is just a cheap way to wrap it. 0xFFF</code> in binary is twelve consecutive 1s. ANDing any number with it throws away everything above bit 11: 4095 & 0xFFF = 4095 // still in range 4096 & 0xFFF = 0 // wraps back cleanly</code></pre> No if-statement, no modulo, just a single bitwise AND. The method is synchronized</code>, so one thread enters at a time on this service instance. The AtomicLong</code> on the sequence counter is belt-and-suspenders given the lock already serializes access. Once the 64-bit integer is generated it gets Base62-encoded into the short code: private static final String ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; public static String encode(long id) { StringBuilder sb = new StringBuilder(); while (id > 0) { sb.append(ALPHABET.charAt((int)(id % 62))); id /= 62; } return sb.reverse().toString(); }</code></pre> A typical Snowflake ID is on the order of 10¹⁸. That encodes to 7 or 8 Base62 characters. Short enough to be a URL slug, and unique enough to never collide across shards. The end result is a query like this resolving in O(log n): SELECT long_url FROM short_urls WHERE internal_id = 286435872092454913;</code></pre> Because internal_id</code> is a sequential BIGINT primary key, MySQL’s InnoDB clustered index keeps rows physically ordered on disk. The B-tree traversal is fast, page cache hit rates are high, and there is no fragmentation from random insertions. The Hard Parts</h2> Sequence overflow, known and unhandled. The current code does not handle generating more than 4,096 IDs within the same millisecond on the same shard. The & 0xFFF</code> mask wraps the counter silently back to zero, which means two calls can produce the same ID if the rate exceeds 4M/s on one shard. The fix is straightforward: if (sequenceNumber == 0) { while (System.currentTimeMillis() <= lastTimestamp) { // spin until the clock moves forward } }</code></pre> I have documented this in the source. At current volume it does not trigger. I am not pretending it is solved. Clock drift, also unhandled. If the system clock moves backward due to an NTP correction or VM live migration, System.currentTimeMillis() - epoch</code> shrinks. The timestamp bits in the new ID become smaller than in the previous one, monotonicity breaks, and you can get duplicate IDs. The standard fix is to track lastTimestamp</code> and throw or wait if the clock regresses. Not implemented. Documented. Duplicate long URLs. Two users submitting the same long URL should get the same short code. Before generating a new ID, the service checks for an existing long_url</code> record. The lookup is on an indexed column so it is cheap, but it is a read-before-write that adds a round trip on every URL creation. A unique constraint on long_url</code> would make this airtight. Click tracking on the redirect path. Every redirect needs to increment click_count</code>. Doing that synchronously stalls the redirect on an UPDATE with a row-level lock. Instead, click increments go into a thread pool and get processed asynchronously. The 302 returns immediately and the counter catches up in the background. The tradeoff is honest: if the JVM dies between the redirect and the write, that click is gone. A click counter is an approximation, not an accounting ledger. I accepted that. Index strategy on redirects. short_code</code> has a unique secondary index and internal_id</code> is the clustered primary key. A redirect traverses the short_code</code> index to locate the row, then does a bookmark lookup into the clustered index to fetch long_url</code>. Two B-tree traversals is the floor for this query shape without a covering index or an in-memory cache in front. If this were serving serious traffic, a Redis layer on short_code -> long_url</code> would collapse the redirect to a single memory read. Not added yet. Running It</h2> git clone https://github.com/siddharth1729/clik.git cd clik ./mvnw spring-boot:run</code></pre> Update src/main/resources/profiles/dev.application.properties</code> with your MySQL credentials. The schema bootstraps itself via Hibernate DDL on first boot. API surface: POST /api/shorten shorten a URL (anonymous) POST /api/shorten + Authorization Bearer shorten a URL (authenticated) GET /{shortCode} redirect to long URL POST /api/users/register create an account</code></pre>Why I Built It</h2> Every explanation of this system at the design level gestures at Base62 encoding and sharding and then moves on. I wanted to know what actually breaks when you implement those things. The ID generator is the crux. Get the bit layout wrong and your index degrades. Miss the sequence overflow case and you silently produce duplicate keys under load. Handle click tracking synchronously and your redirect latency spikes under write contention. None of that shows up on a whiteboard. All of it matters in the code. Source: github.com/siddharth1729/clik</a> Connect with me on LinkedIn</a> Follow on Twitter</a> </blockquote>

Legion: An Ultra-Fast, Multi-Threaded Logging Library

2025-06-14T00:00:00+00:00

By Siddharth Sabron, June 14, 2025

Mcleodganj</figcaption> Image Link</a>
This article presents a comprehensive design document for a high-performance Java logging library, based on deep research and proven architectural patterns.
The development of an ultra-fast</code>, high-throughput</code>, low-dependency</code> logging library for Java applications, particularly for Spring environments, necessitates a rigorous architectural approach. The core value proposition of such a library lies in its ability to minimize performance overhead on application threads while ensuring comprehensive log capture, request tracing, and visualization-friendly JSON output. This objective is achieved through fundamental principles such as asynchronous processing, meticulous zero-allocation strategies, and intelligent concurrency management. The design emphasizes leveraging core Java constructs where possible, while pragmatically acknowledging the irreplaceable role of battle-tested components like the LMAX Disruptor for achieving truly “ultra-fast” performance characteristics. The success of this endeavor hinges on a deep understanding of JVM optimization, concurrent programming paradigms, and rigorous benchmarking.
1. Core Design & Foundation: High Level Architecture</h3> The pursuit of an “ultra-fast” logging library begins with foundational architectural principles that dictate how log events are handled from their inception to their persistence. These principles are designed to ensure minimal impact on the application’s critical path and maximize logging throughput. Asynchronous Processing: Decoupling for Minimal Caller Impact</h4> A cornerstone for achieving “ultra-fast” performance is the immediate decoupling of logging events from the caller thread. This means that the application thread, upon initiating a log request, should relinquish control as quickly as possible, allowing the core business logic to proceed without delay. This asynchronous approach is widely recognized as a technique to significantly improve application logging performance by offloading all I/O operations to separate threads. For instance, Log4j 2’s asynchronous loggers are specifically designed to return control to the application with minimal delay. The underlying reason for this design choice is rooted in the nature of I/O operations. Synchronous I/O, whether to disk or network, inherently blocks the executing thread until the operation completes. If logging were synchronous, the application’s responsiveness and overall throughput would be directly constrained by the speed of log persistence. By immediately handing off the log event to a dedicated queue and returning control, the application thread is shielded from these I/O latencies, thereby maintaining high responsiveness and throughput for its primary functions. This fundamental decoupling is a direct driver for achieving minimal overhead on the main application threads. Batching: Amortizing I/O Costs for High Throughput</h4> To further optimize I/O operations, log events should be processed and written in batches. The cost associated with initiating an I/O operation (e.g., a system call for a file write or a network packet transmission) often has a fixed overhead, regardless of the amount of data being transferred (up to a certain point). By accumulating multiple log events into a single, larger batch before writing, this fixed overhead is amortized over numerous log entries. This significantly reduces the number of I/O calls, leading to a substantial increase in overall logging throughput. For example, asynchronous appenders in Log4j2 are designed to flush to disk at the end of a batch, which is more efficient than immediate flushing for each log event. While batching enhances throughput by reducing the frequency of costly I/O operations, it introduces a slight increase in latency for individual log events. Each event must wait in the buffer until a batch is full or a timeout occurs. This represents a classic trade-off between maximizing throughput and minimizing individual log event latency, a balance that requires careful tuning. Zero-Allocation (Garbage-Free where possible): Minimizing GC Pressure in the Critical Path</h4> A critical aspect of “ultra-fast” performance and consistent low latency is the minimization of new object allocations during the logging critical path. Excessive object creation leads to increased pressure on the Java Garbage Collector (GC), resulting in more frequent and potentially longer GC pauses, which can manifest as latency spikes. Libraries like ZeroLog, though a .NET library, exemplify this principle by aiming for a “complete zero-allocation manner” after initialization to prevent GC triggers. Log4j2 also implements a “garbage-free mode” by reusing objects and buffers to reduce GC pressure. It is important to understand the practical implications of “zero-allocation.” While the aspiration is to eliminate all new object allocations after initialization, achieving this absolutely in a feature-rich logging library can be exceptionally challenging. Log4j2’s “garbage-free mode” demonstrates this nuance, noting that certain scenarios, such as logging messages exceeding a specific character limit, logging numerous parameters, or using certain lambda expressions, may still result in temporary object allocations. This suggests that “garbage-free” in a real-world context often translates to “minimal garbage” or “garbage-free under ideal, frequently occurring conditions.” The design should therefore prioritize object reuse in the most performance-critical paths, such as LogEvent creation and population, while acknowledging and documenting scenarios where some allocation might be unavoidable, or where pooling mechanisms can mitigate their impact. Thread Isolation: Dedicated Logging Workers for Application Stability</h4> To prevent logging operations from adversely affecting the main application’s responsiveness or stability, the threads responsible for processing and writing logs must be isolated from the application’s core threads. This separation ensures that even if logging operations encounter bottlenecks (e.g., slow disk writes, network congestion), the main application threads remain unblocked and continue executing business logic without interruption. Asynchronous logging inherently promotes this thread isolation. The ExecutorService</code> is a fundamental Java utility for managing thread execution and task scheduling, facilitating the creation of dedicated thread pools for logging activities. This approach to thread pool isolation prevents resource exhaustion within the logging subsystem from cascading and affecting other critical services within the application. Similarly, ThreadLocal</code> variables contribute to thread isolation by providing each thread with its own independent storage, preventing data interference between concurrent operations. By containing logging-related performance fluctuations or failures within their own isolated thread pool, the overall system gains improved stability and resilience.
1.2. Core Components Overview</h3> The architecture of this ultra-fast logging library is composed of several interconnected components, each designed for optimal performance and specific responsibilities:

LogEvent (Data Carrier): This Plain Old Java Object (POJO) encapsulates all information pertinent to a single log entry. Its fields include timestampMillis</code>, LogLevel</code>, message</code>, mdcContext</code> (for Mapped Diagnostic Context data), customData</code> (for structured log data), throwable</code> (for exception details), threadName</code>, loggerName</code>, serviceName</code>, and hostName</code>. A critical design consideration for LogEvent is its reusability through pooling, which necessitates a reset()</code> method to clear its state for subsequent use, thereby contributing to garbage-free operation.</li>
LogBuffer (Concurrent Queue): This is the high-performance conduit between producer (application) threads and consumer (logging worker) threads. It is designed as a highly concurrent, non-blocking queue for LogEvent</code> instances. The LMAX Disruptor pattern is the primary candidate for achieving extreme throughput, given its proven capabilities in high-performance inter-thread communication. Alternatively, a carefully tuned ArrayBlockingQueue</code> could serve as a high-performance option if the Disruptor’s external dependency is deemed too significant.</li>
LogProcessor (Worker Thread Pool): This component comprises a pool of dedicated threads. Their primary responsibilities include continuously polling or consuming LogEvent</code> instances from the LogBuffer</code>, formatting these events into the specified JSON schema, and writing the formatted events to designated sinks (e.g., file, console, network). The LogProcessor</code> also manages batching of writes to optimize I/O operations.</li>
Logger Facade: This is the public-facing API that application code interacts with (e.g., MyLogger.info()</code>, MyLogger.error()</code>). It is designed to be exceptionally lightweight, ensuring minimal overhead on the caller thread. Its primary function is to quickly capture log parameters, populate a LogEvent</code>, and offer it to the LogBuffer</code>.</li>
Custom MDC (Mapped Diagnostic Context): A custom, high-performance implementation of a thread-local mechanism. It is used to store request-specific contextual data, such as a unique requestId</code>, which is essential for comprehensive request tracing across distributed systems.</li>
Request Correlation Filter/Interceptor: This is a Spring-specific component responsible for generating and managing the requestId</code> within the custom MDC. It ensures that a unique identifier is associated with each incoming request and propagated throughout its lifecycle, enabling end-to-end traceability of log events.</li> </ul>
1.3. Dependency Strategy: The “Less Libraries” Imperative</h3>
A core tenet of this project is to minimize external dependencies, thereby reducing the library’s footprint, potential for dependency conflicts, and overall complexity.

Core Java: The design heavily relies on fundamental Java APIs, particularly java.util.concurrent</code> for robust concurrency primitives. This approach leverages the highly optimized and battle-tested components within the Java Development Kit (JDK).</li>
JSON Serialization: For structured logging, JSON serialization is often unavoidable. While a strict “zero external dependencies” approach might suggest hand-rolling JSON serialization, this introduces significant complexity and risk in handling edge cases, escaping, and performance. Therefore, a fast, lightweight JSON library like Jackson (specifically its core modules) or Gson is considered. Jackson is generally preferred for its superior performance in high-throughput and multi-threaded scenarios. Limiting to “core modules only” helps maintain the low-dependency objective.</li>
LMAX Disruptor (Conditional): The LMAX Disruptor library is acknowledged as a near necessity if the “ultra-fast” claim requires performance that significantly surpasses traditional queue-based asynchronous logging, akin to Log4j2’s asynchronous capabilities. Building a true Disruptor from scratch is a non-trivial undertaking. However, if careful tuning of a java.util.concurrent.ArrayBlockingQueue</code> proves “fast enough” for the target application’s requirements, prioritizing this built-in Java component would align more closely with the “less libraries” objective.</li>
No other logging frameworks: This library is explicitly designed to be the logging framework, precluding the inclusion of Log4j2, Logback, SLF4J, or any other existing logging solutions as dependencies.</li> </ul> 2. Low-Level Design & Implementation Details: Engineering for Speed</h3>
2.1. LogEvent Class: Object Pooling and reset() for Garbage-Free Operation</h4>
The LogEvent</code> class serves as the data carrier for each log entry. Its design is critical for achieving garbage-free logging.
Fields: The LogEvent</code> should encapsulate comprehensive data: long timestampMillis</code>, LogLevel level</code>, String message</code>, Map<String, String> mdcContext</code>, Map<String, Object> customData</code>, Throwable throwable</code>, String threadName</code>, String loggerName</code>, String serviceName</code>, and String hostName</code>. These fields provide the necessary context for rich, visualization-friendly JSON output.</li> Pooling Mechanism: To minimize new object allocations and reduce GC pressure, a pooling mechanism for LogEvent</code> instances is essential. A ThreadLocal</code> pool of LogEvent</code> objects is a viable approach. When a log method is invoked, an LogEvent</code> instance is retrieved from this thread-local pool, populated with the current log data, and then offered to the LogBuffer</code>. Once the LogEvent</code> has been processed by the LogProcessor</code> (i.e., formatted and written to a sink), it should be returned to the pool for reuse. This recycling process is fundamental to the garbage-free strategy. A crucial aspect of object pooling is the reset()</code> method. Before a LogEvent</code> is reused from the pool, its previous state must be completely cleared to prevent data contamination from prior log entries. This explicit state management is a trade-off: while it reduces GC overhead, it introduces the complexity of manual memory-like management. The benefits of reduced GC pressure, especially for short-lived, frequently created objects, generally outweigh these complexities in performance-critical applications. However, the implementation must be meticulous to avoid pitfalls such as reference leaks (where objects are not returned to the pool) or premature recycling (where objects are returned too early while still referenced), which can lead to hard-to-debug memory issues.</li> </ul> 2.2. Custom MDC Implementation (MyMDC): Efficient ThreadLocal Management and Map Reuse</h4> The Mapped Diagnostic Context (MDC) provides a mechanism to inject contextual information into log entries, such as a requestId</code> for tracing. The custom MyMDC</code> implementation will leverage ThreadLocal</code> for this purpose. Mechanism: MyMDC</code> will internally use a ThreadLocal<Map<String, String>></code> to store key-value pairs of contextual data. Each thread will have its own isolated Map</code> instance, ensuring thread safety without explicit synchronization.</li> Methods: Essential methods for MyMDC</code> include put(key, value)</code>, get(key)</code>, remove(key)</code>, clear()</code>, and getCopyOfContextMap()</code>.</li> Optimization: A primary focus for MyMDC</code> is minimizing Map re-creations. Instead of creating a new Map for each ThreadLocal</code> access or for each LogEvent</code>, the design should aim to reuse the Map</code> instance associated with the ThreadLocal</code>. This can involve clearing and repopulating the existing map. When capturing the MDC context for a LogEvent</code>, a choice must be made: either provide a read-only “view” of the ThreadLocal</code> map or create a defensive copy. Providing a view would eliminate the allocation of a new Map</code> object, directly contributing to garbage-free operation. However, this approach introduces a risk: if the LogEvent</code> is processed asynchronously by a different thread, and the original ThreadLocal</code> map is modified by the application thread before the LogProcessor</code> consumes the event, the log data could become inconsistent. For scenarios demanding absolute immutability of the log event’s context, a defensive copy is safer, albeit at the cost of object allocation. For internal processing where the LogEvent</code> is consumed very quickly and its lifecycle is tightly controlled, a view might be considered for extreme performance, but with rigorous validation. Proper cleanup of ThreadLocal</code> variables is paramount, especially in environments utilizing thread pools (common in Spring applications). Failure to call MyMDC.remove()</code> or MyMDC.clear()</code> in a finally</code> block after a request completes can lead to memory leaks and, critically, data pollution where stale requestIds</code> or other context from a previous request are inadvertently carried over to a new request processed by the same reused thread.</li> </ul> 2.3. LogBuffer Deep Dive: Achieving Extreme Throughput</h4> The LogBuffer</code> is the central component for inter-thread communication, directly influencing the “ultra-fast” performance claim. Option A (Extreme - LMAX Disruptor) For achieving the highest possible throughput and lowest latency, especially in high-volume logging scenarios, integrating the LMAX Disruptor library is often considered a necessity. The Disruptor is not merely a queue; it is a concurrency pattern designed for “mechanical sympathy” with modern hardware, minimizing cache contention and false sharing. Its core components include: Ring Buffer: A pre-allocated, fixed-size circular array that stores LogEvent</code> instances, avoiding runtime memory allocations and contributing to garbage-free operation.</li> Sequencer: Manages the allocation of slots in the Ring Buffer and ensures producers do not overrun consumers. It comes in single-producer and multi-producer implementations.</li> Sequence: A counter used by producers and consumers to track their progress, optimized to prevent false sharing.</li> Sequence Barrier: Coordinates dependencies between consumers and determines event availability.</li> Wait Strategy: Defines how consumers wait for new events, offering trade-offs between CPU utilization and latency (e.g., BusySpinWaitStrategy</code> for lowest latency but high CPU, BlockingWaitStrategy</code> for lower CPU but higher latency).</li> </ul> To integrate, a LogEvent</code> factory would be defined for the Disruptor to pre-allocate LogEvent</code> objects. EventTranslator</code> instances would be used by producer threads to publish LogEvent</code> data to the Ring Buffer, and EventHandler</code> implementations would be used by LogProcessor</code> threads to consume and process these events. Log4j2’s asynchronous loggers leverage the Disruptor, demonstrating its capability to achieve significantly higher throughput and lower latency compared to ArrayBlockingQueue</code>-based solutions. However, the Disruptor’s performance is highly context-dependent. While it excels in single-producer, single-consumer scenarios, its advantages in multi-producer environments, especially without explicit bulk operations, can be less pronounced and even worse than a well-optimized ArrayBlockingQueue</code>. The Disruptor also requires a substantial upfront memory allocation for its ring buffer (e.g., 80-140MB for Log4j2), which is a trade-off for its garbage-free design. This memory footprint must be considered for microservices or memory-constrained environments. Option B (High Performance - Custom ArrayBlockingQueue</code>) As an alternative to the Disruptor, a java.util.concurrent.ArrayBlockingQueue<LogEvent></code> can be used. This is a fixed-capacity, bounded blocking queue. Producer Behavior: Application threads would use queue.offer()</code> to non-blockingly submit LogEvent</code> instances. For “ultra-fast” caller performance, a discard policy is often preferred when the buffer is full, preventing the application thread from blocking. Alternatively, queue.put()</code> could be used with a timeout for critical logs, allowing some blocking but preventing indefinite waits.</li> Consumer Behavior: LogProcessor</code> threads would use queue.poll()</code> (non-blocking) or queue.take()</code> (blocking) to retrieve events. The queue.drainTo()</code> method is particularly useful for fetching multiple events at once, facilitating efficient batch processing by the LogProcessor</code>.</li> </ul> While ArrayBlockingQueue</code> is simpler to implement and avoids the external Disruptor dependency, it typically does not match the peak performance of a finely tuned Disruptor for extreme throughput scenarios. Log4j2’s asynchronous appenders, which use ArrayBlockingQueue</code>, are demonstrably slower than its Disruptor-based asynchronous loggers. Capacity & Full Policies Determining the optimal buffer size is crucial. For the Disruptor, a power-of-two capacity (e.g., 2^N) is typically recommended. For ArrayBlockingQueue</code>, a reasonable fixed size must be chosen. When the LogBuffer</code> is full, the library’s behavior must be explicitly defined: Discard: The fastest option for the caller, where log events are simply dropped. This is often chosen for non-critical logs to ensure the application remains “ultra-fast”.</li> Block: The caller thread waits until space becomes available in the buffer. This ensures data integrity but directly impacts application performance if the logging subsystem cannot keep up.</li> Log a warning: An internal log message can be generated to indicate that the buffer is full and events are being discarded or blocked.</li> </ul> A hybrid approach might be implemented, where low-priority logs are discarded when the buffer is full, while high-priority logs might block or be buffered to a secondary, smaller critical buffer. The choice between Disruptor and ArrayBlockingQueue</code> is not a simple matter of “faster is better.” While Disruptor offers unparalleled low-latency and high-throughput capabilities in specific configurations, its complexity and memory footprint are significant. For multi-producer scenarios, a well-tuned ArrayBlockingQueue</code> with bulk operations can sometimes rival or even surpass Disruptor’s performance. This highlights that “ultra-fast” is highly dependent on the specific workload patterns and the careful tuning of the chosen concurrency mechanism. 2.4. LogProcessor (Worker Threads): Thread Management, Batching Logic, and Graceful Shutdown</h4> The LogProcessor</code> is responsible for consuming LogEvents</code> from the LogBuffer</code> and writing them to the configured sinks. Thread Management: An ExecutorService</code>, such as a FixedThreadPool</code>, is a suitable choice for managing the pool of worker threads. This abstracts the complexities of thread lifecycle management.</li> Loop: Each worker thread in the LogProcessor</code> continuously enters a loop, attempting to poll or consume LogEvent</code> instances from the LogBuffer</code>.</li> Batching: To amortize I/O costs, worker threads collect LogEvents</code> into batches. This can be achieved by collecting a predefined number of events (batchSize</code>) or by waiting for a specified batchTimeout</code> before performing the actual I/O operation. Fine-tuning these parameters is crucial to balance latency and throughput.</li> Error Handling: Robust error handling is essential. If writing to a sink fails (e.g., disk full, network error), the system must react gracefully. Strategies include retry for transient errors, discard for persistent errors, and internal logging for diagnostics without causing a recursive loop.</li> Shutdown Hook: A graceful shutdown mechanism is vital to ensure that all remaining LogEvents</code> in the LogBuffer</code> are processed and flushed before the application terminates. This can be implemented by registering a shutdown hook or by using Spring’s DisposableBean</code> interface or @PreDestroy</code> methods.</li> </ul> 2.5. LogFormatter: Crafting Visualization-Friendly JSON</h4> The LogFormatter</code> transforms a LogEvent</code> into a structured JSON string. Schema Definition The JSON schema must be rich and consistent for external tools. { "timestamp": "ISO_DATE_TIME", "level": "INFO|WARN|ERROR|DEBUG|TRACE", "serviceName": "String", "hostName": "String", "requestId": "UUID_STRING", "threadName": "String", "loggerName": "String", "message": "String", "context": { "userId": "123", "sessionId": "abc" }, "data": { "productId": "PROD-001", "operationTimeMs": 50, "eventIdentifier": "PRODUCT_RETRIEVAL_SUCCESS" }, "exception": { "type": "String", "message": "String", "stackTrace": "String" } }</code></pre> Optimization: StringBuilder</code> and ByteArrayOutputStream</code> Reuse To minimize object allocations, reusing mutable buffers is crucial. StringBuilder</code> Reuse: A StringBuilder</code> instance should be reused per thread. The most efficient way to clear it is setLength(0)</code>.</li> ByteArrayOutputStream</code> Reuse: If converting to bytes, a ByteArrayOutputStream</code> can also be reused by resetting its internal buffer.</li> </ul> Jackson vs. Gson Jackson: Widely regarded as the fastest JSON library in Java, particularly for high-throughput, multi-threaded scenarios. Its performance and memory efficiency make it ideal for logging.</li> Gson: Simpler API but generally slower due to its reliance on reflection.</li> </ul> Given the “ultra-fast” requirement, Jackson is the preferred choice, using only its core modules to maintain the “less libraries” goal. 2.6. LogSink Interface and Implementations: Efficient Data Persistence</h4> The LogSink</code> interface defines the contract for writing log entries. interface LogSink { void write(List<String> batchedLogEntries); void flush(); void close(); }</code></pre> FileLogSink Buffered I/O: BufferedWriter</code> is a standard, efficient choice.</li> Log Rotation: Simple size- or time-based rotation should be implemented.</li> Advanced I/O (java.nio</code>): FileChannel</code> and ByteBuffer</code>: Excellent for high-performance writes with durability control via force()</code>.</li> MappedByteBuffer</code>: Can be “ultra-fast” for frequent writes to large files, but is complex, OS-dependent, and not ideal for the small, frequent appends typical of logging.</li> AsynchronousFileChannel</code>: Adds complexity (callbacks, Future</code>s) that may offer marginal benefit in a dedicated worker thread model.</li> </ul> </li> </ul> NetworkLogSink (e.g., HTTP/Kafka) Minimal Libraries (HttpURLConnection</code>, Sockets): Adheres to “zero dependencies” but requires a massive effort to build a reliable, high-performance client.</li> Kafka Integration (KafkaProducer</code>): The industry-standard client. While an external dependency, it is essential for reliable, high-performance Kafka integration. It’s highly optimized with features for batching, compression, and durability.</li> Retry Mechanisms and Backpressure: For any network sink, robust retry mechanisms (e.g., exponential backoff with jitter) and backpressure handling are crucial for resilience against network failures.</li> </ul> 2.7. MyLogger Facade: The Ultra-Lightweight Entry Point</h4> The MyLogger</code> facade is the public API. It must be exceptionally fast. Caching: MyLogger</code> instances should be cached in a ConcurrentHashMap</code>.</li> Methods: info()</code>, warn()</code>, error()</code>, debug()</code>, trace()</code>, with overloads for structured data.</li> Core Logic: Level Check: The first and most critical step is an efficient if (logger.isInfoEnabled())</code> check to prevent any work for disabled log levels.</li> LogEvent Reuse: Retrieve a LogEvent</code> from a pool.</li> Population: Populate the event with data.</li> Buffer Offer: Non-blockingly offer the event to the LogBuffer</code>.</li> </ol> </li> </ul> 2.8. Spring Integration: RequestIdFilter</code> and Lifecycle Hooks</h4> RequestIdFilter</code>: A Spring WebFilter</code> should: Check for an X-Request-ID</code> header; if not present, generate a new UUID.</li> Store the ID in MyMDC</code>.</li> Crucially, clear the ID from MyMDC</code> in a finally</code> block to prevent thread pollution.</li> </ol> </li> Asynchronous Context Propagation: ThreadLocal</code>-based MDC context does not automatically propagate to new threads (e.g., in a CompletableFuture</code>). Explicit context transfer is required, typically by wrapping the Runnable</code> or Callable</code>.</li> Startup/Shutdown: Use Spring’s ApplicationRunner</code> for initialization and DisposableBean</code> or @PreDestroy</code> for graceful shutdown to flush all logs.</li> </ul> 3. Deep Research & Advanced Optimization: Pushing the Performance Envelope</h3> 3.1. Comprehensive Garbage-Free Strategies</h4> LogEvent Pooling: Consider a global, concurrent object pool if LogEvent</code> objects are returned by consumer threads, which can simplify cleanup and avoid ThreadLocal</code> leak issues.</li> MDC Map Reuse: Re-evaluate the “view vs. copy” trade-off. A read-only view offers zero allocation but carries risks if the underlying map changes. A defensive copy is safer but incurs an allocation. The choice depends on the internal data flow and immutability guarantees.</li> </ul> 3.2. Concurrency Model Deep Dive</h4> LMAX Disruptor Mastery: A deep understanding of the Disruptor’s “mechanical sympathy” and its various WaitStrategy</code> options is key to tuning the trade-off between CPU usage and latency.</li> Contention Analysis: Use profilers (JVisualVM, YourKit, Async-profiler) to identify real-world bottlenecks like lock contention, GC pauses, and I/O constraints.</li> Backpressure: A robust library must gracefully handle overload. The LogBuffer</code> full policy (discard, block, hybrid) is the primary mechanism for this.</li> </ul> 3.3. I/O Optimization: Advanced NIO Nuances</h4> FileChannel</code> and ByteBuffer</code>: Reusing ByteBuffer</code> instances, especially direct buffers, is critical to avoid the “ridiculously slow” performance of frequent allocations.</li> MappedByteBuffer</code>: Re-emphasize that for typical sequential logging, MappedByteBuffer</code> is often not the optimal choice and can be slower and more complex than alternatives.</li> AsynchronousFileChannel</code>: Its benefits may be marginal for a logging library that already offloads I/O to a dedicated worker pool.</li> Batching Strategies: Fine-tuning batchSize</code> and batchTimeout</code> through empirical testing is crucial for balancing latency and throughput.</li> </ul> 3.4. Logging Levels & Filtering</h4> Facade-Level Checks: The if (logger.is...Enabled())</code> guard clause is the single most important performance optimization on the application thread.</li> Configurable Levels: Allow granular, dynamic configuration of log levels per logger name, stored in a ConcurrentHashMap</code> for fast lookups.</li> </ul> 3.5. Visualization & Schema Refinement</h4> Event-Driven Logging: Encourage using an eventIdentifier</code> field to transform logs into a stream of actionable business or system events for better dashboarding.</li> Metric Extraction: Ensure numerical fields (operationTimeMs</code>) are logged as numbers, not strings, to facilitate direct aggregation and analysis.</li> Field Naming Consistency: Adhere to a consistent naming convention (e.g., camelCase) for all JSON fields.</li> </ul> 3.6. Request Unique ID Lifecycle (Traceability)</h4> Generation: A RequestIdFilter</code> should check for an existing X-Request-ID</code> header before generating a new UUID to maintain trace continuity.</li> Propagation: The requestId</code> must be propagated across all communication layers: outgoing HTTP requests, message queues (Kafka, RabbitMQ), and RPC frameworks.</li> Clearing: The most critical step is clearing the requestId</code> from MyMDC</code> in a finally</code> block to prevent thread pollution and incorrect log correlation.</li> </ul> 4. Rigorous Testing & Reliability: Ensuring Production Readiness</h3> 4.1. Benchmarking: Measuring “Ultra-Fast” Performance</h4> JMH (Java Microbenchmark Harness): Use JMH with proper @Fork</code>, @Warmup</code>, and @Measurement</code> configurations. Use @Threads</code> to simulate concurrent load.</li> Key Metrics: Focus on tail latencies (p99, p99.9), not just average latency, as they reveal performance spikes. Also monitor throughput, CPU, and memory.</li> Realistic Simulations: Benchmark against a known baseline like Log4j2’s AsyncLogger under realistic, bursty load conditions to validate performance claims.</li> </ul> 4.2. Failure Modes & Graceful Degradation: Resilience in Adversity</h4> A production-ready library must degrade gracefully without crashing the application. Disk Full Scenarios: Detect disk full conditions, stop writing, log a warning to a fallback sink (like the console), and start discarding messages.</li> Network Sink Unavailability: Use robust retry policies with exponential backoff and jitter. If retries fail, buffer locally (with a bounded buffer) or discard logs to prevent blocking.</li> Backpressure: Use the LogBuffer</code>’s full policy (discard, block, hybrid) to ensure application threads are not affected by a slow logging subsystem.</li> </ul> 4.3. Configuration: Custom Parser vs. Standard Properties/YAML</h4> For a low-dependency library, avoid heavy configuration frameworks. Custom Configuration Parser: Leverage java.util.Properties</code> for simple key-value configs or a minimal, zero-dependency JSON parser for more structured configurations. This provides full control without introducing bloat.</li> </ul> 4.4. Testability: Unit Testing Performance-Critical Paths</h4> JMH for Micro-benchmarking: Use JMH to unit test performance-critical code paths, not just for overall system benchmarking.</li> Concurrency Testing: Rigorously test the LogBuffer</code> and LogProcessor</code> for thread safety, deadlocks, and race conditions under high contention.</li> </ul> Conclusion & Future Outlook</h3> The development of an “ultra-fast,” low-dependency, multi-threaded Java logging library is a challenging but achievable endeavor. The core principles of asynchronous processing, rigorous zero-allocation strategies, intelligent concurrency management, and dedicated thread isolation are fundamental to achieving minimal overhead and high throughput. The success of such a library is measured by its empirical performance and resilience. This necessitates continuous, rigorous benchmarking using tools like JMH, focusing on tail latencies. Furthermore, the library must be designed with robust failure modes and graceful degradation strategies, ensuring that logging issues never compromise the main application’s stability. Future enhancements could include: Expansion of LogSink</code> types (e.g., cloud-specific sinks).</li> More sophisticated filtering and routing capabilities.</li> Deeper integration with distributed tracing systems like OpenTelemetry.</li> Dynamic configuration reloading without application restarts.</li> </ul> By adhering to these principles, a truly “ultra-fast,” high-throughput, low-dependency logging library can be realized, providing invaluable observability for performance-critical Java applications. Works Cited</h3> Asynchronous loggers :: Apache Log4j - Apache Logging Services</a></li> Unlocking Java Asynchronous Programming: Key Concepts, Tools, and Benefits</a></li> Log4j 2 Lock-free Asynchronous Loggers for Low-Latency Logging</a></li> ZeroLog 2.2.0 - NuGet</a></li> Abc-Arbitrage/ZeroLog: A high-performance, zero … - GitHub</a></li> Garbage-free logging :: Apache Log4j</a></li> Log4j – Garbage-free Steady State Logging - Apache Log4j 2</a></li> Log4j 2: The Complete Guide to Modern Java Logging - Dash0</a></li> Thread Pool Isolation | Apache Dubbo</a></li> Java Thread-Local Variables: Explained - DZone</a></li> Java ThreadLocal - Java and Spring Trends</a></li> An Introduction to ThreadLocal in Java | Baeldung</a></li> Jackson vs Gson: Edge Cases in JSON Parsing for Java Apps - DZone</a></li> Efficient JSON Parsing in Java: Jackson vs. Gson vs. JSON-B - Java …</a></li> Why did log4j2 used LMAX Disruptor in Async logger instead of any other built in non blocking data structure? - Stack Overflow</a></li> A Well Known But Forgotten Trick: Object Pooling - High Scalability -</a></li> Object Pooling in Java - memory management - Stack Overflow</a></li> Logback MDC: An Essential Tool For Effective Logging In Java</a></li> Log4j 2 Thread Context - Apache Logging Services</a></li> Java ThreadLocal: Unveiling the Power of Thread-Specific Data - Devzery</a></li> How to clean up ThreadLocals - Stack Overflow</a></li> Adding user info to log entries in a multi-user app using Mapped Diagnostic Context</a></li> Correlation in logging - DEV Community</a></li> Java • ThreadLocal Best Practices - KapreSoft</a></li> LMAX Disruptor User Guide</a></li> Re: Disruptor vs ArrayBlockingQueue tests. Disruptor is better only …</a></li> Create an alternative async logger implementation using JCTools · Issue #2220 · apache/logging-log4j2 - GitHub</a></li> Analysis of Performance Practices in Java Log and Principle Interpretation - Alibaba Cloud</a></li> What does the term “backpressure” mean in Rxjava? - Stack Overflow</a></li> DiscardingAsyncQueueFullPolicy (Apache Log4j Core 2.24.3 API)</a></li> Debugging Batch Applications - Documentation | Spring Cloud Data Flow</a></li> Reuse StringBuilder for Efficiency | Baeldung</a></li> XML Performance Optimization - Speed and Memory Efficiency - Web Reference</a></li> Java File writing I/O performance - TMSVR</a></li> Reading/Writing to/from Files using FileChannel and ByteBuffer in Java - Java Code Geeks</a></li> Boost Java performance with memory-mapped files | Transloadit</a></li> asynchronous - MappedByteBuffer throwing a java.nio …</a></li> Java ByteBuffer performance issue - Stack Overflow</a></li> ByteBuffer.allocateDirect ridiculously slow| JBoss.org Content Archive (Read Only)</a></li> NIO Performance Improvement compared to traditional IO in Java - Stack Overflow</a></li> IO and NIO performance difference and example - Stack Overflow</a></li> Java NIO AsynchronousFileChannel - Jenkov.com</a></li> Troubleshooting Java NIO Non-Blocking IO Performance Pitfalls - Java Tech Blog</a></li> Simple Java Rest Client posting JSON to a server - GitHub Gist</a></li> POST request send JSON data Java HttpUrlConnection - Stack Overflow</a></li> How can I use java to send HTTP post request of a JSON array of strings? - Stack Overflow</a></li> Enable logging to TCP inputs in your Java project - Splunk Dev</a></li> A Guide to Using Raw Sockets - Open Source For You</a></li> Basic Distributed Counter using Java Sockets - Codemia</a></li> mirsamantajbakhsh/RawSocket: A simple code for raw socket in Java using JNetPCAP. - GitHub</a></li> Kafka performance: 7 critical best practices - NetApp Instaclustr</a></li> How to Improve Kafka Performance: A Comprehensive Guide</a></li> Failure Handling Mechanisms in Microservices and Their Importance - DZone</a></li> How Do I Handle Failures in Microservices? - JavaDZone</a></li> LoggingRetryPolicy (DataStax Java Driver for Apache Cassandra - Binary distribution 2.1.10 API)</a></li> Java Client | GridGain Documentation</a></li> Timeouts, retries and backoff with jitter - AWS</a></li> Exponential Backoff in RxJava - Stack Overflow</a></li> Mastering Backpressure in Project Reactor: Common Pitfalls | Java Tech Blog</a></li> Java Logging: Troubleshooting Tips and Best Practices - Last9</a></li> Implementing Correlation ids in Spring Boot (for Distributed Tracing in SOA/Microservices)</a></li> ThreadPool with CompletableFuture (need MDC propagation) : r/SpringBoot - Reddit</a></li> Java Microbenchmark Harness (JMH) - GeeksforGeeks</a></li> ExecutorService with backpressure - java - Stack Overflow</a></li> Java • Logback Mapped Diagnostic Context (MDC) in Action | KapreSoft</a></li> OpenElements/java-logger-benchmark: A benchmark for … - GitHub</a></li> Benchmark - Swift Package Index</a></li> Performance Analysis using JMH | The Backend Guy</a></li> jmh: Run benchmark concurrently - java - Stack Overflow</a></li> java - JMH - How to correctly benchmark Thread Pools? - Stack Overflow</a></li> Is your reported p99 wrong? - Random Notes and Cheat Sheets - Kirk Pepperdine</a></li> Design for graceful degradation | Cloud Architecture Center | Google …</a></li> A guide to graceful degradation in web development - LogRocket Blog</a></li> Logging Best Practices to Reduce Noise and Improve Insights - Last9</a></li> ralfstx/minimal-json: A fast and small JSON parser and … - GitHub</a></li> mmastrac/nanojson: A tiny, compliant JSON parser and … - GitHub</a></li> </ol>

64 Bit Distributed ID Generator

2025-03-01T00:00:00+00:00

Landor ❤️</figcaption>

Image Link</a>

Introduction</h3>

Unique ID</code> generation is a crucial part of any application, because it is used to represents a single unique row in the table. Furthermore, this ID</code> is often used as the primary-key</code> of the table and heavily used in searching a specific row uniquely.

Example Query:

select</span>  *</span> from</span>  short_urls </span>where</span> internal_id </span>=</span> 286435872092454913</span>;</span></span></code></pre>
You might wonder why we are emphasizing the ID so much ?. Couldn’t we just generate a unique value using something like Math.random() + epoch</code> or some random_string_of_some_size + epoch</code> While that could</em> produce unique values, but it would be a poor choice for several reasons, especially at scale:</p>


Meaningful Data:</strong> A simple timestamp (or a random number) doesn’t convey any useful information other</em> than the time of creation.  Our Snowflake-like approach embeds a timestamp and</em> a shard ID, which can be useful for operational purposes (e.g., identifying which shard a URL was created on).</p>
</li>

Database Performance (Crucially):</strong>  The data type of the primary key has a major</em> impact on database performance, particularly for search operations.  Purely numeric IDs (integers or long integers) are vastly superior to strings for primary keys and indexed columns or both as well(A Indexed Primary).</p>
</li>
</ol>
Why Not Random Strings?</h3>
Using random strings (e.g., generated by Math.random() + epoch</code> or random string of some size) as primary keys is generally a bad</em> idea for performance reasons:</p>


Slower Comparisons:</strong> String comparisons are inherently slower than numeric comparisons. Databases can compare integers much more efficiently at the hardware level.</p>
</li>

Larger Index Size:</strong> String primary keys create larger database indexes. Indexes are data structures (usually B-trees or B+trees) that speed up lookups. A larger index means:</p>

More disk I/O</code> to read the index.</li>
More memory consumption to keep parts of the index</code> in cache</code>.</li>
Slower query performance.</li>
</ul>
</li>

Index Fragmentation:</strong> Inserting random strings into a B-tree index leads to fragmentation</em>.  A fragmented index has its data scattered across the disk. This requires more disk seeks (which are slow) to retrieve data. Numeric, sequential</em> IDs minimize fragmentation because new entries are usually appended to the end</em> of the index.</p>
</li>

Poor Locality of Reference:</strong> Random strings have poor locality of reference</em>. Successive ID lookups are likely to access very different parts of the index and the table data, leading to more disk seeks and cache misses.</p>
</li>
</ul>
Why Numeric, Sequential IDs are Better:</h3>
Numeric IDs, especially when generated in a roughly sequential manner (like our Snowflake-like approach), offer substantial advantages:</p>

Fast Comparisons:</strong> Databases can compare numbers extremely quickly.</li>
Smaller Indexes:</strong> Numeric IDs (e.g., 64-bit integers) take up less space than long, random strings, resulting in smaller, faster indexes.</li>
Reduced Fragmentation:</strong> Sequential IDs cause new rows to be inserted at the end of the B-tree index, minimizing fragmentation and improving write performance.</li>
Improved Locality of Reference:</strong> Sequential IDs tend to be clustered together in the index and on disk. This improves cache utilization and reduces disk seeks, speeding up both reads and writes.</li>
</ul>
Database Internals: B-Trees (Simplified)</h3>
Most relational databases (MySQL, PostgreSQL, etc.) use B-trees (or variants like B+trees) to implement indexes. Here’s a simplified explanation:</p>


Structure:</strong> A B-tree is a self-balancing tree data structure.  Data is kept sorted, and searches, insertions, and deletions can be performed in logarithmic time (O(log n)).</p>
</li>

Nodes:</strong> Each node in the B-tree contains multiple keys</em> (in our case, the internal_id</code> values) and pointers</em> to child nodes.</p>
</li>

Searching:</strong> To find a specific ID, the database:</p>

Starts at the root node.</li>
Compares the target ID with the keys in the node.</li>
Follows the appropriate pointer to a child node.</li>
Repeats this process until it finds the ID or determines it doesn’t exist.  Because the keys are sorted within each node</em>, the database can very efficiently narrow down the search.</li>
</ol>
</li>

Insertions:</strong></p>

Sequential IDs:</strong> New keys are usually added to the rightmost</em> leaf node of the B-tree. This is very efficient.</li>
Random Strings:</strong> The new key could need to be inserted anywhere</em> in the B-tree. This might require splitting nodes (if a node is full), leading to multiple write operations and fragmentation.</li>
</ul>
</li>

Time Complexity:</strong> The Time complexity for searching using the internal_id</code> which is primary key, will be O(log n)</strong>. Where ‘n’ is the number of rows in your table. And, If we had chosen a string-based ID, and even if that string column were indexed, the search complexity within the index itself</em> would still involve string comparisons, which are slower than integer comparisons. The complexity would still be logarithmic (O(log n)) in terms of the number of index nodes accessed, but</em> each node comparison would take longer. And, if your are quering on non-indexed string column then it will be O(n*m)</strong>, Where ‘n’ is the number of rows, and ‘m’ is the average length of your string ID.</p>
</li>
</ul>
Method: generateInternalId(int shardId)</code></h3>
The generateInternalId</code> method creates unique IDs without relying on a central coordinating service (like a database auto-increment).  It achieves this by combining a timestamp, a shard ID, and a sequence number into a single 64-bit integer.</p>
ID Structure</h3>
The generated ID is a 64-bit long integer, structured as follows:</p>
+-------------------------------------------------------------------------------------+</span></span>
|    Timestamp (41 bits)   |   ShardID(10) |   Sequence(12)   |      extra bit        |</span></span>
+--------------------------+---------------+------------------+-----------------------+</span></span>
|            63            |      22       |        12        |  0  (Bit positions)   |</span></span>
+-------------------------------------------------------------------------------------+</span></span></code></pre>


Timestamp (41 bits):</strong>  Represents the number of milliseconds since a custom epoch</em>.  The epoch in this implementation is 2023-01-01 00:00:00 UTC</code> (defined by the epoch</code> variable).</p>

Calculation:</strong> System.currentTimeMillis() - epoch</code></li>
Capacity:</strong> 41 bits allow for 2^{41</sup> milliseconds, which is approximately 69.7 years:

2^{41</sup> milliseconds = 2,199,023,255,552 milliseconds</li>}
2,199,023,255,552 ms / 1000 ms/s / 60 s/min / 60 min/hr / 24 hr/day / 365 days/year ≈ 69.7 years</li>
</ul>
</li>
Lifespan:</strong>  The system can generate unique IDs for about 69.7 years. After this, the timestamp would wrap around, potentially causing ID collisions.  To prevent this, you’d need a strategy like re-sharding with a new epoch or implementing collision detection (not done in this simplified version).</li>
</ul>
</li>

Shard ID (10 bits):</strong> Identifies the specific database shard or application instance that generated the ID.  This is essential</em> for distributed systems.  10 bits allow for 2^{10</sup> = 1024 unique shard IDs.</p>
</li>}

Sequence Number (12 bits):</strong> A counter that increments for each ID generated within the same millisecond</em> on the same shard</em>. 12 bits allow for 2^{12</sup> = 4096 unique sequence numbers per millisecond per shard. This handles situations where many IDs are generated very quickly.</p>}

Atomic Increment:</strong>  sequence.getAndIncrement()</code> ensures the sequence number is incremented atomically, preventing race conditions in a multi-threaded environment.</li>
Bitwise AND (& 0xFFF</code>):</strong> Keeps the sequence number within the 12-bit range (0-4095). 0xFFF</code> is the hexadecimal representation of 4095 (binary 111111111111</code>). The bitwise AND effectively masks all but the last 12 bits.</li>
</ul>
</li>
</ul>
Bitwise Operations</h3>
The code uses bitwise operations to combine the timestamp, shard ID, and sequence number:</p>


Left Shift (<<</code>)</strong>:</p>


timestamp << 22</code>:  Shifts the timestamp 22 bits to the left.  This creates 22 empty bits (filled with zeros) on the right-hand side.  These bits are reserved for the shard ID (10 bits) and sequence number (12 bits).  Left shifting by n</em> bits is equivalent to multiplying by 2^{n</em></sup>.</p>}
Timestamp (binary, simplified): 1001...00100</span></span>
Timestamp << 22:                1001...00100000000000000000000000000</span></span></code></pre></li>

shardId << 12</code>: Shifts the shard ID 12 bits to the left, creating 12 empty bits on the right for the sequence number (multiplying by 2^{12</sup>).</p>}
Shard ID (binary): 0000000101</span></span>
Shard ID << 12:    0000000101000000000000</span></span></code></pre></li>
</ul>
</li>

Bitwise OR (|</code>)</strong>: Combines the shifted timestamp, shifted shard ID, and sequence number.  Because the left shifts created non-overlapping sections, the bitwise OR effectively concatenates the three values.</p>
</li>
</ul>
+------------------------------------------------------------------------------------+</span></span>
|       (timestamp << 22)               |       (shardId << 12)    | sequenceNumber  |</span></span>
+---------------------------------------+--------------------------+-----------------+</span></span>
|  1001...0010000000000000000000000000  |  0000000101000000000000  | 000001111011    |</span></span>
+------------------------------------------------------------------------------------+ </span></span></code></pre>
Resulting 64-bit ID: 1001…001000000000101000001111011</p>
Visual Representation</h3>
+--------------------------------------------------------------------------------------+</span></span>
|                                         ID (64 bits)                                 |</span></span>
+--------------------------------------------------------------------------------------+</span></span>
|      Timestamp (41 bits)        |      Shard ID (10 bits)     |    Seq (12 bits)     |</span></span>
+---------------------------------+-----------------------------+----------------------+</span></span>
| 10010011...00100000000000       |          0000000101         |      000001111011    |</span></span>
+---------------------------------+-----------------------------+----------------------+</span></span></code></pre>Example</h3>
Let’s say:</p>

Current time: 2024-03-01 10:30:15.500 UTC</code></li>
Epoch: 2023-01-01 00:00:00.000 UTC</code></li>
Shard ID: 5</code></li>
Sequence: 123</code></li>
</ul>


Timestamp:</strong> 39,486,615,500 milliseconds (difference between current time and epoch).</p>
</li>

Binary Representations (simplified):</strong></p>

Timestamp: 10010011111000000110000110101100000100100</code></li>
Shard ID: 0000000101</code></li>
Sequence Number: 000001111011</code></li>
</ul>
</li>

Bitwise Operations:</strong> The code performs the shifts and OR as described above, resulting in the final 64-bit ID.</p>
</li>
</ol>
Code</h3>
Here is the java</code> implementation of the above explanation:</p>
    private</span> final</span> long</span> epoch</span> =</span> 1672531200000L</span>;</span></span>
    private</span> final</span> AtomicInteger</span> sequence</span> =</span> new</span> AtomicInteger</span>(</span>0</span>)</span>;</span></span>
</span>
    private</span> synchronized</span> long</span> generateInternalId</span>(</span>int</span> shardId</span>)</span> {</span></span>
        long</span> timestamp</span> =</span> System</span>.</span>currentTimeMillis</span>(</span>)</span> -</span> epoch</span>;</span></span>
        int</span> sequenceNumber</span> =</span> sequence</span>.</span>getAndIncrement</span>(</span>)</span> &</span> 0xFFF</span>;</span></span>
        return</span> (</span>timestamp </span><<</span> 22</span>)</span> |</span> (</span>shardId </span><<</span> 12</span>)</span> |</span> sequenceNumber</span>;</span></span>
    }</span></span></code></pre>
Also i have used this in one of my personal project Url Shortener</a></p>


shardId</code>:</strong>  The ID of the database shard (an integer between 0 and 1023, inclusive).</p>
</li>

Return Type</code>:</strong> The generated 64-bit unique internal ID.</p>
</li>
</ul>
Important Considerations</h3>

Sequence Overflow:</strong> This code does not</em> handle the case where more than 4096 IDs are generated within the same millisecond on the same shard.  In a production system, you must</em> address this (e.g., by waiting for the next millisecond, throwing an exception, or using a larger sequence number space).</li>
Clock Drift/Jump:</strong>  This code does not</em> handle system clock changes (clock drift or NTP adjustments).  Clock changes can cause the timestamp to go backward, potentially leading to duplicate IDs.  A production system must</em> handle this (e.g., by tracking the last generated timestamp and waiting or throwing an exception if the clock moves backward).</li>
Shard ID Assignment:</strong> You need a robust mechanism to assign shard IDs and to determine the shard ID for a given operation. Consistent hashing is a common approach.</li>
</ul>
This detailed explanation should cover all aspects of the generateInternalId</code> method and the Snowflake-like ID generation strategy.  Remember to use the more robust version of this method (with clock drift and sequence overflow handling) in a production environment.</p>


If you like the post please connect me on LinkedIn</a>
And Twitter</a></p>
</blockquote>}

A Simple Github Report Generator

2025-02-25T00:00:00+00:00

Introduction</h2>
As working in an early stage startup we(software enginners) have to work on multiple repositories and we baerly maintain log of our work in short what all we have worked on and what all we have done.
I have faced this problem and have been searching for a tool which can help me to generate a report of my github commits activity and how much i have worked on it.
So i have decided to build a tool which can help me to generate a report of my github commits activity and how much i have worked on it.

Features</h2>
This tool offers a range of features to help you analyze and visualize your GitHub commits activity. These include:

Comprehensive Commit Activity Report: Get a detailed report of your GitHub commits activity.</li>
Date Range Filtering: Filter your commits activity by a specific date range to understand your progress over time.</li>
Commit Message Insights: Gain insights into your commits activity based on commit messages to understand the context behind your work.</li> </ul>
The tool generates two types of reports:

report.pdf: A professionally formatted PDF report that includes:

Developer details</li>
Commit summaries</li>
Repository statistics</li>
Detailed commit logs</li>
Color-coded tables for easy analysis</li> </ul> </li>

report.txt: A plain text version of the same report, ideal for quick reference or integration with other tools. </li> </ul>
A preview of the report:</h3>

Here is the link to the repository: Github Report Generator</a>

If you like the post please connect me on LinkedIn</a> Follow me on Twitter</a> </blockquote>

Understanding Isolation in Database

2023-03-31T00:00:00+00:00

Rajiv Chowk Metro Stataion</figcaption>

Image Link</a>

Isolation in database refers to the ability of a database system to allow multiple transactions to access the same data without interfering with each other. Isolation ensures that each transaction sees a consistent view of the data, regardless of the concurrent activities of other transactions.

Types of Read Phenomena

There are three main types of read phenomena

Dirty Reads</a></li>
Non-Repeatable Reads</a></li>
Phantom Reads</a></li> </ul>
Dirty Reads</h2>
Dirty reads occur when a transaction reads data that has been modified by another transaction but not yet committed. This means that the transaction is reading data that is still in an intermediate or “dirty” state, and it may be rolled back later.
To illustrate this, let’s consider a simple example. Suppose we have a table named “employees” with the following columns: “id”, “name”, “salary”, and “department”. Transaction A executes the following query:
SELECT salary FROM employees WHERE id = 1;</code></pre> Meanwhile, Transaction B updates the salary of employee with id 1 using the following query:
UPDATE employees SET salary = 60000 WHERE id = 1;</code></pre> However, Transaction B does not commit the update yet. If we are using Read Uncommitted isolation level, Transaction A can read the new value of the salary column even though Transaction B has not committed the update yet. To demonstrate this, let’s run the following SQL queries:
-- Start Transaction A BEGIN TRANSACTION; SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT salary FROM employees WHERE id = 1; -- End Transaction A -- Start Transaction B BEGIN TRANSACTION; UPDATE employees SET salary = 60000 WHERE id = 1; -- End Transaction B</code></pre> If we are using Read Uncommitted isolation level, the read operation in Transaction A may return the new value of the salary column even though Transaction B has not committed the update yet. To prevent dirty reads, we can use at least the Read Committed isolation level, which ensures that a transaction reads only committed data. In SQL, we can set the isolation level for a transaction using the SET TRANSACTION statement. To set the isolation level to Read Committed, we would use the following query:
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;</code></pre> In conclusion, dirty reads can occur in database systems when a transaction reads data that has been modified by another transaction but not yet committed. To prevent dirty reads, we can use at least the Read Committed isolation level, which ensures that a transaction reads only committed data.
Non-Repeatable Reads</h2>
Non-repeatable reads occur when a transaction reads a row twice but gets different values in each read. This can happen when another transaction modifies the row in between the two reads.
To illustrate this, let’s consider a simple example. Suppose we have a table named “employees” with the following columns: “id”, “name”, “salary”, and “department”. Transaction A executes the following query:
SELECT salary FROM employees WHERE id = 1;</code></pre> Meanwhile, Transaction B updates the salary of employee with id 1 using the following query:
UPDATE employees SET salary = 60000 WHERE id = 1;</code></pre> If we are using Read Committed isolation level, Transaction A will not see the new value of the salary column until Transaction B commits. However, if we use Repeatable Read isolation level, Transaction A may see the new value of the salary column on its subsequent reads. To demonstrate this, let’s run the following SQL queries: -- Start Transaction A BEGIN TRANSACTION; SELECT salary FROM employees WHERE id = 1; -- End Transaction A -- Start Transaction B BEGIN TRANSACTION; UPDATE employees SET salary = 60000 WHERE id = 1; COMMIT; -- End Transaction B -- Start Transaction A again BEGIN TRANSACTION; SELECT salary FROM employees WHERE id = 1; -- End Transaction A again</code></pre> If we are using Repeatable Read isolation level, the second read operation in Transaction A may return a different value for the salary column compared to the first read operation, since the row was updated by Transaction B. To prevent non-repeatable reads, we can use Serializable isolation level, which ensures that a transaction sees a consistent snapshot of the database throughout its execution, even if other transactions modify the same data. In SQL, we can set the isolation level for a transaction using the SET TRANSACTION statement. To set the isolation level to Serializable, we would use the following query: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;</code></pre> In conclusion, non-repeatable reads can occur in database systems when a transaction reads a row twice but gets different values in each read, due to another transaction modifying the row in between. To prevent non-repeatable reads, we can use Serializable isolation level, which ensures that a transaction sees a consistent snapshot of the database throughout its execution. Phantom Reads</h2> Phantom reads refer to the situation where a transaction reads a set of rows that satisfy a certain condition, but when the same transaction repeats the same read operation later, additional rows appear that were not visible before. This happens when another transaction commits a new row that satisfies the same condition. To understand phantom reads better, let’s look at an example: Suppose we have a table named “employees” with the following columns: “id”, “name”, “salary”, and “department”. Let’s say Transaction A executes the following query: SELECT * FROM employees WHERE department = 'Sales';</code></pre> Meanwhile, Transaction B inserts a new row into the employees table with department ‘Sales’ using the following query: INSERT INTO employees (name, salary, department) VALUES ('John Doe', 50000, 'Sales');</code></pre> If we are using Read Committed isolation level, Transaction A will not see the new row inserted by Transaction B until Transaction B commits. However, if we use Repeatable Read isolation level, Transaction A may see the new row inserted by Transaction B on its subsequent reads. To demonstrate this, let’s run the following SQL queries: -- Start Transaction A BEGIN TRANSACTION; SELECT * FROM employees WHERE department = 'Sales'; -- End Transaction A -- Start Transaction B BEGIN TRANSACTION; INSERT INTO employees (name, salary, department) VALUES ('John Doe', 50000, 'Sales'); COMMIT; -- End Transaction B -- Start Transaction A again BEGIN TRANSACTION; SELECT * FROM employees WHERE department = 'Sales'; -- End Transaction A again</code></pre> If we are using Repeatable Read isolation level, the second read operation in Transaction A may return a new row inserted by Transaction B. This is because Repeatable Read isolation level uses a snapshot of the database taken at the beginning of the transaction, and any subsequent changes made by other transactions are not visible to the current transaction. To prevent phantom reads, we can use Serializable isolation level, which ensures that a transaction sees a consistent snapshot of the database throughout its execution, even if other transactions modify the same data. In SQL, we can set the isolation level for a transaction using the SET TRANSACTION statement. To set the isolation level to Serializable, we would use the following query: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;</code></pre> In conclusion, phantom reads can occur in database systems when a transaction reads a set of rows that satisfy a certain condition, but additional rows appear when the same transaction repeats the same read operation later. To prevent phantom reads, we can use Serializable isolation level, which ensures that a transaction sees a consistent snapshot of the database throughout its execution. If you enjoyed reading this article, please subscribe to it and tell your peers about it. If you like the post please connect me on LinkedIn</a> Follow me on Twitter</a> </blockquote>

2. Low-Level Design & Implementation Details: Engineering for Speed</h3>

3. Deep Research & Advanced Optimization: Pushing the Performance Envelope</h3>

4. Rigorous Testing & Reliability: Ensuring Production Readiness</h3>

A preview of the report:</h3>
</p>
Here is the link to the repository: Github Report Generator</a></p>

If you like the post please connect me on LinkedIn</a> Follow me on Twitter</a></p> </blockquote>

References</h3>

Linux Kernel Documentation - Memory Barriers</a></li> </ul>

Increased Cost</h3>
Each HTTPS call incurs a cost, especially if using external services or APIs. With roughly 50 HTTP connections per session, this cost can add up significantly over time.</p>

Impact on Latency and Speed</h3>
Multiple HTTPS calls introduce latency and can slow down the overall charging process. This delay can be noticeable, especially in scenarios where rapid responses are required, such as managing real-time charging sessions or handling high volumes of transactions.</p>

Complexity in Management</h3>
Managing and coordinating multiple HTTPS calls can add complexity to the system architecture. It requires robust error handling, retries, and monitoring mechanisms to ensure smooth operation and reliability.</p>

WebSocket</h2>
WebSocket’s duplex mechanism allows for simultaneous data transfer from both ends over a single connection. This bidirectional communication capability is crucial for real-time updates and efficient interaction between the user and the server.</p>

Fewer Connections</h3>
WebSocket’s persistent nature means fewer connection establishment overheads. Unlike HTTP, where each request requires a new connection and handshake, WebSocket maintains a continuous connection, leading to fewer resources consumed and improved efficiency.</p>

Choosing Database</h2>
Choosing a scalable database like ScyllaDB for a real-time EV charging management system with WebSocket technology offers several advantages:</p>

Real-Time Performance</h3>
ScyllaDB’s high-throughput and low-latency capabilities make it ideal for handling real-time data streams and rapid updates in an event-driven system.</p>

Horizontal Scalability</h3>
ScyllaDB’s distributed architecture allows for seamless horizontal scaling by adding more nodes, ensuring the system can accommodate growing data volumes and user traffic.</p>

Flexible Schema</h3>
With support for flexible schemas, ScyllaDB enables easy adaptation to evolving data structures and simplifies integration with WebSocket-based real-time data streams.</p>

System Design</h2>
Designing a Detailed System Architecture for Real-Time EV Charging Management</p>

WebSocket Communication Layer</h3>
Implement WebSocket protocol for real-time bidirectional communication between EV chargers, users, and the OCPP server. Use WebSocket libraries and frameworks like Socket.IO for efficient WebSocket management and event handling.</p>

ScyllaDB for Data Storage</h3>
Utilize ScyllaDB as the primary database for storing real-time charging session data, user information, and system configurations.</p>

Redis for Scaling and Caching</h3>
Integrate Redis as a caching layer for scaling WebSocket connections and maintaining session states. Use Redis Pub/Sub for real-time notifications and event broadcasting to connected clients</p>

Stateful API Gateway</h3>
Design a stateful API gateway using Redis to manage session states, authenticate users, and handle API requests. Utilize Redis data structures like Hashes and Sets for storing session information, user tokens, and access control lists (ACLs).</p>

Kafka for Event Processing</h3>
Integrate Kafka as a message broker for handling high-throughput event streams and data processing. Use Kafka topics for event-driven architecture, allowing components to publish and subscribe to relevant events.</p>

Current Architecture Design</h2>
Now that we have a clear understanding of our technology, tools, and architecture, let’s simulate the entire system using a straightforward example, step by step. Imagine planning a road trip from start to finish.</p>

Starting a Charging Session</h3>
The user initiates a charging session through the mobile app or web interface, sending a request to the OCPP server via HTTPS or WebSocket API endpoints.</p>

Charger Starts the Session</h3>
The charger receives the data from the WebSocket connection and verifies the session details and authorization. Upon successful validation, the charger starts the charging session, initiating the flow of electrical power to the EV.</p>

Siddharth Sabron

Clik - A Self-Deployable URL Shortener

Legion: An Ultra-Fast, Multi-Threaded Logging Library

64 Bit Distributed ID Generator

A Simple Github Report Generator

Linux Kernel Memory Barriers: A Deep Dive

Designing a Scalable OCPP server

How Discord Used Rust to Scale Elixir to Millions

Authentication Service in TypeScript and Clean Architecture

Understanding Isolation in Database

What caused Discord to switch from Go to Rust?

Siddharth Sabron

Clik - A Self-Deployable URL Shortener

Legion: An Ultra-Fast, Multi-Threaded Logging Library

1.3. Dependency Strategy: The “Less Libraries” Imperative</h3> A core tenet of this project is to minimize external dependencies, thereby reducing the library’s footprint, potential for dependency conflicts, and overall complexity.</p>

2. Low-Level Design & Implementation Details: Engineering for Speed</h3>

2.1. LogEvent Class: Object Pooling and reset() for Garbage-Free Operation</h4> The LogEvent</code> class serves as the data carrier for each log entry. Its design is critical for achieving garbage-free logging.</p>

3. Deep Research & Advanced Optimization: Pushing the Performance Envelope</h3>

4. Rigorous Testing & Reliability: Ensuring Production Readiness</h3>

64 Bit Distributed ID Generator

A Simple Github Report Generator

Features</h2> This tool offers a range of features to help you analyze and visualize your GitHub commits activity. These include:</p>

A preview of the report:</h3> </p> Here is the link to the repository: Github Report Generator</a></p> If you like the post please connect me on LinkedIn</a> Follow me on Twitter</a></p> </blockquote>

Linux Kernel Memory Barriers: A Deep Dive

What are Memory Barriers?</h3> Memory barriers are instructions that impose partial ordering on memory operations. They act as fences, restricting the CPU and compiler from freely reordering accesses across the barrier. There are several types:</p>

References</h3> Linux Kernel Documentation - Memory Barriers</a></li> </ul>

Designing a Scalable OCPP server

Client-server</h2> Now, let’s visualize how your system might look if you opt for a client-server model:</p>

Increased Cost</h3> Each HTTPS call incurs a cost, especially if using external services or APIs. With roughly 50 HTTP connections per session, this cost can add up significantly over time.</p>

Complexity in Management</h3> Managing and coordinating multiple HTTPS calls can add complexity to the system architecture. It requires robust error handling, retries, and monitoring mechanisms to ensure smooth operation and reliability.</p>

WebSocket</h2> WebSocket’s duplex mechanism allows for simultaneous data transfer from both ends over a single connection. This bidirectional communication capability is crucial for real-time updates and efficient interaction between the user and the server.</p>

Fewer Connections</h3> WebSocket’s persistent nature means fewer connection establishment overheads. Unlike HTTP, where each request requires a new connection and handshake, WebSocket maintains a continuous connection, leading to fewer resources consumed and improved efficiency.</p>

Choosing Database</h2> Choosing a scalable database like ScyllaDB for a real-time EV charging management system with WebSocket technology offers several advantages:</p>

Real-Time Performance</h3> ScyllaDB’s high-throughput and low-latency capabilities make it ideal for handling real-time data streams and rapid updates in an event-driven system.</p>

Horizontal Scalability</h3> ScyllaDB’s distributed architecture allows for seamless horizontal scaling by adding more nodes, ensuring the system can accommodate growing data volumes and user traffic.</p>

Flexible Schema</h3> With support for flexible schemas, ScyllaDB enables easy adaptation to evolving data structures and simplifies integration with WebSocket-based real-time data streams.</p>

System Design</h2> Designing a Detailed System Architecture for Real-Time EV Charging Management</p>

WebSocket Communication Layer</h3> Implement WebSocket protocol for real-time bidirectional communication between EV chargers, users, and the OCPP server. Use WebSocket libraries and frameworks like Socket.IO for efficient WebSocket management and event handling.</p>

ScyllaDB for Data Storage</h3> Utilize ScyllaDB as the primary database for storing real-time charging session data, user information, and system configurations.</p>

Redis for Scaling and Caching</h3> Integrate Redis as a caching layer for scaling WebSocket connections and maintaining session states. Use Redis Pub/Sub for real-time notifications and event broadcasting to connected clients</p>

Stateful API Gateway</h3> Design a stateful API gateway using Redis to manage session states, authenticate users, and handle API requests. Utilize Redis data structures like Hashes and Sets for storing session information, user tokens, and access control lists (ACLs).</p>

Kafka for Event Processing</h3> Integrate Kafka as a message broker for handling high-throughput event streams and data processing. Use Kafka topics for event-driven architecture, allowing components to publish and subscribe to relevant events.</p>

Current Architecture Design</h2> Now that we have a clear understanding of our technology, tools, and architecture, let’s simulate the entire system using a straightforward example, step by step. Imagine planning a road trip from start to finish.</p>

Starting a Charging Session</h3> The user initiates a charging session through the mobile app or web interface, sending a request to the OCPP server via HTTPS or WebSocket API endpoints.</p>

Charger Starts the Session</h3> The charger receives the data from the WebSocket connection and verifies the session details and authorization. Upon successful validation, the charger starts the charging session, initiating the flow of electrical power to the EV.</p>

How Discord Used Rust to Scale Elixir to Millions

Authentication Service in TypeScript and Clean Architecture

Prerequisites</h2> Before you begin, make sure you have the following:</p> Node.js (v14 or higher)</li> npm or yarn</li>

Project Setup</h2> To set up the project, follow these steps:</p>

Understanding Isolation in Database

What caused Discord to switch from Go to Rust?

A preview of the report:</h3>
</p>
Here is the link to the repository: Github Report Generator</a></p>

If you like the post please connect me on LinkedIn</a> Follow me on Twitter</a></p> </blockquote>

References</h3>

Linux Kernel Documentation - Memory Barriers</a></li> </ul>

Increased Cost</h3>
Each HTTPS call incurs a cost, especially if using external services or APIs. With roughly 50 HTTP connections per session, this cost can add up significantly over time.</p>

Complexity in Management</h3>
Managing and coordinating multiple HTTPS calls can add complexity to the system architecture. It requires robust error handling, retries, and monitoring mechanisms to ensure smooth operation and reliability.</p>

WebSocket</h2>
WebSocket’s duplex mechanism allows for simultaneous data transfer from both ends over a single connection. This bidirectional communication capability is crucial for real-time updates and efficient interaction between the user and the server.</p>

Fewer Connections</h3>
WebSocket’s persistent nature means fewer connection establishment overheads. Unlike HTTP, where each request requires a new connection and handshake, WebSocket maintains a continuous connection, leading to fewer resources consumed and improved efficiency.</p>

Choosing Database</h2>
Choosing a scalable database like ScyllaDB for a real-time EV charging management system with WebSocket technology offers several advantages:</p>

Real-Time Performance</h3>
ScyllaDB’s high-throughput and low-latency capabilities make it ideal for handling real-time data streams and rapid updates in an event-driven system.</p>

Horizontal Scalability</h3>
ScyllaDB’s distributed architecture allows for seamless horizontal scaling by adding more nodes, ensuring the system can accommodate growing data volumes and user traffic.</p>

Flexible Schema</h3>
With support for flexible schemas, ScyllaDB enables easy adaptation to evolving data structures and simplifies integration with WebSocket-based real-time data streams.</p>

System Design</h2>
Designing a Detailed System Architecture for Real-Time EV Charging Management</p>

WebSocket Communication Layer</h3>
Implement WebSocket protocol for real-time bidirectional communication between EV chargers, users, and the OCPP server. Use WebSocket libraries and frameworks like Socket.IO for efficient WebSocket management and event handling.</p>

ScyllaDB for Data Storage</h3>
Utilize ScyllaDB as the primary database for storing real-time charging session data, user information, and system configurations.</p>

Redis for Scaling and Caching</h3>
Integrate Redis as a caching layer for scaling WebSocket connections and maintaining session states. Use Redis Pub/Sub for real-time notifications and event broadcasting to connected clients</p>

Stateful API Gateway</h3>
Design a stateful API gateway using Redis to manage session states, authenticate users, and handle API requests. Utilize Redis data structures like Hashes and Sets for storing session information, user tokens, and access control lists (ACLs).</p>

Kafka for Event Processing</h3>
Integrate Kafka as a message broker for handling high-throughput event streams and data processing. Use Kafka topics for event-driven architecture, allowing components to publish and subscribe to relevant events.</p>

Current Architecture Design</h2>
Now that we have a clear understanding of our technology, tools, and architecture, let’s simulate the entire system using a straightforward example, step by step. Imagine planning a road trip from start to finish.</p>

Starting a Charging Session</h3>
The user initiates a charging session through the mobile app or web interface, sending a request to the OCPP server via HTTPS or WebSocket API endpoints.</p>

Charger Starts the Session</h3>
The charger receives the data from the WebSocket connection and verifies the session details and authorization. Upon successful validation, the charger starts the charging session, initiating the flow of electrical power to the EV.</p>