In distributed analytical systems, performance relies on two main factors: the efficiency of data movement between processes or network nodes and the efficiency of data processing once it reaches its destination. Apache Arrow's Inter-Process Communication (IPC) framework addresses both challenges. Arrow IPC offers a language-agnostic columnar data format
Discover how compiler-generated futures work behind the scenes—and why they’re both powerful and perplexing.
If the compiler doesn’t force you to Box, you probably don’t need one!
Exploring the why and how of capturing variables inside a closure running in a new thread in Rust
Traditional ETL has long revolved around batch processing pipelines—Spark jobs ingest static, bounded datasets from data lakes or warehouses, perform transformations, and write results to another table or file system. Meanwhile, real-time ingestion platforms such as Apache Kafka, Apache Pulsar, and Redpanda have specialized in high-throughput, low-latency pipelines but
Modern analytics engines need to squeeze every drop of performance out of the CPU. We often hear that SIMD (Single Instruction Multiple Data) can accelerate computation by processing multiple data points in a single CPU instruction. And that’s true—provided the processor isn’t left twiddling its thumbs waiting
This blog post is part of my ongoing series dissecting key concepts from the CMU Databases Courses and presenting them at the Bangalore systems meetup group. Join the discord channel to be part of the community and stay tuned about these sessions. The current series of sessions particularly cover query
The Apache Iceberg project introduced an open table storage format to store structured data in object storage systems (e.g., AWS S3, MinIO) for analytical purposes. Much like the S3 protocol became the de facto standard for accessing and managing object storage, Iceberg’s catalog API has the potential to
In this post, I will argue that today’s query engines, such as Apache Spark, do not effectively interact with lake houses due to limitations in handling object storage semantics, lack of integration with catalog systems like Iceberg for governance and lineage, and inadequate support for lakehouse-specific features. I believe
When working with collections in Rust programming language, especially arrays or vectors, it's common to encounter elements inside arrays wrapped in Option or Result types. Rust provides powerful iterator methods like map and filter_map to manipulate these arrays efficiently. In this blog post, we'll explore
In Rust programming language, lifetimes are a powerful feature that ensures memory safety without needing a garbage collector. However, they can sometimes introduce complexity, mainly when dealing with borrowed references in structs and functions. This complexity can spread throughout your codebase, making it harder to read and maintain. This post
The rapid advancement of large language models (LLMs) has transformed the landscape of artificial intelligence. As these models grow in size and complexity, efficient distributed training and inference have become critical challenges. Apache Ray, a popular distributed computing framework, has gained attention for its flexibility and ease of use. However,