Overview

Relevant source files

• pinot-broker/pom.xml
• pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/GrpcBrokerRequestHandler.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandler.java
• pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/TimeSeriesRequestHandler.java
• pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterHostnamePortTest.java
• pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandlerTest.java
• pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java
• pinot-common/pom.xml
• pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java
• pinot-common/src/main/java/org/apache/pinot/common/utils/config/QueryOptionsUtils.java
• pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
• pinot-common/src/test/java/org/apache/pinot/common/utils/config/QueryOptionsUtilsTest.java
• pinot-common/src/test/java/org/apache/pinot/common/utils/helix/HelixHelperTest.java
• pinot-compatibility-verifier/pom.xml
• pinot-controller/pom.xml
• pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
• pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
• pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java
• pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/MessagingServiceUtils.java
• pinot-controller/src/test/java/org/apache/pinot/controller/ControllerConfTest.java
• pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocatorTest.java
• pinot-core/pom.xml
• pinot-distribution/pinot-assembly.xml
• pinot-distribution/pom.xml
• pinot-integration-test-base/pom.xml
• pinot-integration-tests/pom.xml
• pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
• pinot-minion/pom.xml
• pinot-minion/src/main/java/org/apache/pinot/minion/BaseMinionStarter.java
• pinot-minion/src/main/java/org/apache/pinot/minion/MinionConf.java
• pinot-perf/pom.xml
• pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml
• pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pom.xml
• pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pom.xml
• pinot-plugins/pinot-batch-ingestion/pom.xml
• pinot-plugins/pinot-file-system/pinot-adls/pom.xml
• pinot-plugins/pinot-file-system/pinot-gcs/pom.xml
• pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml
• pinot-plugins/pinot-file-system/pinot-s3/pom.xml
• pinot-plugins/pinot-file-system/pom.xml
• pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml
• pinot-plugins/pinot-input-format/pinot-avro/pom.xml
• pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml
• pinot-plugins/pinot-input-format/pinot-csv/pom.xml
• pinot-plugins/pinot-input-format/pinot-json/pom.xml
• pinot-plugins/pinot-input-format/pinot-orc/pom.xml
• pinot-plugins/pinot-input-format/pinot-parquet/pom.xml
• pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
• pinot-plugins/pinot-input-format/pinot-thrift/pom.xml
• pinot-plugins/pinot-input-format/pom.xml
• pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml
• pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml
• pinot-plugins/pinot-stream-ingestion/pinot-kinesis/pom.xml
• pinot-plugins/pinot-stream-ingestion/pinot-pulsar/pom.xml
• pinot-plugins/pinot-stream-ingestion/pom.xml
• pinot-plugins/pom.xml
• pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java
• pinot-query-planner/src/main/java/org/apache/pinot/query/catalog/PinotCatalog.java
• pinot-query-planner/src/main/java/org/apache/pinot/query/context/PlannerContext.java
• pinot-server/pom.xml
• pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java
• pinot-spi/pom.xml
• pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
• pinot-tools/pom.xml
• pinot-tools/src/main/java/org/apache/pinot/tools/MultiDirQuickstart.java
• pom.xml

Purpose and Scope

Apache Pinot is a distributed, real-time OLAP (Online Analytical Processing) datastore designed for low-latency analytics at scale. This document provides a high-level overview of Pinot's architecture, core components, and how they interact to support both batch and real-time data ingestion with fast query execution.

For detailed information about specific subsystems:

• System architecture and component interactions: see System Architecture and Components
• Key terminology and concepts: see Key Concepts and Terminology
• Build system and plugin architecture: see Build System and Plugins
• Cluster management with Helix: see Cluster Management and Coordination
• Configuration system: see Configuration Management
• Data ingestion mechanisms: see Data Ingestion
• Query processing engines: see Query Processing

Sources: pom.xml1-38

---

Four-Tier Architecture

Pinot operates as a distributed system with four primary component tiers that work together to provide scalable, real-time analytics:

Diagram: Pinot Four-Tier Architecture with Core Classes

Tier	Primary Class	Maven Module	Purpose
Broker	BaseBrokerStarter	pinot-broker	Query gateway; receives SQL queries, routes to servers, reduces results
Controller	BaseControllerStarter	pinot-controller	Cluster orchestrator; manages tables, schemas, segments, and metadata
Server	BaseServerStarter	pinot-server	Data node; hosts segments, executes queries, ingests real-time data
Minion	BaseMinionStarter	pinot-minion	Background task executor; performs segment optimization, rebalancing

Sources: pom.xml40-64 pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java120-136 pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java165-166 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java158-159 pinot-minion/src/main/java/org/apache/pinot/minion/BaseMinionStarter.java1-22

---

Component Lifecycle and Initialization

Each Pinot component follows a common initialization pattern through the ServiceStartable interface:

Diagram: Component Initialization Sequence

All four component types share this initialization flow:

1. Configuration Loading: Each component reads from a PinotConfiguration object containing properties from configuration files and cluster configs
2. Plugin Registration: The PluginManager loads plugins for file systems, stream ingestion, input formats, and metrics
3. Helix Connection: Components connect to Apache Helix as either participants (Broker, Server, Minion) or controllers (Controller)
4. Resource Manager Setup: Component-specific managers are initialized (e.g., PinotHelixResourceManager for Controller, InstanceDataManager for Server)
5. Admin API Start: Each component exposes REST and optionally gRPC endpoints for administration and health checks

Sources: pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java188-244 pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java232-244 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java188-192

---

Coordination via Apache Helix

Pinot uses Apache Helix for distributed cluster coordination. Helix manages cluster state, resource assignments, and state transitions across all components.

Diagram: Helix Integration Points in Pinot Components

Helix Key Concepts in Pinot

• IdealState: Defines the desired assignment of segments to servers, managed by the Controller via PinotHelixResourceManager
• ExternalView: Reflects the actual current state of segment assignments, used by Brokers for query routing via BrokerRoutingManager
• InstanceConfig: Contains server metadata including host, port, tags, and capacity information
• PropertyStore: ZooKeeper-backed store for table configs (/CONFIGS), schemas (/SCHEMAS), and segment metadata (/SEGMENTS)
• State Model: SegmentOnlineOfflineStateModel (for offline tables) and includes CONSUMING state (for real-time tables) to manage segment lifecycle transitions

Sources: pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java1-32 pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java165-180 pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java101-205

---

Query Execution Flow

Pinot supports two query execution engines: single-stage (scatter-gather) and multi-stage (distributed SQL).

Diagram: Query Execution Path Through Broker and Server

Query Handling Components

Component	Class	Purpose
SQL Parser	CalciteSqlParser	Parses SQL into SqlNode AST using Apache Calcite
Query Compiler	QueryEnvironment	Compiles SqlNode to logical plan, then to physical DispatchableSubPlan
Single-Stage Handler	BaseSingleStageBrokerRequestHandler	Scatter-gather for simple aggregations and filters
Multi-Stage Handler	MultiStageBrokerRequestHandler	Complex queries with joins, window functions, sub-queries
Query Dispatcher	QueryDispatcher	Distributes multi-stage query plans to workers
Query Router	QueryRouter	Routes requests to appropriate servers based on segment assignment

Sources: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java76-144 pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java1-97 pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java114-152 pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java108-157

---

Data Ingestion Pipeline

Pinot supports both batch and real-time streaming ingestion:

Diagram: Data Ingestion and Storage Flow

Ingestion Components by Type

Batch Ingestion:

• RecordReader Interface: Plugin-based readers for Avro (pinot-avro), Parquet (pinot-parquet), ORC (pinot-orc), JSON (pinot-json), CSV (pinot-csv), Protobuf (pinot-protobuf), Thrift (pinot-thrift)
• Segment Generators: Spark-based (pinot-batch-ingestion-spark-2.4, pinot-batch-ingestion-spark-3), Hadoop-based (pinot-batch-ingestion-hadoop), or standalone
• Segment Upload: Controller receives segments via FileUploadDownloadClient and stores metadata in ZooKeeper

Real-Time Ingestion:

• Stream Plugins: Kafka 2.x (pinot-kafka-2.0), Kafka 3.x (pinot-kafka-3.0), Kinesis (pinot-kinesis), Pulsar (pinot-pulsar)
• RealtimeTableDataManager: Manages consuming segments on servers
• RealtimeSegmentDataManager: Coordinates consumption from a single partition, building MutableSegmentImpl in memory
• SegmentCompletionProtocol: Distributed consensus protocol ensuring exactly-once semantics when segments complete

Sources: pom.xml640-697 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java234-260 pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java95-99

---

Plugin Architecture

Pinot's extensibility is built on a Service Provider Interface (SPI) pattern with plugins loaded via Java's ServiceLoader:

Diagram: Plugin Architecture with SPI Contracts and Implementations

Plugin Loading Mechanism

Plugins are:

1. Packaged as shaded JARs with all dependencies included (see shade.phase.prop properties in plugin POMs)
2. Discovered via META-INF/services files following Java ServiceLoader convention
3. Loaded by PluginManager during component initialization
4. Initialized with PinotConfiguration allowing plugin-specific configuration

Example from S3 plugin POM:

This tells Maven to shade the plugin JAR, bundling AWS SDK dependencies so they don't conflict with Pinot core dependencies.

Sources: pom.xml640-778 pinot-plugins/pom.xml1-20 pinot-plugins/pinot-file-system/pinot-adls/pom.xml29-54 pinot-plugins/pinot-file-system/pinot-s3/pom.xml1-21 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/pom.xml29-69 pinot-plugins/pinot-input-format/pinot-parquet/pom.xml1-23 pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml1-19

---

Maven Module Structure

The Pinot codebase is organized into multiple Maven modules:

Module	Purpose	Key Dependencies
pinot-spi	Service Provider Interfaces for plugins	Commons, Jackson
pinot-segment-spi	Segment format and index SPIs	pinot-spi
pinot-common	Common utilities, metrics, request/response models	pinot-segment-spi, Helix, Calcite
pinot-segment-local	Segment creation and local indexing	pinot-common
pinot-core	Query execution, operators, data structures	pinot-segment-local
pinot-broker	Broker implementation	pinot-core
pinot-server	Server implementation	pinot-core
pinot-controller	Controller implementation	pinot-core
pinot-minion	Minion implementation	pinot-core
pinot-query-planner	Multi-stage query planning	pinot-common, Calcite
pinot-query-runtime	Multi-stage query execution	pinot-query-planner
pinot-plugins	All plugin modules	Various
pinot-tools	CLI tools and utilities	All components
pinot-distribution	Assembly and packaging	pinot-tools
pinot-integration-tests	End-to-end tests	All components

Sources: pom.xml40-64 pinot-spi/pom.xml24-34 pinot-common/pom.xml22-31 pinot-core/pom.xml22-34 pinot-broker/pom.xml1-27 pinot-server/pom.xml1-24 pinot-controller/pom.xml1-28 pinot-tools/pom.xml22-34 pinot-distribution/pom.xml22-36 pinot-integration-tests/pom.xml22-36

---

Configuration System

Pinot uses a hierarchical configuration system with multiple precedence levels:

Diagram: Configuration Hierarchy and Precedence

Configuration Priority Order (Highest to Lowest)

1. Query Options: Per-query settings via SET statements or query parameters (e.g., SET timeoutMs = 10000)
2. Table Config: Per-table settings in TableConfig JSON stored in ZooKeeper under /CONFIGS
3. Cluster Config: Cluster-wide settings in Helix ClusterConfig, applied via ServiceStartableUtils.applyClusterConfig()
4. Component Config: Component-specific configuration files (e.g., pinot-broker.conf)

Key Configuration Classes

• CommonConstants: Defines all configuration keys with defaults (e.g., CommonConstants.Broker.CONFIG_OF_BROKER_TIMEOUT_MS)
• ControllerConf: Controller-specific configuration wrapper extending PinotConfiguration
• ServerConf: Server-specific configuration
• BrokerConf: Implicit via PinotConfiguration passed to BaseBrokerStarter
• QueryOptionsUtils: Resolves case-insensitive query options and validates values

Sources: pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java30-327 pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java52-103 pinot-common/src/main/java/org/apache/pinot/common/utils/config/QueryOptionsUtils.java1-84 pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java124-135

---

Distribution and Deployment

Pinot is packaged as a binary distribution using the Maven Assembly plugin:

Diagram: Binary Distribution Structure

Key Distribution Components

The distribution is assembled by pinot-assembly.xml:

1. Executable Scripts: Shell scripts in bin/ that invoke Java with appropriate classpath and JVM options
2. Core Libraries: All non-plugin dependencies in lib/
3. Plugin JARs: Shaded plugin JARs in plugins/ subdirectories, loaded separately to avoid conflicts
4. Configuration Templates: Sample configurations in conf/

Each start script (e.g., start-controller.sh) wraps pinot-admin.sh with component-specific class:

• Controller: org.apache.pinot.tools.admin.PinotController
• Broker: org.apache.pinot.tools.admin.PinotBroker
• Server: org.apache.pinot.tools.admin.PinotServer
• Minion: org.apache.pinot.tools.admin.PinotMinion

Sources: pinot-distribution/pinot-assembly.xml1-14 pinot-distribution/pom.xml28-46 pinot-tools/pom.xml204-297

---

Integration Testing

Pinot includes comprehensive integration tests that demonstrate end-to-end functionality:

Diagram: Integration Test Cluster Lifecycle

The OfflineClusterIntegrationTest demonstrates:

• Starting a full Pinot cluster (ZooKeeper, Controller, Broker, Server)
• Creating tables and uploading segments
• Executing queries and validating results against H2 database
• Testing index creation (inverted, range, bloom filter, star-tree)
• Testing table configuration updates and reloads
• Testing segment operations (refresh, replace, delete)

This test serves as both validation and a reference implementation for cluster setup.

Sources: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java107-277 pinot-integration-tests/pom.xml22-36

---

Summary

Apache Pinot is a distributed OLAP system with:

• Four-tier architecture: Broker (query gateway), Controller (cluster orchestrator), Server (data + compute), Minion (background tasks)
• Dual query engines: Single-stage scatter-gather for simple queries, multi-stage for complex SQL with joins
• Flexible ingestion: Batch via Spark/Hadoop, real-time via Kafka/Kinesis/Pulsar with exactly-once semantics
• Plugin-based extensibility: File systems, stream sources, input formats, and metrics all pluggable via SPI
• Helix-based coordination: All cluster state management, segment assignment, and health monitoring via Apache Helix
• Hierarchical configuration: From component configs to cluster configs to table configs to query options

This overview provides the foundation for understanding Pinot's architecture. For deeper dives into specific subsystems, refer to the linked sections at the top of this document.