Apache Impala Guide

Important Notice
© 2010-2018 Cloudera, Inc. All rights reserved.

Cloudera, the Cloudera logo, and any other product or

service names or slogans contained in this document are trademarks of Cloudera and
its suppliers or licensors, and may not be copied, imitated or used, in whole or in part,
without the prior written permission of Cloudera or the applicable trademark holder. If
this documentation includes code, including but not limited to, code examples, Cloudera
makes this available to you under the terms of the Apache License, Version 2.0, including
any required notices. A copy of the Apache License Version 2.0, including any notices,
is included herein. A copy of the Apache License Version 2.0 can also be found here:
https://opensource.org/licenses/Apache-2.0

Hadoop and the Hadoop elephant logo are trademarks of the Apache Software
Foundation. All other trademarks, registered trademarks, product names and company
names or logos mentioned in this document are the property of their respective owners.
Reference to any products, services, processes or other information, by trade name,
trademark, manufacturer, supplier or otherwise does not constitute or imply
endorsement, sponsorship or recommendation thereof by us.

Complying with all applicable copyright laws is the responsibility of the user. Without
limiting the rights under copyright, no part of this document may be reproduced, stored
in or introduced into a retrieval system, or transmitted in any form or by any means
(electronic, mechanical, photocopying, recording, or otherwise), or for any purpose,
without the express written permission of Cloudera.

Cloudera may have patents, patent applications, trademarks, copyrights, or other

intellectual property rights covering subject matter in this document. Except as expressly
provided in any written license agreement from Cloudera, the furnishing of this document
does not give you any license to these patents, trademarks copyrights, or other
intellectual property. For information about patents covering Cloudera products, see
http://tiny.cloudera.com/patents.

The information in this document is subject to change without notice. Cloudera shall
not be liable for any damages resulting from technical errors or omissions which may
be present in this document, or from use of this document.

Cloudera, Inc.
395 Page Mill Road
Palo Alto, CA 94306
[email protected]
US: 1-888-789-1488
Intl: 1-650-362-0488
www.cloudera.com

Release Information

Version: Cloudera Impala Impala 2.4.x / CDH 5.6.x

Date: June 25, 2018
Table of Contents

Introducing Apache Impala....................................................................................16

Impala Benefits..................................................................................................................................................16
How Impala Works with CDH.............................................................................................................................16
Primary Impala Features....................................................................................................................................17

Impala Concepts and Architecture..........................................................................18

Components of the Impala Server.....................................................................................................................18
The Impala Daemon.............................................................................................................................................................18
The Impala Statestore..........................................................................................................................................................18
The Impala Catalog Service..................................................................................................................................................19
Developing Impala Applications.........................................................................................................................19
Overview of the Impala SQL Dialect.....................................................................................................................................20
Overview of Impala Programming Interfaces......................................................................................................................21
How Impala Fits Into the Hadoop Ecosystem.....................................................................................................21
How Impala Works with Hive...............................................................................................................................................21
Overview of Impala Metadata and the Metastore..............................................................................................................21
How Impala Uses HDFS........................................................................................................................................................22
How Impala Uses HBase......................................................................................................................................................22

Planning for Impala Deployment............................................................................23

Impala Requirements.........................................................................................................................................23
Product Compatibility Matrix...............................................................................................................................................23
Supported Operating Systems..............................................................................................................................................23
Hive Metastore and Related Configuration.........................................................................................................................23
Java Dependencies...............................................................................................................................................................24
Networking Configuration Requirements............................................................................................................................24
Hardware Requirements......................................................................................................................................................24
User Account Requirements.................................................................................................................................................24
Guidelines for Designing Impala Schemas.........................................................................................................25

Installing Impala.....................................................................................................27
What is Included in an Impala Installation.........................................................................................................27
Installing Impala with Cloudera Manager..........................................................................................................27
Installing Impala without Cloudera Manager.....................................................................................................28
Managing Impala...................................................................................................30
Post-Installation Configuration for Impala.........................................................................................................30
Configuring Impala to Work with ODBC.............................................................................................................31
Downloading the ODBC Driver.............................................................................................................................................32
Configuring the ODBC Port..................................................................................................................................................32
Example of Setting Up an ODBC Application for Impala......................................................................................................32
Notes about JDBC and ODBC Interaction with Impala SQL Features...................................................................................33
Configuring Impala to Work with JDBC..............................................................................................................34
Configuring the JDBC Port....................................................................................................................................................34
Choosing the JDBC Driver.....................................................................................................................................................34
Enabling Impala JDBC Support on Client Systems................................................................................................................35
Establishing JDBC Connections.............................................................................................................................................36
Notes about JDBC and ODBC Interaction with Impala SQL Features...................................................................................37
Kudu Considerations for DML Statements...........................................................................................................................37

Upgrading Impala...................................................................................................38
Upgrading Impala through Cloudera Manager - Parcels....................................................................................38
Upgrading Impala through Cloudera Manager - Packages.................................................................................39
Upgrading Impala without Cloudera Manager...................................................................................................40

Starting Impala......................................................................................................42
Starting Impala through Cloudera Manager.......................................................................................................42
Starting Impala from the Command Line...........................................................................................................42
Modifying Impala Startup Options.....................................................................................................................43
Configuring Impala Startup Options through Cloudera Manager.......................................................................................43
Configuring Impala Startup Options through the Command Line.......................................................................................43
Checking the Values of Impala Configuration Options........................................................................................................46
Startup Options for impalad Daemon..................................................................................................................................46
Startup Options for statestored Daemon.............................................................................................................................46
Startup Options for catalogd Daemon.................................................................................................................................46

Impala Tutorials.....................................................................................................47
Tutorials for Getting Started...............................................................................................................................47
Explore a New Impala Instance............................................................................................................................................47
Load CSV Data from Local Files............................................................................................................................................52
Point an Impala Table at Existing Data Files........................................................................................................................53
Describe the Impala Table....................................................................................................................................................55
Query the Impala Table........................................................................................................................................................55
Data Loading and Querying Examples.................................................................................................................................56
Advanced Tutorials.............................................................................................................................................58
Attaching an External Partitioned Table to an HDFS Directory Structure............................................................................58
Switching Back and Forth Between Impala and Hive...........................................................................................................60
Cross Joins and Cartesian Products with the CROSS JOIN Operator....................................................................................61
Dealing with Parquet Files with Unknown Schema............................................................................................62

Impala Administration...........................................................................................75
Admission Control and Query Queuing..............................................................................................................75
Overview of Impala Admission Control................................................................................................................................75
How Impala Admission Control Relates to YARN.................................................................................................................76
How Impala Schedules and Enforces Limits on Concurrent Queries....................................................................................76
How Admission Control works with Impala Clients (JDBC, ODBC, HiveServer2)..................................................................77
Configuring Admission Control............................................................................................................................................77
Guidelines for Using Admission Control...............................................................................................................................82
Integrated Resource Management with YARN...................................................................................................83
The Llama Daemon..............................................................................................................................................................83
How Resource Limits Are Enforced.......................................................................................................................................84
impala-shell Query Options for Resource Management......................................................................................................84
Limitations of Resource Management for Impala...............................................................................................................84
Setting Timeout Periods for Daemons, Queries, and Sessions...........................................................................84
Increasing the Statestore Timeout.......................................................................................................................................84
Setting the Idle Query and Idle Session Timeouts for impalad.............................................................................................84
Cancelling a Query...............................................................................................................................................................85
Using Impala through a Proxy for High Availability............................................................................................85
Overview of Proxy Usage and Load Balancing for Impala...................................................................................................86
Special Proxy Considerations for Clusters Using Kerberos...................................................................................................87
Example of Configuring HAProxy Load Balancer for Impala................................................................................................88
Managing Disk Space for Impala Data................................................................................................................89

Impala Security......................................................................................................91
Security Guidelines for Impala...........................................................................................................................91
Securing Impala Data and Log Files....................................................................................................................92
Installation Considerations for Impala Security..................................................................................................93
Securing the Hive Metastore Database..............................................................................................................93
Securing the Impala Web User Interface............................................................................................................93
Configuring TLS/SSL for Impala..........................................................................................................................94
Using Cloudera Manager.....................................................................................................................................................94
Using the Command Line.....................................................................................................................................................95
Using TLS/SSL with Business Intelligence Tools....................................................................................................................95
Enabling Sentry Authorization for Impala..........................................................................................................95
The Sentry Privilege Model..................................................................................................................................................96
Starting the impalad Daemon with Sentry Authorization Enabled......................................................................................96
Using Impala with the Sentry Service (CDH 5.1 or higher only)...........................................................................................97
Using Impala with the Sentry Policy File..............................................................................................................................98
Setting Up Schema Objects for a Secure Impala Deployment............................................................................................102
Privilege Model and Object Hierarchy...............................................................................................................................102
Debugging Failed Sentry Authorization Requests..............................................................................................................105
Managing Sentry for Impala through Cloudera Manager.................................................................................................106
The DEFAULT Database in a Secure Deployment...............................................................................................................106
Impala Authentication......................................................................................................................................106
Enabling Kerberos Authentication for Impala....................................................................................................................106
Enabling LDAP Authentication for Impala..........................................................................................................................109
Using Multiple Authentication Methods with Impala........................................................................................................111
Configuring Impala Delegation for Hue and BI Tools.........................................................................................................112
Auditing Impala Operations.............................................................................................................................112
Durability and Performance Considerations for Impala Auditing......................................................................................112
Format of the Audit Log Files.............................................................................................................................................113
Which Operations Are Audited..........................................................................................................................................113
Reviewing the Audit Logs...................................................................................................................................................114
Viewing Lineage Information for Impala Data..................................................................................................114

Impala SQL Language Reference...........................................................................116

Comments........................................................................................................................................................116
Data Types........................................................................................................................................................116
ARRAY Complex Type (CDH 5.5 or higher only)..................................................................................................................117
BIGINT Data Type...............................................................................................................................................................120
BOOLEAN Data Type..........................................................................................................................................................121
CHAR Data Type (CDH 5.2 or higher only)..........................................................................................................................122
DECIMAL Data Type (CDH 5.1 or higher only)....................................................................................................................125
DOUBLE Data Type.............................................................................................................................................................133
FLOAT Data Type................................................................................................................................................................134
INT Data Type....................................................................................................................................................................135
MAP Complex Type (CDH 5.5 or higher only).....................................................................................................................136
REAL Data Type..................................................................................................................................................................139
SMALLINT Data Type..........................................................................................................................................................140
STRING Data Type..............................................................................................................................................................141
STRUCT Complex Type (CDH 5.5 or higher only)................................................................................................................142
TIMESTAMP Data Type.......................................................................................................................................................148
TINYINT Data Type.............................................................................................................................................................153
VARCHAR Data Type (CDH 5.2 or higher only)...................................................................................................................154
Complex Types (CDH 5.5 or higher only)............................................................................................................................156
Literals..............................................................................................................................................................184
Numeric Literals.................................................................................................................................................................184
String Literals.....................................................................................................................................................................185
Boolean Literals.................................................................................................................................................................187
Timestamp Literals.............................................................................................................................................................187
NULL...................................................................................................................................................................................187
SQL Operators..................................................................................................................................................188
Arithmetic Operators.........................................................................................................................................................188
BETWEEN Operator...........................................................................................................................................................191
Comparison Operators.......................................................................................................................................................192
EXISTS Operator.................................................................................................................................................................193
IN Operator........................................................................................................................................................................196
IS NULL Operator...............................................................................................................................................................198
LIKE Operator.....................................................................................................................................................................199
Logical Operators...............................................................................................................................................................199
REGEXP Operator...............................................................................................................................................................202
RLIKE Operator...................................................................................................................................................................203
Impala Schema Objects and Object Names.....................................................................................................204
Overview of Impala Aliases................................................................................................................................................204
Overview of Impala Databases..........................................................................................................................................205
Overview of Impala Functions...........................................................................................................................................206
Overview of Impala Identifiers...........................................................................................................................................207
Overview of Impala Tables.................................................................................................................................................208
Overview of Impala Views.................................................................................................................................................210
Impala SQL Statements....................................................................................................................................214
DDL Statements.................................................................................................................................................................214
DML Statements................................................................................................................................................................215
ALTER TABLE Statement.....................................................................................................................................................215
ALTER VIEW Statement......................................................................................................................................................224
COMPUTE STATS Statement...............................................................................................................................................226
CREATE DATABASE Statement............................................................................................................................................231
CREATE FUNCTION Statement...........................................................................................................................................232
CREATE ROLE Statement (CDH 5.2 or higher only).............................................................................................................235
CREATE TABLE Statement...................................................................................................................................................235
CREATE VIEW Statement....................................................................................................................................................243
DESCRIBE Statement..........................................................................................................................................................245
DROP DATABASE Statement...............................................................................................................................................252
DROP FUNCTION Statement..............................................................................................................................................254
DROP ROLE Statement (CDH 5.2 or higher only)................................................................................................................254
DROP STATS Statement......................................................................................................................................................255
DROP TABLE Statement......................................................................................................................................................258
DROP VIEW Statement.......................................................................................................................................................259
EXPLAIN Statement............................................................................................................................................................260
GRANT Statement (CDH 5.2 or higher only).......................................................................................................................263
INSERT Statement..............................................................................................................................................................263
INVALIDATE METADATA Statement....................................................................................................................................271
LOAD DATA Statement.......................................................................................................................................................274
REFRESH Statement...........................................................................................................................................................277
REVOKE Statement (CDH 5.2 or higher only).....................................................................................................................279
SELECT Statement..............................................................................................................................................................280
SET Statement....................................................................................................................................................................304
SHOW Statement...............................................................................................................................................................322
TRUNCATE TABLE Statement (CDH 5.5 or higher only)......................................................................................................335
USE Statement...................................................................................................................................................................337
Impala Built-In Functions.................................................................................................................................338
Impala Mathematical Functions........................................................................................................................................339
Impala Bit Functions..........................................................................................................................................................351
Impala Type Conversion Functions.....................................................................................................................................360
Impala Date and Time Functions.......................................................................................................................................363
Impala Conditional Functions............................................................................................................................................391
Impala String Functions.....................................................................................................................................................394
Impala Miscellaneous Functions........................................................................................................................................403
Impala Aggregate Functions..............................................................................................................................................403
Impala Analytic Functions..................................................................................................................................................429
Impala User-Defined Functions (UDFs)..............................................................................................................................447
SQL Differences Between Impala and Hive......................................................................................................461
HiveQL Features not Available in Impala...........................................................................................................................462
Semantic Differences Between Impala and HiveQL Features............................................................................................463
Porting SQL from Other Database Systems to Impala......................................................................................464
Porting DDL and DML Statements.....................................................................................................................................464
Porting Data Types from Other Database Systems............................................................................................................464
SQL Statements to Remove or Adapt.................................................................................................................................467
SQL Constructs to Doublecheck..........................................................................................................................................468
Next Porting Steps after Verifying Syntax and Semantics..................................................................................................469

Using the Impala Shell (impala-shell Command)...................................................470

impala-shell Configuration Options..................................................................................................................470
Summary of impala-shell Configuration Options...............................................................................................................470
impala-shell Configuration File..........................................................................................................................................473
Connecting to impalad through impala-shell...................................................................................................474
Running Commands and SQL Statements in impala-shell................................................................................475
impala-shell Command Reference...................................................................................................................476

Tuning Impala for Performance............................................................................479

Impala Performance Guidelines and Best Practices.........................................................................................479
Performance Considerations for Join Queries..................................................................................................482
How Joins Are Processed when Statistics Are Unavailable................................................................................................483
Overriding Join Reordering with STRAIGHT_JOIN..............................................................................................................483
Examples of Join Order Optimization.................................................................................................................................484
Table and Column Statistics..............................................................................................................................489
Overview of Table Statistics...............................................................................................................................................489
Overview of Column Statistics............................................................................................................................................490
How Table and Column Statistics Work for Partitioned Tables..........................................................................................491
Overview of Incremental Statistics.....................................................................................................................................493
Generating Table and Column Statistics (COMPUTE STATS Statement).............................................................................494
Detecting Missing Statistics...............................................................................................................................................494
Keeping Statistics Up to Date.............................................................................................................................................496
Setting the NUMROWS Value Manually through ALTER TABLE.........................................................................................496
Examples of Using Table and Column Statistics with Impala.............................................................................................497
Benchmarking Impala Queries.........................................................................................................................500
Controlling Impala Resource Usage..................................................................................................................500
Using HDFS Caching with Impala (CDH 5.1 or higher only)..............................................................................501
Overview of HDFS Caching for Impala...............................................................................................................................501
Setting Up HDFS Caching for Impala..................................................................................................................................501
Enabling HDFS Caching for Impala Tables and Partitions..................................................................................................502
Loading and Removing Data with HDFS Caching Enabled.................................................................................................503
Administration for HDFS Caching with Impala...................................................................................................................504
Performance Considerations for HDFS Caching with Impala.............................................................................................505
Testing Impala Performance.............................................................................................................................506
Understanding Impala Query Performance - EXPLAIN Plans and Query Profiles.............................................507
Using the EXPLAIN Plan for Performance Tuning...............................................................................................................507
Using the SUMMARY Report for Performance Tuning.......................................................................................................508
Using the Query Profile for Performance Tuning...............................................................................................................509
Detecting and Correcting HDFS Block Skew Conditions...................................................................................514

Scalability Considerations for Impala....................................................................516

Scalability Considerations for the Impala Statestore........................................................................................516
SQL Operations that Spill to Disk......................................................................................................................517
Limits on Query Size and Complexity...............................................................................................................520
Scalability Considerations for Impala I/O.........................................................................................................521
Scalability Considerations for Table Layout......................................................................................................521

Partitioning for Impala Tables...............................................................................522

When to Use Partitioned Tables.......................................................................................................................522
SQL Statements for Partitioned Tables.............................................................................................................522
Static and Dynamic Partitioning Clauses..........................................................................................................523
Permissions for Partition Subdirectories..........................................................................................................523
Partition Pruning for Queries...........................................................................................................................524
Checking if Partition Pruning Happens for a Query...........................................................................................................524
What SQL Constructs Work with Partition Pruning............................................................................................................524
Partition Key Columns......................................................................................................................................525
Setting Different File Formats for Partitions.....................................................................................................526
Managing Partitions.........................................................................................................................................526

How Impala Works with Hadoop File Formats......................................................527

Choosing the File Format for a Table................................................................................................................528
Using Text Data Files with Impala Tables..........................................................................................................528
Query Performance for Impala Text Tables........................................................................................................................529
Creating Text Tables...........................................................................................................................................................529
Data Files for Text Tables...................................................................................................................................................530
Loading Data into Impala Text Tables................................................................................................................................531
Using LZO-Compressed Text Files.......................................................................................................................................531
Using gzip, bzip2, or Snappy-Compressed Text Files..........................................................................................................534
Using the Parquet File Format with Impala Tables...........................................................................................535
Creating Parquet Tables in Impala.....................................................................................................................................535
Loading Data into Parquet Tables......................................................................................................................................536
Query Performance for Impala Parquet Tables..................................................................................................................538
Snappy and GZip Compression for Parquet Data Files.......................................................................................................539
Parquet Tables for Impala Complex Types.........................................................................................................................541
Exchanging Parquet Data Files with Other Hadoop Components......................................................................................541
How Parquet Data Files Are Organized..............................................................................................................................544
Compacting Data Files for Parquet Tables.........................................................................................................................545
Schema Evolution for Parquet Tables.................................................................................................................................546
Data Type Considerations for Parquet Tables....................................................................................................................546
Using the Avro File Format with Impala Tables................................................................................................547
Creating Avro Tables..........................................................................................................................................................547
Using a Hive-Created Avro Table in Impala........................................................................................................................549
Specifying the Avro Schema through JSON........................................................................................................................550
Loading Data into an Avro Table........................................................................................................................................550
Enabling Compression for Avro Tables...............................................................................................................................550
How Impala Handles Avro Schema Evolution....................................................................................................................550
Data Type Considerations for Avro Tables..........................................................................................................................551
Using the RCFile File Format with Impala Tables..............................................................................................552
Creating RCFile Tables and Loading Data..........................................................................................................................552
Enabling Compression for RCFile Tables.............................................................................................................................553
Using the SequenceFile File Format with Impala Tables..................................................................................554
Creating SequenceFile Tables and Loading Data...............................................................................................................554
Enabling Compression for SequenceFile Tables..................................................................................................................555

Using Impala to Query HBase Tables....................................................................557

Overview of Using HBase with Impala.............................................................................................................557
Configuring HBase for Use with Impala............................................................................................................557
Supported Data Types for HBase Columns.......................................................................................................558
Performance Considerations for the Impala-HBase Integration......................................................................558
Use Cases for Querying HBase through Impala................................................................................................562
Loading Data into an HBase Table....................................................................................................................562
Limitations and Restrictions of the Impala and HBase Integration..................................................................562
Examples of Querying HBase Tables from Impala............................................................................................563

Using Impala to Query the Amazon S3 Filesystem.................................................566

Specifying Impala Credentials to Access Data in S3.........................................................................................566
Loading Data into S3 for Impala Queries..........................................................................................................566
Creating Impala Databases, Tables, and Partitions for Data Stored on S3.......................................................567
Internal and External Tables Located on S3......................................................................................................568
Running and Tuning Impala Queries for Data Stored on S3.............................................................................570
Understanding and Tuning Impala Query Performance for S3 Data..................................................................................570
Restrictions on Impala Support for S3..............................................................................................................570

Using Impala with Isilon Storage..........................................................................572

Required Configurations...................................................................................................................................572

Using Impala Logging...........................................................................................574

Locations and Names of Impala Log Files.........................................................................................................574
Managing Impala Logs through Cloudera Manager or Manually.....................................................................575
Rotating Impala Logs........................................................................................................................................575
Reviewing Impala Logs.....................................................................................................................................575
Understanding Impala Log Contents................................................................................................................576
Setting Logging Levels......................................................................................................................................576
Redacting Sensitive Information from Impala Log Files...................................................................................577

Troubleshooting Impala.......................................................................................578
Troubleshooting Impala SQL Syntax Issues......................................................................................................578
Troubleshooting I/O Capacity Problems...........................................................................................................578
Impala Troubleshooting Quick Reference........................................................................................................579
Impala Web User Interface for Debugging.......................................................................................................580
Debug Web UI for impalad................................................................................................................................................581

Ports Used by Impala...........................................................................................583

Impala Reserved Words........................................................................................585

List of Current Reserved Words........................................................................................................................585
Planning for Future Reserved Words...............................................................................................................587
Impala Frequently Asked Questions.....................................................................590
Transition to Apache Governance....................................................................................................................590
Trying Impala....................................................................................................................................................590
Impala System Requirements...........................................................................................................................591
Supported and Unsupported Functionality In Impala......................................................................................592
How do I?.........................................................................................................................................................593
Impala Performance.........................................................................................................................................594
Impala Use Cases.............................................................................................................................................596
Questions about Impala And Hive....................................................................................................................597
Impala Availability............................................................................................................................................598
Impala Internals...............................................................................................................................................598
SQL...................................................................................................................................................................601
Partitioned Tables.............................................................................................................................................602
HBase...............................................................................................................................................................602

Impala Release Notes...........................................................................................604

Impala Release Notes.......................................................................................................................................604
New Features in Apache Impala.......................................................................................................................604
Further Information Available in Standalone CDH Release Notes......................................................................................604
New Features in Impala Version 2.4.x / CDH 5.6.x.............................................................................................................604
New Features in Impala Version 2.3.x / CDH 5.5.x.............................................................................................................604
New Features in Impala Version 2.2.9 / CDH 5.4.9............................................................................................................607
New Features in Impala Version 2.2.8 / CDH 5.4.8............................................................................................................607
New Features in Impala Version 2.2.7 / CDH 5.4.7............................................................................................................607
New Features in Impala Version 2.2.x for CDH 5.4.5.........................................................................................................607
New Features in Impala 2.2.x for CDH 5.4.3 and 5.4.4......................................................................................................607
New Features in Impala for CDH 5.4.x...............................................................................................................................607
New Features in Impala Version 2.2.0 / CDH 5.4.0............................................................................................................608
New Features in Impala Version 2.1.7 / CDH 5.3.9............................................................................................................610
New Features in Impala Version 2.1.6 / CDH 5.3.8............................................................................................................610
New Features in Impala Version 2.1.5 / CDH 5.3.6............................................................................................................610
New Features in Impala Version 2.1.4 / CDH 5.3.4............................................................................................................610
New Features in Impala Version 2.1.3 / CDH 5.3.3............................................................................................................610
New Features in Impala Version 2.1.2 / CDH 5.3.2............................................................................................................610
New Features in Impala Version 2.1.1 / CDH 5.3.1............................................................................................................611
New Features in Impala Version 2.1.0 / CDH 5.3.0............................................................................................................611
New Features in Impala Version 2.0.5 / CDH 5.2.6............................................................................................................611
New Features in Impala Version 2.0.4 / CDH 5.2.5............................................................................................................611
New Features in Impala Version 2.0.3 / CDH 5.2.4............................................................................................................611
New Features in Impala Version 2.0.2 / CDH 5.2.3............................................................................................................611
New Features in Impala Version 2.0.1 / CDH 5.2.1............................................................................................................612
New Features in Impala Version 2.0.0 / CDH 5.2.0............................................................................................................612
New Features in Impala Version 1.4.4 / CDH 5.1.5............................................................................................................614
New Features in Impala Version 1.4.3 / CDH 5.1.4............................................................................................................614
New Features in Impala Version 1.4.2 / CDH 5.1.3............................................................................................................614
New Features in Impala Version 1.4.1 / CDH 5.1.2............................................................................................................614
New Features in Impala Version 1.4.0 / CDH 5.1.0............................................................................................................614
New Features in Impala Version 1.3.3 / CDH 5.0.5............................................................................................................616
New Features in Impala Version 1.3.2 / CDH 5.0.4............................................................................................................616
New Features in Impala Version 1.3.1 / CDH 5.0.3............................................................................................................616
New Features in Impala Version 1.3.0 / CDH 5.0.0............................................................................................................617
New Features in Impala Version 1.2.4...............................................................................................................................617
New Features in Impala Version 1.2.3...............................................................................................................................618
New Features in Impala Version 1.2.2...............................................................................................................................618
New Features in Impala Version 1.2.1...............................................................................................................................619
New Features in Impala Version 1.2.0 (Beta).....................................................................................................................621
New Features in Impala Version 1.1.1...............................................................................................................................622
New Features in Impala Version 1.1..................................................................................................................................622
New Features in Impala Version 1.0.1...............................................................................................................................623
New Features in Impala Version 1.0..................................................................................................................................624
New Features in Version 0.7 of the Impala Beta Release...................................................................................................624
New Features in Version 0.6 of the Impala Beta Release...................................................................................................624
New Features in Version 0.5 of the Impala Beta Release...................................................................................................624
New Features in Version 0.4 of the Impala Beta Release...................................................................................................624
New Features in Version 0.3 of the Impala Beta Release...................................................................................................625
New Features in Version 0.2 of the Impala Beta Release...................................................................................................625
Incompatible Changes and Limitations in Apache Impala................................................................................625
Further Information Available in Standalone CDH Release Notes......................................................................................625
Incompatible Changes Introduced in Impala for CDH 5.6.x / Impala 2.4.x........................................................................625
Incompatible Changes Introduced in Impala for CDH 5.5.x / Impala 2.3.x........................................................................625
Incompatible Changes Introduced in Impala for CDH 5.4.x (CDH 5.4.1 through CDH 5.4.9)..............................................626
Incompatible Changes Introduced in Impala 2.2.0 / CDH 5.4.0.........................................................................................626
Incompatible Changes Introduced in Cloudera Impala 2.1.7 / CDH 5.3.9..........................................................................626
Incompatible Changes Introduced in Impala 2.1.6 / CDH 5.3.8.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.5 / CDH 5.3.6.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.4 / CDH 5.3.4.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.3 / CDH 5.3.3.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.2 / CDH 5.3.2.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.1 / CDH 5.3.1.........................................................................................627
Incompatible Changes Introduced in Impala 2.1.0 / CDH 5.3.0.........................................................................................627
Incompatible Changes Introduced in Impala 2.0.5 / CDH 5.2.6.........................................................................................628
Incompatible Changes Introduced in Impala 2.0.4 / CDH 5.2.5.........................................................................................628
Incompatible Changes Introduced in Impala 2.0.3 / CDH 5.2.4.........................................................................................628
Incompatible Changes Introduced in Impala 2.0.2 / CDH 5.2.3.........................................................................................628
Incompatible Changes Introduced in Impala 2.0.1 / CDH 5.2.1.........................................................................................628
Incompatible Changes Introduced in Impala 2.0.0 / CDH 5.2.0.........................................................................................628
Incompatible Changes Introduced in Impala 1.4.4 / CDH 5.1.5.........................................................................................630
Incompatible Changes Introduced in Impala 1.4.3 / CDH 5.1.4.........................................................................................630
Incompatible Changes Introduced in Impala 1.4.2 / CDH 5.1.3.........................................................................................630
Incompatible Changes Introduced in Impala 1.4.1 / CDH 5.1.2.........................................................................................630
Incompatible Changes Introduced in Impala 1.4.0 / CDH 5.1.0.........................................................................................630
Incompatible Changes Introduced in Impala 1.3.3 / CDH 5.0.5.........................................................................................631
Incompatible Changes Introduced in Impala 1.3.2 / CDH 5.0.4.........................................................................................631
Incompatible Changes Introduced in Impala 1.3.1 / CDH 5.0.3.........................................................................................631
Incompatible Changes Introduced in Impala 1.3.0 / CDH 5.0.0.........................................................................................631
Incompatible Changes Introduced in Impala 1.2.4............................................................................................................632
Incompatible Changes Introduced in Impala 1.2.3............................................................................................................632
Incompatible Changes Introduced in Impala 1.2.2............................................................................................................632
Incompatible Changes Introduced in Impala 1.2.1............................................................................................................633
Incompatible Changes Introduced in Impala 1.2.0 (Beta)..................................................................................................633
Incompatible Changes Introduced in Impala 1.1.1............................................................................................................634
Incompatible Change Introduced in Impala 1.1.................................................................................................................634
Incompatible Changes Introduced in Impala 1.0...............................................................................................................634
Incompatible Change Introduced in Version 0.7 of the Cloudera Impala Beta Release.....................................................635
Incompatible Change Introduced in Version 0.6 of the Cloudera Impala Beta Release.....................................................635
Incompatible Change Introduced in Version 0.4 of the Cloudera Impala Beta Release.....................................................635
Incompatible Change Introduced in Version 0.3 of the Cloudera Impala Beta Release.....................................................635
Known Issues and Workarounds in Impala......................................................................................................635
Further Information Available in Standalone CDH Release Notes......................................................................................635
Impala Known Issues: Crashes and Hangs.........................................................................................................................635
Impala Known Issues: Performance...................................................................................................................................636
Impala Known Issues: Usability..........................................................................................................................................636
Impala Known Issues: JDBC and ODBC Drivers..................................................................................................................636
Impala Known Issues: Security...........................................................................................................................................636
Impala Known Issues: Supportability.................................................................................................................................637
Impala Known Issues: Resources........................................................................................................................................637
Impala Known Issues: Correctness.....................................................................................................................................638
Impala Known Issues: Metadata........................................................................................................................................639
Impala Known Issues: Interoperability...............................................................................................................................640
Impala Known Issues: Limitations......................................................................................................................................642
Impala Known Issues: Miscellaneous / Older Issues..........................................................................................................642
Fixed Issues in Apache Impala..........................................................................................................................643
Further Information Available in Standalone CDH Release Notes......................................................................................644
Issues Fixed in Impala for CDH 5.6.0..................................................................................................................................644
Issues Fixed in Impala for CDH 5.5.2..................................................................................................................................644
Issues Fixed in Impala for CDH 5.5.1..................................................................................................................................647
Issues Fixed in Impala for CDH 5.5.0..................................................................................................................................647
Issues Fixed in Impala for CDH 5.4.9..................................................................................................................................647
Issues Fixed in Impala for CDH 5.4.8..................................................................................................................................649
Issues Fixed in Impala for CDH 5.4.7..................................................................................................................................650
Issues Fixed in Impala for CDH 5.4.5..................................................................................................................................652
Issues Fixed in Impala for CDH 5.4.3..................................................................................................................................653
Issues Fixed in Impala for CDH 5.4.1..................................................................................................................................654
Issues Fixed in the 2.2.0 Release / CDH 5.4.0.....................................................................................................................654
Issues Fixed in the 2.1.7 Release / CDH 5.3.9.....................................................................................................................656
Issues Fixed in the 2.1.6 Release / CDH 5.3.8.....................................................................................................................657
Issues Fixed in the 2.1.5 Release / CDH 5.3.6.....................................................................................................................658
Issues Fixed in the 2.1.4 Release / CDH 5.3.4.....................................................................................................................658
Issues Fixed in the 2.1.3 Release / CDH 5.3.3.....................................................................................................................659
Issues Fixed in the 2.1.2 Release / CDH 5.3.2.....................................................................................................................660
Issues Fixed in the 2.1.1 Release / CDH 5.3.1.....................................................................................................................661
Issues Fixed in the 2.1.0 Release / CDH 5.3.0.....................................................................................................................661
Issues Fixed in the 2.0.5 Release / CDH 5.2.6.....................................................................................................................662
Issues Fixed in the 2.0.4 Release / CDH 5.2.5.....................................................................................................................662
Issues Fixed in the 2.0.3 Release / CDH 5.2.4.....................................................................................................................663
Issues Fixed in the 2.0.2 Release / CDH 5.2.3.....................................................................................................................663
Issues Fixed in the 2.0.1 Release / CDH 5.2.1.....................................................................................................................664
Issues Fixed in the 2.0.0 Release / CDH 5.2.0.....................................................................................................................664
Issues Fixed in the 1.4.4 Release / CDH 5.1.5.....................................................................................................................666
Issues Fixed in the 1.4.3 Release / CDH 5.1.4.....................................................................................................................667
Issues Fixed in the 1.4.2 Release / CDH 5.1.3.....................................................................................................................667
Issues Fixed in the 1.4.1 Release / CDH 5.1.2.....................................................................................................................667
Issues Fixed in the 1.4.0 Release / CDH 5.1.0.....................................................................................................................668
Issues Fixed in the 1.3.3 Release / CDH 5.0.5.....................................................................................................................670
Issues Fixed in the 1.3.2 Release / CDH 5.0.4.....................................................................................................................670
Issues Fixed in the 1.3.1 Release / CDH 5.0.3.....................................................................................................................670
Issues Fixed in the 1.3.0 Release / CDH 5.0.0.....................................................................................................................672
Issues Fixed in the 1.2.4 Release........................................................................................................................................674
Issues Fixed in the 1.2.3 Release........................................................................................................................................675
Issues Fixed in the 1.2.2 Release........................................................................................................................................676
Issues Fixed in the 1.2.1 Release........................................................................................................................................677
Issues Fixed in the 1.2.0 Beta Release................................................................................................................................677
Issues Fixed in the 1.1.1 Release........................................................................................................................................677
Issues Fixed in the 1.1.0 Release........................................................................................................................................679
Issues Fixed in the 1.0.1 Release........................................................................................................................................679
Issues Fixed in the 1.0 GA Release.....................................................................................................................................681
Issues Fixed in Version 0.7 of the Beta Release..................................................................................................................684
Issues Fixed in Version 0.6 of the Beta Release..................................................................................................................685
Issues Fixed in Version 0.5 of the Beta Release..................................................................................................................686
Issues Fixed in Version 0.4 of the Beta Release..................................................................................................................687
Issues Fixed in Version 0.3 of the Beta Release..................................................................................................................687
Issues Fixed in Version 0.2 of the Beta Release..................................................................................................................688

Appendix: Apache License, Version 2.0.................................................................689

Introducing Apache Impala

Introducing Apache Impala

Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS, HBase, or the
Amazon Simple Storage Service (S3). In addition to using the same unified storage platform, Impala also uses the same
metadata, SQL syntax (Hive SQL), ODBC driver, and user interface (Impala query UI in Hue) as Apache Hive. This provides
a familiar and unified platform for real-time or batch-oriented queries.
Impala is an addition to tools available for querying big data. Impala does not replace the batch processing frameworks
built on MapReduce such as Hive. Hive and other frameworks built on MapReduce are best suited for long running
batch jobs, such as those involving batch processing of Extract, Transform, and Load (ETL) type jobs.

Note: Impala graduated from the Apache Incubator on November 15, 2017. In places where the
documentation formerly referred to “Cloudera Impala”, now the official name is “Apache Impala”.

Impala Benefits
Impala provides:
• Familiar SQL interface that data scientists and analysts already know.
• Ability to query high volumes of data (“big data”) in Apache Hadoop.
• Distributed queries in a cluster environment, for convenient scaling and to make use of cost-effective commodity
hardware.
• Ability to share data files between different components with no copy or export/import step; for example, to
write with Pig, transform with Hive and query with Impala. Impala can read from and write to Hive tables, enabling
simple data interchange using Impala for analytics on Hive-produced data.
• Single system for big data processing and analytics, so customers can avoid costly modeling and ETL just for
analytics.

How Impala Works with CDH

The following graphic illustrates how Impala is positioned in the broader Cloudera environment:

The Impala solution is composed of the following components:

• Clients - Entities including Hue, ODBC clients, JDBC clients, and the Impala Shell can all interact with Impala. These
interfaces are typically used to issue queries or complete administrative tasks such as connecting to Impala.

16 | Apache Impala Guide

 Introducing Apache Impala

• Hive Metastore - Stores information about the data available to Impala. For example, the metastore lets Impala
know what databases are available and what the structure of those databases is. As you create, drop, and alter
schema objects, load data into tables, and so on through Impala SQL statements, the relevant metadata changes
are automatically broadcast to all Impala nodes by the dedicated catalog service introduced in Impala 1.2.
• Impala - This process, which runs on DataNodes, coordinates and executes queries. Each instance of Impala can
receive, plan, and coordinate queries from Impala clients. Queries are distributed among Impala nodes, and these
nodes then act as workers, executing parallel query fragments.
• HBase and HDFS - Storage for data to be queried.
Queries executed using Impala are handled as follows:
1. User applications send SQL queries to Impala through ODBC or JDBC, which provide standardized querying
interfaces. The user application may connect to any impalad in the cluster. This impalad becomes the coordinator
for the query.
2. Impala parses the query and analyzes it to determine what tasks need to be performed by impalad instances
across the cluster. Execution is planned for optimal efficiency.
3. Services such as HDFS and HBase are accessed by local impalad instances to provide data.
4. Each impalad returns data to the coordinating impalad, which sends these results to the client.

Primary Impala Features

Impala provides support for:
• Most common SQL-92 features of Hive Query Language (HiveQL) including SELECT, joins, and aggregate functions.
• HDFS, HBase, and Amazon Simple Storage System (S3) storage, including:
– HDFS file formats: delimited text files, Parquet, Avro, SequenceFile, and RCFile.
– Compression codecs: Snappy, GZIP, Deflate, BZIP.
• Common data access interfaces including:
– JDBC driver.
– ODBC driver.
– Hue Beeswax and the Impala Query UI.
• impala-shell command-line interface.
• Kerberos authentication.

Apache Impala Guide | 17

Impala Concepts and Architecture

Impala Concepts and Architecture

The following sections provide background information to help you become productive using Impala and its features.
Where appropriate, the explanations include context to help understand how aspects of Impala relate to other
technologies you might already be familiar with, such as relational database management systems and data warehouses,
or other Hadoop components such as Hive, HDFS, and HBase.

Components of the Impala Server

The Impala server is a distributed, massively parallel processing (MPP) database engine. It consists of different daemon
processes that run on specific hosts within your CDH cluster.

The Impala Daemon

The core Impala component is a daemon process that runs on each DataNode of the cluster, physically represented
by the impalad process. It reads and writes to data files; accepts queries transmitted from the impala-shell
command, Hue, JDBC, or ODBC; parallelizes the queries and distributes work across the cluster; and transmits
intermediate query results back to the central coordinator node.
You can submit a query to the Impala daemon running on any DataNode, and that instance of the daemon serves as
the coordinator node for that query. The other nodes transmit partial results back to the coordinator, which constructs
the final result set for a query. When running experiments with functionality through the impala-shell command,
you might always connect to the same Impala daemon for convenience. For clusters running production workloads,
you might load-balance by submitting each query to a different Impala daemon in round-robin style, using the JDBC
or ODBC interfaces.
The Impala daemons are in constant communication with the statestore, to confirm which nodes are healthy and can
accept new work.
They also receive broadcast messages from the catalogd daemon (introduced in Impala 1.2) whenever any Impala
node in the cluster creates, alters, or drops any type of object, or when an INSERT or LOAD DATA statement is processed
through Impala. This background communication minimizes the need for REFRESH or INVALIDATE METADATA
statements that were needed to coordinate metadata across nodes prior to Impala 1.2.
Related information: Modifying Impala Startup Options on page 43, Starting Impala on page 42, Setting the Idle Query
and Idle Session Timeouts for impalad on page 84, Ports Used by Impala on page 583, Using Impala through a Proxy
for High Availability on page 85

The Impala Statestore

The Impala component known as the statestore checks on the health of Impala daemons on all the DataNodes in a
cluster, and continuously relays its findings to each of those daemons. It is physically represented by a daemon process
named statestored; you only need such a process on one host in the cluster. If an Impala daemon goes offline due
to hardware failure, network error, software issue, or other reason, the statestore informs all the other Impala daemons
so that future queries can avoid making requests to the unreachable node.
Because the statestore's purpose is to help when things go wrong and to broadcast metadata to coordinators, it is not
always critical to the normal operation of an Impala cluster. If the statestore is not running or becomes unreachable,
the Impala daemons continue running and distributing work among themselves as usual when working with the data
known to Impala. The cluster just becomes less robust if other Impala daemons fail, and metadata becomes less
consistent as it changes while the statestore is offline. When the statestore comes back online, it re-establishes
communication with the Impala daemons and resumes its monitoring and broadcasting functions.
If you issue a DDL statement while the statestore is down, the queries that access the new object the DDL created will
fail.

18 | Apache Impala Guide

 Impala Concepts and Architecture

Most considerations for load balancing and high availability apply to the impalad daemon. The statestored and
catalogd daemons do not have special requirements for high availability, because problems with those daemons do
not result in data loss. If those daemons become unavailable due to an outage on a particular host, you can stop the
Impala service, delete the Impala StateStore and Impala Catalog Server roles, add the roles on a different host, and
restart the Impala service.
Related information:
Scalability Considerations for the Impala Statestore on page 516, Modifying Impala Startup Options on page 43, Starting
Impala on page 42, Increasing the Statestore Timeout on page 84, Ports Used by Impala on page 583

The Impala Catalog Service

The Impala component known as the catalog service relays the metadata changes from Impala SQL statements to all
the Impala daemons in a cluster. It is physically represented by a daemon process named catalogd; you only need
such a process on one host in the cluster. Because the requests are passed through the statestore daemon, it makes
sense to run the statestored and catalogd services on the same host.
The catalog service avoids the need to issue REFRESH and INVALIDATE METADATA statements when the metadata
changes are performed by statements issued through Impala. When you create a table, load data, and so on through
Hive, you do need to issue REFRESH or INVALIDATE METADATA on an Impala node before executing a query there.
This feature touches a number of aspects of Impala:
• See Installing Impala on page 27, Upgrading Impala on page 38 and Starting Impala on page 42, for usage
information for the catalogd daemon.
• The REFRESH and INVALIDATE METADATA statements are not needed when the CREATE TABLE, INSERT, or
other table-changing or data-changing operation is performed through Impala. These statements are still needed
if such operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the
statements only need to be issued on one Impala node rather than on all nodes. See REFRESH Statement on page
277 and INVALIDATE METADATA Statement on page 271 for the latest usage information for those statements.

By default, the metadata loading and caching on startup happens asynchronously, so Impala can begin accepting
requests promptly. To enable the original behavior, where Impala waited until all metadata was loaded before accepting
any requests, set the catalogd configuration option --load_catalog_in_background=false.
Most considerations for load balancing and high availability apply to the impalad daemon. The statestored and
catalogd daemons do not have special requirements for high availability, because problems with those daemons do
not result in data loss. If those daemons become unavailable due to an outage on a particular host, you can stop the
Impala service, delete the Impala StateStore and Impala Catalog Server roles, add the roles on a different host, and
restart the Impala service.

Note:
In Impala 1.2.4 and higher, you can specify a table name with INVALIDATE METADATA after the table
is created in Hive, allowing you to make individual tables visible to Impala without doing a full reload
of the catalog metadata. Impala 1.2.4 also includes other changes to make the metadata broadcast
mechanism faster and more responsive, especially during Impala startup. See New Features in Impala
Version 1.2.4 on page 617 for details.

Related information: Modifying Impala Startup Options on page 43, Starting Impala on page 42, Ports Used by Impala
on page 583

Developing Impala Applications

The core development language with Impala is SQL. You can also use Java or other languages to interact with Impala
through the standard JDBC and ODBC interfaces used by many business intelligence tools. For specialized kinds of
analysis, you can supplement the SQL built-in functions by writing user-defined functions (UDFs) in C++ or Java.

Apache Impala Guide | 19

Impala Concepts and Architecture

Overview of the Impala SQL Dialect

The Impala SQL dialect is highly compatible with the SQL syntax used in the Apache Hive component (HiveQL). As such,
it is familiar to users who are already familiar with running SQL queries on the Hadoop infrastructure. Currently, Impala
SQL supports a subset of HiveQL statements, data types, and built-in functions. Impala also includes additional built-in
functions for common industry features, to simplify porting SQL from non-Hadoop systems.
For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the
SQL dialect might seem familiar:
• The SELECT statement includes familiar clauses such as WHERE, GROUP BY, ORDER BY, and WITH. You will find
familiar notions such as joins, built-in functions for processing strings, numbers, and dates, aggregate functions,
subqueries, and comparison operators such as IN() and BETWEEN. The SELECT statement is the place where
SQL standards compliance is most important.
• From the data warehousing world, you will recognize the notion of partitioned tables. One or more columns serve
as partition keys, and the data is physically arranged so that queries that refer to the partition key columns in the
WHERE clause can skip partitions that do not match the filter conditions. For example, if you have 10 years worth
of data and use a clause such as WHERE year = 2015, WHERE year > 2010, or WHERE year IN (2014,
2015), Impala skips all the data for non-matching years, greatly reducing the amount of I/O for the query.

• In Impala 1.2 and higher, UDFs let you perform custom comparisons and transformation logic during SELECT and
INSERT...SELECT statements.

For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the
SQL dialect might require some learning and practice for you to become proficient in the Hadoop environment:
• Impala SQL is focused on queries and includes relatively little DML. There is no UPDATE or DELETE statement.
Stale data is typically discarded (by DROP TABLE or ALTER TABLE ... DROP PARTITION statements) or
replaced (by INSERT OVERWRITE statements).
• All data creation is done by INSERT statements, which typically insert data in bulk by querying from other tables.
There are two variations, INSERT INTO which appends to the existing data, and INSERT OVERWRITE which
replaces the entire contents of a table or partition (similar to TRUNCATE TABLE followed by a new INSERT).
Although there is an INSERT ... VALUES syntax to create a small number of values in a single statement, it is
far more efficient to use the INSERT ... SELECT to copy and transform large amounts of data from one table
to another in a single operation.
• You often construct Impala table definitions and data files in some other environment, and then attach Impala so
that it can run real-time queries. The same data files and table metadata are shared with other components of
the Hadoop ecosystem. In particular, Impala can access tables created by Hive or data inserted by Hive, and Hive
can access tables and data produced by Impala. Many other Hadoop components can write files in formats such
as Parquet and Avro, that can then be queried by Impala.
• Because Hadoop and Impala are focused on data warehouse-style operations on large data sets, Impala SQL
includes some idioms that you might find in the import utilities for traditional database systems. For example,
you can create a table that reads comma-separated or tab-separated text files, specifying the separator in the
CREATE TABLE statement. You can create external tables that read existing data files but do not move or transform
them.
• Because Impala reads large quantities of data that might not be perfectly tidy and predictable, it does not require
length constraints on string data types. For example, you can define a database column as STRING with unlimited
length, rather than CHAR(1) or VARCHAR(64). (Although in Impala 2.0 and later, you can also use
length-constrained CHAR and VARCHAR types.)

Related information: Impala SQL Language Reference on page 116, especially Impala SQL Statements on page 214 and
Impala Built-In Functions on page 338

20 | Apache Impala Guide

 Impala Concepts and Architecture

Overview of Impala Programming Interfaces

You can connect and submit requests to the Impala daemons through:
• The impala-shell interactive command interpreter.
• The Hue web-based user interface.
• JDBC.
• ODBC.
With these options, you can use Impala in heterogeneous environments, with JDBC or ODBC applications running on
non-Linux platforms. You can also use Impala on combination with various Business Intelligence tools that use the
JDBC and ODBC interfaces.
Each impalad daemon process, running on separate nodes in a cluster, listens to several ports for incoming requests.
Requests from impala-shell and Hue are routed to the impalad daemons through the same port. The impalad
daemons listen on separate ports for JDBC and ODBC requests.

How Impala Fits Into the Hadoop Ecosystem

Impala makes use of many familiar components within the Hadoop ecosystem. Impala can interchange data with other
Hadoop components, as both a consumer and a producer, so it can fit in flexible ways into your ETL and ELT pipelines.

How Impala Works with Hive

A major Impala goal is to make SQL-on-Hadoop operations fast and efficient enough to appeal to new categories of
users and open up Hadoop to new types of use cases. Where practical, it makes use of existing Apache Hive infrastructure
that many Hadoop users already have in place to perform long-running, batch-oriented SQL queries.
In particular, Impala keeps its table definitions in a traditional MySQL or PostgreSQL database known as the metastore,
the same database where Hive keeps this type of data. Thus, Impala can access tables defined or loaded by Hive, as
long as all columns use Impala-supported data types, file formats, and compression codecs.
The initial focus on query features and performance means that Impala can read more types of data with the SELECT
statement than it can write with the INSERT statement. To query data using the Avro, RCFile, or SequenceFile file
formats, you load the data using Hive.
The Impala query optimizer can also make use of table statistics and column statistics. Originally, you gathered this
information with the ANALYZE TABLE statement in Hive; in Impala 1.2.2 and higher, use the Impala COMPUTE STATS
statement instead. COMPUTE STATS requires less setup, is more reliable, and does not require switching back and
forth between impala-shell and the Hive shell.

Overview of Impala Metadata and the Metastore

As discussed in How Impala Works with Hive on page 21, Impala maintains information about table definitions in a
central database known as the metastore. Impala also tracks other metadata for the low-level characteristics of data
files:
• The physical locations of blocks within HDFS.
For tables with a large volume of data and/or many partitions, retrieving all the metadata for a table can be
time-consuming, taking minutes in some cases. Thus, each Impala node caches all of this metadata to reuse for future
queries against the same table.
If the table definition or the data in the table is updated, all other Impala daemons in the cluster must receive the
latest metadata, replacing the obsolete cached metadata, before issuing a query against that table. In Impala 1.2 and
higher, the metadata update is automatic, coordinated through the catalogd daemon, for all DDL and DML statements
issued through Impala. See The Impala Catalog Service on page 19 for details.
For DDL and DML issued through Hive, or changes made manually to files in HDFS, you still use the REFRESH statement
(when new data files are added to existing tables) or the INVALIDATE METADATA statement (for entirely new tables,
or after dropping a table, performing an HDFS rebalance operation, or deleting data files). Issuing INVALIDATE

Apache Impala Guide | 21

Impala Concepts and Architecture

METADATA by itself retrieves metadata for all the tables tracked by the metastore. If you know that only specific tables
have been changed outside of Impala, you can issue REFRESH table_name for each affected table to only retrieve
the latest metadata for those tables.

How Impala Uses HDFS

Impala uses the distributed filesystem HDFS as its primary data storage medium. Impala relies on the redundancy
provided by HDFS to guard against hardware or network outages on individual nodes. Impala table data is physically
represented as data files in HDFS, using familiar HDFS file formats and compression codecs. When data files are present
in the directory for a new table, Impala reads them all, regardless of file name. New data is added in files with names
controlled by Impala.

How Impala Uses HBase

HBase is an alternative to HDFS as a storage medium for Impala data. It is a database storage system built on top of
HDFS, without built-in SQL support. Many Hadoop users already have it configured and store large (often sparse) data
sets in it. By defining tables in Impala and mapping them to equivalent tables in HBase, you can query the contents of
the HBase tables through Impala, and even perform join queries including both Impala and HBase tables. See Using
Impala to Query HBase Tables on page 557 for details.

22 | Apache Impala Guide

 Planning for Impala Deployment

Planning for Impala Deployment

Before you set up Impala in production, do some planning to make sure that your hardware setup has sufficient capacity,
that your cluster topology is optimal for Impala queries, and that your schema design and ETL processes follow the
best practices for Impala.

Impala Requirements
To perform as expected, Impala depends on the availability of the software, hardware, and configurations described
in the following sections.

Product Compatibility Matrix

The ultimate source of truth about compatibility between various versions of CDH, Cloudera Manager, and various
CDH components is the online Product Compatibility Matrix.
For Impala, see the Impala compatibility matrix page.

Supported Operating Systems

The relevant supported operating systems and versions for Impala are the same as for the corresponding CDH 5
platforms. For details, see the Supported Operating Systems page for CDH 5.

Hive Metastore and Related Configuration

Impala can interoperate with data stored in Hive, and uses the same infrastructure as Hive for tracking metadata about
schema objects such as tables and columns. The following components are prerequisites for Impala:
• MySQL or PostgreSQL, to act as a metastore database for both Impala and Hive.

Note:
Installing and configuring a Hive metastore is an Impala requirement. Impala does not work
without the metastore database. For the process of installing and configuring the metastore, see
Installing Impala on page 27.
Always configure a Hive metastore service rather than connecting directly to the metastore
database. The Hive metastore service is required to interoperate between possibly different levels
of metastore APIs used by CDH and Impala, and avoids known issues with connecting directly to
the metastore database. The Hive metastore service is set up for you by default if you install
through Cloudera Manager 4.5 or higher.
A summary of the metastore installation process is as follows:
• Install a MySQL or PostgreSQL database. Start the database if it is not started after installation.
• Download the MySQL connector or the PostgreSQL connector and place it in the
/usr/share/java/ directory.
• Use the appropriate command line tool for your database to create the metastore database.
• Use the appropriate command line tool for your database to grant privileges for the metastore
database to the hive user.
• Modify hive-site.xml to include information matching your particular database: its URL,
user name, and password. You will copy the hive-site.xml file to the Impala Configuration
Directory later in the Impala installation process.

• Optional: Hive. Although only the Hive metastore database is required for Impala to function, you might install
Hive on some client machines to create and load data into tables that use certain file formats. See How Impala

Apache Impala Guide | 23

Planning for Impala Deployment

Works with Hadoop File Formats on page 527 for details. Hive does not need to be installed on the same DataNodes
as Impala; it just needs access to the same metastore database.

Java Dependencies
Although Impala is primarily written in C++, it does use Java to communicate with various Hadoop components:
• The officially supported JVM for Impala is the Oracle JVM. Other JVMs might cause issues, typically resulting in a
failure at impalad startup. In particular, the JamVM used by default on certain levels of Ubuntu systems can
cause impalad to fail to start.
• Internally, the impalad daemon relies on the JAVA_HOME environment variable to locate the system Java libraries.
Make sure the impalad service is not run from an environment with an incorrect setting for this variable.
• All Java dependencies are packaged in the impala-dependencies.jar file, which is located at
/usr/lib/impala/lib/. These map to everything that is built under fe/target/dependency.

Networking Configuration Requirements

As part of ensuring best performance, Impala attempts to complete tasks on local data, as opposed to using network
connections to work with remote data. To support this goal, Impala matches the hostname provided to each Impala
daemon with the IP address of each DataNode by resolving the hostname flag to an IP address. For Impala to work
with local data, use a single IP interface for the DataNode and the Impala daemon on each machine. Ensure that the
Impala daemon's hostname flag resolves to the IP address of the DataNode. For single-homed machines, this is usually
automatic, but for multi-homed machines, ensure that the Impala daemon's hostname resolves to the correct interface.
Impala tries to detect the correct hostname at start-up, and prints the derived hostname at the start of the log in a
message of the form:

Using hostname: impala-daemon-1.example.com

In the majority of cases, this automatic detection works correctly. If you need to explicitly set the hostname, do so by
setting the --hostname flag.

Hardware Requirements
During join operations, portions of data from each joined table are loaded into memory. Data sets can be very large,
so ensure your hardware has sufficient memory to accommodate the joins you anticipate completing.
While requirements vary according to data set size, the following is generally recommended:
• CPU - Impala version 2.2 and higher uses the SSSE3 instruction set, which is included in newer processors.

Note: This required level of processor is the same as in Impala version 1.x. The Impala 2.0 and
2.1 releases had a stricter requirement for the SSE4.1 instruction set, which has now been relaxed.

• Memory - 128 GB or more recommended, ideally 256 GB or more. If the intermediate results during query
processing on a particular node exceed the amount of memory available to Impala on that node, the query writes
temporary work data to disk, which can lead to long query times. Note that because the work is parallelized, and
intermediate results for aggregate queries are typically smaller than the original data, Impala can query and join
tables that are much larger than the memory available on an individual node.
• Storage - DataNodes with 12 or more disks each. I/O speeds are often the limiting factor for disk performance
with Impala. Ensure that you have sufficient disk space to store the data Impala will be querying.

User Account Requirements

Impala creates and uses a user and group named impala. Do not delete this account or group and do not modify the
account's or group's permissions and rights. Ensure no existing systems obstruct the functioning of these accounts and
groups. For example, if you have scripts that delete user accounts not in a white-list, add these accounts to the list of
permitted accounts.

24 | Apache Impala Guide

 Planning for Impala Deployment

For correct file deletion during DROP TABLE operations, Impala must be able to move files to the HDFS trashcan. You
might need to create an HDFS directory /user/impala, writeable by the impala user, so that the trashcan can be
created. Otherwise, data files might remain behind after a DROP TABLE statement.
Impala should not run as root. Best Impala performance is achieved using direct reads, but root is not permitted to
use direct reads. Therefore, running Impala as root negatively affects performance.
By default, any user can connect to Impala and access all the associated databases and tables. You can enable
authorization and authentication based on the Linux OS user who connects to the Impala server, and the associated
groups for that user. Impala Security on page 91 for details. These security features do not change the underlying file
permission requirements; the impala user still needs to be able to access the data files.

Guidelines for Designing Impala Schemas

The guidelines in this topic help you to construct an optimized and scalable schema, one that integrates well with your
existing data management processes. Use these guidelines as a checklist when doing any proof-of-concept work,
porting exercise, or before deploying to production.
If you are adapting an existing database or Hive schema for use with Impala, read the guidelines in this section and
then see Porting SQL from Other Database Systems to Impala on page 464 for specific porting and compatibility tips.

Prefer binary file formats over text-based formats.

To save space and improve memory usage and query performance, use binary file formats for any large or intensively
queried tables. Parquet file format is the most efficient for data warehouse-style analytic queries. Avro is the other
binary file format that Impala supports, that you might already have as part of a Hadoop ETL pipeline.
Although Impala can create and query tables with the RCFile and SequenceFile file formats, such tables are relatively
bulky due to the text-based nature of those formats, and are not optimized for data warehouse-style queries due to
their row-oriented layout. Impala does not support INSERT operations for tables with these file formats.
Guidelines:
• For an efficient and scalable format for large, performance-critical tables, use the Parquet file format.
• To deliver intermediate data during the ETL process, in a format that can also be used by other Hadoop components,
Avro is a reasonable choice.
• For convenient import of raw data, use a text table instead of RCFile or SequenceFile, and convert to Parquet in
a later stage of the ETL process.

Use Snappy compression where practical.

Snappy compression involves low CPU overhead to decompress, while still providing substantial space savings. In cases
where you have a choice of compression codecs, such as with the Parquet and Avro file formats, use Snappy compression
unless you find a compelling reason to use a different codec.

Prefer numeric types over strings.

If you have numeric values that you could treat as either strings or numbers (such as YEAR, MONTH, and DAY for partition
key columns), define them as the smallest applicable integer types. For example, YEAR can be SMALLINT, MONTH and
DAY can be TINYINT. Although you might not see any difference in the way partitioned tables or text files are laid out
on disk, using numeric types will save space in binary formats such as Parquet, and in memory when doing queries,
particularly resource-intensive queries such as joins.

Partition, but don't over-partition.

Partitioning is an important aspect of performance tuning for Impala. Follow the procedures in Partitioning for Impala
Tables on page 522 to set up partitioning for your biggest, most intensively queried tables.
If you are moving to Impala from a traditional database system, or just getting started in the Big Data field, you might
not have enough data volume to take advantage of Impala parallel queries with your existing partitioning scheme. For

Apache Impala Guide | 25

Planning for Impala Deployment

example, if you have only a few tens of megabytes of data per day, partitioning by YEAR, MONTH, and DAY columns
might be too granular. Most of your cluster might be sitting idle during queries that target a single day, or each node
might have very little work to do. Consider reducing the number of partition key columns so that each partition directory
contains several gigabytes worth of data.
For example, consider a Parquet table where each data file is 1 HDFS block, with a maximum block size of 1 GB. (In
Impala 2.0 and later, the default Parquet block size is reduced to 256 MB. For this exercise, let's assume you have
bumped the size back up to 1 GB by setting the query option PARQUET_FILE_SIZE=1g.) if you have a 10-node cluster,
you need 10 data files (up to 10 GB) to give each node some work to do for a query. But each core on each machine
can process a separate data block in parallel. With 16-core machines on a 10-node cluster, a query could process up
to 160 GB fully in parallel. If there are only a few data files per partition, not only are most cluster nodes sitting idle
during queries, so are most cores on those machines.
You can reduce the Parquet block size to as low as 128 MB or 64 MB to increase the number of files per partition and
improve parallelism. But also consider reducing the level of partitioning so that analytic queries have enough data to
work with.

Always compute stats after loading data.

Impala makes extensive use of statistics about data in the overall table and in each column, to help plan
resource-intensive operations such as join queries and inserting into partitioned Parquet tables. Because this information
is only available after data is loaded, run the COMPUTE STATS statement on a table after loading or replacing data in
a table or partition.
Having accurate statistics can make the difference between a successful operation, or one that fails due to an
out-of-memory error or a timeout. When you encounter performance or capacity issues, always use the SHOW STATS
statement to check if the statistics are present and up-to-date for all tables in the query.
When doing a join query, Impala consults the statistics for each joined table to determine their relative sizes and to
estimate the number of rows produced in each join stage. When doing an INSERT into a Parquet table, Impala consults
the statistics for the source table to determine how to distribute the work of constructing the data files for each
partition.
See COMPUTE STATS Statement on page 226 for the syntax of the COMPUTE STATS statement, and Table and Column
Statistics on page 489 for all the performance considerations for table and column statistics.

Verify sensible execution plans with EXPLAIN and SUMMARY.

Before executing a resource-intensive query, use the EXPLAIN statement to get an overview of how Impala intends
to parallelize the query and distribute the work. If you see that the query plan is inefficient, you can take tuning steps
such as changing file formats, using partitioned tables, running the COMPUTE STATS statement, or adding query hints.
For information about all of these techniques, see Tuning Impala for Performance on page 479.
After you run a query, you can see performance-related information about how it actually ran by issuing the SUMMARY
command in impala-shell. Prior to Impala 1.4, you would use the PROFILE command, but its highly technical output
was only useful for the most experienced users. SUMMARY, new in Impala 1.4, summarizes the most useful information
for all stages of execution, for all nodes rather than splitting out figures for each node.

26 | Apache Impala Guide

 Installing Impala

Installing Impala
Impala is an open-source add-on to the Cloudera Enterprise Core that returns rapid responses to queries.

Note:
Under CDH 5, Impala is included as part of the CDH installation and no separate steps are needed.
Therefore, the instruction steps in this section apply to CDH 4 only.

What is Included in an Impala Installation

Impala is made up of a set of components that can be installed on multiple nodes throughout your cluster. The key
installation step for performance is to install the impalad daemon (which does most of the query processing work)
on all DataNodes in the cluster.
The Impala package installs these binaries:
• impalad - The Impala daemon. Plans and executes queries against HDFS, HBase, and Amazon S3 data. Run one
impalad process on each node in the cluster that has a DataNode.
• statestored - Name service that tracks location and status of all impalad instances in the cluster. Run one
instance of this daemon on a node in your cluster. Most production deployments run this daemon on the namenode.
• catalogd - Metadata coordination service that broadcasts changes from Impala DDL and DML statements to all
affected Impala nodes, so that new tables, newly loaded data, and so on are immediately visible to queries
submitted through any Impala node. (Prior to Impala 1.2, you had to run the REFRESH or INVALIDATE METADATA
statement on each node to synchronize changed metadata. Now those statements are only required if you perform
the DDL or DML through an external mechanism such as Hive or by uploading data to the Amazon S3 filesystem.)
Run one instance of this daemon on a node in your cluster, preferably on the same host as the statestored
daemon.
• impala-shell - Command-line interface for issuing queries to the Impala daemon. You install this on one or
more hosts anywhere on your network, not necessarily DataNodes or even within the same cluster as Impala. It
can connect remotely to any instance of the Impala daemon.

Before doing the installation, ensure that you have all necessary prerequisites. See Impala Requirements on page 23
for details.

Installing Impala with Cloudera Manager

Before installing Impala through the Cloudera Manager interface, make sure all applicable nodes have the appropriate
hardware configuration and levels of operating system and CDH. See Impala Requirements on page 23 for details.

Note:
To install the latest Impala under CDH 4, upgrade Cloudera Manager to 4.8 or higher. Cloudera Manager
4.8 is the first release that can manage the Impala catalog service introduced in Impala 1.2. Cloudera
Manager 4.8 requires this service to be present, so if you upgrade to Cloudera Manager 4.8, also
upgrade Impala to the most recent version at the same time.

For information on installing Impala in a Cloudera Manager-managed environment, see Installing Impala.

Apache Impala Guide | 27

Installing Impala

Managing your Impala installation through Cloudera Manager has a number of advantages. For example, when you
make configuration changes to CDH components using Cloudera Manager, it automatically applies changes to the
copies of configuration files, such as hive-site.xml, that Impala keeps under /etc/impala/conf. It also sets up
the Hive Metastore service that is required for Impala running under CDH 4.1.
In some cases, depending on the level of Impala, CDH, and Cloudera Manager, you might need to add particular
component configuration details in some of the free-form option fields on the Impala configuration pages within
Cloudera Manager. In Cloudera Manager 4, these fields are labelled Safety Valve; in Cloudera Manager 5, they are
called Advanced Configuration Snippet.

Installing Impala without Cloudera Manager

Before installing Impala manually, make sure all applicable nodes have the appropriate hardware configuration, levels
of operating system and CDH, and any other software prerequisites. See Impala Requirements on page 23 for details.
You can install Impala across many hosts or on one host:
• Installing Impala across multiple machines creates a distributed configuration. For best performance, install Impala
on all DataNodes.
• Installing Impala on a single machine produces a pseudo-distributed cluster.
To install Impala on a host:
1. Install CDH as described in the Installation section of the CDH 5 Installation Guide.
2. Install the Hive metastore somewhere in your cluster, as described in the Hive Installation topic in the CDH 5
Installation Guide. As part of this process, you configure the Hive metastore to use an external database as a
metastore. Impala uses this same database for its own table metadata. You can choose either a MySQL or PostgreSQL
database as the metastore. The process for configuring each type of database is described in the CDH Installation
Guide).
Cloudera recommends setting up a Hive metastore service rather than connecting directly to the metastore
database; this configuration is required when running Impala under CDH 4.1. Make sure the
/etc/impala/conf/hive-site.xml file contains the following setting, substituting the appropriate hostname
for metastore_server_host:

<property>
<name>hive.metastore.uris</name>
<value>thrift://metastore_server_host:9083</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>3600</value>
<description>MetaStore Client socket timeout in seconds</description>
</property>

3. (Optional) If you installed the full Hive component on any host, you can verify that the metastore is configured
properly by starting the Hive console and querying for the list of available tables. Once you confirm that the console
starts, exit the console to continue the installation:

$ hive
Hive history file=/tmp/root/hive_job_log_root_201207272011_678722950.txt
hive> show tables;
table1
table2
hive> quit;
$

4. Confirm that your package management command is aware of the Impala repository settings, as described in
Impala Requirements on page 23. (For CDH 4, this is a different repository than for CDH.) You might need to
download a repo or list file into a system directory underneath /etc.
5. Use one of the following sets of commands to install the Impala package:

28 | Apache Impala Guide

 Installing Impala

For RHEL, Oracle Linux, or CentOS systems:

$ sudo yum install impala # Binaries for daemons

$ sudo yum install impala-server # Service start/stop script
$ sudo yum install impala-state-store # Service start/stop script
$ sudo yum install impala-catalog # Service start/stop script

For SUSE systems:

$ sudo zypper install impala # Binaries for daemons

$ sudo zypper install impala-server # Service start/stop script
$ sudo zypper install impala-state-store # Service start/stop script
$ sudo zypper install impala-catalog # Service start/stop script

For Debian or Ubuntu systems:

$ sudo apt-get install impala # Binaries for daemons

$ sudo apt-get install impala-server # Service start/stop script
$ sudo apt-get install impala-state-store # Service start/stop script
$ sudo apt-get install impala-catalog # Service start/stop script

Note: Cloudera recommends that you not install Impala on any HDFS NameNode. Installing
Impala on NameNodes provides no additional data locality, and executing queries with such a
configuration might cause memory contention and negatively impact the HDFS NameNode.

6. Copy the client hive-site.xml, core-site.xml, hdfs-site.xml, and hbase-site.xml configuration files
to the Impala configuration directory, which defaults to /etc/impala/conf. Create this directory if it does not
already exist.
7. Use one of the following commands to install impala-shell on the machines from which you want to issue
queries. You can install impala-shell on any supported machine that can connect to DataNodes that are running
impalad.

For RHEL/CentOS systems:

$ sudo yum install impala-shell

For SUSE systems:

$ sudo zypper install impala-shell

For Debian/Ubuntu systems:

$ sudo apt-get install impala-shell

8. Complete any required or recommended configuration, as described in Post-Installation Configuration for Impala
on page 30. Some of these configuration changes are mandatory. (They are applied automatically when you install
using Cloudera Manager.)
Once installation and configuration are complete, see Starting Impala on page 42 for how to activate the software on
the appropriate nodes in your cluster.
If this is your first time setting up and using Impala in this cluster, run through some of the exercises in Impala Tutorials
on page 47 to verify that you can do basic operations such as creating tables and querying them.

Apache Impala Guide | 29

Managing Impala

Managing Impala
This section explains how to configure Impala to accept connections from applications that use popular programming
APIs:
• Post-Installation Configuration for Impala on page 30
• Configuring Impala to Work with ODBC on page 31
• Configuring Impala to Work with JDBC on page 34
This type of configuration is especially useful when using Impala in combination with Business Intelligence tools, which
use these standard interfaces to query different kinds of database and Big Data systems.
You can also configure these other aspects of Impala:
• Impala Security on page 91
• Modifying Impala Startup Options on page 43

Post-Installation Configuration for Impala

This section describes the mandatory and recommended configuration settings for Impala. If Impala is installed using
Cloudera Manager, some of these configurations are completed automatically; you must still configure short-circuit
reads manually. If you installed Impala without Cloudera Manager, or if you want to customize your environment,
consider making the changes described in this topic.
In some cases, depending on the level of Impala, CDH, and Cloudera Manager, you might need to add particular
component configuration details in one of the free-form fields on the Impala configuration pages within Cloudera
Manager. In Cloudera Manager 4, these fields are labelled Safety Valve; in Cloudera Manager 5, they are called
Advanced Configuration Snippet.
• You must enable short-circuit reads, whether or not Impala was installed through Cloudera Manager. This setting
goes in the Impala configuration settings, not the Hadoop-wide settings.
• If you installed Impala in an environment that is not managed by Cloudera Manager, you must enable block location
tracking, and you can optionally enable native checksumming for optimal performance.
• If you deployed Impala using Cloudera Manager see Testing Impala Performance on page 506 to confirm proper
configuration.

Mandatory: Short-Circuit Reads

Enabling short-circuit reads allows Impala to read local data directly from the file system. This removes the need to
communicate through the DataNodes, improving performance. This setting also minimizes the number of additional
copies of data. Short-circuit reads requires libhadoop.so (the Hadoop Native Library) to be accessible to both the
server and the client. libhadoop.so is not available if you have installed from a tarball. You must install from an
.rpm, .deb, or parcel to use short-circuit local reads.

Note: If you use Cloudera Manager, you can enable short-circuit reads through a checkbox in the
user interface and that setting takes effect for Impala as well.

To configure DataNodes for short-circuit reads:

1. Copy the client core-site.xml and hdfs-site.xml configuration files from the Hadoop configuration directory
to the Impala configuration directory. The default Impala configuration location is /etc/impala/conf.
2. On all Impala nodes, configure the following properties in Impala's copy of hdfs-site.xml as shown:

<property>
<name>dfs.client.read.shortcircuit</name>

30 | Apache Impala Guide

 Managing Impala

<value>true</value>
</property>

<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hdfs-sockets/dn</value>
</property>

<property>
<name>dfs.client.file-block-storage-locations.timeout.millis</name>
<value>10000</value>
</property>

3. If /var/run/hadoop-hdfs/ is group-writable, make sure its group is root.

Note: If you are also going to enable block location tracking, you can skip copying configuration
files and restarting DataNodes and go straight to Optional: Block Location Tracking. Configuring
short-circuit reads and block location tracking require the same process of copying files and
restarting services, so you can complete that process once when you have completed all
configuration changes. Whether you copy files and restart services now or during configuring
block location tracking, short-circuit reads are not enabled until you complete those final steps.

4. After applying these changes, restart all DataNodes.

Mandatory: Block Location Tracking

Enabling block location metadata allows Impala to know which disk data blocks are located on, allowing better utilization
of the underlying disks. Impala will not start unless this setting is enabled.
To enable block location tracking:
1. For each DataNode, adding the following to the hdfs-site.xml file:

<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>

2. Copy the client core-site.xml and hdfs-site.xml configuration files from the Hadoop configuration directory
to the Impala configuration directory. The default Impala configuration location is /etc/impala/conf.
3. After applying these changes, restart all DataNodes.

Optional: Native Checksumming

Enabling native checksumming causes Impala to use an optimized native library for computing checksums, if that library
is available.
To enable native checksumming:
If you installed CDH from packages, the native checksumming library is installed and setup correctly. In such a case,
no additional steps are required. Conversely, if you installed by other means, such as with tarballs, native checksumming
may not be available due to missing shared objects. Finding the message "Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable" in the Impala logs
indicates native checksumming may be unavailable. To enable native checksumming, you must build and install
libhadoop.so (the Hadoop Native Library).

Configuring Impala to Work with ODBC

Third-party products can be designed to integrate with Impala using ODBC. For the best experience, ensure any
third-party product you intend to use is supported. Verifying support includes checking that the versions of Impala,

Apache Impala Guide | 31

Managing Impala

ODBC, the operating system, and the third-party product have all been approved for use together. Before configuring
your systems to use ODBC, download a connector. You may need to sign in and accept license agreements before
accessing the pages required for downloading ODBC connectors.

Downloading the ODBC Driver

Important: As of late 2015, most business intelligence applications are certified with the 2.x ODBC
drivers. Although the instructions on this page cover both the 2.x and 1.x drivers, expect to use the
2.x drivers exclusively for most ODBC applications connecting to Impala.

See the documentation page for installation instructions.

Configuring the ODBC Port

Versions 2.5 and 2.0 of the Cloudera ODBC Connector, currently certified for some but not all BI applications, use the
HiveServer2 protocol, corresponding to Impala port 21050. Impala supports Kerberos authentication with all the
supported versions of the driver, and requires ODBC 2.05.13 for Impala or higher for LDAP username/password
authentication.
Version 1.x of the Cloudera ODBC Connector uses the original HiveServer1 protocol, corresponding to Impala port
21000.

Example of Setting Up an ODBC Application for Impala

To illustrate the outline of the setup process, here is a transcript of a session to set up all required drivers and a business
intelligence application that uses the ODBC driver, under Mac OS X. Each .dmg file runs a GUI-based installer, first for
the underlying IODBC driver needed for non-Windows systems, then for the Cloudera ODBC Connector, and finally for
the BI tool itself.

$ ls -1
Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf
BI_Tool_Installer.dmg
iodbc-sdk-3.52.7-macosx-10.5.dmg
ClouderaImpalaODBC.dmg
$ open iodbc-sdk-3.52.7-macosx-10.dmg
Install the IODBC driver using its installer
$ open ClouderaImpalaODBC.dmg
Install the Cloudera ODBC Connector using its installer
$ installer_dir=$(pwd)
$ cd /opt/cloudera/impalaodbc
$ ls -1
Cloudera ODBC Driver for Impala Install Guide.pdf
Readme.txt
Setup
lib
ErrorMessages
Release Notes.txt
Tools
$ cd Setup
$ ls
odbc.ini odbcinst.ini
$ cp odbc.ini ~/.odbc.ini
$ vi ~/.odbc.ini
$ cat ~/.odbc.ini
[ODBC]
# Specify any global ODBC configuration here such as ODBC tracing.

[ODBC Data Sources]

Sample Cloudera Impala DSN=Cloudera ODBC Driver for Impala

[Sample Cloudera Impala DSN]

# Description: DSN Description.

# This key is not necessary and is only to give a description of the data source.
Description=Cloudera ODBC Driver for Impala DSN

32 | Apache Impala Guide

 Managing Impala

# Driver: The location where the ODBC driver is installed to.

Driver=/opt/cloudera/impalaodbc/lib/universal/libclouderaimpalaodbc.dylib

# The DriverUnicodeEncoding setting is only used for SimbaDM

# When set to 1, SimbaDM runs in UTF-16 mode.
# When set to 2, SimbaDM runs in UTF-8 mode.
#DriverUnicodeEncoding=2

# Values for HOST, PORT, KrbFQDN, and KrbServiceName should be set here.
# They can also be specified on the connection string.
HOST=hostname.sample.example.com
PORT=21050
Schema=default

# The authentication mechanism.

# 0 - No authentication (NOSASL)
# 1 - Kerberos authentication (SASL)
# 2 - Username authentication (SASL)
# 3 - Username/password authentication (SASL)
# 4 - Username/password authentication with SSL (SASL)
# 5 - No authentication with SSL (NOSASL)
# 6 - Username/password authentication (NOSASL)
AuthMech=0

# Kerberos related settings.

KrbFQDN=
KrbRealm=
KrbServiceName=

# Username/password authentication with SSL settings.

UID=
PWD
CAIssuedCertNamesMismatch=1
TrustedCerts=/opt/cloudera/impalaodbc/lib/universal/cacerts.pem

# Specify the proxy user ID to use.

#DelegationUID=

# General settings
TSaslTransportBufSize=1000
RowsFetchedPerBlock=10000
SocketTimeout=0
StringColumnLength=32767
UseNativeQuery=0
$ pwd
/opt/cloudera/impalaodbc/Setup
$ cd $installer_dir
$ open BI_Tool_Installer.dmg
Install the BI tool using its installer
$ ls /Applications | grep BI_Tool
BI_Tool.app
$ open -a BI_Tool.app
In the BI tool, connect to a data source using port 21050

Notes about JDBC and ODBC Interaction with Impala SQL Features
Most Impala SQL features work equivalently through the impala-shell interpreter of the JDBC or ODBC APIs. The
following are some exceptions to keep in mind when switching between the interactive shell and applications using
the APIs:

Note: If your JDBC or ODBC application connects to Impala through a load balancer such as haproxy,
be cautious about reusing the connections. If the load balancer has set up connection timeout values,
either check the connection frequently so that it never sits idle longer than the load balancer timeout
value, or check the connection validity before using it and create a new one if the connection has
been closed.

Apache Impala Guide | 33

Managing Impala

• The Impala complex types (STRUCT, ARRAY, or MAP) are available in CDH 5.5 / Impala 2.3 and higher. To use these
types with JDBC requires version 2.5.28 or higher of the Cloudera JDBC Connector for Impala. To use these types
with ODBC requires version 2.5.30 or higher of the Cloudera ODBC Connector for Impala. Consider upgrading all
JDBC and ODBC drivers at the same time you upgrade from CDH 5.5 or higher.
• Although the result sets from queries involving complex types consist of all scalar values, the queries involve join
notation and column references that might not be understood by a particular JDBC or ODBC connector. Consider
defining a view that represents the flattened version of a table containing complex type columns, and pointing
the JDBC or ODBC application at the view. See Complex Types (CDH 5.5 or higher only) on page 156 for details.

Configuring Impala to Work with JDBC

Impala supports the standard JDBC interface, allowing access from commercial Business Intelligence tools and custom
software written in Java or other programming languages. The JDBC driver allows you to access Impala from a Java
program that you write, or a Business Intelligence or similar tool that uses JDBC to communicate with various database
products.
Setting up a JDBC connection to Impala involves the following steps:
• Verifying the communication port where the Impala daemons in your cluster are listening for incoming JDBC
requests.
• Installing the JDBC driver on every system that runs the JDBC-enabled application.
• Specifying a connection string for the JDBC application to access one of the servers running the impalad daemon,
with the appropriate security settings.

Configuring the JDBC Port

The default port used by JDBC 2.0 and later (as well as ODBC 2.x) is 21050. Impala server accepts JDBC connections
through this same port 21050 by default. Make sure this port is available for communication with other hosts on your
network, for example, that it is not blocked by firewall software. If your JDBC client software connects to a different
port, specify that alternative port number with the --hs2_port option when starting impalad. See Starting Impala
on page 42 for details about Impala startup options. See Ports Used by Impala on page 583 for information about all
ports used for communication between Impala and clients or between Impala components.

Choosing the JDBC Driver

In Impala 2.0 and later, you have the choice between the Cloudera JDBC Connector and the Hive 0.13 JDBC driver.
Cloudera recommends using the Cloudera JDBC Connector where practical.
If you are already using JDBC applications with an earlier Impala release, you must update your JDBC driver to one of
these choices, because the Hive 0.12 driver that was formerly the only choice is not compatible with Impala 2.0 and
later.
Both the Cloudera JDBC 2.5 Connector and the Hive JDBC driver provide a substantial speed increase for JDBC applications
with Impala 2.0 and higher, for queries that return large result sets.
Complex type considerations:
The Impala complex types (STRUCT, ARRAY, or MAP) are available in CDH 5.5 / Impala 2.3 and higher. To use these
types with JDBC requires version 2.5.28 or higher of the Cloudera JDBC Connector for Impala. To use these types with
ODBC requires version 2.5.30 or higher of the Cloudera ODBC Connector for Impala. Consider upgrading all JDBC and
ODBC drivers at the same time you upgrade from CDH 5.5 or higher.
Although the result sets from queries involving complex types consist of all scalar values, the queries involve join
notation and column references that might not be understood by a particular JDBC or ODBC connector. Consider
defining a view that represents the flattened version of a table containing complex type columns, and pointing the
JDBC or ODBC application at the view. See Complex Types (CDH 5.5 or higher only) on page 156 for details.

34 | Apache Impala Guide

 Managing Impala

Enabling Impala JDBC Support on Client Systems

Using the Cloudera JDBC Connector (recommended)

You download and install the Cloudera JDBC 2.5 connector on any Linux, Windows, or Mac system where you intend
to run JDBC-enabled applications. From the Cloudera Connectors download page, you choose the appropriate protocol
(JDBC or ODBC) and target product (Impala or Hive). The ease of downloading and installing on non-CDH systems makes
this connector a convenient choice for organizations with heterogeneous environments.

Using the Hive JDBC Driver

You install the Hive JDBC driver (hive-jdbc package) through the Linux package manager, on hosts within the CDH
cluster. The driver consists of several Java JAR files. The same driver can be used by Impala and Hive.
To get the JAR files, install the Hive JDBC driver on each CDH-enabled host in the cluster that will run JDBC applications.
Follow the instructions for CDH 5.

Note: The latest JDBC driver, corresponding to Hive 0.13, provides substantial performance
improvements for Impala queries that return large result sets. Impala 2.0 and later are compatible
with the Hive 0.13 driver. If you already have an older JDBC driver installed, and are running Impala
2.0 or higher, consider upgrading to the latest Hive JDBC driver for best performance with JDBC
applications.

If you are using JDBC-enabled applications on hosts outside the CDH cluster, you cannot use the CDH install procedure
on the non-CDH hosts. Install the JDBC driver on at least one CDH host using the preceding procedure. Then download
the JAR files to each client machine that will use JDBC with Impala:

commons-logging-X.X.X.jar
hadoop-common.jar
hive-common-X.XX.X-cdhX.X.X.jar
hive-jdbc-X.XX.X-cdhX.X.X.jar
hive-metastore-X.XX.X-cdhX.X.X.jar
hive-service-X.XX.X-cdhX.X.X.jar
httpclient-X.X.X.jar
httpcore-X.X.X.jar
libfb303-X.X.X.jar
libthrift-X.X.X.jar
log4j-X.X.XX.jar
slf4j-api-X.X.X.jar
slf4j-logXjXX-X.X.X.jar

To enable JDBC support for Impala on the system where you run the JDBC application:
1. Download the JAR files listed above to each client machine.

Note: For Maven users, see this sample github page for an example of the dependencies you
could add to a pom file instead of downloading the individual JARs.

2. Store the JAR files in a location of your choosing, ideally a directory already referenced in your CLASSPATH setting.
For example:
• On Linux, you might use a location such as /opt/jars/.
• On Windows, you might use a subdirectory underneath C:\Program Files.
3. To successfully load the Impala JDBC driver, client programs must be able to locate the associated JAR files. This
often means setting the CLASSPATH for the client process to include the JARs. Consult the documentation for
your JDBC client for more details on how to install new JDBC drivers, but some examples of how to set CLASSPATH
variables include:

Apache Impala Guide | 35

Managing Impala

• On Linux, if you extracted the JARs to /opt/jars/, you might issue the following command to prepend the
JAR files path to an existing classpath:

export CLASSPATH=/opt/jars/*.jar:$CLASSPATH

• On Windows, use the System Properties control panel item to modify the Environment Variables for your
system. Modify the environment variables to include the path to which you extracted the files.

Note: If the existing CLASSPATH on your client machine refers to some older version of the
Hive JARs, ensure that the new JARs are the first ones listed. Either put the new JAR files
earlier in the listings, or delete the other references to Hive JAR files.

Establishing JDBC Connections

The JDBC driver class depends on which driver you select.

Note: If your JDBC or ODBC application connects to Impala through a load balancer such as haproxy,
be cautious about reusing the connections. If the load balancer has set up connection timeout values,
either check the connection frequently so that it never sits idle longer than the load balancer timeout
value, or check the connection validity before using it and create a new one if the connection has
been closed.

Using the Cloudera JDBC Connector (recommended)

Depending on the level of the JDBC API your application is targeting, you can use the following fully-qualified class
names (FQCNs):
• com.cloudera.impala.jdbc41.Driver
• com.cloudera.impala.jdbc41.DataSource
• com.cloudera.impala.jdbc4.Driver
• com.cloudera.impala.jdbc4.DataSource
• com.cloudera.impala.jdbc3.Driver
• com.cloudera.impala.jdbc3.DataSource
The connection string has the following format:

jdbc:impala://Host:Port[/Schema];Property1=Value;Property2=Value;...

The port value is typically 21050 for Impala.

For full details about the classes and the connection string (especially the property values available for the connection
string), download the appropriate driver documentation for your platform from the Impala JDBC Connector download
page.

Using the Hive JDBC Driver

For example, with the Hive JDBC driver, the class name is org.apache.hive.jdbc.HiveDriver. Once you have
configured Impala to work with JDBC, you can establish connections between the two. To do so for a cluster that does
not use Kerberos authentication, use a connection string of the form jdbc:hive2://host:port/;auth=noSasl.
For example, you might use:

jdbc:hive2://myhost.example.com:21050/;auth=noSasl

36 | Apache Impala Guide

 Managing Impala

To connect to an instance of Impala that requires Kerberos authentication, use a connection string of the form
jdbc:hive2://host:port/;principal=principal_name. The principal must be the same user principal you
used when starting Impala. For example, you might use:

jdbc:hive2://myhost.example.com:21050/;principal=impala/[email protected]

To connect to an instance of Impala that requires LDAP authentication, use a connection string of the form
jdbc:hive2://host:port/db_name;user=ldap_userid;password=ldap_password. For example, you might
use:

jdbc:hive2://myhost.example.com:21050/test_db;user=fred;password=xyz123

Note:
Currently, the Hive JDBC driver does not support connections that use both Kerberos authentication
and SSL encryption. To use both of these security features with Impala through a JDBC application,
use the Cloudera JDBC Connector as the JDBC driver.

Notes about JDBC and ODBC Interaction with Impala SQL Features
Most Impala SQL features work equivalently through the impala-shell interpreter of the JDBC or ODBC APIs. The
following are some exceptions to keep in mind when switching between the interactive shell and applications using
the APIs:
• Complex type considerations:
– Queries involving the complex types (ARRAY, STRUCT, and MAP) require notation that might not be available
in all levels of JDBC and ODBC drivers. If you have trouble querying such a table due to the driver level or
inability to edit the queries used by the application, you can create a view that exposes a “flattened” version
of the complex columns and point the application at the view. See Complex Types (CDH 5.5 or higher only)
on page 156 for details.
– The complex types available in CDH 5.5 / Impala 2.3 and higher are supported by the JDBC getColumns()
API. Both MAP and ARRAY are reported as the JDBC SQL Type ARRAY, because this is the closest matching Java
SQL type. This behavior is consistent with Hive. STRUCT types are reported as the JDBC SQL Type STRUCT.
To be consistent with Hive's behavior, the TYPE_NAME field is populated with the primitive type name for
scalar types, and with the full toSql() for complex types. The resulting type names are somewhat inconsistent,
because nested types are printed differently than top-level types. For example, the following list shows how
toSQL() for Impala types are translated to TYPE_NAME values:

DECIMAL(10,10) becomes DECIMAL

CHAR(10) becomes CHAR
VARCHAR(10) becomes VARCHAR
ARRAY<DECIMAL(10,10)> becomes ARRAY<DECIMAL(10,10)>
ARRAY<CHAR(10)> becomes ARRAY<CHAR(10)>
ARRAY<VARCHAR(10)> becomes ARRAY<VARCHAR(10)>

Kudu Considerations for DML Statements

Currently, Impala INSERT, UPDATE, or other DML statements issued through the JDBC interface against a Kudu table
do not return JDBC error codes for conditions such as duplicate primary key columns. Therefore, for applications that
issue a high volume of DML statements, prefer to use the Kudu Java API directly rather than a JDBC application.

Apache Impala Guide | 37

Upgrading Impala

Upgrading Impala
Upgrading Impala involves stopping Impala services, using your operating system's package management tool to
upgrade Impala to the latest version, and then restarting Impala services.

Note:
• Each version of CDH 5 has an associated version of Impala, When you upgrade from CDH 4 to
CDH 5, you get whichever version of Impala comes with the associated level of CDH. Depending
on the version of Impala you were running on CDH 4, this could install a lower level of Impala on
CDH 5. For example, if you upgrade to CDH 5.0 from CDH 4 plus Impala 1.4, the CDH 5.0 installation
comes with Impala 1.3. Always check the associated level of Impala before upgrading to a specific
version of CDH 5. Where practical, upgrade from CDH 4 to the latest CDH 5, which also has the
latest Impala.
• When you upgrade Impala, also upgrade Cloudera Manager if necessary:
– Users running Impala on CDH 5 must upgrade to Cloudera Manager 5.0.0 or higher.
– Users running Impala on CDH 4 must upgrade to Cloudera Manager 4.8 or higher. Cloudera
Manager 4.8 includes management support for the Impala catalog service, and is the minimum
Cloudera Manager version you can use.
– Cloudera Manager is continually updated with configuration settings for features introduced
in the latest Impala releases.
• If you are upgrading from CDH 5 beta to CDH 5.0 production, make sure you are using the
appropriate CDH 5 repositories shown on the CDH version and packaging page, then follow the
procedures throughout the rest of this section.
• Every time you upgrade to a new major or minor Impala release, see Incompatible Changes and
Limitations in Apache Impala on page 625 in the Release Notes for any changes needed in your
source code, startup scripts, and so on.
• Also check Known Issues and Workarounds in Impala on page 635 in the Release Notes for any
issues or limitations that require workarounds.

Upgrading Impala through Cloudera Manager - Parcels

Parcels are an alternative binary distribution format available in Cloudera Manager 4.5 and higher.

Important: In CDH 5, there is not a separate Impala parcel; Impala is part of the main CDH 5 parcel.
Each level of CDH 5 has a corresponding version of Impala, and you upgrade Impala by upgrading CDH.
See the CDH 5 upgrade instructions and choose the instructions for parcels. The remainder of this
section only covers parcel upgrades for Impala under CDH 4.

To upgrade Impala for CDH 4 in a Cloudera Managed environment, using parcels:

1. If you originally installed using packages and now are switching to parcels, remove all the Impala-related packages
first. You can check which packages are installed using one of the following commands, depending on your operating
system:

rpm -qa # RHEL, Oracle Linux, CentOS, Debian

dpkg --get-selections # Debian

38 | Apache Impala Guide

 Upgrading Impala

and then remove the packages using one of the following commands:

sudo yum remove pkg_names # RHEL, Oracle Linux, CentOS

sudo zypper remove pkg_names # SLES
sudo apt-get purge pkg_names # Ubuntu, Debian

2. Connect to the Cloudera Manager Admin Console.

3. Go to the Hosts > Parcels tab. You should see a parcel with a newer version of Impala that you can upgrade to.
4. Click Download, then Distribute. (The button changes as each step completes.)
5. Click Activate.
6. When prompted, click Restart to restart the Impala service.

Upgrading Impala through Cloudera Manager - Packages

To upgrade Impala in a Cloudera Managed environment, using packages:
1. Connect to the Cloudera Manager Admin Console.
2. In the Services tab, click the Impala service.
3. Click Actions and click Stop.
4. Use one of the following sets of commands to update Impala on each Impala node in your cluster:
For RHEL, Oracle Linux, or CentOS systems:

$ sudo yum update impala

$ sudo yum update hadoop-lzo-cdh4 # Optional; if this package is already installed

For SUSE systems:

$ sudo zypper update impala

$ sudo zypper update hadoop-lzo-cdh4 # Optional; if this package is already installed

For Debian or Ubuntu systems:

$ sudo apt-get install impala

$ sudo apt-get install hadoop-lzo-cdh4 # Optional; if this package is already installed

5. Use one of the following sets of commands to update Impala shell on each node on which it is installed:
For RHEL, Oracle Linux, or CentOS systems:

$ sudo yum update impala-shell

For SUSE systems:

$ sudo zypper update impala-shell

For Debian or Ubuntu systems:

$ sudo apt-get install impala-shell

6. Connect to the Cloudera Manager Admin Console.

7. In the Services tab, click the Impala service.
8. Click Actions and click Start.

Apache Impala Guide | 39

Upgrading Impala

Upgrading Impala without Cloudera Manager

To upgrade Impala on a cluster not managed by Cloudera Manager, run these Linux commands on the appropriate
hosts in your cluster:
1. Stop Impala services.
a. Stop impalad on each Impala node in your cluster:

$ sudo service impala-server stop

b. Stop any instances of the state store in your cluster:

$ sudo service impala-state-store stop

c. Stop any instances of the catalog service in your cluster:

$ sudo service impala-catalog stop

2. Check if there are new recommended or required configuration settings to put into place in the configuration
files, typically under /etc/impala/conf. See Post-Installation Configuration for Impala on page 30 for settings
related to performance and scalability.
3. Use one of the following sets of commands to update Impala on each Impala node in your cluster:
For RHEL, Oracle Linux, or CentOS systems:

$ sudo yum update impala-server

$ sudo yum update hadoop-lzo-cdh4 # Optional; if this package is already installed
$ sudo yum update impala-catalog # New in Impala 1.2; do yum install when upgrading from
1.1.

For SUSE systems:

$ sudo zypper update impala-server

$ sudo zypper update hadoop-lzo-cdh4 # Optional; if this package is already installed
$ sudo zypper update impala-catalog # New in Impala 1.2; do zypper install when upgrading
from 1.1.

For Debian or Ubuntu systems:

$ sudo apt-get install impala-server

$ sudo apt-get install hadoop-lzo-cdh4 # Optional; if this package is already installed
$ sudo apt-get install impala-catalog # New in Impala 1.2.

4. Use one of the following sets of commands to update Impala shell on each node on which it is installed:
For RHEL, Oracle Linux, or CentOS systems:

$ sudo yum update impala-shell

For SUSE systems:

$ sudo zypper update impala-shell

For Debian or Ubuntu systems:

$ sudo apt-get install impala-shell

40 | Apache Impala Guide

 Upgrading Impala

5. Depending on which release of Impala you are upgrading from, you might find that the symbolic links
/etc/impala/conf and /usr/lib/impala/sbin are missing. If so, see Known Issues and Workarounds in
Impala on page 635 for the procedure to work around this problem.
6. Restart Impala services:
a. Restart the Impala state store service on the desired nodes in your cluster. Expect to see a process named
statestored if the service started successfully.

$ sudo service impala-state-store start

$ ps ax | grep [s]tatestored
6819 ? Sl 0:07 /usr/lib/impala/sbin/statestored -log_dir=/var/log/impala
-state_store_port=24000

Restart the state store service before the Impala server service to avoid “Not connected” errors when you
run impala-shell.
b. Restart the Impala catalog service on whichever host it runs on in your cluster. Expect to see a process named
catalogd if the service started successfully.

$ sudo service impala-catalog restart

$ ps ax | grep [c]atalogd
6068 ? Sl 4:06 /usr/lib/impala/sbin/catalogd

c. Restart the Impala daemon service on each node in your cluster. Expect to see a process named impalad if
the service started successfully.

$ sudo service impala-server start

$ ps ax | grep [i]mpalad
7936 ? Sl 0:12 /usr/lib/impala/sbin/impalad -log_dir=/var/log/impala
-state_store_port=24000 -use_statestore
-state_store_host=127.0.0.1 -be_port=22000

Note:
If the services did not start successfully (even though the sudo service command might display
[OK]), check for errors in the Impala log file, typically in /var/log/impala.

Apache Impala Guide | 41

Starting Impala

Starting Impala
To activate Impala if it is installed but not yet started:
1. Set any necessary configuration options for the Impala services. See Modifying Impala Startup Options on page
43 for details.
2. Start one instance of the Impala statestore. The statestore helps Impala to distribute work efficiently, and to
continue running in the event of availability problems for other Impala nodes. If the statestore becomes unavailable,
Impala continues to function.
3. Start one instance of the Impala catalog service.
4. Start the main Impala service on one or more DataNodes, ideally on all DataNodes to maximize local processing
and avoid network traffic due to remote reads.
Once Impala is running, you can conduct interactive experiments using the instructions in Impala Tutorials on page 47
and try Using the Impala Shell (impala-shell Command) on page 470.

Starting Impala through Cloudera Manager

If you installed Impala with Cloudera Manager, use Cloudera Manager to start and stop services. The Cloudera Manager
GUI is a convenient way to check that all services are running, to set configuration options using form fields in a browser,
and to spot potential issues such as low disk space before they become serious. Cloudera Manager automatically starts
all the Impala-related services as a group, in the correct order. See the Cloudera Manager Documentation for details.

Note:
Currently, Impala UDFs and UDAs are not persisted in the metastore database. Information about
these functions is held in the memory of the catalogd daemon. You must reload them by running
the CREATE FUNCTION statements again each time you restart the catalogd daemon.

Starting Impala from the Command Line

To start the Impala state store and Impala from the command line or a script, you can either use the service command
or you can start the daemons directly through the impalad, statestored, and catalogd executables.
Start the Impala statestore and then start impalad instances. You can modify the values the service initialization
scripts use when starting the statestore and Impala by editing /etc/default/impala.
Start the statestore service using a command similar to the following:

$ sudo service impala-state-store start

Start the catalog service using a command similar to the following:

$ sudo service impala-catalog start

Start the Impala service on each DataNode using a command similar to the following:

$ sudo service impala-server start

42 | Apache Impala Guide

 Starting Impala

Note:
Currently, Impala UDFs and UDAs are not persisted in the metastore database. Information about
these functions is held in the memory of the catalogd daemon. You must reload them by running
the CREATE FUNCTION statements again each time you restart the catalogd daemon.

If any of the services fail to start, review:

• Reviewing Impala Logs on page 575
• Troubleshooting Impala on page 578

Modifying Impala Startup Options

The configuration options for the Impala-related daemons let you choose which hosts and ports to use for the services
that run on a single host, specify directories for logging, control resource usage and security, and specify other aspects
of the Impala software.

Configuring Impala Startup Options through Cloudera Manager

If you manage your cluster through Cloudera Manager, configure the settings for all the Impala-related daemons by
navigating to this page: Clusters > Services > Impala > Configuration > View and Edit. See the Cloudera Manager
documentation for instructions about how to configure Impala through Cloudera Manager.
If the Cloudera Manager interface does not yet have a form field for a newly added option, or if you need to use special
options for debugging and troubleshooting, the Advanced option page for each daemon includes one or more fields
where you can enter option names directly. In Cloudera Manager 4, these fields are labelled Safety Valve; in Cloudera
Manager 5, they are called Advanced Configuration Snippet. There is also a free-form field for query options, on the
top-level Impala Daemon options page.

Configuring Impala Startup Options through the Command Line

When you run Impala in a non-Cloudera Manager environment, the Impala server, statestore, and catalog services
start up using values provided in a defaults file, /etc/default/impala.
This file includes information about many resources used by Impala. Most of the defaults included in this file should
be effective in most cases. For example, typically you would not change the definition of the CLASSPATH variable, but
you would always set the address used by the statestore server. Some of the content you might modify includes:

IMPALA_STATE_STORE_HOST=127.0.0.1
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/var/log/impala
IMPALA_CATALOG_SERVICE_HOST=...
IMPALA_STATE_STORE_HOST=...

export IMPALA_STATE_STORE_ARGS=${IMPALA_STATE_STORE_ARGS:- \
-log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}}
IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-be_port=${IMPALA_BACKEND_PORT}"
export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}

Apache Impala Guide | 43

Starting Impala

To use alternate values, edit the defaults file, then restart all the Impala-related services so that the changes take
effect. Restart the Impala server using the following commands:

$ sudo service impala-server restart

Stopping Impala Server: [ OK ]
Starting Impala Server: [ OK ]

Restart the Impala statestore using the following commands:

$ sudo service impala-state-store restart

Stopping Impala State Store Server: [ OK ]
Starting Impala State Store Server: [ OK ]

Restart the Impala catalog service using the following commands:

$ sudo service impala-catalog restart

Stopping Impala Catalog Server: [ OK ]
Starting Impala Catalog Server: [ OK ]

Some common settings to change include:

• Statestore address. Cloudera recommends the statestore be on a separate host not running the impalad daemon.
In that recommended configuration, the impalad daemon cannot refer to the statestore server using the loopback
address. If the statestore is hosted on a machine with an IP address of 192.168.0.27, change:

IMPALA_STATE_STORE_HOST=127.0.0.1

to:

IMPALA_STATE_STORE_HOST=192.168.0.27

• Catalog server address (including both the hostname and the port number). Update the value of the
IMPALA_CATALOG_SERVICE_HOST variable. Cloudera recommends the catalog server be on the same host as
the statestore. In that recommended configuration, the impalad daemon cannot refer to the catalog server using
the loopback address. If the catalog service is hosted on a machine with an IP address of 192.168.0.27, add the
following line:

IMPALA_CATALOG_SERVICE_HOST=192.168.0.27:26000

The /etc/default/impala defaults file currently does not define an IMPALA_CATALOG_ARGS environment
variable, but if you add one it will be recognized by the service startup/shutdown script. Add a definition for this
variable to /etc/default/impala and add the option -catalog_service_host=hostname. If the port is
different than the default 26000, also add the option -catalog_service_port=port.
• Memory limits. You can limit the amount of memory available to Impala. For example, to allow Impala to use no
more than 70% of system memory, change:

export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
-be_port=${IMPALA_BACKEND_PORT}}

to:

export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
-log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} -mem_limit=70%}

44 | Apache Impala Guide

 Starting Impala

You can specify the memory limit using absolute notation such as 500m or 2G, or as a percentage of physical
memory such as 60%.

Note: Queries that exceed the specified memory limit are aborted. Percentage limits are based
on the physical memory of the machine and do not consider cgroups.

• Core dump enablement. To enable core dumps on systems not managed by Cloudera Manager, change:

export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}

to:

export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-true}

On systems managed by Cloudera Manager, enable the Enable Core Dump setting for the Impala service.

Note:
• The location of core dump files may vary according to your operating system configuration.
• Other security settings may prevent Impala from writing core dumps even when this option
is enabled.
• On systems managed by Cloudera Manager, the default location for core dumps is on a
temporary filesystem, which can lead to out-of-space issues if the core dumps are large,
frequent, or not removed promptly. To specify an alternative location for the core dumps,
filter the Impala configuration settings to find the core_dump_dir option, which is available
in Cloudera Manager 5.4.3 and higher. This option lets you specify a different directory for
core dumps for each of the Impala-related daemons.

• Authorization using the open source Sentry plugin. Specify the -server_name and
-authorization_policy_file options as part of the IMPALA_SERVER_ARGS and IMPALA_STATE_STORE_ARGS
settings to enable the core Impala support for authentication. See Starting the impalad Daemon with Sentry
Authorization Enabled on page 96 for details.
• Auditing for successful or blocked Impala queries, another aspect of security. Specify the
-audit_event_log_dir=directory_path option and optionally the
-max_audit_event_log_file_size=number_of_queries and -abort_on_failed_audit_event options
as part of the IMPALA_SERVER_ARGS settings, for each Impala node, to enable and customize auditing. See
Auditing Impala Operations on page 112 for details.
• Password protection for the Impala web UI, which listens on port 25000 by default. This feature involves adding
some or all of the --webserver_password_file, --webserver_authentication_domain, and
--webserver_certificate_file options to the IMPALA_SERVER_ARGS and IMPALA_STATE_STORE_ARGS
settings. See Security Guidelines for Impala on page 91 for details.
• Another setting you might add to IMPALA_SERVER_ARGS is a comma-separated list of query options and values:

-default_query_options='option=value,option=value,...'

These options control the behavior of queries performed by this impalad instance. The option values you specify
here override the default values for Impala query options, as shown by the SET statement in impala-shell.
• During troubleshooting, Cloudera Support might direct you to change other values, particularly for
IMPALA_SERVER_ARGS, to work around issues or gather debugging information.

Apache Impala Guide | 45

Starting Impala

Note:
These startup options for the impalad daemon are different from the command-line options for the
impala-shell command. For the impala-shell options, see impala-shell Configuration Options
on page 470.

Checking the Values of Impala Configuration Options

You can check the current runtime value of all these settings through the Impala web interface, available by default
at http://impala_hostname:25000/varz for the impalad daemon, http://impala_hostname:25010/varz
for the statestored daemon, or http://impala_hostname:25020/varz for the catalogd daemon. In the
Cloudera Manager interface, you can see the link to the appropriate service_name Web UI page when you look at the
status page for a specific daemon on a specific host.

Startup Options for impalad Daemon

The impalad daemon implements the main Impala service, which performs query processing and reads and writes
the data files.

Startup Options for statestored Daemon

The statestored daemon implements the Impala statestore service, which monitors the availability of Impala services
across the cluster, and handles situations such as nodes becoming unavailable or becoming available again.

Startup Options for catalogd Daemon

The catalogd daemon implements the Impala catalog service, which broadcasts metadata changes to all the Impala
nodes when Impala creates a table, inserts data, or performs other kinds of DDL and DML operations.
By default, the metadata loading and caching on startup happens asynchronously, so Impala can begin accepting
requests promptly. To enable the original behavior, where Impala waited until all metadata was loaded before accepting
any requests, set the catalogd configuration option --load_catalog_in_background=false.

46 | Apache Impala Guide

 Impala Tutorials

Impala Tutorials
This section includes tutorial scenarios that demonstrate how to begin using Impala once the software is installed. It
focuses on techniques for loading data, because once you have some data in tables and can query that data, you can
quickly progress to more advanced Impala features.

Note:
Where practical, the tutorials take you from “ground zero” to having the desired Impala tables and
data. In some cases, you might need to download additional files from outside sources, set up additional
software components, modify commands or scripts to fit your own configuration, or substitute your
own sample data.

Before trying these tutorial lessons, install Impala using one of these procedures:
• If you already have a CDH environment set up and just need to add Impala to it, follow the installation process
described in Installing Impala on page 27. Make sure to also install the Hive metastore service if you do not already
have Hive configured.
• To set up Impala and all its prerequisites at once, in a minimal configuration that you can use for small-scale
experiments, set up the Cloudera QuickStart VM, which includes CDH and Impala on CentOS. Use this single-node
VM to try out basic SQL functionality, not anything related to performance and scalability. For more information,
see the Cloudera QuickStart VM.

Tutorials for Getting Started

These tutorials demonstrate the basics of using Impala. They are intended for first-time users, and for trying out Impala
on any new cluster to make sure the major components are working correctly.

Explore a New Impala Instance

This tutorial demonstrates techniques for finding your way around the tables and databases of an unfamiliar (possibly
empty) Impala instance.
When you connect to an Impala instance for the first time, you use the SHOW DATABASES and SHOW TABLES statements
to view the most common types of objects. Also, call the version() function to confirm which version of Impala you
are running; the version number is important when consulting documentation and dealing with support issues.
A completely empty Impala instance contains no tables, but still has two databases:
• default, where new tables are created when you do not specify any other database.
• _impala_builtins, a system database used to hold all the built-in functions.
The following example shows how to see the available databases, and the tables in each. If the list of databases or
tables is long, you can use wildcard notation to locate specific databases or tables based on their names.

$ impala-shell -i localhost --quiet

Starting Impala Shell without Kerberos authentication
Welcome to the Impala shell. Press TAB twice to see a list of available commands.

Copyright (c) 2012 Cloudera, Inc. All rights reserved.

(Shell build version: Impala Shell v...

[localhost:21000] > select version();
+-------------------------------------------
| version()
+-------------------------------------------
| impalad version ...
| Built on ...

Apache Impala Guide | 47

Impala Tutorials

+-------------------------------------------
[localhost:21000] > show databases;
+--------------------------+
| name |
+--------------------------+
| _impala_builtins |
| ctas |
| d1 |
| d2 |
| d3 |
| default |
| explain_plans |
| external_table |
| file_formats |
| tpc |
+--------------------------+
[localhost:21000] > select current_database();
+--------------------+
| current_database() |
+--------------------+
| default |
+--------------------+
[localhost:21000] > show tables;
+-------+
| name |
+-------+
| ex_t |
| t1 |
+-------+
[localhost:21000] > show tables in d3;

[localhost:21000] > show tables in tpc;

+------------------------+
| name |
+------------------------+
| city |
| customer |
| customer_address |
| customer_demographics |
| household_demographics |
| item |
| promotion |
| store |
| store2 |
| store_sales |
| ticket_view |
| time_dim |
| tpc_tables |
+------------------------+
[localhost:21000] > show tables in tpc like 'customer*';
+-----------------------+
| name |
+-----------------------+
| customer |
| customer_address |
| customer_demographics |
+-----------------------+

Once you know what tables and databases are available, you descend into a database with the USE statement. To
understand the structure of each table, you use the DESCRIBE command. Once inside a database, you can issue
statements such as INSERT and SELECT that operate on particular tables.
The following example explores a database named TPC whose name we learned in the previous example. It shows
how to filter the table names within a database based on a search string, examine the columns of a table, and run
queries to examine the characteristics of the table data. For example, for an unfamiliar table you might want to know
the number of rows, the number of different values for a column, and other properties such as whether the column

48 | Apache Impala Guide

 Impala Tutorials

contains any NULL values. When sampling the actual data values from a table, use a LIMIT clause to avoid excessive
output if the table contains more rows or distinct values than you expect.

[localhost:21000] > use tpc;

[localhost:21000] > show tables like '*view*';
+-------------+
| name |
+-------------+
| ticket_view |
+-------------+
[localhost:21000] > describe city;
+-------------+--------+---------+
| name | type | comment |
+-------------+--------+---------+
| id | int | |
| name | string | |
| countrycode | string | |
| district | string | |
| population | int | |
+-------------+--------+---------+
[localhost:21000] > select count(*) from city;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
[localhost:21000] > desc customer;
+------------------------+--------+---------+
| name | type | comment |
+------------------------+--------+---------+
| c_customer_sk | int | |
| c_customer_id | string | |
| c_current_cdemo_sk | int | |
| c_current_hdemo_sk | int | |
| c_current_addr_sk | int | |
| c_first_shipto_date_sk | int | |
| c_first_sales_date_sk | int | |
| c_salutation | string | |
| c_first_name | string | |
| c_last_name | string | |
| c_preferred_cust_flag | string | |
| c_birth_day | int | |
| c_birth_month | int | |
| c_birth_year | int | |
| c_birth_country | string | |
| c_login | string | |
| c_email_address | string | |
| c_last_review_date | string | |
+------------------------+--------+---------+
[localhost:21000] > select count(*) from customer;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+
[localhost:21000] > select count(distinct c_birth_month) from customer;
+-------------------------------+
| count(distinct c_birth_month) |
+-------------------------------+
| 12 |
+-------------------------------+
[localhost:21000] > select count(*) from customer where c_email_address is null;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
[localhost:21000] > select distinct c_salutation from customer limit 10;
+--------------+
| c_salutation |
+--------------+
| Mr. |
| Ms. |

Apache Impala Guide | 49

Impala Tutorials

| Dr. |
| |
| Miss |
| Sir |
| Mrs. |
+--------------+

When you graduate from read-only exploration, you use statements such as CREATE DATABASE and CREATE TABLE
to set up your own database objects.
The following example demonstrates creating a new database holding a new table. Although the last example ended
inside the TPC database, the new EXPERIMENTS database is not nested inside TPC; all databases are arranged in a
single top-level list.

[localhost:21000] > create database experiments;

[localhost:21000] > show databases;
+--------------------------+
| name |
+--------------------------+
| _impala_builtins |
| ctas |
| d1 |
| d2 |
| d3 |
| default |
| experiments |
| explain_plans |
| external_table |
| file_formats |
| tpc |
+--------------------------+
[localhost:21000] > show databases like 'exp*';
+---------------+
| name |
+---------------+
| experiments |
| explain_plans |
+---------------+

The following example creates a new table, T1. To illustrate a common mistake, it creates this table inside the wrong
database, the TPC database where the previous example ended. The ALTER TABLE statement lets you move the table
to the intended database, EXPERIMENTS, as part of a rename operation. The USE statement is always needed to switch
to a new database, and the current_database() function confirms which database the session is in, to avoid these
kinds of mistakes.

[localhost:21000] > create table t1 (x int);

[localhost:21000] > show tables;

+------------------------+
| name |
+------------------------+
| city |
| customer |
| customer_address |
| customer_demographics |
| household_demographics |
| item |
| promotion |
| store |
| store2 |
| store_sales |
| t1 |
| ticket_view |
| time_dim |
| tpc_tables |
+------------------------+
[localhost:21000] > select current_database();
+--------------------+
| current_database() |

50 | Apache Impala Guide

 Impala Tutorials

+--------------------+
| tpc |
+--------------------+
[localhost:21000] > alter table t1 rename to experiments.t1;
[localhost:21000] > use experiments;
[localhost:21000] > show tables;
+------+
| name |
+------+
| t1 |
+------+
[localhost:21000] > select current_database();
+--------------------+
| current_database() |
+--------------------+
| experiments |
+--------------------+

For your initial experiments with tables, you can use ones with just a few columns and a few rows, and text-format
data files.

Note: As you graduate to more realistic scenarios, you will use more elaborate tables with many
columns, features such as partitioning, and file formats such as Parquet. When dealing with realistic
data volumes, you will bring in data using LOAD DATA or INSERT ... SELECT statements to operate
on millions or billions of rows at once.

The following example sets up a couple of simple tables with a few rows, and performs queries involving sorting,
aggregate functions and joins.

[localhost:21000] > insert into t1 values (1), (3), (2), (4);

[localhost:21000] > select x from t1 order by x desc;
+---+
| x |
+---+
| 4 |
| 3 |
| 2 |
| 1 |
+---+
[localhost:21000] > select min(x), max(x), sum(x), avg(x) from t1;
+--------+--------+--------+--------+
| min(x) | max(x) | sum(x) | avg(x) |
+--------+--------+--------+--------+
| 1 | 4 | 10 | 2.5 |
+--------+--------+--------+--------+

[localhost:21000] > create table t2 (id int, word string);

[localhost:21000] > insert into t2 values (1, "one"), (3, "three"), (5, 'five');
[localhost:21000] > select word from t1 join t2 on (t1.x = t2.id);
+-------+
| word |
+-------+
| one |
| three |
+-------+

After completing this tutorial, you should now know:

• How to tell which version of Impala is running on your system.
• How to find the names of databases in an Impala instance, either displaying the full list or searching for specific
names.
• How to find the names of tables in an Impala database, either displaying the full list or searching for specific names.
• How to switch between databases and check which database you are currently in.
• How to learn the column names and types of a table.
• How to create databases and tables, insert small amounts of test data, and run simple queries.

Apache Impala Guide | 51

Impala Tutorials

Load CSV Data from Local Files

This scenario illustrates how to create some very small tables, suitable for first-time users to experiment with Impala
SQL features. TAB1 and TAB2 are loaded with data from files in HDFS. A subset of data is copied from TAB1 into TAB3.
Populate HDFS with the data you want to query. To begin this process, create one or more new subdirectories underneath
your user directory in HDFS. The data for each table resides in a separate subdirectory. Substitute your own user name
for cloudera where appropriate. This example uses the -p option with the mkdir operation to create any necessary
parent directories if they do not already exist.

$ whoami
cloudera
$ hdfs dfs -ls /user
Found 3 items
drwxr-xr-x - cloudera cloudera 0 2013-04-22 18:54 /user/cloudera
drwxrwx--- - mapred mapred 0 2013-03-15 20:11 /user/history
drwxr-xr-x - hue supergroup 0 2013-03-15 20:10 /user/hive

$ hdfs dfs -mkdir -p /user/cloudera/sample_data/tab1 /user/cloudera/sample_data/tab2

Here is some sample data, for two tables named TAB1 and TAB2.
Copy the following content to .csv files in your local filesystem:
tab1.csv:

1,true,123.123,2012-10-24 08:55:00
2,false,1243.5,2012-10-25 13:40:00
3,false,24453.325,2008-08-22 09:33:21.123
4,false,243423.325,2007-05-12 22:32:21.33454
5,true,243.325,1953-04-22 09:11:33

tab2.csv:

1,true,12789.123
2,false,1243.5
3,false,24453.325
4,false,2423.3254
5,true,243.325
60,false,243565423.325
70,true,243.325
80,false,243423.325
90,true,243.325

Put each .csv file into a separate HDFS directory using commands like the following, which use paths available in the
Impala Demo VM:

$ hdfs dfs -put tab1.csv /user/cloudera/sample_data/tab1

$ hdfs dfs -ls /user/cloudera/sample_data/tab1
Found 1 items
-rw-r--r-- 1 cloudera cloudera 192 2013-04-02 20:08
/user/cloudera/sample_data/tab1/tab1.csv

$ hdfs dfs -put tab2.csv /user/cloudera/sample_data/tab2

$ hdfs dfs -ls /user/cloudera/sample_data/tab2
Found 1 items
-rw-r--r-- 1 cloudera cloudera 158 2013-04-02 20:09
/user/cloudera/sample_data/tab2/tab2.csv

The name of each data file is not significant. In fact, when Impala examines the contents of the data directory for the
first time, it considers all files in the directory to make up the data of the table, regardless of how many files there are
or what the files are named.
To understand what paths are available within your own HDFS filesystem and what the permissions are for the various
directories and files, issue hdfs dfs -ls / and work your way down the tree doing -ls operations for the various
directories.

52 | Apache Impala Guide

 Impala Tutorials

Use the impala-shell command to create tables, either interactively or through a SQL script.
The following example shows creating three tables. For each table, the example shows creating columns with various
attributes such as Boolean or integer types. The example also includes commands that provide information about how
the data is formatted, such as rows terminating with commas, which makes sense in the case of importing data from
a .csv file. Where we already have .csv files containing data in the HDFS directory tree, we specify the location of
the directory containing the appropriate .csv file. Impala considers all the data from all the files in that directory to
represent the data for the table.

DROP TABLE IF EXISTS tab1;

-- The EXTERNAL clause means the data is located outside the central location
-- for Impala data files and is preserved when the associated Impala table is dropped.
-- We expect the data to already exist in the directory specified by the LOCATION clause.
CREATE EXTERNAL TABLE tab1
(
id INT,
col_1 BOOLEAN,
col_2 DOUBLE,
col_3 TIMESTAMP
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/cloudera/sample_data/tab1';

DROP TABLE IF EXISTS tab2;

-- TAB2 is an external table, similar to TAB1.
CREATE EXTERNAL TABLE tab2
(
id INT,
col_1 BOOLEAN,
col_2 DOUBLE
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/cloudera/sample_data/tab2';

DROP TABLE IF EXISTS tab3;

-- Leaving out the EXTERNAL clause means the data will be managed
-- in the central Impala data directory tree. Rather than reading
-- existing data files when the table is created, we load the
-- data after creating the table.
CREATE TABLE tab3
(
id INT,
col_1 BOOLEAN,
col_2 DOUBLE,
month INT,
day INT
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

Note: Getting through these CREATE TABLE statements successfully is an important validation step
to confirm everything is configured correctly with the Hive metastore and HDFS permissions. If you
receive any errors during the CREATE TABLE statements:
• Make sure you followed the installation instructions closely, in Installing Impala on page 27.
• Make sure the hive.metastore.warehouse.dir property points to a directory that Impala
can write to. The ownership should be hive:hive, and the impala user should also be a member
of the hive group.
• If the value of hive.metastore.warehouse.dir is different in the Cloudera Manager dialogs
and in the Hive shell, you might need to designate the hosts running impalad with the “gateway”
role for Hive, and deploy the client configuration files to those hosts.

Point an Impala Table at Existing Data Files

A convenient way to set up data for Impala to access is to use an external table, where the data already exists in a set
of HDFS files and you just point the Impala table at the directory containing those files. For example, you might run in

Apache Impala Guide | 53

Impala Tutorials

impala-shell a *.sql file with contents similar to the following, to create an Impala table that accesses an existing
data file used by Hive.
The following examples set up 2 tables, referencing the paths and sample data supplied with the Cloudera QuickStart
VM. For historical reasons, the data physically resides in an HDFS directory tree under /user/hive, although this
particular data is entirely managed by Impala rather than Hive. When we create an external table, we specify the
directory containing one or more data files, and Impala queries the combined content of all the files inside that directory.
Here is how we examine the directories and files within the HDFS filesystem:

$ cd ~/cloudera/datasets
$ ./tpcds-setup.sh
... Downloads and unzips the kit, builds the data and loads it into HDFS ...
$ hdfs dfs -ls /user/hive/tpcds/customer
Found 1 items
-rw-r--r-- 1 cloudera supergroup 13209372 2013-03-22 18:09
/user/hive/tpcds/customer/customer.dat
$ hdfs dfs -cat /user/hive/tpcds/customer/customer.dat | more
1|AAAAAAAABAAAAAAA|980124|7135|32946|2452238|2452208|Mr.|Javier|Lewis|Y|9|12|1936|CHILE||Javie
[email protected]|2452508|
2|AAAAAAAACAAAAAAA|819667|1461|31655|2452318|2452288|Dr.|Amy|Moses|Y|9|4|1966|TOGO||Amy.Moses@
Ovk9KjHH.com|2452318|
3|AAAAAAAADAAAAAAA|1473522|6247|48572|2449130|2449100|Miss|Latisha|Hamilton|N|18|9|1979|NIUE||
[email protected]|2452313|
4|AAAAAAAAEAAAAAAA|1703214|3986|39558|2450030|2450000|Dr.|Michael|White|N|7|6|1983|MEXICO||Mic
[email protected]|2452361|
5|AAAAAAAAFAAAAAAA|953372|4470|36368|2449438|2449408|Sir|Robert|Moran|N|8|5|1956|FIJI||Robert.
[email protected]|2452469|
...

Here is a SQL script to set up Impala tables pointing to some of these data files in HDFS. (The script in the VM sets up
tables like this through Hive; ignore those tables for purposes of this demonstration.) Save the following as
customer_setup.sql:

--
-- store_sales fact table and surrounding dimension tables only
--
create database tpcds;
use tpcds;

drop table if exists customer;

create external table customer
(
c_customer_sk int,
c_customer_id string,
c_current_cdemo_sk int,
c_current_hdemo_sk int,
c_current_addr_sk int,
c_first_shipto_date_sk int,
c_first_sales_date_sk int,
c_salutation string,
c_first_name string,
c_last_name string,
c_preferred_cust_flag string,
c_birth_day int,
c_birth_month int,
c_birth_year int,
c_birth_country string,
c_login string,
c_email_address string,
c_last_review_date string
)
row format delimited fields terminated by '|'
location '/user/hive/tpcds/customer';

drop table if exists customer_address;

create external table customer_address
(
ca_address_sk int,
ca_address_id string,
ca_street_number string,

54 | Apache Impala Guide

 Impala Tutorials

ca_street_name string,
ca_street_type string,
ca_suite_number string,
ca_city string,
ca_county string,
ca_state string,
ca_zip string,
ca_country string,
ca_gmt_offset float,
ca_location_type string
)
row format delimited fields terminated by '|'
location '/user/hive/tpcds/customer_address';

We would run this script with a command such as:

impala-shell -i localhost -f customer_setup.sql

Describe the Impala Table

Now that you have updated the database metadata that Impala caches, you can confirm that the expected tables are
accessible by Impala and examine the attributes of one of the tables. We created these tables in the database named
default. If the tables were in a database other than the default, we would issue a command use db_name to switch
to that database before examining or querying its tables. We could also qualify the name of a table by prepending the
database name, for example default.customer and default.customer_name.

[impala-host:21000] > show databases

Query finished, fetching results ...
default
Returned 1 row(s) in 0.00s
[impala-host:21000] > show tables
Query finished, fetching results ...
customer
customer_address
Returned 2 row(s) in 0.00s
[impala-host:21000] > describe customer_address
+------------------+--------+---------+
| name | type | comment |
+------------------+--------+---------+
| ca_address_sk | int | |
| ca_address_id | string | |
| ca_street_number | string | |
| ca_street_name | string | |
| ca_street_type | string | |
| ca_suite_number | string | |
| ca_city | string | |
| ca_county | string | |
| ca_state | string | |
| ca_zip | string | |
| ca_country | string | |
| ca_gmt_offset | float | |
| ca_location_type | string | |
+------------------+--------+---------+
Returned 13 row(s) in 0.01

Query the Impala Table

You can query data contained in the tables. Impala coordinates the query execution across a single node or multiple
nodes depending on your configuration, without the overhead of running MapReduce jobs to perform the intermediate
processing.
There are a variety of ways to execute queries on Impala:
• Using the impala-shell command in interactive mode:

$ impala-shell -i impala-host
Connected to localhost:21000

Apache Impala Guide | 55

Impala Tutorials

[impala-host:21000] > select count(*) from customer_address;

50000
Returned 1 row(s) in 0.37s

• Passing a set of commands contained in a file:

$ impala-shell -i impala-host -f myquery.sql

Connected to localhost:21000
50000
Returned 1 row(s) in 0.19s

• Passing a single command to the impala-shell command. The query is executed, the results are returned, and
the shell exits. Make sure to quote the command, preferably with single quotation marks to avoid shell expansion
of characters such as *.

$ impala-shell -i impala-host -q 'select count(*) from customer_address'

Connected to localhost:21000
50000
Returned 1 row(s) in 0.29s

Data Loading and Querying Examples

This section describes how to create some sample tables and load data into them. These tables can then be queried
using the Impala shell.

Loading Data
Loading data involves:
• Establishing a data set. The example below uses .csv files.
• Creating tables to which to load data.
• Loading the data into the tables you created.

Sample Queries
To run these sample queries, create a SQL query file query.sql, copy and paste each query into the query file, and
then run the query file using the shell. For example, to run query.sql on impala-host, you might use the command:

impala-shell.sh -i impala-host -f query.sql

The examples and results below assume you have loaded the sample data into the tables as described above.
Example: Examining Contents of Tables
Let's start by verifying that the tables do contain the data we expect. Because Impala often deals with tables containing
millions or billions of rows, when examining tables of unknown size, include the LIMIT clause to avoid huge amounts
of unnecessary output, as in the final query. (If your interactive query starts displaying an unexpected volume of data,
press Ctrl-C in impala-shell to cancel the query.)

SELECT * FROM tab1;

SELECT * FROM tab2;
SELECT * FROM tab2 LIMIT 5;

Results:

+----+-------+------------+-------------------------------+
| id | col_1 | col_2 | col_3 |
+----+-------+------------+-------------------------------+
| 1 | true | 123.123 | 2012-10-24 08:55:00 |
| 2 | false | 1243.5 | 2012-10-25 13:40:00 |
| 3 | false | 24453.325 | 2008-08-22 09:33:21.123000000 |
| 4 | false | 243423.325 | 2007-05-12 22:32:21.334540000 |

56 | Apache Impala Guide

 Impala Tutorials

| 5 | true | 243.325 | 1953-04-22 09:11:33 |

+----+-------+------------+-------------------------------+

+----+-------+---------------+
| id | col_1 | col_2 |
+----+-------+---------------+
| 1 | true | 12789.123 |
| 2 | false | 1243.5 |
| 3 | false | 24453.325 |
| 4 | false | 2423.3254 |
| 5 | true | 243.325 |
| 60 | false | 243565423.325 |
| 70 | true | 243.325 |
| 80 | false | 243423.325 |
| 90 | true | 243.325 |
+----+-------+---------------+

+----+-------+-----------+
| id | col_1 | col_2 |
+----+-------+-----------+
| 1 | true | 12789.123 |
| 2 | false | 1243.5 |
| 3 | false | 24453.325 |
| 4 | false | 2423.3254 |
| 5 | true | 243.325 |
+----+-------+-----------+

Example: Aggregate and Join

SELECT tab1.col_1, MAX(tab2.col_2), MIN(tab2.col_2)

FROM tab2 JOIN tab1 USING (id)
GROUP BY col_1 ORDER BY 1 LIMIT 5;

Results:

+-------+-----------------+-----------------+
| col_1 | max(tab2.col_2) | min(tab2.col_2) |
+-------+-----------------+-----------------+
| false | 24453.325 | 1243.5 |
| true | 12789.123 | 243.325 |
+-------+-----------------+-----------------+

Example: Subquery, Aggregate and Joins

SELECT tab2.*
FROM tab2,
(SELECT tab1.col_1, MAX(tab2.col_2) AS max_col2
FROM tab2, tab1
WHERE tab1.id = tab2.id
GROUP BY col_1) subquery1
WHERE subquery1.max_col2 = tab2.col_2;

Results:

+----+-------+-----------+
| id | col_1 | col_2 |
+----+-------+-----------+
| 1 | true | 12789.123 |
| 3 | false | 24453.325 |
+----+-------+-----------+

Example: INSERT Query

INSERT OVERWRITE TABLE tab3

SELECT id, col_1, col_2, MONTH(col_3), DAYOFMONTH(col_3)
FROM tab1 WHERE YEAR(col_3) = 2012;

Apache Impala Guide | 57

Impala Tutorials

Query TAB3 to check the result:

SELECT * FROM tab3;

Results:

+----+-------+---------+-------+-----+
| id | col_1 | col_2 | month | day |
+----+-------+---------+-------+-----+
| 1 | true | 123.123 | 10 | 24 |
| 2 | false | 1243.5 | 10 | 25 |
+----+-------+---------+-------+-----+

Advanced Tutorials
These tutorials walk you through advanced scenarios or specialized features.

Attaching an External Partitioned Table to an HDFS Directory Structure

This tutorial shows how you might set up a directory tree in HDFS, put data files into the lowest-level subdirectories,
and then use an Impala external table to query the data files from their original locations.
The tutorial uses a table with web log data, with separate subdirectories for the year, month, day, and host. For
simplicity, we use a tiny amount of CSV data, loading the same data into each partition.
First, we make an Impala partitioned table for CSV data, and look at the underlying HDFS directory structure to
understand the directory structure to re-create elsewhere in HDFS. The columns field1, field2, and field3
correspond to the contents of the CSV data files. The year, month, day, and host columns are all represented as
subdirectories within the table structure, and are not part of the CSV files. We use STRING for each of these columns
so that we can produce consistent subdirectory names, with leading zeros for a consistent length.

create database external_partitions;

use external_partitions;
create table logs (field1 string, field2 string, field3 string)
partitioned by (year string, month string , day string, host string)
row format delimited fields terminated by ',';
insert into logs partition (year="2013", month="07", day="28", host="host1") values
("foo","foo","foo");
insert into logs partition (year="2013", month="07", day="28", host="host2") values
("foo","foo","foo");
insert into logs partition (year="2013", month="07", day="29", host="host1") values
("foo","foo","foo");
insert into logs partition (year="2013", month="07", day="29", host="host2") values
("foo","foo","foo");
insert into logs partition (year="2013", month="08", day="01", host="host1") values
("foo","foo","foo");

Back in the Linux shell, we examine the HDFS directory structure. (Your Impala data directory might be in a different
location; for historical reasons, it is sometimes under the HDFS path /user/hive/warehouse.) We use the hdfs
dfs -ls command to examine the nested subdirectories corresponding to each partitioning column, with separate
subdirectories at each level (with = in their names) representing the different values for each partitioning column.
When we get to the lowest level of subdirectory, we use the hdfs dfs -cat command to examine the data file and
see CSV-formatted data produced by the INSERT statement in Impala.

$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db

Found 1 items
drwxrwxrwt - impala hive 0 2013-08-07 12:24
/user/impala/warehouse/external_partitions.db/logs
$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs
Found 1 items
drwxr-xr-x - impala hive 0 2013-08-07 12:24
/user/impala/warehouse/external_partitions.db/logs/year=2013
$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013

58 | Apache Impala Guide

 Impala Tutorials

Found 2 items
drwxr-xr-x - impala hive 0 2013-08-07 12:23
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07
drwxr-xr-x - impala hive 0 2013-08-07 12:24
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=08
$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07
Found 2 items
drwxr-xr-x - impala hive 0 2013-08-07 12:22
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28
drwxr-xr-x - impala hive 0 2013-08-07 12:23
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=29
$ hdfs dfs -ls
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28
Found 2 items
drwxr-xr-x - impala hive 0 2013-08-07 12:21
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host1
drwxr-xr-x - impala hive 0 2013-08-07 12:22
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host2
$ hdfs dfs -ls
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host1
Found 1 items
-rw-r--r-- 3 impala hive 12 2013-08-07 12:21
/user/impala/warehouse/external_partiti
ons.db/logs/year=2013/month=07/day=28/host=host1/3981726974111751120--8907184999369517436_822630111_data.0
$ hdfs dfs -cat
/user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/\
host=host1/3981726974111751120--8 907184999369517436_822630111_data.0
foo,foo,foo

Still in the Linux shell, we use hdfs dfs -mkdir to create several data directories outside the HDFS directory tree
that Impala controls (/user/impala/warehouse in this example, maybe different in your case). Depending on your
configuration, you might need to log in as a user with permission to write into this HDFS directory tree; for example,
the commands shown here were run while logged in as the hdfs user.

$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host1

$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host2
$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host1
$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=29/host=host1
$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=08/day=01/host=host1

We make a tiny CSV file, with values different than in the INSERT statements used earlier, and put a copy within each
subdirectory that we will use as an Impala partition.

$ cat >dummy_log_data
bar,baz,bletch
$ hdfs dfs -mkdir -p
/user/impala/data/external_partitions/year=2013/month=08/day=01/host=host1
$ hdfs dfs -mkdir -p
/user/impala/data/external_partitions/year=2013/month=07/day=28/host=host1
$ hdfs dfs -mkdir -p
/user/impala/data/external_partitions/year=2013/month=07/day=28/host=host2
$ hdfs dfs -mkdir -p
/user/impala/data/external_partitions/year=2013/month=07/day=29/host=host1
$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=28/host=host1
$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=28/host=host2
$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=29/host=host1
$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=08/day=01/host=host1

Back in the impala-shell interpreter, we move the original Impala-managed table aside, and create a new external
table with a LOCATION clause pointing to the directory under which we have set up all the partition subdirectories
and data files.

use external_partitions;
alter table logs rename to logs_original;
create external table logs (field1 string, field2 string, field3 string)
partitioned by (year string, month string, day string, host string)

Apache Impala Guide | 59

Impala Tutorials

row format delimited fields terminated by ','

location '/user/impala/data/logs';

Because partition subdirectories and data files come and go during the data lifecycle, you must identify each of the
partitions through an ALTER TABLE statement before Impala recognizes the data files they contain.

alter table logs add partition (year="2013",month="07",day="28",host="host1")

alter table log_type add partition (year="2013",month="07",day="28",host="host2");
alter table log_type add partition (year="2013",month="07",day="29",host="host1");
alter table log_type add partition (year="2013",month="08",day="01",host="host1");

We issue a REFRESH statement for the table, always a safe practice when data files have been manually added, removed,
or changed. Then the data is ready to be queried. The SELECT * statement illustrates that the data from our trivial
CSV file was recognized in each of the partitions where we copied it. Although in this case there are only a few rows,
we include a LIMIT clause on this test query just in case there is more data than we expect.

refresh log_type;
select * from log_type limit 100;
+--------+--------+--------+------+-------+-----+-------+
| field1 | field2 | field3 | year | month | day | host |
+--------+--------+--------+------+-------+-----+-------+
| bar | baz | bletch | 2013 | 07 | 28 | host1 |
| bar | baz | bletch | 2013 | 08 | 01 | host1 |
| bar | baz | bletch | 2013 | 07 | 29 | host1 |
| bar | baz | bletch | 2013 | 07 | 28 | host2 |
+--------+--------+--------+------+-------+-----+-------+

Switching Back and Forth Between Impala and Hive

Sometimes, you might find it convenient to switch to the Hive shell to perform some data loading or transformation
operation, particularly on file formats such as RCFile, SequenceFile, and Avro that Impala currently can query but not
write to.
Whenever you create, drop, or alter a table or other kind of object through Hive, the next time you switch back to the
impala-shell interpreter, issue a one-time INVALIDATE METADATA statement so that Impala recognizes the new
or changed object.
Whenever you load, insert, or change data in an existing table through Hive (or even through manual HDFS operations
such as the hdfs command), the next time you switch back to the impala-shell interpreter, issue a one-time
REFRESH table_name statement so that Impala recognizes the new or changed data.

For examples showing how this process works for the REFRESH statement, look at the examples of creating RCFile and
SequenceFile tables in Impala, loading data through Hive, and then querying the data through Impala. See Using the
RCFile File Format with Impala Tables on page 552 and Using the SequenceFile File Format with Impala Tables on page
554 for those examples.
For examples showing how this process works for the INVALIDATE METADATA statement, look at the example of
creating and loading an Avro table in Hive, and then querying the data through Impala. See Using the Avro File Format
with Impala Tables on page 547 for that example.

Note:
Originally, Impala did not support UDFs, but this feature is available in Impala starting in Impala 1.2.
Some INSERT ... SELECT transformations that you originally did through Hive can now be done
through Impala. See Impala User-Defined Functions (UDFs) on page 447 for details.
Prior to Impala 1.2, the REFRESH and INVALIDATE METADATA statements needed to be issued on
each Impala node to which you connected and issued queries. In Impala 1.2 and higher, when you
issue either of those statements on any Impala node, the results are broadcast to all the Impala nodes
in the cluster, making it truly a one-step operation after each round of DDL or ETL operations in Hive.

60 | Apache Impala Guide

 Impala Tutorials

Cross Joins and Cartesian Products with the CROSS JOIN Operator
Originally, Impala restricted join queries so that they had to include at least one equality comparison between the
columns of the tables on each side of the join operator. With the huge tables typically processed by Impala, any
miscoded query that produced a full Cartesian product as a result set could consume a huge amount of cluster resources.
In Impala 1.2.2 and higher, this restriction is lifted when you use the CROSS JOIN operator in the query. You still
cannot remove all WHERE clauses from a query like SELECT * FROM t1 JOIN t2 to produce all combinations of
rows from both tables. But you can use the CROSS JOIN operator to explicitly request such a Cartesian product.
Typically, this operation is applicable for smaller tables, where the result set still fits within the memory of a single
Impala node.
The following example sets up data for use in a series of comic books where characters battle each other. At first, we
use an equijoin query, which only allows characters from the same time period and the same planet to meet.

[localhost:21000] > create table heroes (name string, era string, planet string);
[localhost:21000] > create table villains (name string, era string, planet string);
[localhost:21000] > insert into heroes values
> ('Tesla','20th century','Earth'),
> ('Pythagoras','Antiquity','Earth'),
> ('Zopzar','Far Future','Mars');
Inserted 3 rows in 2.28s
[localhost:21000] > insert into villains values
> ('Caligula','Antiquity','Earth'),
> ('John Dillinger','20th century','Earth'),
> ('Xibulor','Far Future','Venus');
Inserted 3 rows in 1.93s
[localhost:21000] > select concat(heroes.name,' vs. ',villains.name) as battle
> from heroes join villains
> where heroes.era = villains.era and heroes.planet = villains.planet;
+--------------------------+
| battle |
+--------------------------+
| Tesla vs. John Dillinger |
| Pythagoras vs. Caligula |
+--------------------------+
Returned 2 row(s) in 0.47s

Readers demanded more action, so we added elements of time travel and space travel so that any hero could face
any villain. Prior to Impala 1.2.2, this type of query was impossible because all joins had to reference matching values
between the two tables:

[localhost:21000] > -- Cartesian product not possible in Impala 1.1.

> select concat(heroes.name,' vs. ',villains.name) as battle from
heroes join villains;
ERROR: NotImplementedException: Join between 'heroes' and 'villains' requires at least
one conjunctive equality predicate between the two tables

With Impala 1.2.2, we rewrite the query slightly to use CROSS JOIN rather than JOIN, and now the result set includes
all combinations:

[localhost:21000] > -- Cartesian product available in Impala 1.2.2 with the CROSS JOIN
syntax.
> select concat(heroes.name,' vs. ',villains.name) as battle from
heroes cross join villains;
+-------------------------------+
| battle |
+-------------------------------+
| Tesla vs. Caligula |
| Tesla vs. John Dillinger |
| Tesla vs. Xibulor |
| Pythagoras vs. Caligula |
| Pythagoras vs. John Dillinger |
| Pythagoras vs. Xibulor |
| Zopzar vs. Caligula |
| Zopzar vs. John Dillinger |
| Zopzar vs. Xibulor |

Apache Impala Guide | 61

Impala Tutorials

+-------------------------------+
Returned 9 row(s) in 0.33s

The full combination of rows from both tables is known as the Cartesian product. This type of result set is often used
for creating grid data structures. You can also filter the result set by including WHERE clauses that do not explicitly
compare columns between the two tables. The following example shows how you might produce a list of combinations
of year and quarter for use in a chart, and then a shorter list with only selected quarters.

[localhost:21000] > create table x_axis (x int);

[localhost:21000] > create table y_axis (y int);
[localhost:21000] > insert into x_axis values (1),(2),(3),(4);
Inserted 4 rows in 2.14s
[localhost:21000] > insert into y_axis values (2010),(2011),(2012),(2013),(2014);
Inserted 5 rows in 1.32s
[localhost:21000] > select y as year, x as quarter from x_axis cross join y_axis;
+------+---------+
| year | quarter |
+------+---------+
| 2010 | 1 |
| 2011 | 1 |
| 2012 | 1 |
| 2013 | 1 |
| 2014 | 1 |
| 2010 | 2 |
| 2011 | 2 |
| 2012 | 2 |
| 2013 | 2 |
| 2014 | 2 |
| 2010 | 3 |
| 2011 | 3 |
| 2012 | 3 |
| 2013 | 3 |
| 2014 | 3 |
| 2010 | 4 |
| 2011 | 4 |
| 2012 | 4 |
| 2013 | 4 |
| 2014 | 4 |
+------+---------+
Returned 20 row(s) in 0.38s
[localhost:21000] > select y as year, x as quarter from x_axis cross join y_axis where
x in (1,3);
+------+---------+
| year | quarter |
+------+---------+
| 2010 | 1 |
| 2011 | 1 |
| 2012 | 1 |
| 2013 | 1 |
| 2014 | 1 |
| 2010 | 3 |
| 2011 | 3 |
| 2012 | 3 |
| 2013 | 3 |
| 2014 | 3 |
+------+---------+
Returned 10 row(s) in 0.39s

Dealing with Parquet Files with Unknown Schema

As data pipelines start to include more aspects such as NoSQL or loosely specified schemas, you might encounter
situations where you have data files (particularly in Parquet format) where you do not know the precise table definition.
This tutorial shows how you can build an Impala table around data that comes from non-Impala or even non-SQL
sources, where you do not have control of the table layout and might not be familiar with the characteristics of the
data.

62 | Apache Impala Guide

 Impala Tutorials

The data used in this tutorial represents airline on-time arrival statistics, from October 1987 through April 2008. See
the details on the 2009 ASA Data Expo web site. You can also see the explanations of the columns; for purposes of this
exercise, wait until after following the tutorial before examining the schema, to better simulate a real-life situation
where you cannot rely on assumptions and assertions about the ranges and representations of data values.
We will download Parquet files containing this data from the Ibis blog. First, we download and unpack the data files.
There are 8 files totalling 1.4 GB. Each file is less than 256 MB.

$ wget -O airlines_parquet.tar.gz
https://www.dropbox.com/s/ol9x51tqp6cv4yc/airlines_parquet.tar.gz?dl=0
...
Length: 1245204740 (1.2G) [application/octet-stream]
Saving to: “airlines_parquet.tar.gz”

2015-08-12 17:14:24 (23.6 MB/s) - “airlines_parquet.tar.gz” saved [1245204740/1245204740]

$ tar xvzf airlines_parquet.tar.gz

airlines_parquet/
airlines_parquet/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
airlines_parquet/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
airlines_parquet/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
airlines_parquet/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
airlines_parquet/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
airlines_parquet/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
airlines_parquet/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
airlines_parquet/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
$ cd airlines_parquet/
$ du -kch *.parq
253M 93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
65M 93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
156M 93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
240M 93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
253M 93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
16M 93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
177M 93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
213M 93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
1.4G total

Next, we put the Parquet data files in HDFS, all together in a single directory, with permissions on the directory and
the files so that the impala user will be able to read them.

Note: After unpacking, we saw the largest Parquet file was 253 MB. When copying Parquet files into
HDFS for Impala to use, for maximum query performance, make sure that each file resides in a single
HDFS data block. Therefore, we pick a size larger than any single file and specify that as the block size,
using the argument -Ddfs.block.size=256m on the hdfs dfs -put command.

$ hdfs dfs -mkdir -p hdfs://demo_host.example.com:8020/user/impala/staging/airlines

$ hdfs dfs -Ddfs.block.size=256m -put *.parq /user/impala/staging/airlines
$ hdfs dfs -ls /user/impala/staging
Found 1 items
drwxrwxrwx - hdfs supergroup 0 2015-08-12 13:52 /user/impala/staging/airlines
$ hdfs dfs -ls hdfs://demo_host.example.com:8020/user/impala/staging/airlines
Found 8 items
-rw-r--r-- 3 jrussell supergroup 265107489 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
-rw-r--r-- 3 jrussell supergroup 67544715 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
-rw-r--r-- 3 jrussell supergroup 162556490 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
-rw-r--r-- 3 jrussell supergroup 251603518 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
-rw-r--r-- 3 jrussell supergroup 265186603 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
-rw-r--r-- 3 jrussell supergroup 16663754 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
-rw-r--r-- 3 jrussell supergroup 185511677 2015-08-12 17:18
/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq

Apache Impala Guide | 63

Impala Tutorials

-rw-r--r-- 3 jrussell supergroup 222794621 2015-08-12 17:18

/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq

With the files in an accessible location in HDFS, we create a database table that uses the data in those files. The CREATE
EXTERNAL syntax and the LOCATION attribute point Impala at the appropriate HDFS directory. The LIKE PARQUET
'path_to_any_parquet_file' clause means we skip the list of column names and types; Impala automatically
gets the column names and data types straight from the data files. (Currently, this technique only works for Parquet
files.) We ignore the warning about lack of READ_WRITE access to the files in HDFS; the impala user can read the
files, which will be sufficient for us to experiment with queries and perform some copy and transform operations into
other tables.

$ impala-shell -i localhost
Starting Impala Shell without Kerberos authentication
Connected to localhost:21000
Server version: impalad version 2.2.0-cdh5 RELEASE (build
2ffd73a4255cefd521362ffe1cfb37463f67f75c)
Welcome to the Impala shell. Press TAB twice to see a list of available commands.

Copyright (c) 2012 Cloudera, Inc. All rights reserved.

(Shell build version: Impala Shell v2.1.2-cdh5 (92438b7) built on Tue Feb 24 12:36:33
PST 2015)
[localhost:21000] > create database airline_data;
[localhost:21000] > use airline_data;
[localhost:21000] > create external table airlines_external
> like parquet
'hdfs://demo_host.example.com:8020/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq'

> stored as parquet location

'hdfs://demo_host.example.com:8020/user/impala/staging/airlines';
WARNINGS: Impala does not have READ_WRITE access to path
'hdfs://demo_host.example.com:8020/user/impala/staging'

With the table created, we examine its physical and logical characteristics to confirm that the data is really there and
in a format and shape that we can work with. The SHOW TABLE STATS statement gives a very high-level summary of
the table, showing how many files and how much total data it contains. Also, it confirms that the table is expecting all
the associated data files to be in Parquet format. (The ability to work with all kinds of HDFS data files in different
formats means that it is possible to have a mismatch between the format of the data files, and the format that the
table expects the data files to be in.) The SHOW FILES statement confirms that the data in the table has the expected
number, names, and sizes of the original Parquet files. The DESCRIBE statement (or its abbreviation DESC) confirms
the names and types of the columns that Impala automatically created after reading that metadata from the Parquet
file. The DESCRIBE FORMATTED statement prints out some extra detail along with the column definitions; the pieces
we care about for this exercise are the containing database for the table, the location of the associated data files in
HDFS, the fact that it's an external table so Impala will not delete the HDFS files when we finish the experiments and
drop the table, and the fact that the table is set up to work exclusively with files in the Parquet format.

[localhost:21000] > show table stats airlines_external;

+-------+--------+--------+--------------+-------------------+---------+-------------------+
| #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental
stats |
+-------+--------+--------+--------------+-------------------+---------+-------------------+
| -1 | 8 | 1.34GB | NOT CACHED | NOT CACHED | PARQUET | false
|
+-------+--------+--------+--------------+-------------------+---------+-------------------+
[localhost:21000] > show files in airlines_external;
+----------------------------------------------------------------------------------------+----------+-----------+
| path
| size | partition |
+----------------------------------------------------------------------------------------+----------+-----------+
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
| 252.83MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
| 64.42MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
| 155.03MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq

64 | Apache Impala Guide

 Impala Tutorials

| 239.95MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
| 252.90MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
| 15.89MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
| 176.92MB | |
| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
| 212.47MB | |
+----------------------------------------------------------------------------------------+----------+-----------+
[localhost:21000] > describe airlines_external;
+---------------------+--------+---------------------------------------------------+
| name | type | comment |
+---------------------+--------+---------------------------------------------------+
| year | int | inferred from: optional int32 year |
| month | int | inferred from: optional int32 month |
| day | int | inferred from: optional int32 day |
| dayofweek | int | inferred from: optional int32 dayofweek |
| dep_time | int | inferred from: optional int32 dep_time |
| crs_dep_time | int | inferred from: optional int32 crs_dep_time |
| arr_time | int | inferred from: optional int32 arr_time |
| crs_arr_time | int | inferred from: optional int32 crs_arr_time |
| carrier | string | inferred from: optional binary carrier |
| flight_num | int | inferred from: optional int32 flight_num |
| tail_num | int | inferred from: optional int32 tail_num |
| actual_elapsed_time | int | inferred from: optional int32 actual_elapsed_time |
| crs_elapsed_time | int | inferred from: optional int32 crs_elapsed_time |
| airtime | int | inferred from: optional int32 airtime |
| arrdelay | int | inferred from: optional int32 arrdelay |
| depdelay | int | inferred from: optional int32 depdelay |
| origin | string | inferred from: optional binary origin |
| dest | string | inferred from: optional binary dest |
| distance | int | inferred from: optional int32 distance |
| taxi_in | int | inferred from: optional int32 taxi_in |
| taxi_out | int | inferred from: optional int32 taxi_out |
| cancelled | int | inferred from: optional int32 cancelled |
| cancellation_code | string | inferred from: optional binary cancellation_code |
| diverted | int | inferred from: optional int32 diverted |
| carrier_delay | int | inferred from: optional int32 carrier_delay |
| weather_delay | int | inferred from: optional int32 weather_delay |
| nas_delay | int | inferred from: optional int32 nas_delay |
| security_delay | int | inferred from: optional int32 security_delay |
| late_aircraft_delay | int | inferred from: optional int32 late_aircraft_delay |
+---------------------+--------+---------------------------------------------------+
[localhost:21000] > desc formatted airlines_external;
+------------------------------+-------------------------------
| name | type
+------------------------------+-------------------------------
...
| # Detailed Table Information | NULL
| Database: | airline_data
| Owner: | jrussell
...
| Location: | /user/impala/staging/airlines
| Table Type: | EXTERNAL_TABLE
...
| # Storage Information | NULL
| SerDe Library: | parquet.hive.serde.ParquetHiveSerDe
| InputFormat: | parquet.hive.DeprecatedParquetInputFormat
| OutputFormat: | parquet.hive.DeprecatedParquetOutputFormat
...

Now that we are confident that the connections are solid between the Impala table and the underlying Parquet files,
we run some initial queries to understand the characteristics of the data: the overall number of rows, and the ranges
and how many different values are in certain columns. For convenience in understanding the magnitude of the COUNT(*)
result, we run another query dividing the number of rows by 1 million, demonstrating that there are 123 million rows
in the table.

[localhost:21000] > select count(*) from airlines_external;

+-----------+
| count(*) |

Apache Impala Guide | 65

Impala Tutorials

+-----------+
| 123534969 |
+-----------+
Fetched 1 row(s) in 1.32s
[localhost:21000] > select count(*) / 1e6 as 'millions of rows' from airlines_external;
+------------------+
| millions of rows |
+------------------+
| 123.534969 |
+------------------+
Fetched 1 row(s) in 1.24s

The NDV() function stands for “number of distinct values”, which for performance reasons is an estimate when there
are lots of different values in the column, but is precise when the cardinality is less than 16 K. Use NDV() calls for this
kind of exploration rather than COUNT(DISTINCT colname), because Impala can evaluate multiple NDV() functions
in a single query, but only a single instance of COUNT DISTINCT. Here we see that there are modest numbers of
different airlines, flight numbers, and origin and destination airports. Two things jump out from this query: the number
of tail_num values is much smaller than we might have expected, and there are more destination airports than origin
airports. Let's dig further. What we find is that most tail_num values are NULL. It looks like this was an experimental
column that wasn't filled in accurately. We make a mental note that if we use this data as a starting point, we'll ignore
this column. We also find that certain airports are represented in the ORIGIN column but not the DEST column; now
we know that we cannot rely on the assumption that those sets of airport codes are identical.

Note: A slight digression for some performance tuning. Notice how the first SELECT DISTINCT
DEST query takes almost 40 seconds. We expect all queries on such a small data set, less than 2 GB,
to take a few seconds at most. The reason is because the expression NOT IN (SELECT origin
FROM airlines_external) produces an intermediate result set of 123 million rows, then runs 123
million comparisons on each data node against the tiny set of destination airports. The way the NOT
IN operator works internally means that this intermediate result set with 123 million rows might be
transmitted across the network to each data node in the cluster. Applying another DISTINCT inside
the NOT IN subquery means that the intermediate result set is only 340 items, resulting in much less
network traffic and fewer comparison operations. The more efficient query with the added DISTINCT
is approximately 7 times as fast.

[localhost:21000] > select ndv(carrier), ndv(flight_num), ndv(tail_num),

> ndv(origin), ndv(dest) from airlines_external;
+--------------+-----------------+---------------+-------------+-----------+
| ndv(carrier) | ndv(flight_num) | ndv(tail_num) | ndv(origin) | ndv(dest) |
+--------------+-----------------+---------------+-------------+-----------+
| 29 | 9086 | 3 | 340 | 347 |
+--------------+-----------------+---------------+-------------+-----------+
[localhost:21000] > select tail_num, count(*) as howmany from airlines_external
> group by tail_num;
+----------+-----------+
| tail_num | howmany |
+----------+-----------+
| 715 | 1 |
| 0 | 406405 |
| 112 | 6562 |
| NULL | 123122001 |
+----------+-----------+
Fetched 1 row(s) in 5.18s
[localhost:21000] > select distinct dest from airlines_external
> where dest not in (select origin from airlines_external);
+------+
| dest |
+------+
| LBF |
| CBM |
| RCA |
| SKA |
| LAR |
+------+
Fetched 5 row(s) in 39.64s
[localhost:21000] > select distinct dest from airlines_external

66 | Apache Impala Guide

 Impala Tutorials

> where dest not in (select distinct origin from airlines_external);

+------+
| dest |
+------+
| LBF |
| RCA |
| CBM |
| SKA |
| LAR |
+------+
Fetched 5 row(s) in 5.59s
[localhost:21000] > select distinct origin from airlines_external
> where origin not in (select distinct dest from airlines_external);
Fetched 0 row(s) in 5.37s

Next, we try doing a simple calculation, with results broken down by year. This reveals that some years have no data
in the AIRTIME column. That means we might be able to use that column in queries involving certain date ranges, but
we cannot count on it to always be reliable. The question of whether a column contains any NULL values, and if so
what is their number, proportion, and distribution, comes up again and again when doing initial exploration of a data
set.

[localhost:21000] > select year, sum(airtime) from airlines_external

> group by year order by year desc;
+------+--------------+
| year | sum(airtime) |
+------+--------------+
| 2008 | 713050445 |
| 2007 | 748015545 |
| 2006 | 720372850 |
| 2005 | 708204026 |
| 2004 | 714276973 |
| 2003 | 665706940 |
| 2002 | 549761849 |
| 2001 | 590867745 |
| 2000 | 583537683 |
| 1999 | 561219227 |
| 1998 | 538050663 |
| 1997 | 536991229 |
| 1996 | 519440044 |
| 1995 | 513364265 |
| 1994 | NULL |
| 1993 | NULL |
| 1992 | NULL |
| 1991 | NULL |
| 1990 | NULL |
| 1989 | NULL |
| 1988 | NULL |
| 1987 | NULL |
+------+--------------+

With the notion of NULL values in mind, let's come back to the TAILNUM column that we discovered had a lot of NULLs.
Let's quantify the NULL and non-NULL values in that column for better understanding. First, we just count the overall
number of rows versus the non-NULL values in that column. That initial result gives the appearance of relatively few
non-NULL values, but we can break it down more clearly in a single query. Once we have the COUNT(*) and the
COUNT(colname) numbers, we can encode that initial query in a WITH clause, then run a followon query that performs
multiple arithmetic operations on those values. Seeing that only one-third of one percent of all rows have non-NULL
values for the TAILNUM column clearly illustrates that that column won't be of much use.

[localhost:21000] > select count(*) as 'rows', count(tail_num) as 'non-null tail numbers'

> from airlines_external;

+-----------+-----------------------+
| rows | non-null tail numbers |
+-----------+-----------------------+
| 123534969 | 412968 |
+-----------+-----------------------+
Fetched 1 row(s) in 1.51s
[localhost:21000] > with t1 as

Apache Impala Guide | 67

Impala Tutorials

> (select count(*) as 'rows', count(tail_num) as 'nonnull'

> from airlines_external)
> select `rows`, `nonnull`, `rows` - `nonnull` as 'nulls',
> (`nonnull` / `rows`) * 100 as 'percentage non-null'
> from t1;
+-----------+---------+-----------+---------------------+
| rows | nonnull | nulls | percentage non-null |
+-----------+---------+-----------+---------------------+
| 123534969 | 412968 | 123122001 | 0.3342923897119365 |
+-----------+---------+-----------+---------------------+

By examining other columns using these techniques, we can form a mental picture of the way data is distributed
throughout the table, and which columns are most significant for query purposes. For this tutorial, we focus mostly
on the fields likely to hold discrete values, rather than columns such as ACTUAL_ELAPSED_TIME whose names suggest
they hold measurements. We would dig deeper into those columns once we had a clear picture of which questions
were worthwhile to ask, and what kinds of trends we might look for. For the final piece of initial exploration, let's look
at the YEAR column. A simple GROUP BY query shows that it has a well-defined range, a manageable number of distinct
values, and relatively even distribution of rows across the different years.

[localhost:21000] > select min(year), max(year), ndv(year) from airlines_external;

+-----------+-----------+-----------+
| min(year) | max(year) | ndv(year) |
+-----------+-----------+-----------+
| 1987 | 2008 | 22 |
+-----------+-----------+-----------+
Fetched 1 row(s) in 2.03s
[localhost:21000] > select year, count(*) howmany from airlines_external
> group by year order by year desc;
+------+---------+
| year | howmany |
+------+---------+
| 2008 | 7009728 |
| 2007 | 7453215 |
| 2006 | 7141922 |
| 2005 | 7140596 |
| 2004 | 7129270 |
| 2003 | 6488540 |
| 2002 | 5271359 |
| 2001 | 5967780 |
| 2000 | 5683047 |
| 1999 | 5527884 |
| 1998 | 5384721 |
| 1997 | 5411843 |
| 1996 | 5351983 |
| 1995 | 5327435 |
| 1994 | 5180048 |
| 1993 | 5070501 |
| 1992 | 5092157 |
| 1991 | 5076925 |
| 1990 | 5270893 |
| 1989 | 5041200 |
| 1988 | 5202096 |
| 1987 | 1311826 |
+------+---------+
Fetched 22 row(s) in 2.13s

We could go quite far with the data in this initial raw format, just as we downloaded it from the web. If the data set
proved to be useful and worth persisting in Impala for extensive queries, we might want to copy it to an internal table,
letting Impala manage the data files and perhaps reorganizing a little for higher efficiency. In this next stage of the
tutorial, we copy the original data into a partitioned table, still in Parquet format. Partitioning based on the YEAR
column lets us run queries with clauses such as WHERE year = 2001 or WHERE year BETWEEN 1989 AND 1999,
which can dramatically cut down on I/O by ignoring all the data from years outside the desired range. Rather than
reading all the data and then deciding which rows are in the matching years, Impala can zero in on only the data files
from specific YEAR partitions. To do this, Impala physically reorganizes the data files, putting the rows from each year
into data files in a separate HDFS directory for each YEAR value. Along the way, we'll also get rid of the TAIL_NUM
column that proved to be almost entirely NULL.

68 | Apache Impala Guide

 Impala Tutorials

The first step is to create a new table with a layout very similar to the original AIRLINES_EXTERNAL table. We'll do
that by reverse-engineering a CREATE TABLE statement for the first table, then tweaking it slightly to include a
PARTITION BY clause for YEAR, and excluding the TAIL_NUM column. The SHOW CREATE TABLE statement gives us
the starting point.

[localhost:21000] > show create table airlines_external;

+-------------------------------------------------------------------------------------
| result
+-------------------------------------------------------------------------------------
| CREATE EXTERNAL TABLE airline_data.airlines_external (
| year INT COMMENT 'inferred from: optional int32 year',
| month INT COMMENT 'inferred from: optional int32 month',
| day INT COMMENT 'inferred from: optional int32 day',
| dayofweek INT COMMENT 'inferred from: optional int32 dayofweek',
| dep_time INT COMMENT 'inferred from: optional int32 dep_time',
| crs_dep_time INT COMMENT 'inferred from: optional int32 crs_dep_time',
| arr_time INT COMMENT 'inferred from: optional int32 arr_time',
| crs_arr_time INT COMMENT 'inferred from: optional int32 crs_arr_time',
| carrier STRING COMMENT 'inferred from: optional binary carrier',
| flight_num INT COMMENT 'inferred from: optional int32 flight_num',
| tail_num INT COMMENT 'inferred from: optional int32 tail_num',
| actual_elapsed_time INT COMMENT 'inferred from: optional int32 actual_elapsed_time',
| crs_elapsed_time INT COMMENT 'inferred from: optional int32 crs_elapsed_time',
| airtime INT COMMENT 'inferred from: optional int32 airtime',
| arrdelay INT COMMENT 'inferred from: optional int32 arrdelay',
| depdelay INT COMMENT 'inferred from: optional int32 depdelay',
| origin STRING COMMENT 'inferred from: optional binary origin',
| dest STRING COMMENT 'inferred from: optional binary dest',
| distance INT COMMENT 'inferred from: optional int32 distance',
| taxi_in INT COMMENT 'inferred from: optional int32 taxi_in',
| taxi_out INT COMMENT 'inferred from: optional int32 taxi_out',
| cancelled INT COMMENT 'inferred from: optional int32 cancelled',
| cancellation_code STRING COMMENT 'inferred from: optional binary cancellation_code',
| diverted INT COMMENT 'inferred from: optional int32 diverted',
| carrier_delay INT COMMENT 'inferred from: optional int32 carrier_delay',
| weather_delay INT COMMENT 'inferred from: optional int32 weather_delay',
| nas_delay INT COMMENT 'inferred from: optional int32 nas_delay',
| security_delay INT COMMENT 'inferred from: optional int32 security_delay',
| late_aircraft_delay INT COMMENT 'inferred from: optional int32 late_aircraft_delay'
| )
| STORED AS PARQUET
| LOCATION 'hdfs://a1730.example.com:8020/user/impala/staging/airlines'
| TBLPROPERTIES ('numFiles'='0', 'COLUMN_STATS_ACCURATE'='false',
| 'transient_lastDdlTime'='1439425228', 'numRows'='-1', 'totalSize'='0',
| 'rawDataSize'='-1')
+-------------------------------------------------------------------------------------
Fetched 1 row(s) in 0.03s
[localhost:21000] > quit;

Although we could edit that output into a new SQL statement, all the ASCII box characters make such editing
inconvenient. To get a more stripped-down CREATE TABLE to start with, we restart the impala-shell command
with the -B option, which turns off the box-drawing behavior.

[localhost:21000] > quit;

Goodbye jrussell
$ impala-shell -i localhost -B -d airline_data;
Starting Impala Shell without Kerberos authentication
Connected to localhost:21000
Server version: impalad version 2.2.0-cdh5 RELEASE (build
2ffd73a4255cefd521362ffe1cfb37463f67f75c)
Welcome to the Impala shell. Press TAB twice to see a list of available commands.

Copyright (c) 2012 Cloudera, Inc. All rights reserved.

(Shell build version: Impala Shell v2.1.2-cdh5 (92438b7) built on Tue Feb 24 12:36:33
PST 2015)
[localhost:21000] > show create table airlines_external;
"CREATE EXTERNAL TABLE airline_data.airlines_external (
year INT COMMENT 'inferred from: optional int32 year',
month INT COMMENT 'inferred from: optional int32 month',
day INT COMMENT 'inferred from: optional int32 day',

Apache Impala Guide | 69

Impala Tutorials

dayofweek INT COMMENT 'inferred from: optional int32 dayofweek',

dep_time INT COMMENT 'inferred from: optional int32 dep_time',
crs_dep_time INT COMMENT 'inferred from: optional int32 crs_dep_time',
arr_time INT COMMENT 'inferred from: optional int32 arr_time',
crs_arr_time INT COMMENT 'inferred from: optional int32 crs_arr_time',
carrier STRING COMMENT 'inferred from: optional binary carrier',
flight_num INT COMMENT 'inferred from: optional int32 flight_num',
tail_num INT COMMENT 'inferred from: optional int32 tail_num',
actual_elapsed_time INT COMMENT 'inferred from: optional int32 actual_elapsed_time',
crs_elapsed_time INT COMMENT 'inferred from: optional int32 crs_elapsed_time',
airtime INT COMMENT 'inferred from: optional int32 airtime',
arrdelay INT COMMENT 'inferred from: optional int32 arrdelay',
depdelay INT COMMENT 'inferred from: optional int32 depdelay',
origin STRING COMMENT 'inferred from: optional binary origin',
dest STRING COMMENT 'inferred from: optional binary dest',
distance INT COMMENT 'inferred from: optional int32 distance',
taxi_in INT COMMENT 'inferred from: optional int32 taxi_in',
taxi_out INT COMMENT 'inferred from: optional int32 taxi_out',
cancelled INT COMMENT 'inferred from: optional int32 cancelled',
cancellation_code STRING COMMENT 'inferred from: optional binary cancellation_code',
diverted INT COMMENT 'inferred from: optional int32 diverted',
carrier_delay INT COMMENT 'inferred from: optional int32 carrier_delay',
weather_delay INT COMMENT 'inferred from: optional int32 weather_delay',
nas_delay INT COMMENT 'inferred from: optional int32 nas_delay',
security_delay INT COMMENT 'inferred from: optional int32 security_delay',
late_aircraft_delay INT COMMENT 'inferred from: optional int32 late_aircraft_delay'
)
STORED AS PARQUET
LOCATION 'hdfs://a1730.example.com:8020/user/impala/staging/airlines'
TBLPROPERTIES ('numFiles'='0', 'COLUMN_STATS_ACCURATE'='false',
'transient_lastDdlTime'='1439425228', 'numRows'='-1', 'totalSize'='0',
'rawDataSize'='-1')"
Fetched 1 row(s) in 0.01s

After copying and pasting the CREATE TABLE statement into a text editor for fine-tuning, we quit and restart
impala-shell without the -B option, to switch back to regular output.

Next we run the CREATE TABLE statement that we adapted from the SHOW CREATE TABLE output. We kept the
STORED AS PARQUET clause because we want to rearrange the data somewhat but still keep it in the high-performance
Parquet format. The LOCATION and TBLPROPERTIES clauses are not relevant for this new table, so we edit those out.
Because we are going to partition the new table based on the YEAR column, we move that column name (and its type)
into a new PARTITIONED BY clause.

[localhost:21000] > CREATE TABLE airline_data.airlines

> (
> month INT,
> day INT,
> dayofweek INT,
> dep_time INT,
> crs_dep_time INT,
> arr_time INT,
> crs_arr_time INT,
> carrier STRING,
> flight_num INT,
> actual_elapsed_time INT,
> crs_elapsed_time INT,
> airtime INT,
> arrdelay INT,
> depdelay INT,
> origin STRING,
> dest STRING,
> distance INT,
> taxi_in INT,
> taxi_out INT,
> cancelled INT,
> cancellation_code STRING,
> diverted INT,
> carrier_delay INT,
> weather_delay INT,
> nas_delay INT,
> security_delay INT,

70 | Apache Impala Guide

 Impala Tutorials

> late_aircraft_delay INT

> )
> STORED AS PARQUET
> PARTITIONED BY (year INT);
Fetched 0 row(s) in 0.10s

Next, we copy all the rows from the original table into this new one with an INSERT statement. (We edited the CREATE
TABLE statement to make an INSERT statement with the column names in the same order.) The only change is to add
a PARTITION(year) clause, and move the YEAR column to the very end of the SELECT list of the INSERT statement.
Specifying PARTITION(year), rather than a fixed value such as PARTITION(year=2000), means that Impala figures
out the partition value for each row based on the value of the very last column in the SELECT list. This is the first SQL
statement that legitimately takes any substantial time, because the rows from different years are shuffled around the
cluster; the rows that go into each partition are collected on one node, before being written to one or more new data
files.

[localhost:21000] > INSERT INTO airline_data.airlines

> PARTITION (year)
> SELECT
> month,
> day,
> dayofweek,
> dep_time,
> crs_dep_time,
> arr_time,
> crs_arr_time,
> carrier,
> flight_num,
> actual_elapsed_time,
> crs_elapsed_time,
> airtime,
> arrdelay,
> depdelay,
> origin,
> dest,
> distance,
> taxi_in,
> taxi_out,
> cancelled,
> cancellation_code,
> diverted,
> carrier_delay,
> weather_delay,
> nas_delay,
> security_delay,
> late_aircraft_delay,
> year
> FROM airline_data.airlines_external;
Inserted 123534969 row(s) in 202.70s

Once partitioning or join queries come into play, it's important to have statistics that Impala can use to optimize queries
on the corresponding tables. The COMPUTE INCREMENTAL STATS statement is the way to collect statistics for
partitioned tables. Then the SHOW TABLE STATS statement confirms that the statistics are in place for each partition,
and also illustrates how many files and how much raw data is in each partition.

[localhost:21000] > compute incremental stats airlines;

+-------------------------------------------+
| summary |
+-------------------------------------------+
| Updated 22 partition(s) and 27 column(s). |
+-------------------------------------------+
[localhost:21000] > show table stats airlines;
+-------+-----------+--------+----------+--------------+------------+---------+-------------------+
| year | #Rows | #Files | Size | Bytes Cached | Cache Repl | Format |
Incremental stats |
+-------+-----------+--------+----------+--------------+------------+---------+-----
| 1987 | 1311826 | 1 | 9.32MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1988 | 5202096 | 1 | 37.04MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1989 | 5041200 | 1 | 36.25MB | NOT CACHED | NOT CACHED | PARQUET | true

Apache Impala Guide | 71

Impala Tutorials

| 1990 | 5270893 | 1 | 38.39MB | NOT CACHED | NOT CACHED | PARQUET | true

| 1991 | 5076925 | 1 | 37.23MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1992 | 5092157 | 1 | 36.85MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1993 | 5070501 | 1 | 37.16MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1994 | 5180048 | 1 | 38.31MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1995 | 5327435 | 1 | 53.14MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1996 | 5351983 | 1 | 53.64MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1997 | 5411843 | 1 | 54.41MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1998 | 5384721 | 1 | 54.01MB | NOT CACHED | NOT CACHED | PARQUET | true
| 1999 | 5527884 | 1 | 56.32MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2000 | 5683047 | 1 | 58.15MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2001 | 5967780 | 1 | 60.65MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2002 | 5271359 | 1 | 57.99MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2003 | 6488540 | 1 | 81.33MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2004 | 7129270 | 1 | 103.19MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2005 | 7140596 | 1 | 102.61MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2006 | 7141922 | 1 | 106.03MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2007 | 7453215 | 1 | 112.15MB | NOT CACHED | NOT CACHED | PARQUET | true
| 2008 | 7009728 | 1 | 105.76MB | NOT CACHED | NOT CACHED | PARQUET | true
| Total | 123534969 | 22 | 1.30GB | 0B | | |
+-------+-----------+--------+----------+--------------+------------+---------+-----

At this point, we go through a quick thought process to sanity check the partitioning we did. All the partitions have
exactly one file, which is on the low side. A query that includes a clause WHERE year=2004 will only read a single data
block; that data block will be read and processed by a single data node; therefore, for a query targeting a single year,
all the other nodes in the cluster will sit idle while all the work happens on a single machine. It's even possible that by
chance (depending on HDFS replication factor and the way data blocks are distributed across the cluster), that multiple
year partitions selected by a filter such as WHERE year BETWEEN 1999 AND 2001 could all be read and processed
by the same data node. The more data files each partition has, the more parallelism you can get and the less probability
of “hotspots” occurring on particular nodes, therefore a bigger performance boost by having a big CDH cluster.
However, the more data files, the less data goes in each one. The overhead of dividing the work in a parallel query
might not be worth it if each node is only reading a few megabytes. 50 or 100 megabytes is a decent size for a Parquet
data block; 9 or 37 megabytes is on the small side. Which is to say, the data distribution we ended up with based on
this partitioning scheme is on the borderline between sensible (reasonably large files) and suboptimal (few files in
each partition). The way to see how well it works in practice is to run the same queries against the original flat table
and the new partitioned table, and compare times.
Spoiler: in this case, with my particular 4-node cluster with its specific distribution of data blocks and my particular
exploratory queries, queries against the partitioned table do consistently run faster than the same queries against the
unpartitioned table. But I could not be sure that would be the case without some real measurements. Here are some
queries I ran to draw that conclusion, first against AIRLINES_EXTERNAL (no partitioning), then against AIRLINES
(partitioned by year). The AIRLINES queries are consistently faster. Changing the volume of data, changing the size
of the cluster, running queries that did or didn't refer to the partition key columns, or other factors could change the
results to favor one table layout or the other.

Note: If you find the volume of each partition is only in the low tens of megabytes, consider lowering
the granularity of partitioning. For example, instead of partitioning by year, month, and day, partition
by year and month or even just by year. The ideal layout to distribute work efficiently in a parallel
query is many tens or even hundreds of megabytes per Parquet file, and the number of Parquet files
in each partition somewhat higher than the number of data nodes.

[localhost:21000] > select sum(airtime) from airlines_external;

+--------------+
| sum(airtime) |
+--------------+
| 8662859484 |
+--------------+
Fetched 1 row(s) in 2.02s
[localhost:21000] > select sum(airtime) from airlines;
+--------------+
| sum(airtime) |
+--------------+

72 | Apache Impala Guide

 Impala Tutorials

| 8662859484 |
+--------------+
Fetched 1 row(s) in 1.21s

[localhost:21000] > select sum(airtime) from airlines_external where year = 2005;

+--------------+
| sum(airtime) |
+--------------+
| 708204026 |
+--------------+
Fetched 1 row(s) in 2.61s
[localhost:21000] > select sum(airtime) from airlines where year = 2005;
+--------------+
| sum(airtime) |
+--------------+
| 708204026 |
+--------------+
Fetched 1 row(s) in 1.19s

[localhost:21000] > select sum(airtime) from airlines_external where dayofweek = 1;

+--------------+
| sum(airtime) |
+--------------+
| 1264945051 |
+--------------+
Fetched 1 row(s) in 2.82s
[localhost:21000] > select sum(airtime) from airlines where dayofweek = 1;
+--------------+
| sum(airtime) |
+--------------+
| 1264945051 |
+--------------+
Fetched 1 row(s) in 1.61s

Now we can finally do some serious analysis with this data set that, remember, a few minutes ago all we had were
some raw data files and we didn't even know what columns they contained. Let's see whether the “air time” of a flight
tends to be different depending on the day of the week. We can see that the average is a little higher on day number
6; perhaps Saturday is a busy flying day and planes have to circle for longer at the destination airport before landing.

[localhost:21000] > select dayofweek, avg(airtime) from airlines

> group by dayofweek order by dayofweek;
+-----------+-------------------+
| dayofweek | avg(airtime) |
+-----------+-------------------+
| 1 | 102.1560425016671 |
| 2 | 102.1582931538807 |
| 3 | 102.2170009256653 |
| 4 | 102.37477661846 |
| 5 | 102.2697358763511 |
| 6 | 105.3627448363705 |
| 7 | 103.4144351202054 |
+-----------+-------------------+
Fetched 7 row(s) in 2.25s

To see if the apparent trend holds up over time, let's do the same breakdown by day of week, but also split up by year.
Now we can see that day number 6 consistently has a higher average air time in each year. We can also see that the
average air time increased over time across the board. And the presence of NULL for this column in years 1987 to 1994
shows that queries involving this column need to be restricted to a date range of 1995 and higher.

[localhost:21000] > select year, dayofweek, avg(airtime) from airlines

> group by year, dayofweek order by year desc, dayofweek;
+------+-----------+-------------------+
| year | dayofweek | avg(airtime) |
+------+-----------+-------------------+
| 2008 | 1 | 103.1821651651355 |
| 2008 | 2 | 103.2149301386094 |
| 2008 | 3 | 103.0585076622796 |
| 2008 | 4 | 103.4671383539038 |
| 2008 | 5 | 103.5575385182659 |

Apache Impala Guide | 73

Impala Tutorials

| 2008 | 6 | 107.4006306562128 |
| 2008 | 7 | 104.8648851041755 |
| 2007 | 1 | 102.2196114337825 |
| 2007 | 2 | 101.9317791906348 |
| 2007 | 3 | 102.0964767689043 |
| 2007 | 4 | 102.6215927201686 |
| 2007 | 5 | 102.4289399000661 |
| 2007 | 6 | 105.1477448215756 |
| 2007 | 7 | 103.6305945644095 |
...
| 1996 | 1 | 99.33860750862108 |
| 1996 | 2 | 99.54225446396656 |
| 1996 | 3 | 99.41129336113134 |
| 1996 | 4 | 99.5110373340348 |
| 1996 | 5 | 99.22120745027595 |
| 1996 | 6 | 101.1717447111921 |
| 1996 | 7 | 99.95410136133704 |
| 1995 | 1 | 96.93779698300494 |
| 1995 | 2 | 96.93458674589712 |
| 1995 | 3 | 97.00972311337051 |
| 1995 | 4 | 96.90843832024412 |
| 1995 | 5 | 96.78382115425562 |
| 1995 | 6 | 98.70872826057003 |
| 1995 | 7 | 97.85570478374616 |
| 1994 | 1 | NULL |
| 1994 | 2 | NULL |
| 1994 | 3 | NULL |
...
| 1987 | 5 | NULL |
| 1987 | 6 | NULL |
| 1987 | 7 | NULL |
+------+-----------+-------------------+

74 | Apache Impala Guide

 Impala Administration

Impala Administration
As an administrator, you monitor Impala's use of resources and take action when necessary to keep Impala running
smoothly and avoid conflicts with other Hadoop components running on the same cluster. When you detect that an
issue has happened or could happen in the future, you reconfigure Impala or other components such as HDFS or even
the hardware of the cluster itself to resolve or avoid problems.
Related tasks:
As an administrator, you can expect to perform installation, upgrade, and configuration tasks for Impala on all machines
in a cluster. See Installing Impala on page 27, Upgrading Impala on page 38, and Managing Impala on page 30 for
details.
For security tasks typically performed by administrators, see Impala Security on page 91.
Administrators also decide how to allocate cluster resources so that all Hadoop components can run smoothly together.
For Impala, this task primarily involves:
• Deciding how many Impala queries can run concurrently and with how much memory, through the admission
control feature. See Admission Control and Query Queuing on page 75 for details.
• Dividing cluster resources such as memory between Impala and other components, using YARN for overall resource
management, and Llama to mediate resource requests from Impala to YARN. See Integrated Resource Management
with YARN on page 83 for details.

Admission Control and Query Queuing

Admission control is an Impala feature that imposes limits on concurrent SQL queries, to avoid resource usage spikes
and out-of-memory conditions on busy CDH clusters. It is a form of “throttling”. New queries are accepted and executed
until certain conditions are met, such as too many queries or too much total memory used across the cluster. When
one of these thresholds is reached, incoming queries wait to begin execution. These queries are queued and are
admitted (that is, begin executing) when the resources become available.
In addition to the threshold values for currently executing queries, you can place limits on the maximum number of
queries that are queued (waiting) and a limit on the amount of time they might wait before returning with an error.
These queue settings let you ensure that queries do not wait indefinitely, so that you can detect and correct “starvation”
scenarios.
Enable this feature if your cluster is underutilized at some times and overutilized at others. Overutilization is indicated
by performance bottlenecks and queries being cancelled due to out-of-memory conditions, when those same queries
are successful and perform well during times with less concurrent load. Admission control works as a safeguard to
avoid out-of-memory conditions during heavy concurrent usage.

Note: Though Impala can be used together with YARN via simple configuration of Static Service Pools
in Cloudera Manager, the use of the general-purpose component Llama for integrated resource
management within YARN is no longer supported with CDH 5.5 / Impala 2.3 and higher.

Overview of Impala Admission Control

On a busy CDH cluster, you might find there is an optimal number of Impala queries that run concurrently. Because
Impala queries are typically I/O-intensive, you might not find any throughput benefit in running more concurrent
queries when the I/O capacity is fully utilized. Because Impala by default cancels queries that exceed the specified
memory limit, running multiple large-scale queries at once can result in having to re-run some queries that are cancelled.
The admission control feature lets you set a cluster-wide upper limit on the number of concurrent Impala queries and
on the memory used by those queries. Any additional queries are queued until the earlier ones finish, rather than

Apache Impala Guide | 75

Impala Administration

being cancelled or running slowly and causing contention. As other queries finish, the queued queries are allowed to
proceed.
For details on the internal workings of admission control, see How Impala Schedules and Enforces Limits on Concurrent
Queries on page 76.

How Impala Admission Control Relates to YARN

The admission control feature is similar in some ways to the YARN resource management framework, and they can be
used separately or together. This section describes some similarities and differences, to help you decide when to use
one, the other, or both together.
Admission control is a lightweight, decentralized system that is suitable for workloads consisting primarily of Impala
queries and other SQL statements. It sets “soft” limits that smooth out Impala memory usage during times of heavy
load, rather than taking an all-or-nothing approach that cancels jobs that are too resource-intensive.
Because the admission control system is not aware of other Hadoop workloads such as MapReduce jobs, you might
use YARN with static service pools on heterogeneous CDH 5 clusters where resources are shared between Impala and
other Hadoop components. Devote a percentage of cluster resources to Impala, allocate another percentage for
MapReduce and other batch-style workloads; let admission control handle the concurrency and memory usage for the
Impala work within the cluster, and let YARN manage the remainder of work within the cluster.
The Impala admission control feature uses the same configuration mechanism as the YARN resource manager to map
users to pools and authenticate them.
For full details about using Impala with YARN, see Integrated Resource Management with YARN on page 83.

How Impala Schedules and Enforces Limits on Concurrent Queries

The admission control system is decentralized, embedded in each Impala daemon and communicating through the
statestore mechanism. Although the limits you set for memory usage and number of concurrent queries apply
cluster-wide, each Impala daemon makes its own decisions about whether to allow each query to run immediately or
to queue it for a less-busy time. These decisions are fast, meaning the admission control mechanism is low-overhead,
but might be imprecise during times of heavy load. There could be times when the query queue contained more queries
than the specified limit, or when the estimated of memory usage for a query is not exact and the overall memory usage
exceeds the specified limit. Thus, you typically err on the high side for the size of the queue, because there is not a big
penalty for having a large number of queued queries; and you typically err on the low side for the memory limit, to
leave some headroom for queries to use more memory than expected, without being cancelled as a result.
At any time, the set of queued queries could include queries submitted through multiple different Impala daemon
hosts. All the queries submitted through a particular host will be executed in order, so a CREATE TABLE followed by
an INSERT on the same table would succeed. Queries submitted through different hosts are not guaranteed to be
executed in the order they were received. Therefore, if you are using load-balancing or other round-robin scheduling
where different statements are submitted through different hosts, set up all table structures ahead of time so that
the statements controlled by the queuing system are primarily queries, where order is not significant. Or, if a sequence
of statements needs to happen in strict order (such as an INSERT followed by a SELECT), submit all those statements
through a single session, while connected to the same Impala daemon host.
The limit on the number of concurrent queries is a “soft” one, To achieve high throughput, Impala makes quick decisions
at the host level about which queued queries to dispatch. Therefore, Impala might slightly exceed the limit from time
to time.
To avoid a large backlog of queued requests, you can also set an upper limit on the size of the queue for queries that
are delayed. When the number of queued queries exceeds this limit, further queries are cancelled rather than being
queued. You can also configure a timeout period, after which queued queries are cancelled, to avoid indefinite waits.
If a cluster reaches this state where queries are cancelled due to too many concurrent requests or long waits for query
execution to begin, that is a signal for an administrator to take action, either by provisioning more resources, scheduling
work on the cluster to smooth out the load, or by doing Impala performance tuning to enable higher throughput.

76 | Apache Impala Guide

 Impala Administration

How Admission Control works with Impala Clients (JDBC, ODBC, HiveServer2)
Most aspects of admission control work transparently with client interfaces such as JDBC and ODBC:
• If a SQL statement is put into a queue rather than running immediately, the API call blocks until the statement is
dequeued and begins execution. At that point, the client program can request to fetch results, which might also
block until results become available.
• If a SQL statement is cancelled because it has been queued for too long or because it exceeded the memory limit
during execution, the error is returned to the client program with a descriptive error message.
If you want to submit queries to different resource pools through the REQUEST_POOL query option, as described in
REQUEST_POOL Query Option on page 320, In Impala 2.0 and higher you can change that query option through a SQL
SET statement that you submit from the client application, in the same session. Prior to Impala 2.0, that option was
only settable for a session through the impala-shell SET command, or cluster-wide through an impalad startup
option.
Admission control has the following limitations or special behavior when used with JDBC or ODBC applications:
• The MEM_LIMIT query option, sometimes useful to work around problems caused by inaccurate memory estimates
for complicated queries, is only settable through the impala-shell interpreter and cannot be used directly
through JDBC or ODBC applications.
• Admission control does not use the other resource-related query options, RESERVATION_REQUEST_TIMEOUT or
V_CPU_CORES. Those query options only apply to the YARN resource management framework.

Configuring Admission Control

The configuration options for admission control range from the simple (a single resource pool with a single set of
options) to the complex (multiple resource pools with different options, each pool handling queries for a different set
of users and groups). You can configure the settings through the Cloudera Manager user interface, or on a system
without Cloudera Manager by editing configuration files or through startup options to the impalad daemon.

Impala Service Flags for Admission Control (Advanced)

The following Impala configuration options let you adjust the settings of the admission control feature. When supplying
the options on the command line, prepend the option name with --.
default_pool_max_queued
Purpose: Maximum number of requests allowed to be queued before rejecting requests. Because this limit applies
cluster-wide, but each Impala node makes independent decisions to run queries immediately or queue them, it is
a soft limit; the overall number of queued queries might be slightly higher during times of heavy load. A negative
value or 0 indicates requests are always rejected once the maximum concurrent requests are executing. Ignored if
fair_scheduler_config_path and llama_site_path are set.

Type: int64
Default: 200
default_pool_max_requests
Purpose: Maximum number of concurrent outstanding requests allowed to run before incoming requests are
queued. Because this limit applies cluster-wide, but each Impala node makes independent decisions to run queries
immediately or queue them, it is a soft limit; the overall number of concurrent queries might be slightly higher
during times of heavy load. A negative value indicates no limit. Ignored if fair_scheduler_config_path and
llama_site_path are set.

Type: int64
Default: 200
default_pool_mem_limit
Purpose: Maximum amount of memory (across the entire cluster) that all outstanding requests in this pool can use
before new requests to this pool are queued. Specified in bytes, megabytes, or gigabytes by a number followed by

Apache Impala Guide | 77

Impala Administration

the suffix b (optional), m, or g, either uppercase or lowercase. You can specify floating-point values for megabytes
and gigabytes, to represent fractional numbers such as 1.5. You can also specify it as a percentage of the physical
memory by specifying the suffix %. 0 or no setting indicates no limit. Defaults to bytes if no unit is given. Because
this limit applies cluster-wide, but each Impala node makes independent decisions to run queries immediately or
queue them, it is a soft limit; the overall memory used by concurrent queries might be slightly higher during times
of heavy load. Ignored if fair_scheduler_config_path and llama_site_path are set.

Note: Impala relies on the statistics produced by the COMPUTE STATS statement to estimate
memory usage for each query. See COMPUTE STATS Statement on page 226 for guidelines about
how and when to use this statement.

Type: string
Default: "" (empty string, meaning unlimited)
disable_admission_control
Purpose: Turns off the admission control feature entirely, regardless of other configuration option settings.
Type: Boolean
Default: true
disable_pool_max_requests
Purpose: Disables all per-pool limits on the maximum number of running requests.
Type: Boolean
Default: false
disable_pool_mem_limits
Purpose: Disables all per-pool mem limits.
Type: Boolean
Default: false
fair_scheduler_allocation_path
Purpose: Path to the fair scheduler allocation file (fair-scheduler.xml).
Type: string
Default: "" (empty string)
Usage notes: Admission control only uses a small subset of the settings that can go in this file, as described below.
For details about all the Fair Scheduler configuration settings, see the Apache wiki.
llama_site_path
Purpose: Path to the Llama configuration file (llama-site.xml). If set, fair_scheduler_allocation_path
must also be set.
Type: string
Default: "" (empty string)
Usage notes: Admission control only uses a small subset of the settings that can go in this file, as described below.
For details about all the Llama configuration settings, see the documentation on Github.
queue_wait_timeout_ms
Purpose: Maximum amount of time (in milliseconds) that a request waits to be admitted before timing out.
Type: int64
Default: 60000

78 | Apache Impala Guide

 Impala Administration

Configuring Admission Control Using Cloudera Manager

In Cloudera Manager, you can configure pools to manage queued Impala queries, and the options for the limit on
number of concurrent queries and how to handle queries that exceed the limit. For details, see Managing Resources
with Cloudera Manager.
See Examples of Admission Control Configurations on page 79 for a sample setup for admission control under Cloudera
Manager.

Configuring Admission Control Using the Command Line

If you do not use Cloudera Manager, you use a combination of startup options for the Impala daemon, and optionally
editing or manually constructing the configuration files fair-scheduler.xml and llama-site.xml.
For a straightforward configuration using a single resource pool named default, you can specify configuration options
on the command line and skip the fair-scheduler.xml and llama-site.xml configuration files.
For an advanced configuration with multiple resource pools using different settings, set up the fair-scheduler.xml
and llama-site.xml configuration files manually. Provide the paths to each one using the Impala daemon
command-line options, --fair_scheduler_allocation_path and --llama_site_path respectively.
The Impala admission control feature only uses the Fair Scheduler configuration settings to determine how to map
users and groups to different resource pools. For example, you might set up different resource pools with separate
memory limits, and maximum number of concurrent and queued queries, for different categories of users within your
organization. For details about all the Fair Scheduler configuration settings, see the Apache wiki.
The Impala admission control feature only uses a small subset of possible settings from the llama-site.xml
configuration file:

llama.am.throttling.maximum.placed.reservations.queue_name
llama.am.throttling.maximum.queued.reservations.queue_name

For details about all the Llama configuration settings, see Llama Default Configuration.
See Example Admission Control Configurations Using Configuration Files on page 81 for sample configuration files for
admission control using multiple resource pools, without Cloudera Manager.

Examples of Admission Control Configurations

Example Admission Control Configurations Using Cloudera Manager

For full instructions about configuring dynamic resource pools through Cloudera Manager, see
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html. The following
examples demonstrate some important points related to the Impala admission control feature.
The following figure shows a sample of the Dynamic Resource Pools page in Cloudera Manager, accessed through the
Clusters > Cluster name > Resource Management > Dynamic Resource Pools menu choice and then the Configuration
tab. Numbers from all the resource pools are combined into the topmost root pool. The default pool is for users
who are not assigned any other pool by the user-to-pool mapping settings. The development and production pools
show how you can set different limits for different classes of users, for total memory, number of concurrent queries,
and number of queries that can be queued.

Apache Impala Guide | 79

Impala Administration

Figure 1: Sample Settings for Cloudera Manager Dynamic Resource Pools Page

The following figure shows a sample of the Placement Rules page in Cloudera Manager, accessed through the Clusters >
Cluster name > Resource Management > Dynamic Resource Pools menu choice and then the Configuration > Placement
Rules tabs. The settings demonstrate a reasonable configuration of a pool named default to service all requests
where the specified resource pool does not exist, is not explicitly set, or the user or group is not authorized for the
specified pool.

80 | Apache Impala Guide

 Impala Administration

Figure 2: Sample Settings for Cloudera Manager Placement Rules Page

Example Admission Control Configurations Using Configuration Files

For clusters not managed by Cloudera Manager, here are sample fair-scheduler.xml and llama-site.xml files
that define resource pools equivalent to the ones in the preceding Cloudera Manager dialog. These sample files are
stripped down: in a real deployment they might contain other settings for use with various aspects of the YARN and
Llama components. The settings shown here are the significant ones for the Impala admission control feature.
fair-scheduler.xml:
Although Impala does not use the vcores value, you must still specify it to satisfy YARN requirements for the file
contents.
Each <aclSubmitApps> tag (other than the one for root) contains a comma-separated list of users, then a space,
then a comma-separated list of groups; these are the users and groups allowed to submit Impala statements to the
corresponding resource pool.
If you leave the <aclSubmitApps> element empty for a pool, nobody can submit directly to that pool; child pools
can specify their own <aclSubmitApps> values to authorize users and groups to submit to those pools.

<allocations>
<queue name="root">
<aclSubmitApps> </aclSubmitApps>
<queue name="default">
<maxResources>50000 mb, 0 vcores</maxResources>
<aclSubmitApps>*</aclSubmitApps>
</queue>
<queue name="development">
<maxResources>200000 mb, 0 vcores</maxResources>
<aclSubmitApps>user1,user2 dev,ops,admin</aclSubmitApps>
</queue>
<queue name="production">
<maxResources>1000000 mb, 0 vcores</maxResources>
<aclSubmitApps> ops,admin</aclSubmitApps>
</queue>
</queue>

Apache Impala Guide | 81

Impala Administration

<queuePlacementPolicy>
<rule name="specified" create="false"/>
<rule name="default" />
</queuePlacementPolicy>
</allocations>

llama-site.xml:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>
<property>
<name>llama.am.throttling.maximum.placed.reservations.root.default</name>
<value>10</value>
</property>
<property>
<name>llama.am.throttling.maximum.queued.reservations.root.default</name>
<value>50</value>
</property>
<property>
<name>llama.am.throttling.maximum.placed.reservations.root.development</name>
<value>50</value>
</property>
<property>
<name>llama.am.throttling.maximum.queued.reservations.root.development</name>
<value>100</value>
</property>
<property>
<name>llama.am.throttling.maximum.placed.reservations.root.production</name>
<value>100</value>
</property>
<property>
<name>llama.am.throttling.maximum.queued.reservations.root.production</name>
<value>200</value>
</property>
</configuration>

Guidelines for Using Admission Control

To see how admission control works for particular queries, examine the profile output for the query. This information
is available through the PROFILE statement in impala-shell immediately after running a query in the shell, on the
queries page of the Impala debug web UI, or in the Impala log file (basic information at log level 1, more detailed
information at log level 2). The profile output contains details about the admission decision, such as whether the query
was queued or not and which resource pool it was assigned to. It also includes the estimated and actual memory usage
for the query, so you can fine-tune the configuration for the memory limits of the resource pools.
Where practical, use Cloudera Manager to configure the admission control parameters. The Cloudera Manager GUI is
much simpler than editing the configuration files directly. In Cloudera Manager 4, the admission control settings are
not available directly, but you can use the impalad safety valve field to configure appropriate startup options.
Remember that the limits imposed by admission control are “soft” limits. Although the limits you specify for number
of concurrent queries and amount of memory apply cluster-wide, the decentralized nature of this mechanism means
that each Impala node makes its own decisions about whether to allow queries to run immediately or to queue them.
These decisions rely on information passed back and forth between nodes by the statestore service. If a sudden surge
in requests causes more queries than anticipated to run concurrently, then as a fallback, the overall Impala memory
limit and the Linux cgroups mechanism serve as hard limits to prevent overallocation of memory, by cancelling queries
if necessary.
If you have trouble getting a query to run because its estimated memory usage is too high, you can override the estimate
by setting the MEM_LIMIT query option in impala-shell, then issuing the query through the shell in the same session.
The MEM_LIMIT value is treated as the estimated amount of memory, overriding the estimate that Impala would
generate based on table and column statistics. This value is used only for making admission control decisions, and is
not pre-allocated by the query.
In impala-shell, you can also specify which resource pool to direct queries to by setting the REQUEST_POOL query
option. (This option was named YARN_POOL during the CDH 5 beta period.)

82 | Apache Impala Guide

 Impala Administration

The statements affected by the admission control feature are primarily queries, but also include statements that write
data such as INSERT and CREATE TABLE AS SELECT. Most write operations in Impala are not resource-intensive,
but inserting into a Parquet table can require substantial memory due to buffering 1 GB of data before writing out
each Parquet data block. See Loading Data into Parquet Tables on page 536 for instructions about inserting data
efficiently into Parquet tables.
Although admission control does not scrutinize memory usage for other kinds of DDL statements, if a query is queued
due to a limit on concurrent queries or memory usage, subsequent statements in the same session are also queued
so that they are processed in the correct order:

-- This query could be queued to avoid out-of-memory at times of heavy load.

select * from huge_table join enormous_table using (id);
-- If so, this subsequent statement in the same session is also queued
-- until the previous statement completes.
drop table huge_table;

If you set up different resource pools for different users and groups, consider reusing any classifications and hierarchy
you developed for use with Sentry security. See Enabling Sentry Authorization for Impala on page 95 for details.
For details about all the Fair Scheduler configuration settings, see Fair Scheduler Configuration, in particular the tags
such as <queue> and <aclSubmitApps> to map users and groups to particular resource pools (queues).

Integrated Resource Management with YARN

Note: Though Impala can be used together with YARN via simple configuration of Static Service Pools
in Cloudera Manager, the use of the general-purpose component Llama for integrated resource
management within YARN is no longer supported with CDH 5.5 / Impala 2.3 and higher.

You can limit the CPU and memory resources used by Impala, to manage and prioritize workloads on clusters that run
jobs from many Hadoop components.
Requests from Impala to YARN go through an intermediary service called Llama. When the resource requests are
granted, Impala starts the query and places all relevant execution threads into the cgroup containers and sets up the
memory limit on each host. If sufficient resources are not available, the Impala query waits until other jobs complete
and the resources are freed. During query processing, as the need for additional resources arises, Llama can “expand”
already-requested resources, to avoid over-allocating at the start of the query.
After a query is finished, Llama caches the resources (for example, leaving memory allocated) in case they are needed
for subsequent Impala queries. This caching mechanism avoids the latency involved in making a whole new set of
resource requests for each query. If the resources are needed by YARN for other types of jobs, Llama returns them.
While the delays to wait for resources might make individual queries seem less responsive on a heavily loaded cluster,
the resource management feature makes the overall performance of the cluster smoother and more predictable,
without sudden spikes in utilization due to memory paging, CPUs pegged at 100%, and so on.

The Llama Daemon

Llama is a system that mediates resource management between Impala and Hadoop YARN. Llama enables Impala to
reserve, use, and release resource allocations in a Hadoop cluster. Llama is only required if resource management is
enabled in Impala.
By default, YARN allocates resources bit-by-bit as needed by MapReduce jobs. Impala needs all resources available at
the same time, so that intermediate results can be exchanged between cluster nodes, and queries do not stall partway
through waiting for new resources to be allocated. Llama is the intermediary process that ensures all requested
resources are available before each Impala query actually begins.
For management through Cloudera Manager, see The Impala Llama ApplicationMaster.

Apache Impala Guide | 83

Impala Administration

How Resource Limits Are Enforced

• If Cloudera Manager Static Partitioning is used, it creates a cgroup in which Impala runs. This cgroup limits CPU,
network, and IO according to the static partitioning policy.
• Limits on memory usage are enforced by Impala's process memory limit (the MEM_LIMIT query option setting).
The admission control feature checks this setting to decide how many queries can be safely run at the same time.
Then the Impala daemon enforces the limit by activating the spill-to-disk mechanism when necessary, or cancelling
a query altogether if the limit is exceeded at runtime.

impala-shell Query Options for Resource Management

Before issuing SQL statements through the impala-shell interpreter, you can use the SET command to configure
the following parameters related to resource management:
• EXPLAIN_LEVEL Query Option on page 309
• MEM_LIMIT Query Option on page 317

Limitations of Resource Management for Impala

The MEM_LIMIT query option, and the other resource-related query options, are settable through the ODBC or JDBC
interfaces in Impala 2.0 and higher. This is a former limitation that is now lifted.

Setting Timeout Periods for Daemons, Queries, and Sessions

Depending on how busy your CDH cluster is, you might increase or decrease various timeout values. Increase timeouts
if Impala is cancelling operations prematurely, when the system is responding slower than usual but the operations
are still successful if given extra time. Decrease timeouts if operations are idle or hanging for long periods, and the idle
or hung operations are consuming resources and reducing concurrency.

Increasing the Statestore Timeout

If you have an extensive Impala schema, for example with hundreds of databases, tens of thousands of tables, and so
on, you might encounter timeout errors during startup as the Impala catalog service broadcasts metadata to all the
Impala nodes using the statestore service. To avoid such timeout errors on startup, increase the statestore timeout
value from its default of 10 seconds. Specify the timeout value using the
-statestore_subscriber_timeout_seconds option for the statestore service, using the configuration instructions
in Modifying Impala Startup Options on page 43. The symptom of this problem is messages in the impalad log such
as:

Connection with state-store lost

Trying to re-register with state-store

See Scalability Considerations for the Impala Statestore on page 516 for more details about statestore operation and
settings on clusters with a large number of Impala-related objects such as tables and partitions.

Setting the Idle Query and Idle Session Timeouts for impalad
To keep long-running queries or idle sessions from tying up cluster resources, you can set timeout intervals for both
individual queries, and entire sessions.

84 | Apache Impala Guide

 Impala Administration

Note:
The timeout clock for queries and sessions only starts ticking when the query or session is idle. For
queries, this means the query has results ready but is waiting for a client to fetch the data. A query
can run for an arbitrary time without triggering a timeout, because the query is computing results
rather than sitting idle waiting for the results to be fetched. The timeout period is intended to prevent
unclosed queries from consuming resources and taking up slots in the admission count of running
queries, potentially preventing other queries from starting.
For sessions, this means that no query has been submitted for some period of time.

Specify the following startup options for the impalad daemon:

• The --idle_query_timeout option specifies the time in seconds after which an idle query is cancelled. This
could be a query whose results were all fetched but was never closed, or one whose results were partially fetched
and then the client program stopped requesting further results. This condition is most likely to occur in a client
program using the JDBC or ODBC interfaces, rather than in the interactive impala-shell interpreter. Once the
query is cancelled, the client program cannot retrieve any further results.
You can reduce the idle query timeout by using the QUERY_TIMEOUT_S query option. Any value specified for the
--idle_query_timeout startup option serves as an upper limit for the QUERY_TIMEOUT_S query option. See
QUERY_TIMEOUT_S Query Option (CDH 5.2 or higher only) on page 320 for details.
• The --idle_session_timeout option specifies the time in seconds after which an idle session is expired. A
session is idle when no activity is occurring for any of the queries in that session, and the session has not started
any new queries. Once a session is expired, you cannot issue any new query requests to it. The session remains
open, but the only operation you can perform is to close it. The default value of 0 means that sessions never
expire.

For instructions on changing impalad startup options, see Modifying Impala Startup Options on page 43.

Note:
To avoid excessive polling, Impala checks periodically for idle sessions and queries to cancel. The actual
idle time before cancellation might be up to 50% greater than the specified configuration setting. For
example, if the timeout setting was 60, the session or query might be cancelled after being idle between
60 and 90 seconds.

Cancelling a Query
Sometimes, an Impala query might run for an unexpectedly long time, tying up resources in the cluster. You can cancel
the query explicitly, independent of the timeout period, by going into the web UI for the impalad host (on port 25000
by default), and using the link on the /queries tab to cancel the running query. Various client applications let you
interactively cancel queries submitted or monitored through those applications. For example, by pressing ^C in
impala-shell, clicking the Cancel button from the Watch page in Hue, clicking Actions > Cancel from the Queries
list in Cloudera Manager, and so on.

Using Impala through a Proxy for High Availability

For most clusters that have multiple users and production availability requirements, you might set up a proxy server
to relay requests to and from Impala.
Currently, the Impala statestore mechanism does not include such proxying and load-balancing features. Set up a
software package of your choice to perform these functions.

Apache Impala Guide | 85

Impala Administration

Note:
Most considerations for load balancing and high availability apply to the impalad daemon. The
statestored and catalogd daemons do not have special requirements for high availability, because
problems with those daemons do not result in data loss. If those daemons become unavailable due
to an outage on a particular host, you can stop the Impala service, delete the Impala StateStore and
Impala Catalog Server roles, add the roles on a different host, and restart the Impala service.

Overview of Proxy Usage and Load Balancing for Impala

Using a load-balancing proxy server for Impala has the following advantages:
• Applications connect to a single well-known host and port, rather than keeping track of the hosts where the
impalad daemon is running.
• If any host running the impalad daemon becomes unavailable, application connection requests still succeed
because you always connect to the proxy server rather than a specific host running the impalad daemon.
• The coordinator node for each Impala query potentially requires more memory and CPU cycles than the other
nodes that process the query. The proxy server can issue queries using round-robin scheduling, so that each
connection uses a different coordinator node. This load-balancing technique lets the Impala nodes share this
additional work, rather than concentrating it on a single machine.
The following setup steps are a general outline that apply to any load-balancing proxy software.
1. Download the load-balancing proxy software. It should only need to be installed and configured on a single host.
Pick a host other than the DataNodes where impalad is running, because the intention is to protect against the
possibility of these DataNodes becoming unavailable.
2. Configure the software (typically by editing a configuration file). In particular:
• Set up a port that the load balancer will listen on to relay Impala requests back and forth.
• Consider enabling “sticky sessions”. Cloudera recommends enabling this setting so that stateless client
applications such as impalad and Hue are not disconnected from long-running queries. Evaluate whether
this setting is appropriate for your combination of workload and client applications.
• For Kerberized clusters, follow the instructions in Special Proxy Considerations for Clusters Using Kerberos
on page 87.

3. Specify the host and port settings for each Impala node. These are the hosts that the load balancer will choose
from when relaying each Impala query. See Ports Used by Impala on page 583 for when to use port 21000, 21050,
or another value depending on what type of connections you are load balancing.

Note:
In particular, if you are using Hue or JDBC-based applications, you typically set up load balancing
for both ports 21000 and 21050, because these client applications connect through port 21050
while the impala-shell command connects through port 21000.

4. Run the load-balancing proxy server, pointing it at the configuration file that you set up.
5. On systems managed by Cloudera Manager, on the page Impala > Configuration > Impala Daemon Default Group,
specify a value for the Impala Daemons Load Balancer field. Specify the address of the load balancer in host:port
format. This setting lets Cloudera Manager route all appropriate Impala-related operations through the proxy
server.
6. For any scripts, jobs, or configuration settings for applications that formerly connected to a specific datanode to
run Impala SQL statements, change the connection information (such as the -i option in impala-shell) to point
to the load balancer instead.

86 | Apache Impala Guide

 Impala Administration

Note: The following sections use the HAProxy software as a representative example of a load balancer
that you can use with Impala. For information specifically about using Impala with the F5 BIG-IP load
balancer, see Impala HA with F5 BIG-IP.

Special Proxy Considerations for Clusters Using Kerberos

In a cluster using Kerberos, applications check host credentials to verify that the host they are connecting to is the
same one that is actually processing the request, to prevent man-in-the-middle attacks. To clarify that the load-balancing
proxy server is legitimate, perform these extra Kerberos setup steps:
1. This section assumes you are starting with a Kerberos-enabled cluster. See Enabling Kerberos Authentication for
Impala on page 106 for instructions for setting up Impala with Kerberos. See the CDH Security Guide for general
steps to set up Kerberos.
2. Choose the host you will use for the proxy server. Based on the Kerberos setup procedure, it should already have
an entry impala/proxy_host@realm in its keytab. If not, go back over the initial Kerberos configuration steps
for the keytab on each host running the impalad daemon.
3. Copy the keytab file from the proxy host to all other hosts in the cluster that run the impalad daemon. (For
optimal performance, impalad should be running on all DataNodes in the cluster.) Put the keytab file in a secure
location on each of these other hosts.
4. On systems not managed by Cloudera Manager, add an entry impala/actual_hostname@realm to the keytab
on each host running the impalad daemon.
5. For each impalad node, merge the existing keytab with the proxy’s keytab using ktutil, producing a new keytab
file. For example:

$ ktutil
ktutil: read_kt proxy.keytab
ktutil: read_kt impala.keytab
ktutil: write_kt proxy_impala.keytab
ktutil: quit

Note: On systems managed by Cloudera Manager 5.1.0 and later, the keytab merging happens
automatically. To verify that Cloudera Manager has merged the keytabs, run the command:

klist -k keytabfile

which lists the credentials for both principal and be_principal on all nodes.

6. Make sure that the impala user has permission to read this merged keytab file.
7. Change some configuration settings for each host in the cluster that participates in the load balancing. Follow the
appropriate steps depending on whether you use Cloudera Manager or not:
• In the impalad option definition, or the Cloudera Manager safety valve (Cloudera Manager 4) or advanced
configuration snippet (Cloudera Manager 5), add:

--principal=impala/proxy_host@realm
--be_principal=impala/actual_host@realm
--keytab_file=path_to_merged_keytab

Note:
Every host has a different --be_principal because the actual hostname is different on
each host.
Specify the fully qualified domain name (FQDN) for the proxy host, not the IP address. Use
the exact FQDN as returned by a reverse DNS lookup for the associated IP address.

Apache Impala Guide | 87

Impala Administration

• On a cluster managed by Cloudera Manager, create a role group to set the configuration values from the
preceding step on a per-host basis.
• On a cluster not managed by Cloudera Manager, see Modifying Impala Startup Options on page 43 for the
procedure to modify the startup options.
8. Restart Impala to make the changes take effect. Follow the appropriate steps depending on whether you use
Cloudera Manager or not:
• On a cluster managed by Cloudera Manager, restart the Impala service.
• On a cluster not managed by Cloudera Manager, restart the impalad daemons on all hosts in the cluster, as
well as the statestored and catalogd daemons.

Example of Configuring HAProxy Load Balancer for Impala

If you are not already using a load-balancing proxy, you can experiment with HAProxy a free, open source load balancer.
This example shows how you might install and configure that load balancer on a Red Hat Enterprise Linux system.
• Install the load balancer: yum install haproxy
• Set up the configuration file: /etc/haproxy/haproxy.cfg. See the following section for a sample configuration
file.
• Run the load balancer (on a single host, preferably one not running impalad):

/usr/sbin/haproxy –f /etc/haproxy/haproxy.cfg

• In impala-shell, JDBC applications, or ODBC applications, connect to the listener port of the proxy host, rather
than port 21000 or 21050 on a host actually running impalad. The sample configuration file sets haproxy to listen
on port 25003, therefore you would send all requests to haproxy_host:25003.

This is the sample haproxy.cfg used in this example:

global
# To have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

# turn on stats unix socket

#stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#
# You might need to adjust timing values to prevent timeouts.
#---------------------------------------------------------------------
defaults
mode http

88 | Apache Impala Guide

 Impala Administration

log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
maxconn 3000
contimeout 5000
clitimeout 50000
srvtimeout 50000

#
# This sets up the admin page for HA Proxy at port 25002.
#
listen stats :25002
balance
mode http
stats enable
stats auth username:password

# This is the setup for Impala. Impala client connect to load_balancer_host:25003.

# HAProxy will balance connections among the list of servers listed below.
# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original
ODBC driver.
# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
listen impala :25003
mode tcp
option tcplog
balance leastconn

server symbolic_name_1 impala-host-1.example.com:21000

server symbolic_name_2 impala-host-2.example.com:21000
server symbolic_name_3 impala-host-3.example.com:21000
server symbolic_name_4 impala-host-4.example.com:21000

# Setup for Hue or other JDBC-enabled applications.

# In particular, Hue requires sticky sessions.
# The application connects to load_balancer_host:21051, and HAProxy balances
# connections to the associated hosts, where Impala listens for JDBC
# requests on port 21050.
listen impalajdbc :21051
mode tcp
option tcplog
balance source
server symbolic_name_5 impala-host-1.example.com:21050
server symbolic_name_6 impala-host-2.example.com:21050
server symbolic_name_7 impala-host-3.example.com:21050
server symbolic_name_8 impala-host-4.example.com:21050

Note: If your JDBC or ODBC application connects to Impala through a load balancer such as haproxy,
be cautious about reusing the connections. If the load balancer has set up connection timeout values,
either check the connection frequently so that it never sits idle longer than the load balancer timeout
value, or check the connection validity before using it and create a new one if the connection has
been closed.

Managing Disk Space for Impala Data

Although Impala typically works with many large files in an HDFS storage system with plenty of capacity, there are
times when you might perform some file cleanup to reclaim space, or advise developers on techniques to minimize
space consumption and file duplication.
• Use compact binary file formats where practical. Numeric and time-based data in particular can be stored in more
compact form in binary data files. Depending on the file format, various compression and encoding features can
reduce file size even further. You can specify the STORED AS clause as part of the CREATE TABLE statement, or
ALTER TABLE with the SET FILEFORMAT clause for an existing table or partition within a partitioned table. See

Apache Impala Guide | 89

Impala Administration

How Impala Works with Hadoop File Formats on page 527 for details about file formats, especially Using the Parquet
File Format with Impala Tables on page 535. See CREATE TABLE Statement on page 235 and ALTER TABLE Statement
on page 215 for syntax details.
• You manage underlying data files differently depending on whether the corresponding Impala table is defined as
an internal or external table:
– Use the DESCRIBE FORMATTED statement to check if a particular table is internal (managed by Impala) or
external, and to see the physical location of the data files in HDFS. See DESCRIBE Statement on page 245 for
details.
– For Impala-managed (“internal”) tables, use DROP TABLE statements to remove data files. See DROP TABLE
Statement on page 258 for details.
– For tables not managed by Impala (“external” tables), use appropriate HDFS-related commands such as
hadoop fs, hdfs dfs, or distcp, to create, move, copy, or delete files within HDFS directories that are
accessible by the impala user. Issue a REFRESH table_name statement after adding or removing any files
from the data directory of an external table. See REFRESH Statement on page 277 for details.
– Use external tables to reference HDFS data files in their original location. With this technique, you avoid
copying the files, and you can map more than one Impala table to the same set of data files. When you drop
the Impala table, the data files are left undisturbed. See External Tables on page 209 for details.
– Use the LOAD DATA statement to move HDFS files into the data directory for an Impala table from inside
Impala, without the need to specify the HDFS path of the destination directory. This technique works for both
internal and external tables. See LOAD DATA Statement on page 274 for details.
• Make sure that the HDFS trashcan is configured correctly. When you remove files from HDFS, the space might not
be reclaimed for use by other files until sometime later, when the trashcan is emptied. See DROP TABLE Statement
on page 258 and the FAQ entry Why is space not freed up when I issue DROP TABLE? on page 601 for details. See
User Account Requirements on page 24 for permissions needed for the HDFS trashcan to operate correctly.
• Drop all tables in a database before dropping the database itself. See DROP DATABASE Statement on page 252 for
details.
• Clean up temporary files after failed INSERT statements. If an INSERT statement encounters an error, and you
see a directory named .impala_insert_staging or _impala_insert_staging left behind in the data
directory for the table, it might contain temporary data files taking up space in HDFS. You might be able to salvage
these data files, for example if they are complete but could not be moved into place due to a permission error.
Or, you might delete those files through commands such as hadoop fs or hdfs dfs, to reclaim space before
re-trying the INSERT. Issue DESCRIBE FORMATTED table_name to see the HDFS path where you can check for
temporary files.
• By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored
in the directory /tmp/impala-scratch . These files are removed when the operation finishes. (Multiple concurrent
queries can perform operations that use the “spill to disk” technique, without any name conflicts for these
temporary files.) You can specify a different location by starting the impalad daemon with the
--scratch_dirs="path_to_directory" configuration option or the equivalent configuration option in the
Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The
scratch directories must be on the local filesystem, not in HDFS. You might specify different directory paths for
different hosts, depending on the capacity and speed of the available storage devices. In CDH 5.5 / Impala 2.3 or
higher, Impala successfully starts (with a warning written to the log) if it cannot create or read and write files in
one of the scratch directories. If there is less than 1 GB free on the filesystem where that directory resides, Impala
still runs, but writes a warning message to its log. If Impala encounters an error reading or writing files in a scratch
directory during a query, Impala logs the error and the query fails.
• If you use the Amazon Simple Storage Service (S3) as a place to offload data to reduce the volume of local storage,
Impala 2.2.0 and higher can query the data directly from S3. See Using Impala to Query the Amazon S3 Filesystem
on page 566 for details.

90 | Apache Impala Guide

 Impala Security

Impala Security
Impala includes a fine-grained authorization framework for Hadoop, based on the Sentry open source project. Sentry
authorization was added in Impala 1.1.0. Together with the Kerberos authentication framework, Sentry takes Hadoop
security to a new level needed for the requirements of highly regulated industries such as healthcare, financial services,
and government. Impala also includes an auditing capability; Impala generates the audit data, the Cloudera Navigator
product consolidates the audit data from all nodes in the cluster, and Cloudera Manager lets you filter, visualize, and
produce reports. The auditing feature was added in Impala 1.1.1.
The Impala security features have several objectives. At the most basic level, security prevents accidents or mistakes
that could disrupt application processing, delete or corrupt data, or reveal data to unauthorized users. More advanced
security features and practices can harden the system against malicious users trying to gain unauthorized access or
perform other disallowed operations. The auditing feature provides a way to confirm that no unauthorized access
occurred, and detect whether any such attempts were made. This is a critical set of features for production deployments
in large organizations that handle important or sensitive data. It sets the stage for multi-tenancy, where multiple
applications run concurrently and are prevented from interfering with each other.
The material in this section presumes that you are already familiar with administering secure Linux systems. That is,
you should know the general security practices for Linux and Hadoop, and their associated commands and configuration
files. For example, you should know how to create Linux users and groups, manage Linux group membership, set Linux
and HDFS file permissions and ownership, and designate the default permissions and ownership for new files. You
should be familiar with the configuration of the nodes in your Hadoop cluster, and know how to apply configuration
changes or run a set of commands across all the nodes.
The security features are divided into these broad categories:
authorization
Which users are allowed to access which resources, and what operations are they allowed to perform? Impala relies
on the open source Sentry project for authorization. By default (when authorization is not enabled), Impala does
all read and write operations with the privileges of the impala user, which is suitable for a development/test
environment but not for a secure production environment. When authorization is enabled, Impala uses the OS user
ID of the user who runs impala-shell or other client program, and associates various privileges with each user.
See Enabling Sentry Authorization for Impala on page 95 for details about setting up and managing authorization.
authentication
How does Impala verify the identity of the user to confirm that they really are allowed to exercise the privileges
assigned to that user? Impala relies on the Kerberos subsystem for authentication. See Enabling Kerberos
Authentication for Impala on page 106 for details about setting up and managing authentication.
auditing
What operations were attempted, and did they succeed or not? This feature provides a way to look back and
diagnose whether attempts were made to perform unauthorized operations. You use this information to track
down suspicious activity, and to see where changes are needed in authorization policies. The audit data produced
by this feature is collected by the Cloudera Manager product and then presented in a user-friendly form by the
Cloudera Manager product. See Auditing Impala Operations on page 112 for details about setting up and managing
auditing.

Security Guidelines for Impala

The following are the major steps to harden a cluster running Impala against accidents and mistakes, or malicious
attackers trying to access sensitive data:
• Secure the root account. The root user can tamper with the impalad daemon, read and write the data files in
HDFS, log into other user accounts, and access other system services that are beyond the control of Impala.

Apache Impala Guide | 91

Impala Security

• Restrict membership in the sudoers list (in the /etc/sudoers file). The users who can run the sudo command
can do many of the same things as the root user.
• Ensure the Hadoop ownership and permissions for Impala data files are restricted.
• Ensure the Hadoop ownership and permissions for Impala log files are restricted.
• Ensure that the Impala web UI (available by default on port 25000 on each Impala node) is password-protected.
See Impala Web User Interface for Debugging on page 580 for details.
• Create a policy file that specifies which Impala privileges are available to users in particular Hadoop groups (which
by default map to Linux OS groups). Create the associated Linux groups using the groupadd command if necessary.
• The Impala authorization feature makes use of the HDFS file ownership and permissions mechanism; for background
information, see the CDH HDFS Permissions Guide. Set up users and assign them to groups at the OS level,
corresponding to the different categories of users with different access levels for various databases, tables, and
HDFS locations (URIs). Create the associated Linux users using the useradd command if necessary, and add them
to the appropriate groups with the usermod command.
• Design your databases, tables, and views with database and table structure to allow policy rules to specify simple,
consistent rules. For example, if all tables related to an application are inside a single database, you can assign
privileges for that database and use the * wildcard for the table name. If you are creating views with different
privileges than the underlying base tables, you might put the views in a separate database so that you can use
the * wildcard for the database containing the base tables, while specifying the precise names of the individual
views. (For specifying table or database names, you either specify the exact name or * to mean all the databases
on a server, or all the tables and views in a database.)
• Enable authorization by running the impalad daemons with the -server_name and
-authorization_policy_file options on all nodes. (The authorization feature does not apply to the
statestored daemon, which has no access to schema objects or data files.)

• Set up authentication using Kerberos, to make sure users really are who they say they are.

Securing Impala Data and Log Files

One aspect of security is to protect files from unauthorized access at the filesystem level. For example, if you store
sensitive data in HDFS, you specify permissions on the associated files and directories in HDFS to restrict read and write
permissions to the appropriate users and groups.
If you issue queries containing sensitive values in the WHERE clause, such as financial account numbers, those values
are stored in Impala log files in the Linux filesystem and you must secure those files also. For the locations of Impala
log files, see Using Impala Logging on page 574.
All Impala read and write operations are performed under the filesystem privileges of the impala user. The impala
user must be able to read all directories and data files that you query, and write into all the directories and data files
for INSERT and LOAD DATA statements. At a minimum, make sure the impala user is in the hive group so that it
can access files and directories shared between Impala and Hive. See User Account Requirements on page 24 for more
details.
Setting file permissions is necessary for Impala to function correctly, but is not an effective security practice by itself:
• The way to ensure that only authorized users can submit requests for databases and tables they are allowed to
access is to set up Sentry authorization, as explained in Enabling Sentry Authorization for Impala on page 95. With
authorization enabled, the checking of the user ID and group is done by Impala, and unauthorized access is blocked
by Impala itself. The actual low-level read and write requests are still done by the impala user, so you must have
appropriate file and directory permissions for that user ID.
• You must also set up Kerberos authentication, as described in Enabling Kerberos Authentication for Impala on
page 106, so that users can only connect from trusted hosts. With Kerberos enabled, if someone connects a new

92 | Apache Impala Guide

 Impala Security

host to the network and creates user IDs that match your privileged IDs, they will be blocked from connecting to
Impala at all from that host.

Installation Considerations for Impala Security

Impala 1.1 comes set up with all the software and settings needed to enable security when you run the impalad
daemon with the new security-related options (-server_name and -authorization_policy_file). You do not
need to change any environment variables or install any additional JAR files. In a cluster managed by Cloudera Manager,
you do not need to change any settings in Cloudera Manager.

Securing the Hive Metastore Database

It is important to secure the Hive metastore, so that users cannot access the names or other information about databases
and tables the through the Hive client or by querying the metastore database. Do this by turning on Hive metastore
security, using the instructions in the CDH 5 Security Guide for securing different Hive components:
• Secure the Hive Metastore.
• In addition, allow access to the metastore only from the HiveServer2 server, and then disable local access to the
HiveServer2 server.

Securing the Impala Web User Interface

The instructions in this section presume you are familiar with the .htpasswd mechanism commonly used to
password-protect pages on web servers.
Password-protect the Impala web UI that listens on port 25000 by default. Set up a .htpasswd file in the $IMPALA_HOME
directory, or start both the impalad and statestored daemons with the --webserver_password_file option
to specify a different location (including the filename).
This file should only be readable by the Impala process and machine administrators, because it contains (hashed)
versions of passwords. The username / password pairs are not derived from Unix usernames, Kerberos users, or any
other system. The domain field in the password file must match the domain supplied to Impala by the new command-line
option --webserver_authentication_domain. The default is mydomain.com.
Impala also supports using HTTPS for secure web traffic. To do so, set --webserver_certificate_file to refer
to a valid .pem TLS/SSL certificate file. Impala will automatically start using HTTPS once the TLS/SSL certificate has
been read and validated. A .pem file is basically a private key, followed by a signed TLS/SSL certificate; make sure to
concatenate both parts when constructing the .pem file.
If Impala cannot find or parse the .pem file, it prints an error message and quits.

Note:
If the private key is encrypted using a passphrase, Impala will ask for that passphrase on startup, which
is not useful for a large cluster. In that case, remove the passphrase and make the .pem file readable
only by Impala and administrators.
When you turn on TLS/SSL for the Impala web UI, the associated URLs change from http:// prefixes
to https://. Adjust any bookmarks or application code that refers to those URLs.

Apache Impala Guide | 93

Impala Security

Configuring TLS/SSL for Impala

Impala supports TLS/SSL network encryption, between Impala and client programs, and between the Impala-related
daemons running on different nodes in the cluster. This feature is important when you also use other features such
as Kerberos authentication or Sentry authorization, where credentials are being transmitted back and forth.

Important:
• You can use either Cloudera Manager or the following command-line instructions to complete
this configuration.
• This information applies specifically to CDH 5.6.x. If you use an earlier version of CDH, see the
documentation for that version located at Cloudera Documentation.

Using Cloudera Manager

To configure Impala to listen for Beeswax and HiveServer2 requests on TLS/SSL-secured ports:
1. Open the Cloudera Manager Admin Console and go to the Impala service.
2. Click the Configuration tab.
3. Select Scope > Impala (Service-Wide).
4. Select Category > Security.
5. Edit the following properties:

Table 1: Impala SSL Properties

Property Description
Enable TLS/SSL for Impala Encrypt communication between clients (like ODBC, JDBC, and the Impala shell)
Client Services and the Impala daemon using Transport Layer Security (TLS) (formerly known
as Secure Socket Layer (SSL)).
SSL/TLS Certificate for Clients Local path to the X509 certificate that identifies the Impala daemon to clients
during TLS/SSL connections. This file must be in PEM format.
SSL/TLS Private Key for Clients Local path to the private key that matches the certificate specified in the
Certificate for Clients. This file must be in PEM format.
SSL/TLS Private Key Password A shell command for Impala to run on startup to retrieve the password for a
for Clients password-protected private key file. The output of the command is truncated
to a maximum of 1024 bytes, and any trailing whitespace (such as spaces or
newline characters) is trimmed. If the command exits with an error, Impala does
not start. If the password is incorrect, clients cannot connect to the server
regardless of whether the public key is correct.
SSL/TLS CA Certificate Must be specified for TLS/SSL encryption to be enabled for communication
between internal Impala components.
SSL/TLS Certificate for Impala There are three of these configuration settings, one each for “Impala Daemon”,
component Webserver “Catalog Server”, and “Statestore”. Each of these Impala components has its
own internal web server that powers the associated web UI with diagnostic
information. The configuration setting represents the local path to the X509
certificate that identifies the web server to clients during TLS/SSL connections.
This file must be in PEM format.

6. Click Save Changes to commit the changes.

7. Restart the Impala service.

94 | Apache Impala Guide

 Impala Security

For information on configuring TLS/SSL communication with the impala-shell interpreter, see Configuring TLS/SSL
Communication for the Impala Shell on page 95.

Using the Command Line

To enable SSL for when client applications connect to Impala, add both of the following flags to the impalad startup
options:
• --ssl_server_certificate: the full path to the server certificate, on the local filesystem.
• --ssl_private_key: the full path to the server private key, on the local filesystem.
In CDH 5.5 / Impala 2.3 and higher, Impala can also use SSL for its own internal communication between the impalad,
statestored, and catalogd daemons. To enable this additional SSL encryption, set the --ssl_server_certificate
and --ssl_private_key flags in the startup options for impalad, catalogd, and statestored, and also add the
--ssl_client_ca_certificate flag for all three of those daemons.

Warning: Prior to CDH 5.5.2 / Impala 2.3.2, you could enable Kerberos authentication between Impala
internal components, or SSL encryption between Impala internal components, but not both at the
same time. This restriction has now been lifted. See IMPALA-2598 to see the maintenance releases
for different levels of CDH where the fix has been published.

If either of these flags are set, both must be set. In that case, Impala starts listening for Beeswax and HiveServer2
requests on SSL-secured ports only. (The port numbers stay the same; see Ports Used by Impala on page 583 for details.)
Since Impala uses passphrase-less certificates in PEM format, you can reuse a host's existing Java keystore by converting
it to the PEM format. For instructions, see
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_openssl_jks.html.

Configuring TLS/SSL Communication for the Impala Shell

Typically, a client program has corresponding configuration properties in Cloudera Manager to verify that it is connecting
to the right server. For example, with SSL enabled for Impala, you use the following options when starting the
impala-shell interpreter:

• --ssl: enables TLS/SSL for impala-shell.

• --ca_cert: the local pathname pointing to the third-party CA certificate, or to a copy of the server certificate
for self-signed server certificates.
If --ca_cert is not set, impala-shell enables TLS/SSL, but does not validate the server certificate. This is useful
for connecting to a known-good Impala that is only running over TLS/SSL, when a copy of the certificate is not available
(such as when debugging customer installations).

Using TLS/SSL with Business Intelligence Tools

You can use Kerberos authentication, TLS/SSL encryption, or both to secure connections from JDBC and ODBC applications
to Impala. See Configuring Impala to Work with JDBC on page 34 and Configuring Impala to Work with ODBC on page
31 for details.
Currently, the Hive JDBC driver does not support connections that use both Kerberos authentication and SSL encryption.
To use both of these security features with Impala through a JDBC application, use the Cloudera JDBC Connector as
the JDBC driver.

Enabling Sentry Authorization for Impala

Authorization determines which users are allowed to access which resources, and what operations they are allowed
to perform. In Impala 1.1 and higher, you use the Sentry open source project for authorization. Sentry adds a fine-grained
authorization framework for Hadoop. By default (when authorization is not enabled), Impala does all read and write
operations with the privileges of the impala user, which is suitable for a development/test environment but not for

Apache Impala Guide | 95

Impala Security

a secure production environment. When authorization is enabled, Impala uses the OS user ID of the user who runs
impala-shell or other client program, and associates various privileges with each user.

Note: Sentry is typically used in conjunction with Kerberos authentication, which defines which hosts
are allowed to connect to each server. Using the combination of Sentry and Kerberos prevents malicious
users from being able to connect by creating a named account on an untrusted machine. See Enabling
Kerberos Authentication for Impala on page 106 for details about Kerberos authentication.

The Sentry Privilege Model

Privileges can be granted on different objects in the schema. Any privilege that can be granted is associated with a
level in the object hierarchy. If a privilege is granted on a container object in the hierarchy, the child object automatically
inherits it. This is the same privilege model as Hive and other database systems such as MySQL.
The object hierarchy for Impala covers Server, URI, Database, Table, and Column. (The Table privileges apply to views
as well; anywhere you specify a table name, you can specify a view name instead.) Column-level authorization is
available in CDH 5.5 / Impala 2.3 and higher, as described in
https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html. Previously, you constructed
views to query specific columns and assigned privileges based on the views rather than the base tables.
A restricted set of privileges determines what you can do with each object:
SELECT privilege
Lets you read data from a table or view, for example with the SELECT statement, the INSERT...SELECT syntax,
or CREATE TABLE...LIKE. Also required to issue the DESCRIBE statement or the EXPLAIN statement for a query
against a particular table. Only objects for which a user has this privilege are shown in the output for SHOW
DATABASES and SHOW TABLES statements. The REFRESH statement and INVALIDATE METADATA statements
only access metadata for tables for which the user has this privilege.
INSERT privilege
Lets you write data to a table. Applies to the INSERT and LOAD DATA statements.
ALL privilege
Lets you create or modify the object. Required to run DDL statements such as CREATE TABLE, ALTER TABLE, or
DROP TABLE for a table, CREATE DATABASE or DROP DATABASE for a database, or CREATE VIEW, ALTER VIEW,
or DROP VIEW for a view. Also required for the URI of the “location” parameter for the CREATE EXTERNAL TABLE
and LOAD DATA statements.
Privileges can be specified for a table or view before that object actually exists. If you do not have sufficient privilege
to perform an operation, the error message does not disclose if the object exists or not.
Originally, privileges were encoded in a policy file, stored in HDFS. This mode of operation is still an option, but the
emphasis of privilege management is moving towards being SQL-based. Although currently Impala does not have
GRANT or REVOKE statements, Impala can make use of privileges assigned through GRANT and REVOKE statements
done through Hive. The mode of operation with GRANT and REVOKE statements instead of the policy file requires that
a special Sentry service be enabled; this service stores, retrieves, and manipulates privilege information stored inside
the metastore database.

Starting the impalad Daemon with Sentry Authorization Enabled

To run the impalad daemon with authorization enabled, you add one or more options to the IMPALA_SERVER_ARGS
declaration in the /etc/default/impala configuration file:
• The -server_name option turns on Sentry authorization for Impala. The authorization rules refer to a symbolic
server name, and you specify the name to use as the argument to the -server_name option.
• If you specify just -server_name, Impala uses the Sentry service for authorization, relying on the results of GRANT
and REVOKE statements issued through Hive. (This mode of operation is available in Impala 1.4.0 and higher.)
Prior to Impala 1.4.0, or if you want to continue storing privilege rules in the policy file, also specify the
-authorization_policy_file option as in the following item.

96 | Apache Impala Guide

 Impala Security

• Specifying the -authorization_policy_file option in addition to -server_name makes Impala read privilege
information from a policy file, rather than from the metastore database. The argument to the
-authorization_policy_file option specifies the HDFS path to the policy file that defines the privileges on
different schema objects.
For example, you might adapt your /etc/default/impala configuration to contain lines like the following. To use
the Sentry service rather than the policy file:

IMPALA_SERVER_ARGS=" \
-server_name=server1 \
...

Or to use the policy file, as in releases prior to Impala 1.4:

IMPALA_SERVER_ARGS=" \
-authorization_policy_file=/user/hive/warehouse/auth-policy.ini \
-server_name=server1 \
...

The preceding examples set up a symbolic name of server1 to refer to the current instance of Impala. This symbolic
name is used in the following ways:
• In an environment managed by Cloudera Manager, the server name is specified through Impala (Service-Wide) >
Category > Advanced > Sentry Service and Hive > Service-Wide > Advanced > Sentry Service. The values must
be the same for both, so that Impala and Hive can share the privilege rules. Restart the Impala and Hive services
after setting or changing this value.
• In an environment not managed by Cloudera Manager, you specify this value for the sentry.hive.server
property in the sentry-site.xml configuration file for Hive, as well as in the -server_name option for impalad.
If the impalad daemon is not already running, start it as described in Starting Impala on page 42. If it is already
running, restart it with the command sudo /etc/init.d/impala-server restart. Run the appropriate
commands on all the nodes where impalad normally runs.
• If you use the mode of operation using the policy file, the rules in the [roles] section of the policy file refer to
this same server1 name. For example, the following rule sets up a role report_generator that lets users with
that role query any table in a database named reporting_db on a node where the impalad daemon was started
up with the -server_name=server1 option:

[roles]
report_generator = server=server1->db=reporting_db->table=*->action=SELECT

When impalad is started with one or both of the -server_name=server1 and -authorization_policy_file
options, Impala authorization is enabled. If Impala detects any errors or inconsistencies in the authorization settings
or the policy file, the daemon refuses to start.

Using Impala with the Sentry Service (CDH 5.1 or higher only)
When you use the Sentry service rather than the policy file, you set up privileges through GRANT and REVOKE statement
in either Impala or Hive, then both components use those same privileges automatically. (Impala added the GRANT
and REVOKE statements in Impala 2.0.0 / CDH 5.2.0.)
Hive already had GRANT and REVOKE statements prior to CDH 5.1, but those statements were not production-ready.
CDH 5.1 is the first release where those statements use the Sentry framework and are considered GA level. If you used
the Hive GRANT and REVOKE statements prior to CDH 5.1, you must set up these privileges with the CDH 5.1 versions
of GRANT and REVOKE to take advantage of Sentry authorization.
For information about using the updated Hive GRANT and REVOKE statements, see Sentry service topic in the CDH 5
Security Guide.

Apache Impala Guide | 97

Impala Security

Using Impala with the Sentry Policy File

The policy file is a file that you put in a designated location in HDFS, and is read during the startup of the impalad
daemon when you specify both the -server_name and -authorization_policy_file startup options. It controls
which objects (databases, tables, and HDFS directory paths) can be accessed by the user who connects to impalad,
and what operations that user can perform on the objects.

Note:
In CDH 5 and higher, Cloudera recommends managing privileges through SQL statements, as described
in Using Impala with the Sentry Service (CDH 5.1 or higher only) on page 97. If you are still using policy
files, plan to migrate to the new approach some time in the future.

The location of the policy file is listed in the auth-site.xml configuration file. To minimize overhead, the security
information from this file is cached by each impalad daemon and refreshed automatically, with a default interval of
5 minutes. After making a substantial change to security policies, restart all Impala daemons to pick up the changes
immediately.

Policy File Location and Format

The policy file uses the familiar .ini format, divided into the major sections [groups] and [roles]. There is also
an optional [databases] section, which allows you to specify a specific policy file for a particular database, as explained
in Using Multiple Policy Files for Different Databases on page 102. Another optional section, [users], allows you to
override the OS-level mapping of users to groups; that is an advanced technique primarily for testing and debugging,
and is beyond the scope of this document.
In the [groups] section, you define various categories of users and select which roles are associated with each
category. The group and user names correspond to Linux groups and users on the server where the impalad daemon
runs.
The group and user names in the [groups] section correspond to Linux groups and users on the server where the
impalad daemon runs. When you access Impala through the impalad interpreter, for purposes of authorization, the
user is the logged-in Linux user and the groups are the Linux groups that user is a member of. When you access Impala
through the ODBC or JDBC interfaces, the user and password specified through the connection string are used as login
credentials for the Linux server, and authorization is based on that user name and the associated Linux group
membership.
In the [roles] section, you a set of roles. For each role, you specify precisely the set of privileges is available. That
is, which objects users with that role can access, and what operations they can perform on those objects. This is the
lowest-level category of security information; the other sections in the policy file map the privileges to higher-level
divisions of groups and users. In the [groups] section, you specify which roles are associated with which groups. The
group and user names correspond to Linux groups and users on the server where the impalad daemon runs. The
privileges are specified using patterns like:

server=server_name->db=database_name->table=table_name->action=SELECT
server=server_name->db=database_name->table=table_name->action=CREATE
server=server_name->db=database_name->table=table_name->action=ALL

For the server_name value, substitute the same symbolic name you specify with the impalad -server_name option.
You can use * wildcard characters at each level of the privilege specification to allow access to all such objects. For
example:

server=impala-host.example.com->db=default->table=t1->action=SELECT
server=impala-host.example.com->db=*->table=*->action=CREATE
server=impala-host.example.com->db=*->table=audit_log->action=SELECT
server=impala-host.example.com->db=default->table=t1->action=*

When authorization is enabled, Impala uses the policy file as a whitelist, representing every privilege available to any
user on any object. That is, only operations specified for the appropriate combination of object, role, group, and user

98 | Apache Impala Guide

 Impala Security

are allowed; all other operations are not allowed. If a group or role is defined multiple times in the policy file, the last
definition takes precedence.
To understand the notion of whitelisting, set up a minimal policy file that does not provide any privileges for any object.
When you connect to an Impala node where this policy file is in effect, you get no results for SHOW DATABASES, and
an error when you issue any SHOW TABLES, USE database_name, DESCRIBE table_name, SELECT, and or other
statements that expect to access databases or tables, even if the corresponding databases and tables exist.
The contents of the policy file are cached, to avoid a performance penalty for each query. The policy file is re-checked
by each impalad node every 5 minutes. When you make a non-time-sensitive change such as adding new privileges
or new users, you can let the change take effect automatically a few minutes later. If you remove or reduce privileges,
and want the change to take effect immediately, restart the impalad daemon on all nodes, again specifying the
-server_name and -authorization_policy_file options so that the rules from the updated policy file are
applied.

Examples of Policy File Rules for Security Scenarios

The following examples show rules that might go in the policy file to deal with various authorization-related scenarios.
For illustration purposes, this section shows several very small policy files with only a few rules each. In your environment,
typically you would define many roles to cover all the scenarios involving your own databases, tables, and applications,
and a smaller number of groups, whose members are given the privileges from one or more roles.
A User with No Privileges
If a user has no privileges at all, that user cannot access any schema objects in the system. The error messages do not
disclose the names or existence of objects that the user is not authorized to read.
This is the experience you want a user to have if they somehow log into a system where they are not an authorized
Impala user. In a real deployment with a filled-in policy file, a user might have no privileges because they are not a
member of any of the relevant groups mentioned in the policy file.
Examples of Privileges for Administrative Users
When an administrative user has broad access to tables or databases, the associated rules in the [roles] section
typically use wildcards and/or inheritance. For example, in the following sample policy file, db=* refers to all databases
and db=*->table=* refers to all tables in all databases.
Omitting the rightmost portion of a rule means that the privileges apply to all the objects that could be specified there.
For example, in the following sample policy file, the all_databases role has all privileges for all tables in all databases,
while the one_database role has all privileges for all tables in one specific database. The all_databases role does
not grant privileges on URIs, so a group with that role could not issue a CREATE TABLE statement with a LOCATION
clause. The entire_server role has all privileges on both databases and URIs within the server.

[groups]
supergroup = all_databases
[roles]
read_all_tables = server=server1->db=*->table=*->action=SELECT
all_tables = server=server1->db=*->table=*
all_databases = server=server1->db=*
one_database = server=server1->db=test_db
entire_server = server=server1

A User with Privileges for Specific Databases and Tables

If a user has privileges for specific tables in specific databases, the user can access those things but nothing else. They
can see the tables and their parent databases in the output of SHOW TABLES and SHOW DATABASES, USE the appropriate
databases, and perform the relevant actions (SELECT and/or INSERT) based on the table privileges. To actually create
a table requires the ALL privilege at the database level, so you might define separate roles for the user that sets up a
schema and other users or applications that perform day-to-day operations on the tables.

Apache Impala Guide | 99

Impala Security

The following sample policy file shows some of the syntax that is appropriate as the policy file grows, such as the #
comment syntax, \ continuation syntax, and comma separation for roles assigned to groups or privileges assigned to
roles.

[groups]
cloudera = training_sysadmin, instructor
visitor = student

[roles]
training_sysadmin = server=server1->db=training, \
server=server1->db=instructor_private, \
server=server1->db=lesson_development
instructor = server=server1->db=training->table=*->action=*, \
server=server1->db=instructor_private->table=*->action=*, \
server=server1->db=lesson_development->table=lesson*
# This particular course is all about queries, so the students can SELECT but not INSERT
or CREATE/DROP.
student = server=server1->db=training->table=lesson_*->action=SELECT

Privileges for Working with External Data Files

When data is being inserted through the LOAD DATA statement, or is referenced from an HDFS location outside the
normal Impala database directories, the user also needs appropriate permissions on the URIs corresponding to those
HDFS locations.
In this sample policy file:
• The external_table role lets us insert into and query the Impala table, external_table.sample.
• The staging_dir role lets us specify the HDFS path /user/cloudera/external_data with the LOAD DATA
statement. Remember, when Impala queries or loads data files, it operates on all the files in that directory, not
just a single file, so any Impala LOCATION parameters refer to a directory rather than an individual file.
• We included the IP address and port of the Hadoop name node in the HDFS URI of the staging_dir rule. We
found those details in /etc/hadoop/conf/core-site.xml, under the fs.default.name element. That is
what we use in any roles that specify URIs (that is, the locations of directories in HDFS).
• We start this example after the table external_table.sample is already created. In the policy file for the
example, we have already taken away the external_table_admin role from the cloudera group, and replaced
it with the lesser-privileged external_table role.
• We assign privileges to a subdirectory underneath /user/cloudera in HDFS, because such privileges also apply
to any subdirectories underneath. If we had assigned privileges to the parent directory /user/cloudera, it would
be too likely to mess up other files by specifying a wrong location by mistake.
• The cloudera under the [groups] section refers to the cloudera group. (In the demo VM used for this example,
there is a cloudera user that is a member of a cloudera group.)
Policy file:

[groups]
cloudera = external_table, staging_dir

[roles]
external_table_admin = server=server1->db=external_table
external_table = server=server1->db=external_table->table=sample->action=*
staging_dir =
server=server1->uri=hdfs://127.0.0.1:8020/user/cloudera/external_data->action=*

impala-shell session:

[localhost:21000] > use external_table;

Query: use external_table
[localhost:21000] > show tables;
Query: show tables
Query finished, fetching results ...
+--------+
| name |
+--------+

100 | Apache Impala Guide

 Impala Security

| sample |
+--------+
Returned 1 row(s) in 0.02s

[localhost:21000] > select * from sample;

Query: select * from sample
Query finished, fetching results ...
+-----+
| x |
+-----+
| 1 |
| 5 |
| 150 |
+-----+
Returned 3 row(s) in 1.04s

[localhost:21000] > load data inpath '/user/cloudera/external_data' into table sample;

Query: load data inpath '/user/cloudera/external_data' into table sample
Query finished, fetching results ...
+----------------------------------------------------------+
| summary |
+----------------------------------------------------------+
| Loaded 1 file(s). Total files in destination location: 2 |
+----------------------------------------------------------+
Returned 1 row(s) in 0.26s
[localhost:21000] > select * from sample;
Query: select * from sample
Query finished, fetching results ...
+-------+
| x |
+-------+
| 2 |
| 4 |
| 6 |
| 8 |
| 64738 |
| 49152 |
| 1 |
| 5 |
| 150 |
+-------+
Returned 9 row(s) in 0.22s

[localhost:21000] > load data inpath '/user/cloudera/unauthorized_data' into table

sample;
Query: load data inpath '/user/cloudera/unauthorized_data' into table sample
ERROR: AuthorizationException: User 'cloudera' does not have privileges to access:
hdfs://127.0.0.1:8020/user/cloudera/unauthorized_data

Separating Administrator Responsibility from Read and Write Privileges

Remember that to create a database requires full privilege on that database, while day-to-day operations on tables
within that database can be performed with lower levels of privilege on specific table. Thus, you might set up separate
roles for each database or application: an administrative one that could create or drop the database, and a user-level
one that can access only the relevant tables.
For example, this policy file divides responsibilities between users in 3 different groups:
• Members of the supergroup group have the training_sysadmin role and so can set up a database named
training.
• Members of the cloudera group have the instructor role and so can create, insert into, and query any tables
in the training database, but cannot create or drop the database itself.
• Members of the visitor group have the student role and so can query those tables in the training database.

[groups]
supergroup = training_sysadmin
cloudera = instructor
visitor = student

Apache Impala Guide | 101

Impala Security

[roles]
training_sysadmin = server=server1->db=training
instructor = server=server1->db=training->table=*->action=*
student = server=server1->db=training->table=*->action=SELECT

Using Multiple Policy Files for Different Databases

For an Impala cluster with many databases being accessed by many users and applications, it might be cumbersome
to update the security policy file for each privilege change or each new database, table, or view. You can allow security
to be managed separately for individual databases, by setting up a separate policy file for each database:
• Add the optional [databases] section to the main policy file.
• Add entries in the [databases] section for each database that has its own policy file.
• For each listed database, specify the HDFS path of the appropriate policy file.
For example:

[databases]
# Defines the location of the per-DB policy files for the 'customers' and 'sales'
databases.
customers = hdfs://ha-nn-uri/etc/access/customers.ini
sales = hdfs://ha-nn-uri/etc/access/sales.ini

To enable URIs in per-DB policy files, add the following string in the Cloudera Manager field Impala Service Environment
Advanced Configuration Snippet (Safety Valve):

JAVA_TOOL_OPTIONS="-Dsentry.allow.uri.db.policyfile=true"

Important: Enabling URIs in per-DB policy files introduces a security risk by allowing the owner of
the db-level policy file to grant himself/herself load privileges to anything the impala user has read
permissions for in HDFS (including data in other databases controlled by different db-level policy files).

Setting Up Schema Objects for a Secure Impala Deployment

Remember that in your role definitions, you specify privileges at the level of individual databases and tables, or all
databases or all tables within a database. To simplify the structure of these rules, plan ahead of time how to name
your schema objects so that data with different authorization requirements is divided into separate databases.
If you are adding security on top of an existing Impala deployment, remember that you can rename tables or even
move them between databases using the ALTER TABLE statement. In Impala, creating new databases is a relatively
inexpensive operation, basically just creating a new directory in HDFS.
You can also plan the security scheme and set up the policy file before the actual schema objects named in the policy
file exist. Because the authorization capability is based on whitelisting, a user can only create a new database or table
if the required privilege is already in the policy file: either by listing the exact name of the object being created, or a *
wildcard to match all the applicable objects within the appropriate container.

Privilege Model and Object Hierarchy

Privileges can be granted on different objects in the schema. Any privilege that can be granted is associated with a
level in the object hierarchy. If a privilege is granted on a container object in the hierarchy, the child object automatically
inherits it. This is the same privilege model as Hive and other database systems such as MySQL.
The kinds of objects in the schema hierarchy are:

Server
URI
Database
Table

102 | Apache Impala Guide

 Impala Security

The server name is specified by the -server_name option when impalad starts. Specify the same name for all
impalad nodes in the cluster.

URIs represent the HDFS paths you specify as part of statements such as CREATE EXTERNAL TABLE and LOAD DATA.
Typically, you specify what look like UNIX paths, but these locations can also be prefixed with hdfs:// to make clear
that they are really URIs. To set privileges for a URI, specify the name of a directory, and the privilege applies to all the
files in that directory and any directories underneath it.
In CDH 5.5 / Impala 2.3 and higher, you can specify privileges for individual columns, as described in
https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html. Formerly, to specify read
privileges at this level, you created a view that queried specific columns and/or partitions from a base table, and gave
SELECT privilege on the view but not the underlying table.

URIs must start with either hdfs:// or file://. If a URI starts with anything else, it will cause an exception and the
policy file will be invalid. When defining URIs for HDFS, you must also specify the NameNode. For example:

data_read = server=server1->uri=file:///path/to/dir, \
server=server1->uri=hdfs://namenode:port/path/to/dir

Warning:
Because the NameNode host and port must be specified, Cloudera strongly recommends you use High
Availability (HA). This ensures that the URI will remain constant even if the NameNode changes.

data_read = server=server1->uri=file:///path/to/dir,\
server=server1->uri=hdfs://ha-nn-uri/path/to/dir

Table 2: Valid privilege types and objects they apply to

Privilege Object
INSERT DB, TABLE
SELECT DB, TABLE, COLUMN
ALL SERVER, TABLE, DB, URI

Note:
Although this document refers to the ALL privilege, currently if you use the policy file mode, you do
not use the actual keyword ALL in the policy file. When you code role entries in the policy file:
• To specify the ALL privilege for a server, use a role like server=server_name.
• To specify the ALL privilege for a database, use a role like
server=server_name->db=database_name.
• To specify the ALL privilege for a table, use a role like
server=server_name->db=database_name->table=table_name->action=*.

Operation Scope Privileges Required URI

CREATE DATABASE SERVER ALL
DROP DATABASE DATABASE ALL
CREATE TABLE DATABASE ALL
DROP TABLE TABLE ALL
CREATE VIEW DATABASE; SELECT on TABLE; ALL

Apache Impala Guide | 103

Impala Security

Operation Scope Privileges Required URI

-This operation is allowed if you
have column-level SELECT access
to the columns being used.

ALTER VIEW VIEW/TABLE ALL

-This operation is allowed if you
have column-level SELECT access
to the columns being used.

DROP VIEW VIEW/TABLE ALL

ALTER TABLE .. ADD COLUMNS TABLE ALL
ALTER TABLE .. REPLACE TABLE ALL
COLUMNS
ALTER TABLE .. CHANGE column TABLE ALL
ALTER TABLE .. RENAME TABLE ALL
ALTER TABLE .. SET TABLE ALL
TBLPROPERTIES
ALTER TABLE .. SET FILEFORMAT TABLE ALL
ALTER TABLE .. SET LOCATION TABLE ALL URI
ALTER TABLE .. ADD PARTITION TABLE ALL
ALTER TABLE .. ADD PARTITION TABLE ALL URI
location
ALTER TABLE .. DROP PARTITION TABLE ALL
ALTER TABLE .. PARTITION SET TABLE ALL
FILEFORMAT
SHOW CREATE TABLE TABLE SELECT/INSERT
SHOW PARTITIONS TABLE SELECT/INSERT
SHOW TABLES TABLE SELECT/INSERT
-Output includes all the tables for
which the user has table-level
privileges and all the tables for
which the user has some
column-level privileges.

SHOW GRANT ROLE TABLE SELECT/INSERT

-Output includes an additional
field for any column-level
privileges.

DESCRIBE TABLE TABLE SELECT/INSERT

-Output shows all columns if the
user has table level-privileges or
SELECT privilege on at least one
table column

LOAD DATA TABLE INSERT URI

104 | Apache Impala Guide

 Impala Security

Operation Scope Privileges Required URI

SELECT VIEW/TABLE; COLUMN SELECT
-You can grant the SELECT
privilege on a view to give users
access to specific columns of a
table they do not otherwise have
access to.
-See Column-level Authorization
for details on allowed
column-level operations.

INSERT OVERWRITE TABLE TABLE INSERT

CREATE TABLE .. AS SELECT DATABASE; SELECT on TABLE ALL
-This operation is allowed if you
have column-level SELECT access
to the columns being used.

USE <dbName> Any

CREATE FUNCTION SERVER ALL
ALTER TABLE .. SET TABLE ALL
SERDEPROPERTIES
ALTER TABLE .. PARTITION SET TABLE ALL
SERDEPROPERTIES
EXPLAIN SELECT TABLE; COLUMN SELECT
EXPLAIN INSERT TABLE; COLUMN INSERT
INVALIDATE METADATA SERVER ALL
INVALIDATE METADATA <table TABLE SELECT/INSERT
name>
REFRESH <table name> or TABLE SELECT/INSERT
REFRESH <table name>
PARTITION (<partition_spec>)
DROP FUNCTION SERVER ALL
COMPUTE STATS TABLE ALL

Debugging Failed Sentry Authorization Requests

Sentry logs all facts that lead up to authorization decisions at the debug level. If you do not understand why Sentry is
denying access, the best way to debug is to temporarily turn on debug logging:
• In Cloudera Manager, add log4j.logger.org.apache.sentry=DEBUG to the logging settings for your service
through the corresponding Logging Safety Valve field for the Impala, Hive Server 2, or Solr Server services.
• On systems not managed by Cloudera Manager, add log4j.logger.org.apache.sentry=DEBUG to the
log4j.properties file on each host in the cluster, in the appropriate configuration directory for each service.

Specifically, look for exceptions and messages such as:

FilePermission server..., RequestPermission server...., result [true|false]

Apache Impala Guide | 105

Impala Security

which indicate each evaluation Sentry makes. The FilePermission is from the policy file, while RequestPermission
is the privilege required for the query. A RequestPermission will iterate over all appropriate FilePermission
settings until a match is found. If no matching privilege is found, Sentry returns false indicating “Access Denied” .

Managing Sentry for Impala through Cloudera Manager

To enable the Sentry service for Impala and Hive, set Hive/Impala > Service-Wide > Sentry Service parameter to the
Sentry service. Then restart Impala and Hive. Simply adding Sentry service as a dependency and restarting enables
Impala and Hive to use the Sentry service.
To set the server name to use when granting server level privileges, set the Hive > Service-Wide > Advanced > Server
Name for Sentry Authorization parameter. When using Sentry with the Hive Metastore, you can specify the list of
users that are allowed to bypass Sentry Authorization in Hive Metastore using Hive > Service-Wide > Security > Bypass
Sentry Authorization Users. These are usually service users that already ensure all activity has been authorized.

Note: The Hive/Impala > Service-Wide > Policy File Based Sentry tab contains parameters only
relevant to configuring Sentry using policy files. In particular, make sure that Enable Sentry
Authorization using Policy Files parameter is unchecked when using the Sentry service. Cloudera
Manager throws a validation error if you attempt to configure the Sentry service and policy file at the
same time.

The DEFAULT Database in a Secure Deployment

Because of the extra emphasis on granular access controls in a secure deployment, you should move any important
or sensitive information out of the DEFAULT database into a named database whose privileges are specified in the
policy file. Sometimes you might need to give privileges on the DEFAULT database for administrative reasons; for
example, as a place you can reliably specify with a USE statement when preparing to drop a database.

Impala Authentication
Authentication is the mechanism to ensure that only specified hosts and users can connect to Impala. It also verifies
that when clients connect to Impala, they are connected to a legitimate server. This feature prevents spoofing such as
impersonation (setting up a phony client system with the same account and group names as a legitimate user) and
man-in-the-middle attacks (intercepting application requests before they reach Impala and eavesdropping on sensitive
information in the requests or the results).
Impala supports authentication using either Kerberos or LDAP.

Note: Regardless of the authentication mechanism used, Impala always creates HDFS directories and
data files owned by the same user (typically impala). To implement user-level access to different
databases, tables, columns, partitions, and so on, use the Sentry authorization feature, as explained
in Enabling Sentry Authorization for Impala on page 95.

Once you are finished setting up authentication, move on to authorization, which involves specifying what databases,
tables, HDFS directories, and so on can be accessed by particular users when they connect through Impala. See Enabling
Sentry Authorization for Impala on page 95 for details.

Enabling Kerberos Authentication for Impala

Impala supports Kerberos authentication. For more information on enabling Kerberos authentication, see the topic
on Configuring Hadoop Security in the CDH 5 Security Guide.
When using Impala in a managed environment, Cloudera Manager automatically completes Kerberos configuration.
In an unmanaged environment, create a Kerberos principal for each host running impalad or statestored. Cloudera
recommends using a consistent format, such as impala/_HOST@Your-Realm, but you can use any three-part Kerberos
server principal.

106 | Apache Impala Guide

 Impala Security

In Impala 2.0 and later, user() returns the full Kerberos principal string, such as [email protected], in a Kerberized
environment.

Note: Regardless of the authentication mechanism used, Impala always creates HDFS directories and
data files owned by the same user (typically impala). To implement user-level access to different
databases, tables, columns, partitions, and so on, use the Sentry authorization feature, as explained
in Enabling Sentry Authorization for Impala on page 95.

An alternative form of authentication you can use is LDAP, described in Enabling LDAP Authentication for Impala on
page 109.

Requirements for Using Impala with Kerberos

On version 5 of Red Hat Enterprise Linux and comparable distributions, some additional setup is needed for the
impala-shell interpreter to connect to a Kerberos-enabled Impala cluster:

sudo yum install python-devel openssl-devel python-pip

sudo pip-python install ssl

Important:
• If you plan to use Impala in your cluster, you must configure your KDC to allow tickets to be
renewed, and you must configure krb5.conf to request renewable tickets. Typically, you can
do this by adding the max_renewable_life setting to your realm in kdc.conf, and by adding
the renew_lifetime parameter to the libdefaults section of krb5.conf.
For more information about renewable tickets, see the Kerberos documentation.
• The Impala Web UI does not support Kerberos authentication.
• You cannot use the Impala resource management feature on a cluster that has Kerberos
authentication enabled.

Start all impalad and statestored daemons with the --principal and --keytab-file flags set to the principal
and full path name of the keytab file containing the credentials for the principal.
Impala supports the Cloudera ODBC driver and the Kerberos interface provided. To use Kerberos through the ODBC
driver, the host type must be set depending on the level of the ODBD driver:
• SecImpala for the ODBC 1.0 driver.
• SecBeeswax for the ODBC 1.2 driver.
• Blank for the ODBC 2.0 driver or higher, when connecting to a secure cluster.
• HS2NoSasl for the ODBC 2.0 driver or higher, when connecting to a non-secure cluster.

To enable Kerberos in the Impala shell, start the impala-shell command using the -k flag.
To enable Impala to work with Kerberos security on your Hadoop cluster, make sure you perform the installation and
configuration steps in Authentication in the CDH 5 Security Guide.

Configuring Impala to Support Kerberos Security

Enabling Kerberos authentication for Impala involves steps that can be summarized as follows:
• Creating service principals for Impala and the HTTP service. Principal names take the form:
serviceName/[email protected]
• Creating, merging, and distributing key tab files for these principals.
• Editing /etc/default/impala (in cluster not managed by Cloudera Manager), or editing the Security settings
in the Cloudera Manager interface, to accommodate Kerberos authentication.

Apache Impala Guide | 107

Impala Security

Enabling Kerberos for Impala

1. Create an Impala service principal, specifying the name of the OS user that the Impala daemons run under, the
fully qualified domain name of each node running impalad, and the realm name. For example:

$ kadmin
kadmin: addprinc -requires_preauth -randkey
impala/[email protected]

2. Create an HTTP service principal. For example:

kadmin: addprinc -randkey HTTP/[email protected]

Note: The HTTP component of the service principal must be uppercase as shown in the preceding
example.

3. Create keytab files with both principals. For example:

kadmin: xst -k impala.keytab impala/impala_host.example.com

kadmin: xst -k http.keytab HTTP/impala_host.example.com
kadmin: quit

4. Use ktutil to read the contents of the two keytab files and then write those contents to a new file. For example:

$ ktutil
ktutil: rkt impala.keytab
ktutil: rkt http.keytab
ktutil: wkt impala-http.keytab
ktutil: quit

5. (Optional) Test that credentials in the merged keytab file are valid, and that the “renew until” date is in the future.
For example:

$ klist -e -k -t impala-http.keytab

6. Copy the impala-http.keytab file to the Impala configuration directory. Change the permissions to be only
read for the file owner and change the file owner to the impala user. By default, the Impala user and group are
both named impala. For example:

$ cp impala-http.keytab /etc/impala/conf
$ cd /etc/impala/conf
$ chmod 400 impala-http.keytab
$ chown impala:impala impala-http.keytab

7. Add Kerberos options to the Impala defaults file, /etc/default/impala. Add the options for both the impalad
and statestored daemons, using the IMPALA_SERVER_ARGS and IMPALA_STATE_STORE_ARGS variables. For
example, you might add:

-kerberos_reinit_interval=60
-principal=impala_1/[email protected]
-keytab_file=/var/run/cloudera-scm-agent/process/3212-impala-IMPALAD/impala.keytab

For more information on changing the Impala defaults specified in /etc/default/impala, see Modifying Impala
Startup Options.

Note: Restart impalad and statestored for these configuration changes to take effect.

108 | Apache Impala Guide

 Impala Security

Enabling Kerberos for Impala with a Proxy Server

A common configuration for Impala with High Availability is to use a proxy server to submit requests to the actual
impalad daemons on different hosts in the cluster. This configuration avoids connection problems in case of machine
failure, because the proxy server can route new requests through one of the remaining hosts in the cluster. This
configuration also helps with load balancing, because the additional overhead of being the “coordinator node” for
each query is spread across multiple hosts.
Although you can set up a proxy server with or without Kerberos authentication, typically users set up a secure Kerberized
configuration. For information about setting up a proxy server for Impala, including Kerberos-specific steps, see Using
Impala through a Proxy for High Availability on page 85.

Enabling Impala Delegation for Kerberos Users

See Configuring Impala Delegation for Hue and BI Tools on page 112 for details about the delegation feature that lets
certain users submit queries using the credentials of other users.

Using TLS/SSL with Business Intelligence Tools

You can use Kerberos authentication, TLS/SSL encryption, or both to secure connections from JDBC and ODBC applications
to Impala. See Configuring Impala to Work with JDBC on page 34 and Configuring Impala to Work with ODBC on page
31 for details.
Currently, the Hive JDBC driver does not support connections that use both Kerberos authentication and SSL encryption.
To use both of these security features with Impala through a JDBC application, use the Cloudera JDBC Connector as
the JDBC driver.

Enabling LDAP Authentication for Impala

Authentication is the process of allowing only specified named users to access the server (in this case, the Impala
server). This feature is crucial for any production deployment, to prevent misuse, tampering, or excessive load on the
server. Impala uses LDAP for authentication, verifying the credentials of each user who connects through impala-shell,
Hue, a Business Intelligence tool, JDBC or ODBC application, and so on.

Note: Regardless of the authentication mechanism used, Impala always creates HDFS directories and
data files owned by the same user (typically impala). To implement user-level access to different
databases, tables, columns, partitions, and so on, use the Sentry authorization feature, as explained
in Enabling Sentry Authorization for Impala on page 95.

An alternative form of authentication you can use is Kerberos, described in Enabling Kerberos Authentication for Impala
on page 106.

Requirements for Using Impala with LDAP

Authentication against LDAP servers is available in Impala 1.2.2 and higher. Impala 1.4.0 adds support for secure LDAP
authentication through SSL and TLS.
The Impala LDAP support lets you use Impala with systems such as Active Directory that use LDAP behind the scenes.
Kerberos Authentication for Connections Between Impala Components
Only client->Impala connections can be authenticated by LDAP.
You must use the Kerberos authentication mechanism for connections between internal Impala components, such as
between the impalad, statestored, and catalogd daemons. See Enabling Kerberos Authentication for Impala on
page 106 on how to set up Kerberos for Impala.

Server-Side LDAP Setup

These requirements apply on the server side when configuring and starting Impala:
To enable LDAP authentication, set the following startup options for impalad:

Apache Impala Guide | 109

Impala Security

• --enable_ldap_auth enables LDAP-based authentication between the client and Impala.

• --ldap_uri sets the URI of the LDAP server to use. Typically, the URI is prefixed with ldap://. In Impala 1.4.0
and higher, you can specify secure SSL-based LDAP transport by using the prefix ldaps://. The URI can optionally
specify the port, for example: ldap://ldap_server.cloudera.com:389 or
ldaps://ldap_server.cloudera.com:636. (389 and 636 are the default ports for non-SSL and SSL LDAP
connections, respectively.)
• For ldaps:// connections secured by SSL, --ldap_ca_certificate="/path/to/certificate/pem"
specifies the location of the certificate in standard .PEM format. Store this certificate on the local filesystem, in a
location that only the impala user and other trusted users can read.

Support for Custom Bind Strings

When Impala connects to LDAP it issues a bind call to the LDAP server to authenticate as the connected user. Impala
clients, including the Impala shell, provide the short name of the user to Impala. This is necessary so that Impala can
use Sentry for role-based access, which uses short names.
However, LDAP servers often require more complex, structured usernames for authentication. Impala supports three
ways of transforming the short name (for example, 'henry') to a more complicated string. If necessary, specify one
of the following configuration options when starting the impalad daemon on each DataNode:
• --ldap_domain: Replaces the username with a string username@ldap_domain.
• --ldap_baseDN: Replaces the username with a “distinguished name” (DN) of the form:
uid=userid,ldap_baseDN. (This is equivalent to a Hive option).
• --ldap_bind_pattern: This is the most general option, and replaces the username with the string
ldap_bind_pattern where all instances of the string #UID are replaced with userid. For example, an
ldap_bind_pattern of "user=#UID,OU=foo,CN=bar" with a username of henry will construct a bind name
of "user=henry,OU=foo,CN=bar".
For clusters not managed by Cloudera Manager, specify the option on the impalad command line. For clusters managed
by Cloudera Manager 5.4.0 and higher, search for the configuration field names ldap_domain, ldap_basedn, or
ldap_bind_pattern, fill in and save the appropriate field values, and restart the Impala service. Prior to Cloudera
Manager 5.4.0, these values were filled in using the Impala Daemon Command Line Argument Advanced Configuration
Snippet (Safety Valve) field.
These options are mutually exclusive; Impala does not start if more than one of these options is specified.

Secure LDAP Connections

To avoid sending credentials over the wire in cleartext, you must configure a secure connection between both the
client and Impala, and between Impala and the LDAP server. The secure connection could use SSL or TLS.
Secure LDAP connections through SSL:
For SSL-enabled LDAP connections, specify a prefix of ldaps:// instead of ldap://. Also, the default port for
SSL-enabled LDAP connections is 636 instead of 389.
Secure LDAP connections through TLS:
TLS, the successor to the SSL protocol, is supported by most modern LDAP servers. Unlike SSL connections, TLS
connections can be made on the same server port as non-TLS connections. To secure all connections using TLS, specify
the following flags as startup options to the impalad daemon:
• --ldap_tls tells Impala to start a TLS connection to the LDAP server, and to fail authentication if it cannot be
done.
• --ldap_ca_certificate="/path/to/certificate/pem" specifies the location of the certificate in standard
.PEM format. Store this certificate on the local filesystem, in a location that only the impala user and other trusted
users can read.

LDAP Authentication for impala-shell Interpreter

To connect to Impala using LDAP authentication, you specify command-line options to the impala-shell command
interpreter and enter the password when prompted:

110 | Apache Impala Guide

 Impala Security

• -l enables LDAP authentication.

• -u sets the user. Per Active Directory, the user is the short user name, not the full LDAP distinguished name. If
your LDAP settings include a search base, use the --ldap_bind_pattern on the impalad daemon to translate
the short user name from impala-shell automatically to the fully qualified name.
• impala-shell automatically prompts for the password.
For the full list of available impala-shell options, see impala-shell Configuration Options on page 470.
LDAP authentication for JDBC applications: See Configuring Impala to Work with JDBC on page 34 for the format to
use with the JDBC connection string for servers using LDAP authentication.

Enabling LDAP for Impala in Hue

Enabling LDAP for Impala in Hue Using Cloudera Manager

1. Go to the Hue service.
2. Click the Configuration tab.
3. Select Scope > Hue Server.
4. Select Category > Advanced.
5. Add the following properties to the Hue Server Advanced Configuration Snippet (Safety Valve) for
hue_safety_valve_server.ini property.

[impala]
auth_username=<LDAP username of Hue user to be authenticated>
auth_password=<LDAP password of Hue user to be authenticated>

6. Click Save Changes.

Enabling LDAP for Impala in Hue Using the Command Line

LDAP authentication for the Impala app in Hue can be enabled by setting the following properties under the [impala]
section in hue.ini.

auth_username LDAP username of Hue user to be authenticated.

auth_password LDAP password of Hue user to be authenticated.

These login details are only used by Impala to authenticate to LDAP. The Impala service trusts Hue to have already
validated the user being impersonated, rather than simply passing on the credentials.

Enabling Impala Delegation for LDAP Users

See Configuring Impala Delegation for Hue and BI Tools on page 112 for details about the delegation feature that lets
certain users submit queries using the credentials of other users.

LDAP Restrictions for Impala

The LDAP support is preliminary. It currently has only been tested against Active Directory.

Using Multiple Authentication Methods with Impala

Impala 2.0 and later automatically handles both Kerberos and LDAP authentication. Each impalad daemon can accept
both Kerberos and LDAP requests through the same port. No special actions need to be taken if some users authenticate
through Kerberos and some through LDAP.
Prior to Impala 2.0, you had to configure each impalad to listen on a specific port depending on the kind of
authentication, then configure your network load balancer to forward each kind of request to a DataNode that was
set up with the appropriate authentication type. Once the initial request was made using either Kerberos or LDAP
authentication, Impala automatically handled the process of coordinating the work across multiple nodes and transmitting
intermediate results back to the coordinator node.

Apache Impala Guide | 111

Impala Security

Configuring Impala Delegation for Hue and BI Tools

When users submit Impala queries through a separate application, such as Hue or a business intelligence tool, typically
all requests are treated as coming from the same user. In Impala 1.2 and higher,,Impala supports applications to pass
along credentials for the users that connect to them, known as “delegation”, and to issue Impala queries with the
privileges for those users. Currently, the delegation feature is available only for Impala queries submitted through
application interfaces such as Hue and BI tools; for example, Impala cannot issue queries using the privileges of the
HDFS user.
The delegation feature is enabled by a startup option for impalad: --authorized_proxy_user_config. When
you specify this option, users whose names you specify (such as hue) can delegate the execution of a query to another
user. The query runs with the privileges of the delegated user, not the original user such as hue. The name of the
delegated user is passed using the HiveServer2 configuration property impala.doas.user.
You can specify a list of users that the application user can delegate to, or * to allow a superuser to delegate to any
other user. For example:

impalad --authorized_proxy_user_config 'hue=user1,user2;admin=*' ...

Note: Make sure to use single quotes or escape characters to ensure that any * characters do not
undergo wildcard expansion when specified in command-line arguments.

See Modifying Impala Startup Options on page 43 for details about adding or changing impalad startup options. See
this Cloudera blog post for background information about the delegation capability in HiveServer2.
To set up authentication for the delegated users:
• On the server side, configure either user/password authentication through LDAP, or Kerberos authentication, for
all the delegated users. See Enabling LDAP Authentication for Impala on page 109 or Enabling Kerberos Authentication
for Impala on page 106 for details.
• On the client side, follow the instructions in the “Using User Name and Password” section in the ODBC driver
installation guide. Then search for “delegation” in that same installation guide to learn about the Delegation UID
field and DelegationUID configuration keyword to enable the delegation feature for ODBC-based BI tools.

Auditing Impala Operations

To monitor how Impala data is being used within your organization, ensure that your Impala authorization and
authentication policies are effective, and detect attempts at intrusion or unauthorized access to Impala data, you can
use the auditing feature in Impala 1.2.1 and higher:
• Enable auditing by including the option -audit_event_log_dir=directory_path in your impalad startup
options for a cluster not managed by Cloudera Manager, or configuring Impala Daemon logging in Cloudera
Manager. The log directory must be a local directory on the server, not an HDFS directory.
• Decide how many queries will be represented in each log files. By default, Impala starts a new log file every 5000
queries. To specify a different number, include the option
-max_audit_event_log_file_size=number_of_queries in the impalad startup options.
• Configure the Cloudera Navigator product to collect and consolidate the audit logs from all the hosts in the cluster.
• Use Cloudera Navigator or Cloudera Manager to filter, visualize, and produce reports based on the audit data.
(The Impala auditing feature works with Cloudera Manager 4.7 to 5.1 and Cloudera Navigator 2.1 and higher.)
Check the audit data to ensure that all activity is authorized and detect attempts at unauthorized access.

Durability and Performance Considerations for Impala Auditing

The auditing feature only imposes performance overhead while auditing is enabled.

112 | Apache Impala Guide

 Impala Security

Because any Impala host can process a query, enable auditing on all hosts where the impalad daemon runs. Each
host stores its own log files, in a directory in the local filesystem. The log data is periodically flushed to disk (through
an fsync() system call) to avoid loss of audit data in case of a crash.
The runtime overhead of auditing applies to whichever host serves as the coordinator for the query, that is, the host
you connect to when you issue the query. This might be the same host for all queries, or different applications or users
might connect to and issue queries through different hosts.
To avoid excessive I/O overhead on busy coordinator hosts, Impala syncs the audit log data (using the fsync() system
call) periodically rather than after every query. Currently, the fsync() calls are issued at a fixed interval, every 5
seconds.
By default, Impala avoids losing any audit log data in the case of an error during a logging operation (such as a disk full
error), by immediately shutting down impalad on the host where the auditing problem occurred. You can override
this setting by specifying the option -abort_on_failed_audit_event=false in the impalad startup options.

Format of the Audit Log Files

The audit log files represent the query information in JSON format, one query per line. Typically, rather than looking
at the log files themselves, you use the Cloudera Navigator product to consolidate the log data from all Impala hosts
and filter and visualize the results in useful ways. (If you do examine the raw log data, you might run the files through
a JSON pretty-printer first.)
All the information about schema objects accessed by the query is encoded in a single nested record on the same line.
For example, the audit log for an INSERT ... SELECT statement records that a select operation occurs on the source
table and an insert operation occurs on the destination table. The audit log for a query against a view records the base
table accessed by the view, or multiple base tables in the case of a view that includes a join query. Every Impala
operation that corresponds to a SQL statement is recorded in the audit logs, whether the operation succeeds or fails.
Impala records more information for a successful operation than for a failed one, because an unauthorized query is
stopped immediately, before all the query planning is completed.
The information logged for each query includes:
• Client session state:
– Session ID
– User name
– Network address of the client connection
• SQL statement details:
– Query ID
– Statement Type - DML, DDL, and so on
– SQL statement text
– Execution start time, in local time
– Execution Status - Details on any errors that were encountered
– Target Catalog Objects:
– Object Type - Table, View, or Database
– Fully qualified object name
– Privilege - How the object is being used (SELECT, INSERT, CREATE, and so on)

Which Operations Are Audited

The kinds of SQL queries represented in the audit log are:
• Queries that are prevented due to lack of authorization.
• Queries that Impala can analyze and parse to determine that they are authorized. The audit data is recorded
immediately after Impala finishes its analysis, before the query is actually executed.

Apache Impala Guide | 113

Impala Security

The audit log does not contain entries for queries that could not be parsed and analyzed. For example, a query that
fails due to a syntax error is not recorded in the audit log. The audit log also does not contain queries that fail due to
a reference to a table that does not exist, if you would be authorized to access the table if it did exist.
Certain statements in the impala-shell interpreter, such as CONNECT, SUMMARY, PROFILE, SET, and QUIT, do not
correspond to actual SQL queries, and these statements are not reflected in the audit log.

Reviewing the Audit Logs

You typically do not review the audit logs in raw form. The Cloudera Manager Agent periodically transfers the log
information into a back-end database where it can be examined in consolidated form. See the Cloudera Navigator
documentation for details .

Viewing Lineage Information for Impala Data

Lineage is a feature in the Cloudera Navigator data management component that helps you track where data originated,
and how data propagates through the system through SQL statements such as SELECT, INSERT, and CREATE TABLE
AS SELECT. Impala is covered by the Cloudera Navigator lineage features in CDH 5.4.0 and higher.
This type of tracking is important in high-security configurations, especially in highly regulated industries such as
healthcare, pharmaceuticals, financial services and intelligence. For such kinds of sensitive data, it is important to know
all the places in the system that contain that data or other data derived from it; to verify who has accessed that data;
and to be able to doublecheck that the data used to make a decision was processed correctly and not tampered with.
You interact with this feature through lineage diagrams showing relationships between tables and columns. For
instructions about interpreting lineage diagrams, see
http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_lineage.html.

Column Lineage
Column lineage tracks information in fine detail, at the level of particular columns rather than entire tables.
For example, if you have a table with information derived from web logs, you might copy that data into other tables
as part of the ETL process. The ETL operations might involve transformations through expressions and function calls,
and rearranging the columns into more or fewer tables (normalizing or denormalizing the data). Then for reporting,
you might issue queries against multiple tables and views. In this example, column lineage helps you determine that
data that entered the system as RAW_LOGS.FIELD1 was then turned into WEBSITE_REPORTS.IP_ADDRESS through
an INSERT ... SELECT statement. Or, conversely, you could start with a reporting query against a view, and trace
the origin of the data in a field such as TOP_10_VISITORS.USER_ID back to the underlying table and even further
back to the point where the data was first loaded into Impala.
When you have tables where you need to track or control access to sensitive information at the column level, see
Enabling Sentry Authorization for Impala on page 95 for how to implement column-level security. You set up
authorization using the Sentry framework, create views that refer to specific sets of columns, and then assign
authorization privileges to those views rather than the underlying tables.

Lineage Data for Impala

The lineage feature is enabled by default. When lineage logging is enabled, the serialized column lineage graph is
computed for each query and stored in a specialized log file in JSON format.
Impala records queries in the lineage log if they complete successfully, or fail due to authorization errors. For write
operations such as INSERT and CREATE TABLE AS SELECT, the statement is recorded in the lineage log only if it
successfully completes. Therefore, the lineage feature tracks data that was accessed by successful queries, or that was
attempted to be accessed by unsuccessful queries that were blocked due to authorization failure. These kinds of queries
represent data that really was accessed, or where the attempted access could represent malicious activity.
Impala does not record in the lineage log queries that fail due to syntax errors or that fail or are cancelled before they
reach the stage of requesting rows from the result set.

114 | Apache Impala Guide

 Impala Security

To enable or disable this feature on a system not managed by Cloudera Manager, set or remove the
-lineage_event_log_dir configuration option for the impalad daemon. For information about turning the lineage
feature on and off through Cloudera Manager, see
http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt_impala_lineage_log.html.

Apache Impala Guide | 115

Impala SQL Language Reference

Impala SQL Language Reference

Impala uses SQL as its query language. To protect user investment in skills development and query design, Impala
provides a high degree of compatibility with the Hive Query Language (HiveQL):
• Because Impala uses the same metadata store as Hive to record information about table structure and properties,
Impala can access tables defined through the native Impala CREATE TABLE command, or tables created using
the Hive data definition language (DDL).
• Impala supports data manipulation (DML) statements similar to the DML component of HiveQL.
• Impala provides many built-in functions with the same names and parameter types as their HiveQL equivalents.
Impala supports most of the same statements and clauses as HiveQL, including, but not limited to JOIN, AGGREGATE,
DISTINCT, UNION ALL, ORDER BY, LIMIT and (uncorrelated) subquery in the FROM clause. Impala also supports
INSERT INTO and INSERT OVERWRITE.

Impala supports data types with the same names and semantics as the equivalent Hive data types: string, TINYINT,
SMALLINT, INT, BIGINT, FLOAT, DOUBLE, BOOLEAN, STRING, TIMESTAMP.

For full details about Impala SQL syntax and semantics, see Impala SQL Statements on page 214.
Most HiveQL SELECT and INSERT statements run unmodified with Impala. For information about Hive syntax not
available in Impala, see SQL Differences Between Impala and Hive on page 461.
For a list of the built-in functions available in Impala queries, see Impala Built-In Functions on page 338.

Comments
Impala supports the familiar styles of SQL comments:
• All text from a -- sequence to the end of the line is considered a comment and ignored. This type of comment
can occur on a single line by itself, or after all or part of a statement.
• All text from a /* sequence to the next */ sequence is considered a comment and ignored. This type of comment
can stretch over multiple lines. This type of comment can occur on one or more lines by itself, in the middle of a
statement, or before or after a statement.
For example:

-- This line is a comment about a table.

create table ...;

/*
This is a multi-line comment about a query.
*/
select ...;

select * from t /* This is an embedded comment about a query. */ where ...;

select * from t -- This is a trailing comment within a multi-line command.

where ...;

Data Types
Impala supports a set of data types that you can use for table columns, expression values, and function arguments and
return values.

Note: Currently, Impala supports only scalar types, not composite or nested types. Accessing a table
containing any columns with unsupported types causes an error.

116 | Apache Impala Guide

 Impala SQL Language Reference

For the notation to write literals of each of these data types, see Literals on page 184.
See SQL Differences Between Impala and Hive on page 461 for differences between Impala and Hive data types.

ARRAY Complex Type (CDH 5.5 or higher only)

A complex data type that can represent an arbitrary number of ordered elements. The elements can be scalars or
another complex type (ARRAY, STRUCT, or MAP).
Syntax:

column_name ARRAY < type >

type ::= primitive_type | complex_type

Usage notes:
Because complex types are often used in combination, for example an ARRAY of STRUCT elements, if you are unfamiliar
with the Impala complex types, start with Complex Types (CDH 5.5 or higher only) on page 156 for background information
and usage examples.
The elements of the array have no names. You refer to the value of the array item using the ITEM pseudocolumn, or
its position in the array with the POS pseudocolumn. See ITEM and POS Pseudocolumns on page 170 for information
about these pseudocolumns.
Each row can have a different number of elements (including none) in the array for that row.
When an array contains items of scalar types, you can use aggregation functions on the array elements without using
join notation. For example, you can find the COUNT(), AVG(), SUM(), and so on of numeric array elements, or the
MAX() and MIN() of any scalar array elements by referring to table_name.array_column in the FROM clause of
the query. When you need to cross-reference values from the array with scalar values from the same row, such as by
including a GROUP BY clause to produce a separate aggregated result for each row, then the join clause is required.
A common usage pattern with complex types is to have an array as the top-level type for the column: an array of
structs, an array of maps, or an array of arrays. For example, you can model a denormalized table by creating a column
that is an ARRAY of STRUCT elements; each item in the array represents a row from a table that would normally be
used in a join query. This kind of data structure lets you essentially denormalize tables by associating multiple rows
from one table with the matching row in another table.
You typically do not create more than one top-level ARRAY column, because if there is some relationship between the
elements of multiple arrays, it is convenient to model the data as an array of another complex type element (either
STRUCT or MAP).

You can pass a qualified name to DESCRIBE to specify an ARRAY, STRUCT, or MAP column and visualize its structure
as if it were a table. An ARRAY is shown as a two-column table, with ITEM and POS columns. A STRUCT is shown as a
table with each field representing a column in the table. A MAP is shown as a two-column table, with KEY and VALUE
columns.
Added in: CDH 5.5.0 (Impala 2.3.0)
Restrictions:
• Columns with this data type can only be used in tables or partitions with the Parquet file format.
• Columns with this data type cannot be used as partition key columns in a partitioned table.
• The COMPUTE STATS statement does not produce any statistics for columns of this data type.
• The maximum length of the column definition for any complex type, including declarations for any nested types,
is 4000 characters.
• See Limitations and Restrictions for Complex Types on page 161 for a full list of limitations and associated guidelines
about complex type columns.

Examples:

Apache Impala Guide | 117

Impala SQL Language Reference

Note: Many of the complex type examples refer to tables such as CUSTOMER and REGION adapted
from the tables used in the TPC-H benchmark. See Sample Schema and Data for Experimenting with
Impala Complex Types on page 178 for the table definitions.

The following example shows how to construct a table with various kinds of ARRAY columns, both at the top level and
nested within other complex types. Whenever the ARRAY consists of a scalar value, such as in the PETS column or the
CHILDREN field, you can see that future expansion is limited. For example, you could not easily evolve the schema to
record the kind of pet or the child's birthday alongside the name. Therefore, it is more common to use an ARRAY whose
elements are of STRUCT type, to associate multiple fields with each array element.

Note: Practice the CREATE TABLE and query notation for complex type columns using empty tables,
until you can visualize a complex data structure and construct corresponding SQL statements reliably.

CREATE TABLE array_demo

(
id BIGINT,
name STRING,
-- An ARRAY of scalar type as a top-level column.
pets ARRAY <STRING>,

-- An ARRAY with elements of complex type (STRUCT).

places_lived ARRAY < STRUCT <
place: STRING,
start_year: INT
>>,

-- An ARRAY as a field (CHILDREN) within a STRUCT.

-- (The STRUCT is inside another ARRAY, because it is rare
-- for a STRUCT to be a top-level column.)
marriages ARRAY < STRUCT <
spouse: STRING,
children: ARRAY <STRING>
>>,

-- An ARRAY as the value part of a MAP.

-- The first MAP field (the key) would be a value such as
-- 'Parent' or 'Grandparent', and the corresponding array would
-- represent 2 parents, 4 grandparents, and so on.
ancestors MAP < STRING, ARRAY <STRING> >
)
STORED AS PARQUET;

The following example shows how to examine the structure of a table containing one or more ARRAY columns by using
the DESCRIBE statement. You can visualize each ARRAY as its own two-column table, with columns ITEM and POS.

DESCRIBE array_demo;
+--------------+---------------------------+
| name | type |
+--------------+---------------------------+
| id | bigint |
| name | string |
| pets | array<string> |
| marriages | array<struct< |
| | spouse:string, |
| | children:array<string> |
| | >> |
| places_lived | array<struct< |
| | place:string, |
| | start_year:int |
| | >> |
| ancestors | map<string,array<string>> |
+--------------+---------------------------+

DESCRIBE array_demo.pets;
+------+--------+

118 | Apache Impala Guide

 Impala SQL Language Reference

| name | type |
+------+--------+
| item | string |
| pos | bigint |
+------+--------+

DESCRIBE array_demo.marriages;
+------+--------------------------+
| name | type |
+------+--------------------------+
| item | struct< |
| | spouse:string, |
| | children:array<string> |
| | > |
| pos | bigint |
+------+--------------------------+

DESCRIBE array_demo.places_lived;
+------+------------------+
| name | type |
+------+------------------+
| item | struct< |
| | place:string, |
| | start_year:int |
| | > |
| pos | bigint |
+------+------------------+

DESCRIBE array_demo.ancestors;
+-------+---------------+
| name | type |
+-------+---------------+
| key | string |
| value | array<string> |
+-------+---------------+

The following example shows queries involving ARRAY columns containing elements of scalar or complex types. You
“unpack” each ARRAY column by referring to it in a join query, as if it were a separate table with ITEM and POS columns.
If the array element is a scalar type, you refer to its value using the ITEM pseudocolumn. If the array element is a
STRUCT, you refer to the STRUCT fields using dot notation and the field names. If the array element is another ARRAY
or a MAP, you use another level of join to unpack the nested collection elements.

-- Array of scalar values.

-- Each array element represents a single string, plus we know its position in the array.
SELECT id, name, pets.pos, pets.item FROM array_demo, array_demo.pets;

-- Array of structs.
-- Now each array element has named fields, possibly of different types.
-- You can consider an ARRAY of STRUCT to represent a table inside another table.
SELECT id, name, places_lived.pos, places_lived.item.place, places_lived.item.start_year
FROM array_demo, array_demo.places_lived;

-- The .ITEM name is optional for array elements that are structs.
-- The following query is equivalent to the previous one, with .ITEM
-- removed from the column references.
SELECT id, name, places_lived.pos, places_lived.place, places_lived.start_year
FROM array_demo, array_demo.places_lived;

-- To filter specific items from the array, do comparisons against the .POS or .ITEM
-- pseudocolumns, or names of struct fields, in the WHERE clause.
SELECT id, name, pets.item FROM array_demo, array_demo.pets
WHERE pets.pos in (0, 1, 3);

SELECT id, name, pets.item FROM array_demo, array_demo.pets

WHERE pets.item LIKE 'Mr. %';

SELECT id, name, places_lived.pos, places_lived.place, places_lived.start_year

FROM array_demo, array_demo.places_lived
WHERE places_lived.place like '%California%';

Apache Impala Guide | 119

Impala SQL Language Reference

Related information:
Complex Types (CDH 5.5 or higher only) on page 156, STRUCT Complex Type (CDH 5.5 or higher only) on page 142, MAP
Complex Type (CDH 5.5 or higher only) on page 136

BIGINT Data Type

An 8-byte integer data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name BIGINT

Range: -9223372036854775808 .. 9223372036854775807. There is no UNSIGNED subtype.

Conversions: Impala automatically converts to a floating-point type (FLOAT or DOUBLE) automatically. Use CAST() to
convert to TINYINT, SMALLINT, INT, STRING, or TIMESTAMP. Casting an integer or floating-point value N to TIMESTAMP
produces a value that is N seconds past the start of the epoch date (January 1, 1970). By default, the result value
represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Examples:

CREATE TABLE t1 (x BIGINT);

SELECT CAST(1000 AS BIGINT);

Usage notes:
BIGINT is a convenient type to use for column declarations because you can use any kind of integer values in INSERT
statements and they are promoted to BIGINT where necessary. However, BIGINT also requires the most bytes of
any integer type on disk and in memory, meaning your queries are not as efficient and scalable as possible if you
overuse this type. Therefore, prefer to use the smallest integer type with sufficient range to hold all input values, and
CAST() when necessary to the appropriate type.

For a convenient and automated way to check the bounds of the BIGINT type, call the functions MIN_BIGINT() and
MAX_BIGINT().

If an integer value is too large to be represented as a BIGINT, use a DECIMAL instead with sufficient digits of precision.
NULL considerations: Casting any non-numeric value to this type produces a NULL value.
Partitioning: Prefer to use this type for a partition key column. Impala can process the numeric type more efficiently
than a STRING representation of the value.
HBase considerations: This data type is fully compatible with HBase tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as an 8-byte value.
Added in: Available in all versions of Impala.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Sqoop considerations:
If you use Sqoop to convert RDBMS data to Parquet, be careful with interpreting any resulting values from DATE,
DATETIME, or TIMESTAMP columns. The underlying values are represented as the Parquet INT64 type, which is
represented as BIGINT in the Impala table. The Parquet values represent the time in milliseconds, while Impala
interprets BIGINT as the time in seconds. Therefore, if you have a BIGINT column in a Parquet table that was imported
this way from Sqoop, divide the values by 1000 when interpreting as the TIMESTAMP type.

120 | Apache Impala Guide

 Impala SQL Language Reference

Related information:
Numeric Literals on page 184, TINYINT Data Type on page 153, SMALLINT Data Type on page 140, INT Data Type on page
135, BIGINT Data Type on page 120, DECIMAL Data Type (CDH 5.1 or higher only) on page 125, Impala Mathematical
Functions on page 339

BOOLEAN Data Type

A data type used in CREATE TABLE and ALTER TABLE statements, representing a single true/false choice.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name BOOLEAN

Range: TRUE or FALSE. Do not use quotation marks around the TRUE and FALSE literal values. You can write the literal
values in uppercase, lowercase, or mixed case. The values queried from a table are always returned in lowercase, true
or false.
Conversions: Impala does not automatically convert any other type to BOOLEAN. All conversions must use an explicit
call to the CAST() function.
You can use CAST() to convert any integer or floating-point type to BOOLEAN: a value of 0 represents false, and any
non-zero value is converted to true.

SELECT CAST(42 AS BOOLEAN) AS nonzero_int, CAST(99.44 AS BOOLEAN) AS nonzero_decimal,

CAST(000 AS BOOLEAN) AS zero_int, CAST(0.0 AS BOOLEAN) AS zero_decimal;
+-------------+-----------------+----------+--------------+
| nonzero_int | nonzero_decimal | zero_int | zero_decimal |
+-------------+-----------------+----------+--------------+
| true | true | false | false |
+-------------+-----------------+----------+--------------+

When you cast the opposite way, from BOOLEAN to a numeric type, the result becomes either 1 or 0:

SELECT CAST(true AS INT) AS true_int, CAST(true AS DOUBLE) AS true_double,

CAST(false AS INT) AS false_int, CAST(false AS DOUBLE) AS false_double;
+----------+-------------+-----------+--------------+
| true_int | true_double | false_int | false_double |
+----------+-------------+-----------+--------------+
| 1 | 1 | 0 | 0 |
+----------+-------------+-----------+--------------+

You can cast DECIMAL values to BOOLEAN, with the same treatment of zero and non-zero values as the other numeric
types. You cannot cast a BOOLEAN to a DECIMAL.
You cannot cast a STRING value to BOOLEAN, although you can cast a BOOLEAN value to STRING, returning '1' for
true values and '0' for false values.

Although you can cast a TIMESTAMP to a BOOLEAN or a BOOLEAN to a TIMESTAMP, the results are unlikely to be useful.
Any non-zero TIMESTAMP (that is, any value other than 1970-01-01 00:00:00) becomes TRUE when converted to
BOOLEAN, while 1970-01-01 00:00:00 becomes FALSE. A value of FALSE becomes 1970-01-01 00:00:00 when
converted to BOOLEAN, and TRUE becomes one second past this epoch date, that is, 1970-01-01 00:00:01.
NULL considerations: An expression of this type produces a NULL value if any argument of the expression is NULL.
Partitioning:
Do not use a BOOLEAN column as a partition key. Although you can create such a table, subsequent operations produce
errors:

[localhost:21000] > create table truth_table (assertion string) partitioned by (truth

boolean);
[localhost:21000] > insert into truth_table values ('Pigs can fly',false);

Apache Impala Guide | 121

Impala SQL Language Reference

ERROR: AnalysisException: INSERT into table with BOOLEAN partition column (truth) is
not supported: partitioning.truth_table

Examples:

SELECT 1 < 2;
SELECT 2 = 5;
SELECT 100 < NULL, 100 > NULL;
CREATE TABLE assertions (claim STRING, really BOOLEAN);
INSERT INTO assertions VALUES
("1 is less than 2", 1 < 2),
("2 is the same as 5", 2 = 5),
("Grass is green", true),
("The moon is made of green cheese", false);
SELECT claim FROM assertions WHERE really = TRUE;

HBase considerations: This data type is fully compatible with HBase tables.
Parquet considerations: This type is fully compatible with Parquet tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Related information: Boolean Literals on page 187, SQL Operators on page 188, Impala Conditional Functions on page
391

CHAR Data Type (CDH 5.2 or higher only)

A fixed-length character type, padded with trailing spaces if necessary to achieve the specified length. If values are
longer than the specified length, Impala truncates any trailing characters.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name CHAR(length)

The maximum length you can specify is 255.

Semantics of trailing spaces:
• When you store a CHAR value shorter than the specified length in a table, queries return the value padded with
trailing spaces if necessary; the resulting value has the same length as specified in the column definition.
• If you store a CHAR value containing trailing spaces in a table, those trailing spaces are not stored in the data file.
When the value is retrieved by a query, the result could have a different number of trailing spaces. That is, the
value includes however many spaces are needed to pad it to the specified length of the column.
• If you compare two CHAR values that differ only in the number of trailing spaces, those values are considered
identical.
Partitioning: This type can be used for partition key columns. Because of the efficiency advantage of numeric values
over character-based values, if the partition key is a string representation of a number, prefer to use an integer type
with sufficient range (INT, BIGINT, and so on) where practical.
HBase considerations: This data type cannot be used with HBase tables.
Parquet considerations:
• This type can be read from and written to Parquet files.
• There is no requirement for a particular level of Parquet.
• Parquet files generated by Impala and containing this type can be freely interchanged with other components
such as Hive and MapReduce.

122 | Apache Impala Guide

 Impala SQL Language Reference

• Any trailing spaces, whether implicitly or explicitly specified, are not written to the Parquet data files.
• Parquet data files might contain values that are longer than allowed by the CHAR(n) length limit. Impala ignores
any extra trailing characters when it processes those values during a query.
Text table considerations:
Text data files might contain values that are longer than allowed for a particular CHAR(n) column. Any extra trailing
characters are ignored when Impala processes those values during a query. Text data files can also contain values that
are shorter than the defined length limit, and Impala pads them with trailing spaces up to the specified length. Any
text data files produced by Impala INSERT statements do not include any trailing blanks for CHAR columns.
Avro considerations:
The Avro specification allows string values up to 2**64 bytes in length. Impala queries for Avro tables use 32-bit integers
to hold string lengths. In CDH 5.7 / Impala 2.5 and higher, Impala truncates CHAR and VARCHAR values in Avro tables
to (2**31)-1 bytes. If a query encounters a STRING value longer than (2**31)-1 bytes in an Avro table, the query fails.
In earlier releases, encountering such long values in an Avro table could cause a crash.
Compatibility:
This type is available using Impala 2.0 or higher under CDH 4, or with Impala on CDH 5.2 or higher. There are no
compatibility issues with other components when exchanging data files or running Impala on CDH 4.
Some other database systems make the length specification optional. For Impala, the length is required.
Internal details: Represented in memory as a byte array with the same size as the length specification. Values that are
shorter than the specified length are padded on the right with trailing spaces.
Added in: CDH 5.2.0 (Impala 2.0.0)
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
UDF considerations: This type cannot be used for the argument or return type of a user-defined function (UDF) or
user-defined aggregate function (UDA).
Examples:
These examples show how trailing spaces are not considered significant when comparing or processing CHAR values.
CAST() truncates any longer string to fit within the defined length. If a CHAR value is shorter than the specified length,
it is padded on the right with spaces until it matches the specified length. Therefore, LENGTH() represents the length
including any trailing spaces, and CONCAT() also treats the column value as if it has trailing spaces.

select cast('x' as char(4)) = cast('x ' as char(4)) as "unpadded equal to padded";

+--------------------------+
| unpadded equal to padded |
+--------------------------+
| true |
+--------------------------+

create table char_length(c char(3));

insert into char_length values (cast('1' as char(3))), (cast('12' as char(3))),
(cast('123' as char(3))), (cast('123456' as char(3)));
select concat("[",c,"]") as c, length(c) from char_length;
+-------+-----------+
| c | length(c) |
+-------+-----------+
| [1 ] | 3 |
| [12 ] | 3 |
| [123] | 3 |
| [123] | 3 |
+-------+-----------+

Apache Impala Guide | 123

Impala SQL Language Reference

This example shows a case where data values are known to have a specific length, where CHAR is a logical data type
to use.

create table addresses

(id bigint,
street_name string,
state_abbreviation char(2),
country_abbreviation char(2));

The following example shows how values written by Impala do not physically include the trailing spaces. It creates a
table using text format, with CHAR values much shorter than the declared length, and then prints the resulting data
file to show that the delimited values are not separated by spaces. The same behavior applies to binary-format Parquet
data files.

create table char_in_text (a char(20), b char(30), c char(40))

row format delimited fields terminated by ',';

insert into char_in_text values (cast('foo' as char(20)), cast('bar' as char(30)),

cast('baz' as char(40))), (cast('hello' as char(20)), cast('goodbye' as char(30)),
cast('aloha' as char(40)));

-- Running this Linux command inside impala-shell using the ! shortcut.

!hdfs dfs -cat
'hdfs://127.0.0.1:8020/user/hive/warehouse/impala_doc_testing.db/char_in_text/*.*';
foo,bar,baz
hello,goodbye,aloha

The following example further illustrates the treatment of spaces. It replaces the contents of the previous table with
some values including leading spaces, trailing spaces, or both. Any leading spaces are preserved within the data file,
but trailing spaces are discarded. Then when the values are retrieved by a query, the leading spaces are retrieved
verbatim while any necessary trailing spaces are supplied by Impala.

insert overwrite char_in_text values (cast('trailing ' as char(20)), cast(' leading

and trailing ' as char(30)), cast(' leading' as char(40)));
!hdfs dfs -cat
'hdfs://127.0.0.1:8020/user/hive/warehouse/impala_doc_testing.db/char_in_text/*.*';
trailing, leading and trailing, leading

select concat('[',a,']') as a, concat('[',b,']') as b, concat('[',c,']') as c from

char_in_text;
+------------------------+----------------------------------+--------------------------------------------+
| a | b | c
|
+------------------------+----------------------------------+--------------------------------------------+
| [trailing ] | [ leading and trailing ] | [ leading
] |
+------------------------+----------------------------------+--------------------------------------------+

Restrictions:
Because the blank-padding behavior requires allocating the maximum length for each value in memory, for scalability
reasons avoid declaring CHAR columns that are much longer than typical values in that column.
All data in CHAR and VARCHAR columns must be in a character encoding that is compatible with UTF-8. If you have
binary data from another database system (that is, a BLOB type), use a STRING column to hold it.
When an expression compares a CHAR with a STRING or VARCHAR, the CHAR value is implicitly converted to STRING
first, with trailing spaces preserved.

select cast("foo " as char(5)) = 'foo' as "char equal to string";

+----------------------+
| char equal to string |
+----------------------+
| false |
+----------------------+

124 | Apache Impala Guide

 Impala SQL Language Reference

This behavior differs from other popular database systems. To get the expected result of TRUE, cast the expressions
on both sides to CHAR values of the appropriate length:

select cast("foo " as char(5)) = cast('foo' as char(3)) as "char equal to string";

+----------------------+
| char equal to string |
+----------------------+
| true |
+----------------------+

This behavior is subject to change in future releases.

Related information:
STRING Data Type on page 141, VARCHAR Data Type (CDH 5.2 or higher only) on page 154, String Literals on page 185,
Impala String Functions on page 394

DECIMAL Data Type (CDH 5.1 or higher only)

A numeric data type with fixed scale and precision, used in CREATE TABLE and ALTER TABLE statements. Suitable
for financial and other arithmetic calculations where the imprecise representation and rounding behavior of FLOAT
and DOUBLE make those types impractical.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name DECIMAL[(precision[,scale])]

DECIMAL with no precision or scale values is equivalent to DECIMAL(9,0).

Precision and Scale:

precision represents the total number of digits that can be represented by the column, regardless of the location of
the decimal point. This value must be between 1 and 38. For example, representing integer values up to 9999, and
floating-point values up to 99.99, both require a precision of 4. You can also represent corresponding negative values,
without any change in the precision. For example, the range -9999 to 9999 still only requires a precision of 4.
scale represents the number of fractional digits. This value must be less than or equal to precision. A scale of 0 produces
integral values, with no fractional part. If precision and scale are equal, all the digits come after the decimal point,
making all the values between 0 and 0.999... or 0 and -0.999...
When precision and scale are omitted, a DECIMAL value is treated as DECIMAL(9,0), that is, an integer value ranging
from -999,999,999 to 999,999,999. This is the largest DECIMAL value that can still be represented in 4 bytes. If
precision is specified but scale is omitted, Impala uses a value of zero for the scale.
Both precision and scale must be specified as integer literals, not any other kind of constant expressions.
To check the precision or scale for arbitrary values, you can call the precision() and scale() built-in functions.
For example, you might use these values to figure out how many characters are required for various fields in a report,
or to understand the rounding characteristics of a formula as applied to a particular DECIMAL column.
Range:
The maximum precision value is 38. Thus, the largest integral value is represented by DECIMAL(38,0) (999... with 9
repeated 38 times). The most precise fractional value (between 0 and 1, or 0 and -1) is represented by DECIMAL(38,38),
with 38 digits to the right of the decimal point. The value closest to 0 would be .0000...1 (37 zeros and the final 1). The
value closest to 1 would be .999... (9 repeated 38 times).
For a given precision and scale, the range of DECIMAL values is the same in the positive and negative directions. For
example, DECIMAL(4,2) can represent from -99.99 to 99.99. This is different from other integral numeric types where
the positive and negative bounds differ slightly.
When you use DECIMAL values in arithmetic expressions, the precision and scale of the result value are determined
as follows:

Apache Impala Guide | 125

Impala SQL Language Reference

• For addition and subtraction, the precision and scale are based on the maximum possible result, that is, if all the
digits of the input values were 9s and the absolute values were added together.
• For multiplication, the precision is the sum of the precisions of the input values. The scale is the sum of the scales
of the input values.
• For division, Impala sets the precision and scale to values large enough to represent the whole and fractional parts
of the result.
• For UNION, the scale is the larger of the scales of the input values, and the precision is increased if necessary to
accommodate any additional fractional digits. If the same input value has the largest precision and the largest
scale, the result value has the same precision and scale. If one value has a larger precision but smaller scale, the
scale of the result value is increased. For example, DECIMAL(20,2) UNION DECIMAL(8,6) produces a result
of type DECIMAL(24,6). The extra 4 fractional digits of scale (6-2) are accommodated by extending the precision
by the same amount (20+4).
• To doublecheck, you can always call the PRECISION() and SCALE() functions on the results of an arithmetic
expression to see the relevant values, or use a CREATE TABLE AS SELECT statement to define a column based
on the return type of the expression.

Compatibility:
• Using the DECIMAL type is only supported under CDH 5.1.0 and higher.
• Use the DECIMAL data type in Impala for applications where you used the NUMBER data type in Oracle. The Impala
DECIMAL type does not support the Oracle idioms of * for scale or negative values for precision.

Conversions and casting:

Casting an integer or floating-point value N to TIMESTAMP produces a value that is N seconds past the start of the
epoch date (January 1, 1970). By default, the result value represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Impala automatically converts between DECIMAL and other numeric types where possible. A DECIMAL with zero scale
is converted to or from the smallest appropriate integral type. A DECIMAL with a fractional part is automatically
converted to or from the smallest appropriate floating-point type. If the destination type does not have sufficient
precision or scale to hold all possible values of the source type, Impala raises an error and does not convert the value.
For example, these statements show how expressions of DECIMAL and other types are reconciled to the same type in
the context of UNION queries and INSERT statements:

[localhost:21000] > select cast(1 as int) as x union select cast(1.5 as decimal(9,4))

as x;
+----------------+
| x |
+----------------+
| 1.5000 |
| 1.0000 |
+----------------+
[localhost:21000] > create table int_vs_decimal as select cast(1 as int) as x union
select cast(1.5 as decimal(9,4)) as x;
+-------------------+
| summary |
+-------------------+
| Inserted 2 row(s) |
+-------------------+
[localhost:21000] > desc int_vs_decimal;
+------+---------------+---------+
| name | type | comment |
+------+---------------+---------+
| x | decimal(14,4) | |
+------+---------------+---------+

126 | Apache Impala Guide

 Impala SQL Language Reference

To avoid potential conversion errors, you can use CAST() to convert DECIMAL values to FLOAT, TINYINT, SMALLINT,
INT, BIGINT, STRING, TIMESTAMP, or BOOLEAN. You can use exponential notation in DECIMAL literals or when casting
from STRING, for example 1.0e6 to represent one million.
If you cast a value with more fractional digits than the scale of the destination type, any extra fractional digits are
truncated (not rounded). Casting a value to a target type with not enough precision produces a result of NULL and
displays a runtime warning.

[localhost:21000] > select cast(1.239 as decimal(3,2));

+-----------------------------+
| cast(1.239 as decimal(3,2)) |
+-----------------------------+
| 1.23 |
+-----------------------------+
[localhost:21000] > select cast(1234 as decimal(3));
+----------------------------+
| cast(1234 as decimal(3,0)) |
+----------------------------+
| NULL |
+----------------------------+
WARNINGS: Expression overflowed, returning NULL

When you specify integer literals, for example in INSERT ... VALUES statements or arithmetic expressions, those
numbers are interpreted as the smallest applicable integer type. You must use CAST() calls for some combinations
of integer literals and DECIMAL precision. For example, INT has a maximum value that is 10 digits long, TINYINT has
a maximum value that is 3 digits long, and so on. If you specify a value such as 123456 to go into a DECIMAL column,
Impala checks if the column has enough precision to represent the largest value of that integer type, and raises an
error if not. Therefore, use an expression like CAST(123456 TO DECIMAL(9,0)) for DECIMAL columns with precision
9 or less, CAST(50 TO DECIMAL(2,0)) for DECIMAL columns with precision 2 or less, and so on. For DECIMAL
columns with precision 10 or greater, Impala automatically interprets the value as the correct DECIMAL type; however,
because DECIMAL(10) requires 8 bytes of storage while DECIMAL(9) requires only 4 bytes, only use precision of 10
or higher when actually needed.

[localhost:21000] > create table decimals_9_0 (x decimal);

[localhost:21000] > insert into decimals_9_0 values (1), (2), (4), (8), (16), (1024),
(32768), (65536), (1000000);
ERROR: AnalysisException: Possible loss of precision for target table
'decimal_testing.decimals_9_0'.
Expression '1' (type: INT) would need to be cast to DECIMAL(9,0) for column 'x'
[localhost:21000] > insert into decimals_9_0 values (cast(1 as decimal)), (cast(2 as
decimal)), (cast(4 as decimal)), (cast(8 as decimal)), (cast(16 as decimal)), (cast(1024
as decimal)), (cast(32768 as decimal)), (cast(65536 as decimal)), (cast(1000000 as
decimal));

[localhost:21000] > create table decimals_10_0 (x decimal(10,0));

[localhost:21000] > insert into decimals_10_0 values (1), (2), (4), (8), (16), (1024),
(32768), (65536), (1000000);

Be aware that in memory and for binary file formats such as Parquet or Avro, DECIMAL(10) or higher consumes 8
bytes while DECIMAL(9) (the default for DECIMAL) or lower consumes 4 bytes. Therefore, to conserve space in large
tables, use the smallest-precision DECIMAL type that is appropriate and CAST() literal values where necessary, rather
than declaring DECIMAL columns with high precision for convenience.
To represent a very large or precise DECIMAL value as a literal, for example one that contains more digits than can be
represented by a BIGINT literal, use a quoted string or a floating-point value for the number, and CAST() to the
desired DECIMAL type:

insert into decimals_38_5 values (1), (2), (4), (8), (16), (1024), (32768), (65536),
(1000000),
(cast("999999999999999999999999999999" as decimal(38,5))),
(cast(999999999999999999999999999999. as decimal(38,5)));

Apache Impala Guide | 127

Impala SQL Language Reference

• The result of the SUM() aggregate function on DECIMAL values is promoted to a precision of 38, with the same
precision as the underlying column. Thus, the result can represent the largest possible value at that particular
precision.
• STRING columns, literals, or expressions can be converted to DECIMAL as long as the overall number of digits and
digits to the right of the decimal point fit within the specified precision and scale for the declared DECIMAL type.
By default, a DECIMAL value with no specified scale or precision can hold a maximum of 9 digits of an integer
value. If there are more digits in the string value than are allowed by the DECIMAL scale and precision, the result
is NULL.
The following examples demonstrate how STRING values with integer and fractional parts are represented when
converted to DECIMAL. If the scale is 0, the number is treated as an integer value with a maximum of precision
digits. If the precision is greater than 0, the scale must be increased to account for the digits both to the left and
right of the decimal point. As the precision increases, output values are printed with additional trailing zeros after
the decimal point if needed. Any trailing zeros after the decimal point in the STRING value must fit within the
number of digits specified by the precision.

[localhost:21000] > select cast('100' as decimal); -- Small integer value fits within
9 digits of scale.
+-----------------------------+
| cast('100' as decimal(9,0)) |
+-----------------------------+
| 100 |
+-----------------------------+
[localhost:21000] > select cast('100' as decimal(3,0)); -- Small integer value fits
within 3 digits of scale.
+-----------------------------+
| cast('100' as decimal(3,0)) |
+-----------------------------+
| 100 |
+-----------------------------+
[localhost:21000] > select cast('100' as decimal(2,0)); -- 2 digits of scale is not
enough!
+-----------------------------+
| cast('100' as decimal(2,0)) |
+-----------------------------+
| NULL |
+-----------------------------+
[localhost:21000] > select cast('100' as decimal(3,1)); -- (3,1) = 2 digits left of the
decimal point, 1 to the right. Not enough.
+-----------------------------+
| cast('100' as decimal(3,1)) |
+-----------------------------+
| NULL |
+-----------------------------+
[localhost:21000] > select cast('100' as decimal(4,1)); -- 4 digits total, 1 to the
right of the decimal point.
+-----------------------------+
| cast('100' as decimal(4,1)) |
+-----------------------------+
| 100.0 |
+-----------------------------+
[localhost:21000] > select cast('98.6' as decimal(3,1)); -- (3,1) can hold a 3 digit
number with 1 fractional digit.
+------------------------------+
| cast('98.6' as decimal(3,1)) |
+------------------------------+
| 98.6 |
+------------------------------+
[localhost:21000] > select cast('98.6' as decimal(15,1)); -- Larger scale allows bigger
numbers but still only 1 fractional digit.
+-------------------------------+
| cast('98.6' as decimal(15,1)) |
+-------------------------------+
| 98.6 |
+-------------------------------+
[localhost:21000] > select cast('98.6' as decimal(15,5)); -- Larger precision allows
more fractional digits, outputs trailing zeros.
+-------------------------------+
| cast('98.6' as decimal(15,5)) |

128 | Apache Impala Guide

 Impala SQL Language Reference

+-------------------------------+
| 98.60000 |
+-------------------------------+
[localhost:21000] > select cast('98.60000' as decimal(15,1)); -- Trailing zeros in the
string must fit within 'scale' digits (1 in this case).
+-----------------------------------+
| cast('98.60000' as decimal(15,1)) |
+-----------------------------------+
| NULL |
+-----------------------------------+

• Most built-in arithmetic functions such as SIN() and COS() continue to accept only DOUBLE values because they
are so commonly used in scientific context for calculations of IEEE 754-compliant values. The built-in functions
that accept and return DECIMAL are:
– ABS()
– CEIL()
– COALESCE()
– FLOOR()
– FNV_HASH()
– GREATEST()
– IF()
– ISNULL()
– LEAST()
– NEGATIVE()
– NULLIF()
– POSITIVE()
– PRECISION()
– ROUND()
– SCALE()
– TRUNCATE()
– ZEROIFNULL()

See Impala Built-In Functions on page 338 for details.

• BIGINT, INT, SMALLINT, and TINYINT values can all be cast to DECIMAL. The number of digits to the left of the
decimal point in the DECIMAL type must be sufficient to hold the largest value of the corresponding integer type.
Note that integer literals are treated as the smallest appropriate integer type, meaning there is sometimes a range
of values that require one more digit of DECIMAL scale than you might expect. For integer values, the precision
of the DECIMAL type can be zero; if the precision is greater than zero, remember to increase the scale value by
an equivalent amount to hold the required number of digits to the left of the decimal point.
The following examples show how different integer types are converted to DECIMAL.

[localhost:21000] > select cast(1 as decimal(1,0));

+-------------------------+
| cast(1 as decimal(1,0)) |
+-------------------------+
| 1 |
+-------------------------+
[localhost:21000] > select cast(9 as decimal(1,0));
+-------------------------+
| cast(9 as decimal(1,0)) |
+-------------------------+
| 9 |
+-------------------------+
[localhost:21000] > select cast(10 as decimal(1,0));
+--------------------------+
| cast(10 as decimal(1,0)) |
+--------------------------+
| 10 |

Apache Impala Guide | 129

Impala SQL Language Reference

+--------------------------+
[localhost:21000] > select cast(10 as decimal(1,1));
+--------------------------+
| cast(10 as decimal(1,1)) |
+--------------------------+
| 10.0 |
+--------------------------+
[localhost:21000] > select cast(100 as decimal(1,1));
+---------------------------+
| cast(100 as decimal(1,1)) |
+---------------------------+
| 100.0 |
+---------------------------+
[localhost:21000] > select cast(1000 as decimal(1,1));
+----------------------------+
| cast(1000 as decimal(1,1)) |
+----------------------------+
| 1000.0 |
+----------------------------+

• When a DECIMAL value is converted to any of the integer types, any fractional part is truncated (that is, rounded
towards zero):

[localhost:21000] > create table num_dec_days (x decimal(4,1));

[localhost:21000] > insert into num_dec_days values (1), (2), (cast(4.5 as decimal(4,1)));
[localhost:21000] > insert into num_dec_days values (cast(0.1 as decimal(4,1))), (cast(.9
as decimal(4,1))), (cast(9.1 as decimal(4,1))), (cast(9.9 as decimal(4,1)));
[localhost:21000] > select cast(x as int) from num_dec_days;
+----------------+
| cast(x as int) |
+----------------+
| 1 |
| 2 |
| 4 |
| 0 |
| 0 |
| 9 |
| 9 |
+----------------+

• You cannot directly cast TIMESTAMP or BOOLEAN values to or from DECIMAL values. You can turn a DECIMAL
value into a time-related representation using a two-step process, by converting it to an integer value and then
using that result in a call to a date and time function such as from_unixtime().

[localhost:21000] > select from_unixtime(cast(cast(1000.0 as decimal) as bigint));

+-------------------------------------------------------------+
| from_unixtime(cast(cast(1000.0 as decimal(9,0)) as bigint)) |
+-------------------------------------------------------------+
| 1970-01-01 00:16:40 |
+-------------------------------------------------------------+
[localhost:21000] > select now() + interval cast(x as int) days from num_dec_days; --
x is a DECIMAL column.

[localhost:21000] > create table num_dec_days (x decimal(4,1));

[localhost:21000] > insert into num_dec_days values (1), (2), (cast(4.5 as decimal(4,1)));
[localhost:21000] > select now() + interval cast(x as int) days from num_dec_days; --
The 4.5 value is truncated to 4 and becomes '4 days'.
+--------------------------------------+
| now() + interval cast(x as int) days |
+--------------------------------------+
| 2014-05-13 23:11:55.163284000 |
| 2014-05-14 23:11:55.163284000 |
| 2014-05-16 23:11:55.163284000 |
+--------------------------------------+

130 | Apache Impala Guide

 Impala SQL Language Reference

• Because values in INSERT statements are checked rigorously for type compatibility, be prepared to use CAST()
function calls around literals, column references, or other expressions that you are inserting into a DECIMAL
column.

NULL considerations: Casting any non-numeric value to this type produces a NULL value.
DECIMAL differences from integer and floating-point types:
With the DECIMAL type, you are concerned with the number of overall digits of a number rather than powers of 2 (as
in TINYINT, SMALLINT, and so on). Therefore, the limits with integral values of DECIMAL types fall around 99, 999,
9999, and so on rather than 32767, 65535, 2 32 -1, and so on. For fractional values, you do not need to account for
imprecise representation of the fractional part according to the IEEE-954 standard (as in FLOAT and DOUBLE). Therefore,
when you insert a fractional value into a DECIMAL column, you can compare, sum, query, GROUP BY, and so on that
column and get back the original values rather than some “close but not identical” value.
FLOAT and DOUBLE can cause problems or unexpected behavior due to inability to precisely represent certain fractional
values, for example dollar and cents values for currency. You might find output values slightly different than you
inserted, equality tests that do not match precisely, or unexpected values for GROUP BY columns. DECIMAL can help
reduce unexpected behavior and rounding errors, at the expense of some performance overhead for assignments and
comparisons.
Literals and expressions:
• When you use an integer literal such as 1 or 999 in a SQL statement, depending on the context, Impala will treat
it as either the smallest appropriate DECIMAL type, or the smallest integer type (TINYINT, SMALLINT, INT, or
BIGINT). To minimize memory usage, Impala prefers to treat the literal as the smallest appropriate integer type.

• When you use a floating-point literal such as 1.1 or 999.44 in a SQL statement, depending on the context, Impala
will treat it as either the smallest appropriate DECIMAL type, or the smallest floating-point type (FLOAT or DOUBLE).
To avoid loss of accuracy, Impala prefers to treat the literal as a DECIMAL.

Storage considerations:
• Only the precision determines the storage size for DECIMAL values; the scale setting has no effect on the storage
size.
• Text, RCFile, and SequenceFile tables all use ASCII-based formats. In these text-based file formats, leading zeros
are not stored, but trailing zeros are stored. In these tables, each DECIMAL value takes up as many bytes as there
are digits in the value, plus an extra byte if the decimal point is present and an extra byte for negative values.
Once the values are loaded into memory, they are represented in 4, 8, or 16 bytes as described in the following
list items. The on-disk representation varies depending on the file format of the table.
• Parquet and Avro tables use binary formats, In these tables, Impala stores each value in as few bytes as possible
depending on the precision specified for the DECIMAL column.
– In memory, DECIMAL values with precision of 9 or less are stored in 4 bytes.
– In memory, DECIMAL values with precision of 10 through 18 are stored in 8 bytes.
– In memory, DECIMAL values with precision greater than 18 are stored in 16 bytes.

File format considerations:

• The DECIMAL data type can be stored in any of the file formats supported by Impala, as described in How Impala
Works with Hadoop File Formats on page 527. Impala only writes to tables that use the Parquet and text formats,
so those formats are the focus for file format compatibility.
• Impala can query Avro, RCFile, or SequenceFile tables containing DECIMAL columns, created by other Hadoop
components, on CDH 5 only.
• You can use DECIMAL columns in Impala tables that are mapped to HBase tables. Impala can query and insert
into such tables.
• Text, RCFile, and SequenceFile tables all use ASCII-based formats. In these tables, each DECIMAL value takes up
as many bytes as there are digits in the value, plus an extra byte if the decimal point is present. The binary format
of Parquet or Avro files offers more compact storage for DECIMAL columns.

Apache Impala Guide | 131

Impala SQL Language Reference

• Parquet and Avro tables use binary formats, In these tables, Impala stores each value in 4, 8, or 16 bytes depending
on the precision specified for the DECIMAL column.
UDF considerations: When writing a C++ UDF, use the DecimalVal data type defined in
/usr/include/impala_udf/udf.h.

Partitioning:
You can use a DECIMAL column as a partition key. Doing so provides a better match between the partition key values
and the HDFS directory names than using a DOUBLE or FLOAT partitioning column:
Schema evolution considerations:
• For text-based formats (text, RCFile, and SequenceFile tables), you can issue an ALTER TABLE ... REPLACE
COLUMNS statement to change the precision and scale of an existing DECIMAL column. As long as the values in
the column fit within the new precision and scale, they are returned correctly by a query. Any values that do not
fit within the new precision and scale are returned as NULL, and Impala reports the conversion error. Leading
zeros do not count against the precision value, but trailing zeros after the decimal point do.

[localhost:21000] > create table text_decimals (x string);

[localhost:21000] > insert into text_decimals values ("1"), ("2"), ("99.99"), ("1.234"),
("000001"), ("1.000000000");
[localhost:21000] > select * from text_decimals;
+-------------+
| x |
+-------------+
| 1 |
| 2 |
| 99.99 |
| 1.234 |
| 000001 |
| 1.000000000 |
+-------------+
[localhost:21000] > alter table text_decimals replace columns (x decimal(4,2));
[localhost:21000] > select * from text_decimals;
+-------+
| x |
+-------+
| 1.00 |
| 2.00 |
| 99.99 |
| NULL |
| 1.00 |
| NULL |
+-------+
ERRORS:
Backend 0:Error converting column: 0 TO DECIMAL(4, 2) (Data is: 1.234)
file:
hdfs://127.0.0.1:8020/user/hive/warehouse/decimal_testing.db/text_decimals/634d4bd3aa0
e8420-b4b13bab7f1be787_56794587_data.0
record: 1.234
Error converting column: 0 TO DECIMAL(4, 2) (Data is: 1.000000000)
file:
hdfs://127.0.0.1:8020/user/hive/warehouse/decimal_testing.db/text_decimals/cd40dc68e20
c565a-cc4bd86c724c96ba_311873428_data.0
record: 1.000000000

• For binary formats (Parquet and Avro tables), although an ALTER TABLE ... REPLACE COLUMNS statement
that changes the precision or scale of a DECIMAL column succeeds, any subsequent attempt to query the changed
column results in a fatal error. (The other columns can still be queried successfully.) This is because the metadata
about the columns is stored in the data files themselves, and ALTER TABLE does not actually make any updates
to the data files. If the metadata in the data files disagrees with the metadata in the metastore database, Impala
cancels the query.

132 | Apache Impala Guide

 Impala SQL Language Reference

Examples:

CREATE TABLE t1 (x DECIMAL, y DECIMAL(5,2), z DECIMAL(25,0));

INSERT INTO t1 VALUES (5, 99.44, 123456), (300, 6.7, 999999999);
SELECT x+y, ROUND(y,1), z/98.6 FROM t1;
SELECT CAST(1000.5 AS DECIMAL);

Restrictions:
Currently, the COMPUTE STATS statement under CDH 4 does not store any statistics for DECIMAL columns. When
Impala runs under CDH 5, which has better support for DECIMAL in the metastore database, COMPUTE STATS does
collect statistics for DECIMAL columns and Impala uses the statistics to optimize query performance.
HBase considerations: This data type is fully compatible with HBase tables.
Parquet considerations: This type is fully compatible with Parquet tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Related information:
Numeric Literals on page 184, TINYINT Data Type on page 153, SMALLINT Data Type on page 140, INT Data Type on page
135, BIGINT Data Type on page 120, DECIMAL Data Type (CDH 5.1 or higher only) on page 125, Impala Mathematical
Functions on page 339 (especially PRECISION() and SCALE())

DOUBLE Data Type

A double precision floating-point data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name DOUBLE

Range: 4.94065645841246544e-324d .. 1.79769313486231570e+308, positive or negative

Precision: 15 to 17 significant digits, depending on usage. The number of significant digits does not depend on the
position of the decimal point.
Representation: The values are stored in 8 bytes, using IEEE 754 Double Precision Binary Floating Point format.
Conversions: Impala does not automatically convert DOUBLE to any other type. You can use CAST() to convert DOUBLE
values to FLOAT, TINYINT, SMALLINT, INT, BIGINT, STRING, TIMESTAMP, or BOOLEAN. You can use exponential
notation in DOUBLE literals or when casting from STRING, for example 1.0e6 to represent one million. Casting an
integer or floating-point value N to TIMESTAMP produces a value that is N seconds past the start of the epoch date
(January 1, 1970). By default, the result value represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Usage notes:
The data type REAL is an alias for DOUBLE.
Examples:

CREATE TABLE t1 (x DOUBLE);

SELECT CAST(1000.5 AS DOUBLE);

Apache Impala Guide | 133

Impala SQL Language Reference

Partitioning: Because fractional values of this type are not always represented precisely, when this type is used for a
partition key column, the underlying HDFS directories might not be named exactly as you expect. Prefer to partition
on a DECIMAL column instead.
HBase considerations: This data type is fully compatible with HBase tables.
Parquet considerations: This type is fully compatible with Parquet tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as an 8-byte value.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Restrictions:
Due to the way arithmetic on FLOAT and DOUBLE columns uses high-performance hardware instructions, and distributed
queries can perform these operations in different order for each query, results can vary slightly for aggregate function
calls such as SUM() and AVG() for FLOAT and DOUBLE columns, particularly on large data sets where millions or billions
of values are summed or averaged. For perfect consistency and repeatability, use the DECIMAL data type for such
operations instead of FLOAT or DOUBLE.
The inability to exactly represent certain floating-point values means that DECIMAL is sometimes a better choice than
DOUBLE or FLOAT when precision is critical, particularly when transferring data from other database systems that use
different representations or file formats.
Related information:
Numeric Literals on page 184, Impala Mathematical Functions on page 339, FLOAT Data Type on page 134

FLOAT Data Type

A single precision floating-point data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name FLOAT

Range: 1.40129846432481707e-45 .. 3.40282346638528860e+38, positive or negative

Precision: 6 to 9 significant digits, depending on usage. The number of significant digits does not depend on the position
of the decimal point.
Representation: The values are stored in 4 bytes, using IEEE 754 Single Precision Binary Floating Point format.
Conversions: Impala automatically converts FLOAT to more precise DOUBLE values, but not the other way around.
You can use CAST() to convert FLOAT values to TINYINT, SMALLINT, INT, BIGINT, STRING, TIMESTAMP, or BOOLEAN.
You can use exponential notation in FLOAT literals or when casting from STRING, for example 1.0e6 to represent one
million. Casting an integer or floating-point value N to TIMESTAMP produces a value that is N seconds past the start of
the epoch date (January 1, 1970). By default, the result value represents a date and time in the UTC time zone. If the
setting --use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP
represents a date and time in the local time zone.
Examples:

CREATE TABLE t1 (x FLOAT);

SELECT CAST(1000.5 AS FLOAT);

Partitioning: Because fractional values of this type are not always represented precisely, when this type is used for a
partition key column, the underlying HDFS directories might not be named exactly as you expect. Prefer to partition
on a DECIMAL column instead.

134 | Apache Impala Guide

 Impala SQL Language Reference

HBase considerations: This data type is fully compatible with HBase tables.
Parquet considerations: This type is fully compatible with Parquet tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as a 4-byte value.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Restrictions:
Due to the way arithmetic on FLOAT and DOUBLE columns uses high-performance hardware instructions, and distributed
queries can perform these operations in different order for each query, results can vary slightly for aggregate function
calls such as SUM() and AVG() for FLOAT and DOUBLE columns, particularly on large data sets where millions or billions
of values are summed or averaged. For perfect consistency and repeatability, use the DECIMAL data type for such
operations instead of FLOAT or DOUBLE.
The inability to exactly represent certain floating-point values means that DECIMAL is sometimes a better choice than
DOUBLE or FLOAT when precision is critical, particularly when transferring data from other database systems that use
different representations or file formats.
Related information:
Numeric Literals on page 184, Impala Mathematical Functions on page 339, DOUBLE Data Type on page 133

INT Data Type

A 4-byte integer data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name INT

Range: -2147483648 .. 2147483647. There is no UNSIGNED subtype.

Conversions: Impala automatically converts to a larger integer type (BIGINT) or a floating-point type (FLOAT or DOUBLE)
automatically. Use CAST() to convert to TINYINT, SMALLINT, STRING, or TIMESTAMP. Casting an integer or
floating-point value N to TIMESTAMP produces a value that is N seconds past the start of the epoch date (January 1,
1970). By default, the result value represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Usage notes:
The data type INTEGER is an alias for INT.
For a convenient and automated way to check the bounds of the INT type, call the functions MIN_INT() and
MAX_INT().

If an integer value is too large to be represented as a INT, use a BIGINT instead.

NULL considerations: Casting any non-numeric value to this type produces a NULL value.
Examples:

CREATE TABLE t1 (x INT);

SELECT CAST(1000 AS INT);

Partitioning: Prefer to use this type for a partition key column. Impala can process the numeric type more efficiently
than a STRING representation of the value.
HBase considerations: This data type is fully compatible with HBase tables.

Apache Impala Guide | 135

Impala SQL Language Reference

Parquet considerations:
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as a 4-byte value.
Added in: Available in all versions of Impala.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Related information:
Numeric Literals on page 184, TINYINT Data Type on page 153, SMALLINT Data Type on page 140, INT Data Type on page
135, BIGINT Data Type on page 120, DECIMAL Data Type (CDH 5.1 or higher only) on page 125, Impala Mathematical
Functions on page 339

MAP Complex Type (CDH 5.5 or higher only)

A complex data type representing an arbitrary set of key-value pairs. The key part is a scalar type, while the value part
can be a scalar or another complex type (ARRAY, STRUCT, or MAP).
Syntax:

column_name MAP < primitive_type, type >

type ::= primitive_type | complex_type

Usage notes:
Because complex types are often used in combination, for example an ARRAY of STRUCT elements, if you are unfamiliar
with the Impala complex types, start with Complex Types (CDH 5.5 or higher only) on page 156 for background information
and usage examples.
The MAP complex data type represents a set of key-value pairs. Each element of the map is indexed by a primitive type
such as BIGINT or STRING, letting you define sequences that are not continuous or categories with arbitrary names.
You might find it convenient for modelling data produced in other languages, such as a Python dictionary or Java
HashMap, where a single scalar value serves as the lookup key.
In a big data context, the keys in a map column might represent a numeric sequence of events during a manufacturing
process, or TIMESTAMP values corresponding to sensor observations. The map itself is inherently unordered, so you
choose whether to make the key values significant (such as a recorded TIMESTAMP) or synthetic (such as a random
global universal ID).

Note: Behind the scenes, the MAP type is implemented in a similar way as the ARRAY type. Impala
does not enforce any uniqueness constraint on the KEY values, and the KEY values are processed by
looping through the elements of the MAP rather than by a constant-time lookup. Therefore, this type
is primarily for ease of understanding when importing data and algorithms from non-SQL contexts,
rather than optimizing the performance of key lookups.

You can pass a qualified name to DESCRIBE to specify an ARRAY, STRUCT, or MAP column and visualize its structure
as if it were a table. An ARRAY is shown as a two-column table, with ITEM and POS columns. A STRUCT is shown as a
table with each field representing a column in the table. A MAP is shown as a two-column table, with KEY and VALUE
columns.
Added in: CDH 5.5.0 (Impala 2.3.0)
Restrictions:
• Columns with this data type can only be used in tables or partitions with the Parquet file format.
• Columns with this data type cannot be used as partition key columns in a partitioned table.

136 | Apache Impala Guide

 Impala SQL Language Reference

• The COMPUTE STATS statement does not produce any statistics for columns of this data type.
• The maximum length of the column definition for any complex type, including declarations for any nested types,
is 4000 characters.
• See Limitations and Restrictions for Complex Types on page 161 for a full list of limitations and associated guidelines
about complex type columns.

Examples:

Note: Many of the complex type examples refer to tables such as CUSTOMER and REGION adapted
from the tables used in the TPC-H benchmark. See Sample Schema and Data for Experimenting with
Impala Complex Types on page 178 for the table definitions.

The following example shows a table with various kinds of MAP columns, both at the top level and nested within other
complex types. Each row represents information about a specific country, with complex type fields of various levels
of nesting to represent different information associated with the country: factual measurements such as area and
population, notable people in different categories, geographic features such as cities, points of interest within each
city, and mountains with associated facts. Practice the CREATE TABLE and query notation for complex type columns
using empty tables, until you can visualize a complex data structure and construct corresponding SQL statements
reliably.

create TABLE map_demo

(
country_id BIGINT,

-- Numeric facts about each country, looked up by name.

-- For example, 'Area':1000, 'Population':999999.
-- Using a MAP instead of a STRUCT because there could be
-- a different set of facts for each country.
metrics MAP <STRING, BIGINT>,

-- MAP whose value part is an ARRAY.

-- For example, the key 'Famous Politicians' could represent an array of 10 elements,
-- while the key 'Famous Actors' could represent an array of 20 elements.
notables MAP <STRING, ARRAY <STRING>>,

-- MAP that is a field within a STRUCT.

-- (The STRUCT is inside another ARRAY, because it is rare
-- for a STRUCT to be a top-level column.)
-- For example, city #1 might have points of interest with key 'Zoo',
-- representing an array of 3 different zoos.
-- City #2 might have completely different kinds of points of interest.
-- Because the set of field names is potentially large, and most entries could be blank,
-- a MAP makes more sense than a STRUCT to represent such a sparse data structure.
cities ARRAY < STRUCT <
name: STRING,
points_of_interest: MAP <STRING, ARRAY <STRING>>
>>,

-- MAP that is an element within an ARRAY. The MAP is inside a STRUCT field to associate
-- the mountain name with all the facts about the mountain.
-- The "key" of the map (the first STRING field) represents the name of some fact whose
value
-- can be expressed as an integer, such as 'Height', 'Year First Climbed', and so on.
mountains ARRAY < STRUCT < name: STRING, facts: MAP <STRING, INT > > >
)
STORED AS PARQUET;

DESCRIBE map_demo;
+------------+------------------------------------------------+
| name | type |
+------------+------------------------------------------------+
| country_id | bigint |
| metrics | map<string,bigint> |

Apache Impala Guide | 137

Impala SQL Language Reference

| notables | map<string,array<string>> |
| cities | array<struct< |
| | name:string, |
| | points_of_interest:map<string,array<string>> |
| | >> |
| mountains | array<struct< |
| | name:string, |
| | facts:map<string,int> |
| | >> |
+------------+------------------------------------------------+

DESCRIBE map_demo.metrics;
+-------+--------+
| name | type |
+-------+--------+
| key | string |
| value | bigint |
+-------+--------+

DESCRIBE map_demo.notables;
+-------+---------------+
| name | type |
+-------+---------------+
| key | string |
| value | array<string> |
+-------+---------------+

DESCRIBE map_demo.notables.value;
+------+--------+
| name | type |
+------+--------+
| item | string |
| pos | bigint |
+------+--------+

DESCRIBE map_demo.cities;
+------+------------------------------------------------+
| name | type |
+------+------------------------------------------------+
| item | struct< |
| | name:string, |
| | points_of_interest:map<string,array<string>> |
| | > |
| pos | bigint |
+------+------------------------------------------------+

DESCRIBE map_demo.cities.item.points_of_interest;
+-------+---------------+
| name | type |
+-------+---------------+
| key | string |
| value | array<string> |
+-------+---------------+

DESCRIBE map_demo.cities.item.points_of_interest.value;
+------+--------+
| name | type |
+------+--------+
| item | string |
| pos | bigint |
+------+--------+

DESCRIBE map_demo.mountains;
+------+-------------------------+
| name | type |
+------+-------------------------+
| item | struct< |
| | name:string, |
| | facts:map<string,int> |
| | > |
| pos | bigint |
+------+-------------------------+

138 | Apache Impala Guide

 Impala SQL Language Reference

DESCRIBE map_demo.mountains.item.facts;
+-------+--------+
| name | type |
+-------+--------+
| key | string |
| value | int |
+-------+--------+

The following example shows a table that uses a variety of data types for the MAP “key” field. Typically, you use BIGINT
or STRING to use numeric or character-based keys without worrying about exceeding any size or length constraints.

CREATE TABLE map_demo_obscure

(
id BIGINT,
m1 MAP <INT, INT>,
m2 MAP <SMALLINT, INT>,
m3 MAP <TINYINT, INT>,
m4 MAP <TIMESTAMP, INT>,
m5 MAP <BOOLEAN, INT>,
m6 MAP <CHAR(5), INT>,
m7 MAP <VARCHAR(25), INT>,
m8 MAP <FLOAT, INT>,
m9 MAP <DOUBLE, INT>,
m10 MAP <DECIMAL(12,2), INT>
)
STORED AS PARQUET;

CREATE TABLE celebrities (name STRING, birth_year MAP < STRING, SMALLINT >) STORED AS
PARQUET;
-- A typical row might represent values with 2 different birth years, such as:
-- ("Joe Movie Star", { "real": 1972, "claimed": 1977 })

CREATE TABLE countries (name STRING, famous_leaders MAP < INT, STRING >) STORED AS
PARQUET;
-- A typical row might represent values with different leaders, with key values
corresponding to their numeric sequence, such as:
-- ("United States", { 1: "George Washington", 3: "Thomas Jefferson", 16: "Abraham
Lincoln" })

CREATE TABLE airlines (name STRING, special_meals MAP < STRING, MAP < STRING, STRING >
>) STORED AS PARQUET;
-- A typical row might represent values with multiple kinds of meals, each with several
components:
-- ("Elegant Airlines",
-- {
-- "vegetarian": { "breakfast": "pancakes", "snack": "cookies", "dinner": "rice
pilaf" },
-- "gluten free": { "breakfast": "oatmeal", "snack": "fruit", "dinner": "chicken"
}
-- } )

Related information:
Complex Types (CDH 5.5 or higher only) on page 156, ARRAY Complex Type (CDH 5.5 or higher only) on page 117, STRUCT
Complex Type (CDH 5.5 or higher only) on page 142

REAL Data Type

An alias for the DOUBLE data type. See DOUBLE Data Type on page 133 for details.
Examples:

Apache Impala Guide | 139

Impala SQL Language Reference

These examples show how you can use the type names REAL and DOUBLE interchangeably, and behind the scenes
Impala treats them always as DOUBLE.

[localhost:21000] > create table r1 (x real);

[localhost:21000] > describe r1;
+------+--------+---------+
| name | type | comment |
+------+--------+---------+
| x | double | |
+------+--------+---------+
[localhost:21000] > insert into r1 values (1.5), (cast (2.2 as double));
[localhost:21000] > select cast (1e6 as real);
+---------------------------+
| cast(1000000.0 as double) |
+---------------------------+
| 1000000 |
+---------------------------+

SMALLINT Data Type

A 2-byte integer data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name SMALLINT

Range: -32768 .. 32767. There is no UNSIGNED subtype.

Conversions: Impala automatically converts to a larger integer type (INT or BIGINT) or a floating-point type (FLOAT
or DOUBLE) automatically. Use CAST() to convert to TINYINT, STRING, or TIMESTAMP. Casting an integer or
floating-point value N to TIMESTAMP produces a value that is N seconds past the start of the epoch date (January 1,
1970). By default, the result value represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Usage notes:
For a convenient and automated way to check the bounds of the SMALLINT type, call the functions MIN_SMALLINT()
and MAX_SMALLINT().
If an integer value is too large to be represented as a SMALLINT, use an INT instead.
NULL considerations: Casting any non-numeric value to this type produces a NULL value.
Examples:

CREATE TABLE t1 (x SMALLINT);

SELECT CAST(1000 AS SMALLINT);

Parquet considerations:
Physically, Parquet files represent TINYINT and SMALLINT values as 32-bit integers. Although Impala rejects attempts
to insert out-of-range values into such columns, if you create a new table with the CREATE TABLE ... LIKE PARQUET
syntax, any TINYINT or SMALLINT columns in the original table turn into INT columns in the new table.
Partitioning: Prefer to use this type for a partition key column. Impala can process the numeric type more efficiently
than a STRING representation of the value.
HBase considerations: This data type is fully compatible with HBase tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as a 2-byte value.

140 | Apache Impala Guide

 Impala SQL Language Reference

Added in: Available in all versions of Impala.

Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Related information:
Numeric Literals on page 184, TINYINT Data Type on page 153, SMALLINT Data Type on page 140, INT Data Type on page
135, BIGINT Data Type on page 120, DECIMAL Data Type (CDH 5.1 or higher only) on page 125, Impala Mathematical
Functions on page 339

STRING Data Type

A data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name STRING

Length: Maximum of 32,767 bytes. Do not use any length constraint when declaring STRING columns, as you might
be familiar with from VARCHAR, CHAR, or similar column types from relational database systems. If you do need to
manipulate string values with precise or maximum lengths, in Impala 2.0 and higher you can declare columns as
VARCHAR(max_length) or CHAR(length), but for best performance use STRING where practical.

Character sets: For full support in all Impala subsystems, restrict string values to the ASCII character set. Although
some UTF-8 character data can be stored in Impala and retrieved through queries, UTF-8 strings containing non-ASCII
characters are not guaranteed to work properly in combination with many SQL aspects, including but not limited to:
• String manipulation functions.
• Comparison operators.
• The ORDER BY clause.
• Values in partition key columns.
For any national language aspects such as collation order or interpreting extended ASCII variants such as ISO-8859-1
or ISO-8859-2 encodings, Impala does not include such metadata with the table definition. If you need to sort, manipulate,
or display data depending on those national language characteristics of string data, use logic on the application side.
Conversions:
• Impala does not automatically convert STRING to any numeric type. Impala does automatically convert STRING
to TIMESTAMP if the value matches one of the accepted TIMESTAMP formats; see TIMESTAMP Data Type on page
148 for details.
• You can use CAST() to convert STRING values to TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, or
TIMESTAMP.

• You cannot directly cast a STRING value to BOOLEAN. You can use a CASE expression to evaluate string values
such as 'T', 'true', and so on and return Boolean true and false values as appropriate.
• You can cast a BOOLEAN value to STRING, returning '1' for true values and '0' for false values.

Partitioning:
Although it might be convenient to use STRING columns for partition keys, even when those columns contain numbers,
for performance and scalability it is much better to use numeric columns as partition keys whenever practical. Although
the underlying HDFS directory name might be the same in either case, the in-memory storage for the partition key
columns is more compact, and computations are faster, if partition key columns such as YEAR, MONTH, DAY and so on
are declared as INT, SMALLINT, and so on.
Zero-length strings: For purposes of clauses such as DISTINCT and GROUP BY, Impala considers zero-length strings
(""), NULL, and space to all be different values.

Apache Impala Guide | 141

Impala SQL Language Reference

Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Avro considerations:
The Avro specification allows string values up to 2**64 bytes in length. Impala queries for Avro tables use 32-bit integers
to hold string lengths. In CDH 5.7 / Impala 2.5 and higher, Impala truncates CHAR and VARCHAR values in Avro tables
to (2**31)-1 bytes. If a query encounters a STRING value longer than (2**31)-1 bytes in an Avro table, the query fails.
In earlier releases, encountering such long values in an Avro table could cause a crash.
Column statistics considerations: Because the values of this type have variable size, none of the column statistics fields
are filled in until you run the COMPUTE STATS statement.
Examples:
The following examples demonstrate double-quoted and single-quoted string literals, and required escaping for
quotation marks within string literals:

SELECT 'I am a single-quoted string';

SELECT "I am a double-quoted string";
SELECT 'I\'m a single-quoted string with an apostrophe';
SELECT "I\'m a double-quoted string with an apostrophe";
SELECT 'I am a "short" single-quoted string containing quotes';
SELECT "I am a \"short\" double-quoted string containing quotes";

The following examples demonstrate calls to string manipulation functions to concatenate strings, convert numbers
to strings, or pull out substrings:

SELECT CONCAT("Once upon a time, there were ", CAST(3 AS STRING), ' little pigs.');
SELECT SUBSTR("hello world",7,5);

The following examples show how to perform operations on STRING columns within a table:

CREATE TABLE t1 (s1 STRING, s2 STRING);

INSERT INTO t1 VALUES ("hello", 'world'), (CAST(7 AS STRING), "wonders");
SELECT s1, s2, length(s1) FROM t1 WHERE s2 LIKE 'w%';

Related information:
String Literals on page 185, CHAR Data Type (CDH 5.2 or higher only) on page 122, VARCHAR Data Type (CDH 5.2 or
higher only) on page 154, Impala String Functions on page 394, Impala Date and Time Functions on page 363

STRUCT Complex Type (CDH 5.5 or higher only)

A complex data type, representing multiple fields of a single item. Frequently used as the element type of an ARRAY
or the VALUE part of a MAP.
Syntax:

column_name STRUCT < name : type [COMMENT 'comment_string'], ... >

type ::= primitive_type | complex_type

The names and number of fields within the STRUCT are fixed. Each field can be a different type. A field within a STRUCT
can also be another STRUCT, or an ARRAY or a MAP, allowing you to create nested data structures with a maximum
nesting depth of 100.
A STRUCT can be the top-level type for a column, or can itself be an item within an ARRAY or the value part of the
key-value pair in a MAP.
When a STRUCT is used as an ARRAY element or a MAP value, you use a join clause to bring the ARRAY or MAP elements
into the result set, and then refer to array_name.ITEM.field or map_name.VALUE.field. In the case of a STRUCT
directly inside an ARRAY or MAP, you can omit the .ITEM and .VALUE pseudocolumns and refer directly to
array_name.field or map_name.field.

142 | Apache Impala Guide

 Impala SQL Language Reference

Usage notes:
Because complex types are often used in combination, for example an ARRAY of STRUCT elements, if you are unfamiliar
with the Impala complex types, start with Complex Types (CDH 5.5 or higher only) on page 156 for background information
and usage examples.
A STRUCT is similar conceptually to a table row: it contains a fixed number of named fields, each with a predefined
type. To combine two related tables, while using complex types to minimize repetition, the typical way to represent
that data is as an ARRAY of STRUCT elements.
Because a STRUCT has a fixed number of named fields, it typically does not make sense to have a STRUCT as the type
of a table column. In such a case, you could just make each field of the STRUCT into a separate column of the table.
The STRUCT type is most useful as an item of an ARRAY or the value part of the key-value pair in a MAP. A nested type
column with a STRUCT at the lowest level lets you associate a variable number of row-like objects with each row of
the table.
The STRUCT type is straightforward to reference within a query. You do not need to include the STRUCT column in a
join clause or give it a table alias, as is required for the ARRAY and MAP types. You refer to the individual fields using
dot notation, such as struct_column_name.field_name, without any pseudocolumn such as ITEM or VALUE.
You can pass a qualified name to DESCRIBE to specify an ARRAY, STRUCT, or MAP column and visualize its structure
as if it were a table. An ARRAY is shown as a two-column table, with ITEM and POS columns. A STRUCT is shown as a
table with each field representing a column in the table. A MAP is shown as a two-column table, with KEY and VALUE
columns.
Internal details:
Within the Parquet data file, the values for each STRUCT field are stored adjacent to each other, so that they can be
encoded and compressed using all the Parquet techniques for storing sets of similar or repeated values. The adjacency
applies even when the STRUCT values are part of an ARRAY or MAP. During a query, Impala avoids unnecessary I/O by
reading only the portions of the Parquet data file containing the requested STRUCT fields.
Added in: CDH 5.5.0 (Impala 2.3.0)
Restrictions:
• Columns with this data type can only be used in tables or partitions with the Parquet file format.
• Columns with this data type cannot be used as partition key columns in a partitioned table.
• The COMPUTE STATS statement does not produce any statistics for columns of this data type.
• The maximum length of the column definition for any complex type, including declarations for any nested types,
is 4000 characters.
• See Limitations and Restrictions for Complex Types on page 161 for a full list of limitations and associated guidelines
about complex type columns.

Examples:

Note: Many of the complex type examples refer to tables such as CUSTOMER and REGION adapted
from the tables used in the TPC-H benchmark. See Sample Schema and Data for Experimenting with
Impala Complex Types on page 178 for the table definitions.

The following example shows a table with various kinds of STRUCT columns, both at the top level and nested within
other complex types. Practice the CREATE TABLE and query notation for complex type columns using empty tables,
until you can visualize a complex data structure and construct corresponding SQL statements reliably.

CREATE TABLE struct_demo

(
id BIGINT,
name STRING,

Apache Impala Guide | 143

Impala SQL Language Reference

-- A STRUCT as a top-level column. Demonstrates how the table ID column

-- and the ID field within the STRUCT can coexist without a name conflict.
employee_info STRUCT < employer: STRING, id: BIGINT, address: STRING >,

-- A STRUCT as the element type of an ARRAY.

places_lived ARRAY < STRUCT <street: STRING, city: STRING, country: STRING >>,

-- A STRUCT as the value portion of the key-value pairs in a MAP.

memorable_moments MAP < STRING, STRUCT < year: INT, place: STRING, details: STRING
>>,

-- A STRUCT where one of the fields is another STRUCT.

current_address STRUCT < street_address: STRUCT <street_number: INT, street_name:
STRING, street_type: STRING>, country: STRING, postal_code: STRING >
)
STORED AS PARQUET;

The following example shows how to examine the structure of a table containing one or more STRUCT columns by
using the DESCRIBE statement. You can visualize each STRUCT as its own table, with columns named the same as
each field of the STRUCT. If the STRUCT is nested inside another complex type, such as ARRAY, you can extend the
qualified name passed to DESCRIBE until the output shows just the STRUCT fields.

DESCRIBE struct_demo;
+-------------------+--------------------------+
| name | type |
+-------------------+--------------------------+
| id | bigint |
| name | string |
| employee_info | struct< |
| | employer:string, |
| | id:bigint, |
| | address:string |
| | > |
| places_lived | array<struct< |
| | street:string, |
| | city:string, |
| | country:string |
| | >> |
| memorable_moments | map<string,struct< |
| | year:int, |
| | place:string, |
| | details:string |
| | >> |
| current_address | struct< |
| | street_address:struct< |
| | street_number:int, |
| | street_name:string, |
| | street_type:string |
| | >, |
| | country:string, |
| | postal_code:string |
| | > |
+-------------------+--------------------------+

The top-level column EMPLOYEE_INFO is a STRUCT. Describing table_name.struct_name displays the fields of the
STRUCT as if they were columns of a table:

DESCRIBE struct_demo.employee_info;
+----------+--------+
| name | type |
+----------+--------+
| employer | string |
| id | bigint |
| address | string |
+----------+--------+

144 | Apache Impala Guide

 Impala SQL Language Reference

Because PLACES_LIVED is a STRUCT inside an ARRAY, the initial DESCRIBE shows the structure of the ARRAY:

DESCRIBE struct_demo.places_lived;
+------+------------------+
| name | type |
+------+------------------+
| item | struct< |
| | street:string, |
| | city:string, |
| | country:string |
| | > |
| pos | bigint |
+------+------------------+

Ask for the details of the ITEM field of the ARRAY to see just the layout of the STRUCT:

DESCRIBE struct_demo.places_lived.item;
+---------+--------+
| name | type |
+---------+--------+
| street | string |
| city | string |
| country | string |
+---------+--------+

Likewise, MEMORABLE_MOMENTS has a STRUCT inside a MAP, which requires an extra level of qualified name to see just
the STRUCT part:

DESCRIBE struct_demo.memorable_moments;
+-------+------------------+
| name | type |
+-------+------------------+
| key | string |
| value | struct< |
| | year:int, |
| | place:string, |
| | details:string |
| | > |
+-------+------------------+

For a MAP, ask to see the VALUE field to see the corresponding STRUCT fields in a table-like structure:

DESCRIBE struct_demo.memorable_moments.value;
+---------+--------+
| name | type |
+---------+--------+
| year | int |
| place | string |
| details | string |
+---------+--------+

For a STRUCT inside a STRUCT, we can see the fields of the outer STRUCT:

DESCRIBE struct_demo.current_address;
+----------------+-----------------------+
| name | type |
+----------------+-----------------------+
| street_address | struct< |
| | street_number:int, |
| | street_name:string, |
| | street_type:string |
| | > |
| country | string |
| postal_code | string |

Apache Impala Guide | 145

Impala SQL Language Reference

+----------------+-----------------------+

Then we can use a further qualified name to see just the fields of the inner STRUCT:

DESCRIBE struct_demo.current_address.street_address;
+---------------+--------+
| name | type |
+---------------+--------+
| street_number | int |
| street_name | string |
| street_type | string |
+---------------+--------+

The following example shows how to examine the structure of a table containing one or more STRUCT columns by
using the DESCRIBE statement. You can visualize each STRUCT as its own table, with columns named the same as
each field of the STRUCT. If the STRUCT is nested inside another complex type, such as ARRAY, you can extend the
qualified name passed to DESCRIBE until the output shows just the STRUCT fields.

DESCRIBE struct_demo;
+-------------------+--------------------------+---------+
| name | type | comment |
+-------------------+--------------------------+---------+
| id | bigint | |
| name | string | |
| employee_info | struct< | |
| | employer:string, | |
| | id:bigint, | |
| | address:string | |
| | > | |
| places_lived | array<struct< | |
| | street:string, | |
| | city:string, | |
| | country:string | |
| | >> | |
| memorable_moments | map<string,struct< | |
| | year:int, | |
| | place:string, | |
| | details:string | |
| | >> | |
| current_address | struct< | |
| | street_address:struct< | |
| | street_number:int, | |
| | street_name:string, | |
| | street_type:string | |
| | >, | |
| | country:string, | |
| | postal_code:string | |
| | > | |
+-------------------+--------------------------+---------+

SELECT id, employee_info.id FROM struct_demo;

SELECT id, employee_info.id AS employee_id FROM struct_demo;

SELECT id, employee_info.id AS employee_id, employee_info.employer

FROM struct_demo;

SELECT id, name, street, city, country

FROM struct_demo, struct_demo.places_lived;

SELECT id, name, places_lived.pos, places_lived.street, places_lived.city,

places_lived.country
FROM struct_demo, struct_demo.places_lived;

SELECT id, name, pl.pos, pl.street, pl.city, pl.country

FROM struct_demo, struct_demo.places_lived AS pl;

SELECT id, name, places_lived.pos, places_lived.street, places_lived.city,

146 | Apache Impala Guide

 Impala SQL Language Reference

places_lived.country
FROM struct_demo, struct_demo.places_lived;

SELECT id, name, pos, street, city, country

FROM struct_demo, struct_demo.places_lived;

SELECT id, name, memorable_moments.key,

memorable_moments.value.year,
memorable_moments.value.place,
memorable_moments.value.details
FROM struct_demo, struct_demo.memorable_moments
WHERE memorable_moments.key IN ('Birthday','Anniversary','Graduation');

SELECT id, name, mm.key, mm.value.year, mm.value.place, mm.value.details

FROM struct_demo, struct_demo.memorable_moments AS mm
WHERE mm.key IN ('Birthday','Anniversary','Graduation');

SELECT id, name, memorable_moments.key, memorable_moments.value.year,

memorable_moments.value.place, memorable_moments.value.details
FROM struct_demo, struct_demo.memorable_moments
WHERE key IN ('Birthday','Anniversary','Graduation');

SELECT id, name, key, value.year, value.place, value.details

FROM struct_demo, struct_demo.memorable_moments
WHERE key IN ('Birthday','Anniversary','Graduation');

SELECT id, name, key, year, place, details

FROM struct_demo, struct_demo.memorable_moments
WHERE key IN ('Birthday','Anniversary','Graduation');

SELECT id, name,

current_address.street_address.street_number,
current_address.street_address.street_name,
current_address.street_address.street_type,
current_address.country,
current_address.postal_code
FROM struct_demo;

For example, this table uses a struct that encodes several data values for each phone number associated with a person.
Each person can have a variable-length array of associated phone numbers, and queries can refer to the category field
to locate specific home, work, mobile, and so on kinds of phone numbers.

CREATE TABLE contact_info_many_structs

(
id BIGINT, name STRING,
phone_numbers ARRAY < STRUCT <category:STRING, country_code:STRING, area_code:SMALLINT,
full_number:STRING, mobile:BOOLEAN, carrier:STRING > >
) STORED AS PARQUET;

Because structs are naturally suited to composite values where the fields have different data types, you might use
them to decompose things such as addresses:

CREATE TABLE contact_info_detailed_address

(
id BIGINT, name STRING,
address STRUCT < house_number:INT, street:STRING, street_type:STRING, apartment:STRING,
city:STRING, region:STRING, country:STRING >
);

In a big data context, splitting out data fields such as the number part of the address and the street name could let
you do analysis on each field independently. For example, which streets have the largest number range of addresses,
what are the statistical properties of the street names, which areas have a higher proportion of “Roads”, “Courts” or
“Boulevards”, and so on.
Related information:

Apache Impala Guide | 147

Impala SQL Language Reference

Complex Types (CDH 5.5 or higher only) on page 156, ARRAY Complex Type (CDH 5.5 or higher only) on page 117, MAP
Complex Type (CDH 5.5 or higher only) on page 136

TIMESTAMP Data Type

A data type used in CREATE TABLE and ALTER TABLE statements, representing a point in time.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name TIMESTAMP

Range: Allowed date values range from 1400-01-01 to 9999-12-31; this range is different from the Hive TIMESTAMP
type. Internally, the resolution of the time portion of a TIMESTAMP value is in nanoseconds.
INTERVAL expressions:
You can perform date arithmetic by adding or subtracting a specified number of time units, using the INTERVAL
keyword and the + and - operators or date_add() and date_sub() functions. You can specify units as YEAR[S],
MONTH[S], WEEK[S], DAY[S], HOUR[S], MINUTE[S], SECOND[S], MILLISECOND[S], MICROSECOND[S], and
NANOSECOND[S]. You can only specify one time unit in each interval expression, for example INTERVAL 3 DAYS or
INTERVAL 25 HOURS, but you can produce any granularity by adding together successive INTERVAL values, such as
timestamp_value + INTERVAL 3 WEEKS - INTERVAL 1 DAY + INTERVAL 10 MICROSECONDS.

For example:

select now() + interval 1 day;

select date_sub(now(), interval 5 minutes);
insert into auction_details
select auction_id, auction_start_time, auction_start_time + interval 2 days + interval
12 hours
from new_auctions;

Time zones:
By default, Impala does not store timestamps using the local timezone, to avoid undesired results from unexpected
time zone issues. Timestamps are stored and interpreted relative to UTC, both when written to or read from data files,
or when converted to or from Unix time values through functions such as from_unixtime() or unix_timestamp().
To convert such a TIMESTAMP value to one that represents the date and time in a specific time zone, convert the
original value with the from_utc_timestamp() function.
Because Impala does not assume that TIMESTAMP values are in any particular time zone, you must be conscious of
the time zone aspects of data that you query, insert, or convert.
For consistency with Unix system calls, the TIMESTAMP returned by the now() function represents the local time in
the system time zone, rather than in UTC. To store values relative to the current time in a portable way, convert any
now() return values using the to_utc_timestamp() function first. For example, the following example shows that
the current time in California (where this Impala cluster is located) is shortly after 2 PM. If that value was written to a
data file, and shipped off to a distant server to be analyzed alongside other data from far-flung locations, the dates
and times would not match up precisely because of time zone differences. Therefore, the to_utc_timestamp()
function converts it using a common reference point, the UTC time zone (descended from the old Greenwich Mean
Time standard). The 'PDT' argument indicates that the original value is from the Pacific time zone with Daylight Saving
Time in effect. When servers in all geographic locations run the same transformation on any local date and time values
(with the appropriate time zone argument), the stored data uses a consistent representation. Impala queries can use
functions such as EXTRACT(), MIN(), AVG(), and so on to do time-series analysis on those timestamps.

[localhost:21000] > select now();

+-------------------------------+
| now() |
+-------------------------------+
| 2015-04-09 14:07:46.580465000 |
+-------------------------------+

148 | Apache Impala Guide

 Impala SQL Language Reference

[localhost:21000] > select to_utc_timestamp(now(), 'PDT');

+--------------------------------+
| to_utc_timestamp(now(), 'pdt') |
+--------------------------------+
| 2015-04-09 21:08:07.664547000 |
+--------------------------------+

The converse function, from_utc_timestamp(), lets you take stored TIMESTAMP data or calculated results and
convert back to local date and time for processing on the application side. The following example shows how you might
represent some future date (such as the ending date and time of an auction) in UTC, and then convert back to local
time when convenient for reporting or other processing. The final query in the example tests whether this arbitrary
UTC date and time has passed yet, by converting it back to the local time zone and comparing it against the current
date and time.

[localhost:21000] > select to_utc_timestamp(now() + interval 2 weeks, 'PDT');

+---------------------------------------------------+
| to_utc_timestamp(now() + interval 2 weeks, 'pdt') |
+---------------------------------------------------+
| 2015-04-23 21:08:34.152923000 |
+---------------------------------------------------+
[localhost:21000] > select from_utc_timestamp('2015-04-23 21:08:34.152923000','PDT');
+------------------------------------------------------------+
| from_utc_timestamp('2015-04-23 21:08:34.152923000', 'pdt') |
+------------------------------------------------------------+
| 2015-04-23 14:08:34.152923000 |
+------------------------------------------------------------+
[localhost:21000] > select from_utc_timestamp('2015-04-23 21:08:34.152923000','PDT') <
now();
+--------------------------------------------------------------------+
| from_utc_timestamp('2015-04-23 21:08:34.152923000', 'pdt') < now() |
+--------------------------------------------------------------------+
| false |
+--------------------------------------------------------------------+

If you have data files written by Hive, those TIMESTAMP values represent the local timezone of the host where the
data was written, potentially leading to inconsistent results when processed by Impala. To avoid compatibility problems
or having to code workarounds, you can specify one or both of these impalad startup flags:
-use_local_tz_for_unix_timestamp_conversions=true
-convert_legacy_hive_parquet_utc_timestamps=true. Although
-convert_legacy_hive_parquet_utc_timestamps is turned off by default to avoid performance overhead,
Cloudera recommends turning it on when processing TIMESTAMP columns in Parquet files written by Hive, to avoid
unexpected behavior.
The -use_local_tz_for_unix_timestamp_conversions setting affects conversions from TIMESTAMP to BIGINT,
or from BIGINT to TIMESTAMP. By default, Impala treats all TIMESTAMP values as UTC, to simplify analysis of time-series
data from different geographic regions. When you enable the -use_local_tz_for_unix_timestamp_conversions
setting, these operations treat the input values as if they are in the local tie zone of the host doing the processing. See
Impala Date and Time Functions on page 363 for the list of functions affected by the
-use_local_tz_for_unix_timestamp_conversions setting.

The following sequence of examples shows how the interpretation of TIMESTAMP values in Parquet tables is affected
by the setting of the -convert_legacy_hive_parquet_utc_timestamps setting.
Regardless of the -convert_legacy_hive_parquet_utc_timestamps setting, TIMESTAMP columns in text tables
can be written and read interchangeably by Impala and Hive:

Impala DDL and queries for text table:

[localhost:21000] > create table t1 (x timestamp);

[localhost:21000] > insert into t1 values (now()), (now() + interval 1 day);
[localhost:21000] > select x from t1;
+-------------------------------+
| x |
+-------------------------------+
| 2015-04-07 15:43:02.892403000 |

Apache Impala Guide | 149

Impala SQL Language Reference

| 2015-04-08 15:43:02.892403000 |
+-------------------------------+
[localhost:21000] > select to_utc_timestamp(x, 'PDT') from t1;
+-------------------------------+
| to_utc_timestamp(x, 'pdt') |
+-------------------------------+
| 2015-04-07 22:43:02.892403000 |
| 2015-04-08 22:43:02.892403000 |
+-------------------------------+

Hive query for text table:

hive> select * from t1;

OK
2015-04-07 15:43:02.892403
2015-04-08 15:43:02.892403
Time taken: 1.245 seconds, Fetched: 2 row(s)

When the table uses Parquet format, Impala expects any time zone adjustment to be applied prior to writing, while
TIMESTAMP values written by Hive are adjusted to be in the UTC time zone. When Hive queries Parquet data files that
it wrote, it adjusts the TIMESTAMP values back to the local time zone, while Impala does no conversion. Hive does no
time zone conversion when it queries Impala-written Parquet files.

Impala DDL and queries for Parquet table:

[localhost:21000] > create table p1 stored as parquet as select x from t1;

+-------------------+
| summary |
+-------------------+
| Inserted 2 row(s) |
+-------------------+
[localhost:21000] > select x from p1;
+-------------------------------+
| x |
+-------------------------------+
| 2015-04-07 15:43:02.892403000 |
| 2015-04-08 15:43:02.892403000 |
+-------------------------------+

Hive DDL and queries for Parquet table:

hive> create table h1 (x timestamp) stored as parquet;

OK
hive> insert into h1 select * from p1;
...
OK
Time taken: 35.573 seconds
hive> select x from p1;
OK
2015-04-07 15:43:02.892403
2015-04-08 15:43:02.892403
Time taken: 0.324 seconds, Fetched: 2 row(s)
hive> select x from h1;
OK
2015-04-07 15:43:02.892403
2015-04-08 15:43:02.892403
Time taken: 0.197 seconds, Fetched: 2 row(s)

The discrepancy arises when Impala queries the Hive-created Parquet table. The underlying values in the TIMESTAMP
column are different from the ones written by Impala, even though they were copied from one table to another by an
INSERT ... SELECT statement in Hive. Hive did an implicit conversion from the local time zone to UTC as it wrote
the values to Parquet.

Impala query for TIMESTAMP values from Impala-written and Hive-written data:

[localhost:21000] > select * from p1;

+-------------------------------+
| x |
+-------------------------------+

150 | Apache Impala Guide

 Impala SQL Language Reference

| 2015-04-07 15:43:02.892403000 |
| 2015-04-08 15:43:02.892403000 |
+-------------------------------+
Fetched 2 row(s) in 0.29s
[localhost:21000] > select * from h1;
+-------------------------------+
| x |
+-------------------------------+
| 2015-04-07 22:43:02.892403000 |
| 2015-04-08 22:43:02.892403000 |
+-------------------------------+
Fetched 2 row(s) in 0.41s

Underlying integer values for Impala-written and Hive-written data:

[localhost:21000] > select cast(x as bigint) from p1;

+-------------------+
| cast(x as bigint) |
+-------------------+
| 1428421382 |
| 1428507782 |
+-------------------+
Fetched 2 row(s) in 0.38s
[localhost:21000] > select cast(x as bigint) from h1;
+-------------------+
| cast(x as bigint) |
+-------------------+
| 1428446582 |
| 1428532982 |
+-------------------+
Fetched 2 row(s) in 0.20s

When the -convert_legacy_hive_parquet_utc_timestamps setting is enabled, Impala recognizes the Parquet

data files written by Hive, and applies the same UTC-to-local-timezone conversion logic during the query as Hive uses,
making the contents of the Impala-written P1 table and the Hive-written H1 table appear identical, whether represented
as TIMESTAMP values or the underlying BIGINT integers:

[localhost:21000] > select x from p1;

+-------------------------------+
| x |
+-------------------------------+
| 2015-04-07 15:43:02.892403000 |
| 2015-04-08 15:43:02.892403000 |
+-------------------------------+
Fetched 2 row(s) in 0.37s
[localhost:21000] > select x from h1;
+-------------------------------+
| x |
+-------------------------------+
| 2015-04-07 15:43:02.892403000 |
| 2015-04-08 15:43:02.892403000 |
+-------------------------------+
Fetched 2 row(s) in 0.19s
[localhost:21000] > select cast(x as bigint) from p1;
+-------------------+
| cast(x as bigint) |
+-------------------+
| 1428446582 |
| 1428532982 |
+-------------------+
Fetched 2 row(s) in 0.29s
[localhost:21000] > select cast(x as bigint) from h1;
+-------------------+
| cast(x as bigint) |
+-------------------+
| 1428446582 |
| 1428532982 |
+-------------------+
Fetched 2 row(s) in 0.22s

Conversions:

Apache Impala Guide | 151

Impala SQL Language Reference

Impala automatically converts STRING literals of the correct format into TIMESTAMP values. Timestamp values are
accepted in the format "yyyy-MM-dd HH:mm:ss.SSSSSS", and can consist of just the date, or just the time, with
or without the fractional second portion. For example, you can specify TIMESTAMP values such as '1966-07-30',
'08:30:00', or '1985-09-25 17:45:30.005'. Casting an integer or floating-point value N to TIMESTAMP produces
a value that is N seconds past the start of the epoch date (January 1, 1970). By default, the result value represents a
date and time in the UTC time zone. If the setting --use_local_tz_for_unix_timestamp_conversions=true
is in effect, the resulting TIMESTAMP represents a date and time in the local time zone.
In Impala 1.3 and higher, the FROM_UNIXTIME() and UNIX_TIMESTAMP() functions allow a wider range of format
strings, with more flexibility in element order, repetition of letter placeholders, and separator characters. In CDH 5.5
/ Impala 2.3 and higher, the UNIX_TIMESTAMP() function also allows a numeric timezone offset to be specified as
part of the input string. See Impala Date and Time Functions on page 363 for details.
In Impala 2.2.0 and higher, built-in functions that accept or return integers representing TIMESTAMP values use the
BIGINT type for parameters and return values, rather than INT. This change lets the date and time functions avoid
an overflow error that would otherwise occur on January 19th, 2038 (known as the “Year 2038 problem” or “Y2K38
problem”). This change affects the from_unixtime() and unix_timestamp() functions. You might need to change
application code that interacts with these functions, change the types of columns that store the return values, or add
CAST() calls to SQL statements that call these functions.
Partitioning:
Although you cannot use a TIMESTAMP column as a partition key, you can extract the individual years, months, days,
hours, and so on and partition based on those columns. Because the partition key column values are represented in
HDFS directory names, rather than as fields in the data files themselves, you can also keep the original TIMESTAMP
values if desired, without duplicating data or wasting storage space. See Partition Key Columns on page 525 for more
details on partitioning with date and time values.

[localhost:21000] > create table timeline (event string) partitioned by (happened

timestamp);
ERROR: AnalysisException: Type 'TIMESTAMP' is not supported as partition-column type in
column: happened

Examples:

select cast('1966-07-30' as timestamp);

select cast('1985-09-25 17:45:30.005' as timestamp);
select cast('08:30:00' as timestamp);
select hour('1970-01-01 15:30:00'); -- Succeeds, returns 15.
select hour('1970-01-01 15:30'); -- Returns NULL because seconds field
required.
select hour('1970-01-01 27:30:00'); -- Returns NULL because hour value out of
range.
select dayofweek('2004-06-13'); -- Returns 1, representing Sunday.
select dayname('2004-06-13'); -- Returns 'Sunday'.
select date_add('2004-06-13', 365); -- Returns 2005-06-13 with zeros for hh:mm:ss
fields.
select day('2004-06-13'); -- Returns 13.
select datediff('1989-12-31','1984-09-01'); -- How many days between these 2 dates?
select now(); -- Returns current date and time in local
timezone.

create table dates_and_times (t timestamp);

insert into dates_and_times values
('1966-07-30'), ('1985-09-25 17:45:30.005'), ('08:30:00'), (now());

NULL considerations: Casting any unrecognized STRING value to this type produces a NULL value.
Partitioning: Because this type potentially has so many distinct values, it is often not a sensible choice for a partition
key column. For example, events 1 millisecond apart would be stored in different partitions. Consider using the TRUNC()
function to condense the number of distinct values, and partition on a new column with the truncated values.
HBase considerations: This data type is fully compatible with HBase tables.
Parquet considerations: This type is fully compatible with Parquet tables.

152 | Apache Impala Guide

 Impala SQL Language Reference

Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as a 16-byte value.
Added in: Available in all versions of Impala.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Sqoop considerations:
If you use Sqoop to convert RDBMS data to Parquet, be careful with interpreting any resulting values from DATE,
DATETIME, or TIMESTAMP columns. The underlying values are represented as the Parquet INT64 type, which is
represented as BIGINT in the Impala table. The Parquet values represent the time in milliseconds, while Impala
interprets BIGINT as the time in seconds. Therefore, if you have a BIGINT column in a Parquet table that was imported
this way from Sqoop, divide the values by 1000 when interpreting as the TIMESTAMP type.
Restrictions:
If you cast a STRING with an unrecognized format to a TIMESTAMP, the result is NULL rather than an error. Make sure
to test your data pipeline to be sure any textual date and time values are in a format that Impala TIMESTAMP can
recognize.
Currently, Avro tables cannot contain TIMESTAMP columns. If you need to store date and time values in Avro tables,
as a workaround you can use a STRING representation of the values, convert the values to BIGINT with the
UNIX_TIMESTAMP() function, or create separate numeric columns for individual date and time fields using the
EXTRACT() function.

Related information:
• Timestamp Literals on page 187.
• To convert to or from different date formats, or perform date arithmetic, use the date and time functions described
in Impala Date and Time Functions on page 363. In particular, the from_unixtime() function requires a
case-sensitive format string such as "yyyy-MM-dd HH:mm:ss.SSSS", matching one of the allowed variations
of a TIMESTAMP value (date plus time, only date, only time, optional fractional seconds).
• See SQL Differences Between Impala and Hive on page 461 for details about differences in TIMESTAMP handling
between Impala and Hive.

TINYINT Data Type

A 1-byte integer data type used in CREATE TABLE and ALTER TABLE statements.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name TINYINT

Range: -128 .. 127. There is no UNSIGNED subtype.

Conversions: Impala automatically converts to a larger integer type (SMALLINT, INT, or BIGINT) or a floating-point
type (FLOAT or DOUBLE) automatically. Use CAST() to convert to STRING or TIMESTAMP. Casting an integer or
floating-point value N to TIMESTAMP produces a value that is N seconds past the start of the epoch date (January 1,
1970). By default, the result value represents a date and time in the UTC time zone. If the setting
--use_local_tz_for_unix_timestamp_conversions=true is in effect, the resulting TIMESTAMP represents a
date and time in the local time zone.
Impala does not return column overflows as NULL, so that customers can distinguish between NULL data and overflow
conditions similar to how they do so with traditional database systems. Impala returns the largest or smallest value in
the range for the type. For example, valid values for a tinyint range from -128 to 127. In Impala, a tinyint with a
value of -200 returns -128 rather than NULL. A tinyint with a value of 200 returns 127.
Usage notes:

Apache Impala Guide | 153

Impala SQL Language Reference

For a convenient and automated way to check the bounds of the TINYINT type, call the functions MIN_TINYINT()
and MAX_TINYINT().
If an integer value is too large to be represented as a TINYINT, use a SMALLINT instead.
NULL considerations: Casting any non-numeric value to this type produces a NULL value.
Examples:

CREATE TABLE t1 (x TINYINT);

SELECT CAST(100 AS TINYINT);

Parquet considerations:
Physically, Parquet files represent TINYINT and SMALLINT values as 32-bit integers. Although Impala rejects attempts
to insert out-of-range values into such columns, if you create a new table with the CREATE TABLE ... LIKE PARQUET
syntax, any TINYINT or SMALLINT columns in the original table turn into INT columns in the new table.
HBase considerations: This data type is fully compatible with HBase tables.
Text table considerations: Values of this type are potentially larger in text tables than in tables using Parquet or other
binary formats.
Internal details: Represented in memory as a 1-byte value.
Added in: Available in all versions of Impala.
Column statistics considerations: Because this type has a fixed size, the maximum and average size fields are always
filled in for column statistics, even before you run the COMPUTE STATS statement.
Related information:
Numeric Literals on page 184, TINYINT Data Type on page 153, SMALLINT Data Type on page 140, INT Data Type on page
135, BIGINT Data Type on page 120, DECIMAL Data Type (CDH 5.1 or higher only) on page 125, Impala Mathematical
Functions on page 339

VARCHAR Data Type (CDH 5.2 or higher only)

A variable-length character type, truncated during processing if necessary to fit within the specified length.
Syntax:
In the column definition of a CREATE TABLE statement:

column_name VARCHAR(max_length)

The maximum length you can specify is 65,535.

Partitioning: This type can be used for partition key columns. Because of the efficiency advantage of numeric values
over character-based values, if the partition key is a string representation of a number, prefer to use an integer type
with sufficient range (INT, BIGINT, and so on) where practical.
HBase considerations: This data type cannot be used with HBase tables.
Parquet considerations:
• This type can be read from and written to Parquet files.
• There is no requirement for a particular level of Parquet.
• Parquet files generated by Impala and containing this type can be freely interchanged with other components
such as Hive and MapReduce.
• Parquet data files can contain values that are longer than allowed by the VARCHAR(n) length limit. Impala ignores
any extra trailing characters when it processes those values during a query.
Text table considerations:

154 | Apache Impala Guide

 Impala SQL Language Reference

Text data files can contain values that are longer than allowed by the VARCHAR(n) length limit. Any extra trailing
characters are ignored when Impala processes those values during a query.
Avro considerations:
The Avro specification allows string values up to 2**64 bytes in length. Impala queries for Avro tables use 32-bit integers
to hold string lengths. In CDH 5.7 / Impala 2.5 and higher, Impala truncates CHAR and VARCHAR values in Avro tables
to (2**31)-1 bytes. If a query encounters a STRING value longer than (2**31)-1 bytes in an Avro table, the query fails.
In earlier releases, encountering such long values in an Avro table could cause a crash.
Schema evolution considerations:
You can use ALTER TABLE ... CHANGE to switch column data types to and from VARCHAR. You can convert from
STRING to VARCHAR(n), or from VARCHAR(n) to STRING, or from CHAR(n) to VARCHAR(n), or from VARCHAR(n)
to CHAR(n). When switching back and forth between VARCHAR and CHAR, you can also change the length value. This
schema evolution works the same for tables using any file format. If a table contains values longer than the maximum
length defined for a VARCHAR column, Impala does not return an error. Any extra trailing characters are ignored when
Impala processes those values during a query.
Compatibility:
This type is available on CDH 5.2 or higher.
Internal details: Represented in memory as a byte array with the minimum size needed to represent each value.
Added in: CDH 5.2.0 (Impala 2.0.0)
Column statistics considerations: Because the values of this type have variable size, none of the column statistics fields
are filled in until you run the COMPUTE STATS statement.
Restrictions:
All data in CHAR and VARCHAR columns must be in a character encoding that is compatible with UTF-8. If you have
binary data from another database system (that is, a BLOB type), use a STRING column to hold it.
Examples:
The following examples show how long and short VARCHAR values are treated. Values longer than the maximum
specified length are truncated by CAST(), or when queried from existing data files. Values shorter than the maximum
specified length are represented as the actual length of the value, with no extra padding as seen with CHAR values.

create table varchar_1 (s varchar(1));

create table varchar_4 (s varchar(4));
create table varchar_20 (s varchar(20));

insert into varchar_1 values (cast('a' as varchar(1))), (cast('b' as varchar(1))),

(cast('hello' as varchar(1))), (cast('world' as varchar(1)));
insert into varchar_4 values (cast('a' as varchar(4))), (cast('b' as varchar(4))),
(cast('hello' as varchar(4))), (cast('world' as varchar(4)));
insert into varchar_20 values (cast('a' as varchar(20))), (cast('b' as varchar(20))),
(cast('hello' as varchar(20))), (cast('world' as varchar(20)));

select * from varchar_1;

+---+
| s |
+---+
| a |
| b |
| h |
| w |
+---+
select * from varchar_4;
+------+
| s |
+------+
| a |
| b |
| hell |
| worl |

Apache Impala Guide | 155

Impala SQL Language Reference

+------+
[localhost:21000] > select * from varchar_20;
+-------+
| s |
+-------+
| a |
| b |
| hello |
| world |
+-------+
select concat('[',s,']') as s from varchar_20;
+---------+
| s |
+---------+
| [a] |
| [b] |
| [hello] |
| [world] |
+---------+

The following example shows how identical VARCHAR values compare as equal, even if the columns are defined with
different maximum lengths. Both tables contain 'a' and 'b' values. The longer 'hello' and 'world' values from
the VARCHAR_20 table were truncated when inserted into the VARCHAR_1 table.

select s from varchar_1 join varchar_20 using (s);

+-------+
| s |
+-------+
| a |
| b |
+-------+

The following examples show how VARCHAR values are freely interchangeable with STRING values in contexts such
as comparison operators and built-in functions:

select length(cast('foo' as varchar(100))) as length;

+--------+
| length |
+--------+
| 3 |
+--------+
select cast('xyz' as varchar(5)) > cast('abc' as varchar(10)) as greater;
+---------+
| greater |
+---------+
| true |
+---------+

UDF considerations: This type cannot be used for the argument or return type of a user-defined function (UDF) or
user-defined aggregate function (UDA).
Related information:
STRING Data Type on page 141, CHAR Data Type (CDH 5.2 or higher only) on page 122, String Literals on page 185, Impala
String Functions on page 394

Complex Types (CDH 5.5 or higher only)

Complex types (also referred to as nested types) let you represent multiple data values within a single row/column
position. They differ from the familiar column types such as BIGINT and STRING, known as scalar types or primitive
types, which represent a single data value within a given row/column position. Impala supports the complex types
ARRAY, MAP, and STRUCT in CDH 5.5 / Impala 2.3 and higher. The Hive UNION type is not currently supported.

Once you understand the basics of complex types, refer to the individual type topics when you need to refresh your
memory about syntax and examples:
• ARRAY Complex Type (CDH 5.5 or higher only) on page 117

156 | Apache Impala Guide

 Impala SQL Language Reference

• STRUCT Complex Type (CDH 5.5 or higher only) on page 142

• MAP Complex Type (CDH 5.5 or higher only) on page 136

Benefits of Impala Complex Types

The reasons for using Impala complex types include the following:
• You already have data produced by Hive or other non-Impala component that uses the complex type column
names. You might need to convert the underlying data to Parquet to use it with Impala.
• Your data model originates with a non-SQL programming language or a NoSQL data management system. For
example, if you are representing Python data expressed as nested lists, dictionaries, and tuples, those data
structures correspond closely to Impala ARRAY, MAP, and STRUCT types.
• Your analytic queries involving multiple tables could benefit from greater locality during join processing. By packing
more related data items within each HDFS data block, complex types let join queries avoid the network overhead
of the traditional Hadoop shuffle or broadcast join techniques.

The Impala complex type support produces result sets with all scalar values, and the scalar components of complex
types can be used with all SQL clauses, such as GROUP BY, ORDER BY, all kinds of joins, subqueries, and inline views.
The ability to process complex type data entirely in SQL reduces the need to write application-specific code in Java or
other programming languages to deconstruct the underlying data structures.

Overview of Impala Complex Types

The ARRAY and MAP types are closely related: they represent collections with arbitrary numbers of elements, where
each element is the same type. In contrast, STRUCT groups together a fixed number of items into a single element.
The parts of a STRUCT element (the fields) can be of different types, and each field has a name.
The elements of an ARRAY or MAP, or the fields of a STRUCT, can also be other complex types. You can construct
elaborate data structures with up to 100 levels of nesting. For example, you can make an ARRAY whose elements are
STRUCTs. Within each STRUCT, you can have some fields that are ARRAY, MAP, or another kind of STRUCT. The Impala
documentation uses the terms complex and nested types interchangeably; for simplicity, it primarily uses the term
complex types, to encompass all the properties of these types.
When visualizing your data model in familiar SQL terms, you can think of each ARRAY or MAP as a miniature table, and
each STRUCT as a row within such a table. By default, the table represented by an ARRAY has two columns, POS to
represent ordering of elements, and ITEM representing the value of each element. Likewise, by default, the table
represented by a MAP encodes key-value pairs, and therefore has two columns, KEY and VALUE.
The ITEM and VALUE names are only required for the very simplest kinds of ARRAY and MAP columns, ones that hold
only scalar values. When the elements within the ARRAY or MAP are of type STRUCT rather than a scalar type, then the
result set contains columns with names corresponding to the STRUCT fields rather than ITEM or VALUE.
You write most queries that process complex type columns using familiar join syntax, even though the data for both
sides of the join resides in a single table. The join notation brings together the scalar values from a row with the values
from the complex type columns for that same row. The final result set contains all scalar values, allowing you to do all
the familiar filtering, aggregation, ordering, and so on for the complex data entirely in SQL or using business intelligence
tools that issue SQL queries.
Behind the scenes, Impala ensures that the processing for each row is done efficiently on a single host, without the
network traffic involved in broadcast or shuffle joins. The most common type of join query for tables with complex
type columns is INNER JOIN, which returns results only in those cases where the complex type contains some elements.
Therefore, most query examples in this section use either the INNER JOIN clause or the equivalent comma notation.

Apache Impala Guide | 157

Impala SQL Language Reference

Note:
Although Impala can query complex types that are present in Parquet files, Impala currently cannot
create new Parquet files containing complex types. Therefore, the discussion and examples presume
that you are working with existing Parquet data produced through Hive, Spark, or some other source.
See Constructing Parquet Files with Complex Columns Using Hive on page 179 for examples of
constructing Parquet data files with complex type columns.
For learning purposes, you can create empty tables with complex type columns and practice query
syntax, even if you do not have sample data with the required structure.

Design Considerations for Complex Types

When planning to use Impala complex types, and designing the Impala schema, first learn how this kind of schema
differs from traditional table layouts from the relational database and data warehousing fields. Because you might
have already encountered complex types in a Hadoop context while using Hive for ETL, also learn how to write
high-performance analytic queries for complex type data using Impala SQL syntax.
How Complex Types Differ from Traditional Data Warehouse Schemas
Complex types let you associate arbitrary data structures with a particular row. If you are familiar with schema design
for relational database management systems or data warehouses, a schema with complex types has the following
differences:
• Logically, related values can now be grouped tightly together in the same table.
In traditional data warehousing, related values were typically arranged in one of two ways:
– Split across multiple normalized tables. Foreign key columns specified which rows from each table were
associated with each other. This arrangement avoided duplicate data and therefore the data was compact,
but join queries could be expensive because the related data had to be retrieved from separate locations.
(In the case of distributed Hadoop queries, the joined tables might even be transmitted between different
hosts in a cluster.)
– Flattened into a single denormalized table. Although this layout eliminated some potential performance
issues by removing the need for join queries, the table typically became larger because values were repeated.
The extra data volume could cause performance issues in other parts of the workflow, such as longer ETL
cycles or more expensive full-table scans during queries.

Complex types represent a middle ground that addresses these performance and volume concerns. By physically
locating related data within the same data files, complex types increase locality and reduce the expense of join
queries. By associating an arbitrary amount of data with a single row, complex types avoid the need to repeat
lengthy values such as strings. Because Impala knows which complex type values are associated with each row,
you can save storage by avoiding artificial foreign key values that are only used for joins. The flexibility of the
STRUCT, ARRAY, and MAP types lets you model familiar constructs such as fact and dimension tables from a data
warehouse, and wide tables representing sparse matrixes.

Physical Storage for Complex Types

Physically, the scalar and complex columns in each row are located adjacent to each other in the same Parquet data
file, ensuring that they are processed on the same host rather than being broadcast across the network when
cross-referenced within a query. This co-location simplifies the process of copying, converting, and backing all the
columns up at once. Because of the column-oriented layout of Parquet files, you can still query only the scalar columns
of a table without imposing the I/O penalty of reading the (possibly large) values of the composite columns.
Within each Parquet data file, the constituent parts of complex type columns are stored in column-oriented format:
• Each field of a STRUCT type is stored like a column, with all the scalar values adjacent to each other and encoded,
compressed, and so on using the Parquet space-saving techniques.

158 | Apache Impala Guide

 Impala SQL Language Reference

• For an ARRAY containing scalar values, all those values (represented by the ITEM pseudocolumn) are stored
adjacent to each other.
• For a MAP, the values of the KEY pseudocolumn are stored adjacent to each other. If the VALUE pseudocolumn is
a scalar type, its values are also stored adjacent to each other.
• If an ARRAY element, STRUCT field, or MAP VALUE part is another complex type, the column-oriented storage
applies to the next level down (or the next level after that, and so on for deeply nested types) where the final
elements, fields, or values are of scalar types.

The numbers represented by the POS pseudocolumn of an ARRAY are not physically stored in the data files. They are
synthesized at query time based on the order of the ARRAY elements associated with each row.
File Format Support for Impala Complex Types
Currently, Impala queries support complex type data only in the Parquet file format. See Using the Parquet File Format
with Impala Tables on page 535 for details about the performance benefits and physical layout of this file format.
Each table, or each partition within a table, can have a separate file format, and you can change file format at the table
or partition level through an ALTER TABLE statement. Because this flexibility makes it difficult to guarantee ahead of
time that all the data files for a table or partition are in a compatible format, Impala does not throw any errors when
you change the file format for a table or partition using ALTER TABLE. Any errors come at runtime when Impala
actually processes a table or partition that contains nested types and is not in one of the supported formats. If a query
on a partitioned table only processes some partitions, and all those partitions are in one of the supported formats, the
query succeeds.
Because Impala does not parse the data structures containing nested types for unsupported formats such as text, Avro,
SequenceFile, or RCFile, you cannot use data files in these formats with Impala, even if the query does not refer to the
nested type columns. Also, if a table using an unsupported format originally contained nested type columns, and then
those columns were dropped from the table using ALTER TABLE ... DROP COLUMN, any existing data files in the
table still contain the nested type data and Impala queries on that table will generate errors.
You can perform DDL operations (even CREATE TABLE) for tables involving complex types in file formats other than
Parquet. The DDL support lets you set up intermediate tables in your ETL pipeline, to be populated by Hive, before the
final stage where the data resides in a Parquet table and is queryable by Impala. Also, you can have a partitioned table
with complex type columns that uses a non-Parquet format, and use ALTER TABLE to change the file format to Parquet
for individual partitions. When you put Parquet data files into those partitions, Impala can execute queries against
that data as long as the query does not involve any of the non-Parquet partitions.
If you use the parquet-tools command to examine the structure of a Parquet data file that includes complex types,
you see that both ARRAY and MAP are represented as a Bag in Parquet terminology, with all fields marked Optional
because Impala allows any column to be nullable.
Impala supports either 2-level and 3-level encoding within each Parquet data file. When constructing Parquet data
files outside Impala, use either encoding style but do not mix 2-level and 3-level encoding within the same data file.
Choosing Between Complex Types and Normalized Tables
Choosing between multiple normalized fact and dimension tables, or a single table containing complex types, is an
important design decision.
• If you are coming from a traditional database or data warehousing background, you might be familiar with how
to split up data between tables. Your business intelligence tools might already be optimized for dealing with this
kind of multi-table scenario through join queries.
• If you are pulling data from Impala into an application written in a programming language that has data structures
analogous to the complex types, such as Python or Java, complex types in Impala could simplify data interchange
and improve understandability and reliability of your program logic.
• You might already be faced with existing infrastructure or receive high volumes of data that assume one layout
or the other. For example, complex types are popular with web-oriented applications, for example to keep
information about an online user all in one place for convenient lookup and analysis, or to deal with sparse or
constantly evolving data fields.

Apache Impala Guide | 159

Impala SQL Language Reference

• If some parts of the data change over time while related data remains constant, using multiple normalized tables
lets you replace certain parts of the data without reloading the entire data set. Conversely, if you receive related
data all bundled together, such as in JSON files, using complex types can save the overhead of splitting the related
items across multiple tables.
• From a performance perspective:
– In Parquet tables, Impala can skip columns that are not referenced in a query, avoiding the I/O penalty of
reading the embedded data. When complex types are nested within a column, the data is physically divided
at a very granular level; for example, a query referring to data nested multiple levels deep in a complex type
column does not have to read all the data from that column, only the data for the relevant parts of the column
type hierarchy.
– Complex types avoid the possibility of expensive join queries when data from fact and dimension tables is
processed in parallel across multiple hosts. All the information for a row containing complex types is typically
to be in the same data block, and therefore does not need to be transmitted across the network when joining
fields that are all part of the same row.
– The tradeoff with complex types is that fewer rows fit in each data block. Whether it is better to have more
data blocks with fewer rows, or fewer data blocks with many rows, depends on the distribution of your data
and the characteristics of your query workload. If the complex columns are rarely referenced, using them
might lower efficiency. If you are seeing low parallelism due to a small volume of data (relatively few data
blocks) in each table partition, increasing the row size by including complex columns might produce more
data blocks and thus spread the work more evenly across the cluster. See Scalability Considerations for Impala
on page 516 for more on this advanced topic.

Differences Between Impala and Hive Complex Types

Impala can query Parquet tables containing ARRAY, STRUCT, and MAP columns produced by Hive. There are some
differences to be aware of between the Impala SQL and HiveQL syntax for complex types, primarily for queries.
The syntax for specifying ARRAY, STRUCT, and MAP types in a CREATE TABLE statement is compatible between Impala
and Hive.
Because Impala STRUCT columns include user-specified field names, you use the NAMED_STRUCT() constructor in
Hive rather than the STRUCT() constructor when you populate an Impala STRUCT column using a Hive INSERT
statement.
The Hive UNION type is not currently supported in Impala.
While Impala usually aims for a high degree of compatibility with HiveQL query syntax, Impala syntax differs from Hive
for queries involving complex types. The differences are intended to provide extra flexibility for queries involving these
kinds of tables.
• Impala uses dot notation for referring to element names or elements within complex types, and join notation for
cross-referencing scalar columns with the elements of complex types within the same row, rather than the LATERAL
VIEW clause and EXPLODE() function of HiveQL.
• Using join notation lets you use all the kinds of join queries with complex type columns. For example, you can use
a LEFT OUTER JOIN, LEFT ANTI JOIN, or LEFT SEMI JOIN query to evaluate different scenarios where the
complex columns do or do not contain any elements.
• You can include references to collection types inside subqueries and inline views. For example, you can construct
a FROM clause where one of the “tables” is a subquery against a complex type column, or use a subquery against
a complex type column as the argument to an IN or EXISTS clause.
• The Impala pseudocolumn POS lets you retrieve the position of elements in an array along with the elements
themselves, equivalent to the POSEXPLODE() function of HiveQL. You do not use index notation to retrieve a
single array element in a query; the join query loops through the array elements and you use WHERE clauses to
specify which elements to return.
• Join clauses involving complex type columns do not require an ON or USING clause. Impala implicitly applies the
join key so that the correct array entries or map elements are associated with the correct row from the table.

160 | Apache Impala Guide

 Impala SQL Language Reference

• Impala does not currently support the UNION complex type.

Limitations and Restrictions for Complex Types

Complex type columns can only be used in tables or partitions with the Parquet file format.
Complex type columns cannot be used as partition key columns in a partitioned table.
When you use complex types with the ORDER BY, GROUP BY, HAVING, or WHERE clauses, you cannot refer to the
column name by itself. Instead, you refer to the names of the scalar values within the complex type, such as the ITEM,
POS, KEY, or VALUE pseudocolumns, or the field names from a STRUCT.

The maximum depth of nesting for complex types is 100 levels.

The maximum length of the column definition for any complex type, including declarations for any nested types, is
4000 characters.
For ideal performance and scalability, use small or medium-sized collections, where all the complex columns contain
at most a few hundred megabytes per row. Remember, all the columns of a row are stored in the same HDFS data
block, whose size in Parquet files typically ranges from 256 MB to 1 GB.
Including complex type columns in a table introduces some overhead that might make queries that do not reference
those columns somewhat slower than Impala queries against tables without any complex type columns. Expect at
most a 2x slowdown compared to tables that do not have any complex type columns.
Currently, the COMPUTE STATS statement does not collect any statistics for columns containing complex types. Impala
uses heuristics to construct execution plans involving complex type columns.
Currently, Impala built-in functions and user-defined functions cannot accept complex types as parameters or produce
them as function return values. (When the complex type values are materialized in an Impala result set, the result set
contains the scalar components of the values, such as the POS or ITEM for an ARRAY, the KEY or VALUE for a MAP, or
the fields of a STRUCT; these scalar data items can be used with built-in functions and UDFs as usual.)
Impala currently cannot write new data files containing complex type columns. Therefore, although the SELECT
statement works for queries involving complex type columns, you cannot use a statement form that writes data, such
as CREATE TABLE AS SELECT or INSERT ... SELECT. To create data files containing complex type data, use the
Hive INSERT statement, or another ETL mechanism such as MapReduce jobs, Spark jobs, Pig, and so on.
Currently, Impala can query complex type columns only from Parquet tables or Parquet partitions within partitioned
tables. Although you can use complex types in tables with Avro, text, and other file formats as part of your ETL pipeline,
for example as intermediate tables populated through Hive, doing analytics through