|
| 1 | +# Hive Regression Local Environment |
| 2 | + |
| 3 | +This document records the local Docker and regression configuration needed to run: |
| 4 | + |
| 5 | +```bash |
| 6 | +regression-test/suites/external_table_p0/hive/ |
| 7 | +``` |
| 8 | + |
| 9 | +## Required Thirdparty Components |
| 10 | + |
| 11 | +Run Hive2 and Hive3 through the thirdparty launcher: |
| 12 | + |
| 13 | +```bash |
| 14 | +./docker/thirdparties/run-thirdparties-docker.sh -c hive2,hive3 --hive-mode refresh |
| 15 | +``` |
| 16 | + |
| 17 | +The Hive startup script may also start `mysql` automatically. Both |
| 18 | +`docker-compose/hive/hive-2x_settings.env` and |
| 19 | +`docker-compose/hive/hive-3x_settings.env` default `JFS_CLUSTER_META` to: |
| 20 | + |
| 21 | +```bash |
| 22 | +mysql://root:123456@(127.0.0.1:3316)/juicefs_meta |
| 23 | +``` |
| 24 | + |
| 25 | +Because of that setting, `run-thirdparties-docker.sh` treats local MySQL 5.7 as |
| 26 | +an implicit dependency for Hive. |
| 27 | + |
| 28 | +Each Hive version starts its own compose services: |
| 29 | + |
| 30 | +- Hive2: `hive2-server`, `hive2-metastore`, `hadoop2-namenode`, |
| 31 | + `hadoop2-datanode`, `hive2-metastore-postgresql` |
| 32 | +- Hive3: `hive3-server`, `hive3-metastore`, `hadoop3-namenode`, |
| 33 | + `hadoop3-datanode`, `hive3-metastore-postgresql` |
| 34 | +- MySQL 5.7: required by the default JuiceFS metadata URI when using the local |
| 35 | + `127.0.0.1:3316` endpoint |
| 36 | + |
| 37 | +The Hive p0 directory does not require these components by default: |
| 38 | + |
| 39 | +- `pg` |
| 40 | +- `iceberg` |
| 41 | +- `minio` |
| 42 | +- `hudi` |
| 43 | +- `kerberos` |
| 44 | +- `ranger` |
| 45 | + |
| 46 | +`test_external_catalog_hive.groovy` has a Ranger-specific branch guarded by |
| 47 | +`enableRangerTest`; if this option is not set to `true`, that branch is skipped. |
| 48 | + |
| 49 | +## Current Running Containers Observed |
| 50 | + |
| 51 | +The following relevant containers were running when checked with |
| 52 | +`sudo docker ps`: |
| 53 | + |
| 54 | +- Hive3 stack: `hive3-server`, `hive3-metastore`, `hadoop3-datanode`, |
| 55 | + `hadoop3-namenode`, `hive3-metastore-postgresql`; all healthy |
| 56 | +- Hive2 stack: `hive2-server`, `hive2-metastore`, `hadoop2-datanode`, |
| 57 | + `hadoop2-namenode`, `hive2-metastore-postgresql`; all healthy |
| 58 | +- MySQL 5.7: `mysql-doris--syt--mysql_57-1`, healthy, published as |
| 59 | + `127.0.0.1:3316 -> 3306` |
| 60 | + |
| 61 | +Other Doris thirdparty containers were also running, but they are not required |
| 62 | +for `external_table_p0/hive/`: |
| 63 | + |
| 64 | +- PostgreSQL 14 on `5442` |
| 65 | +- Iceberg REST, Spark, Postgres, and MinIO |
| 66 | +- Oracle on `1521` |
| 67 | +- ClickHouse on `8123` |
| 68 | +- SQLServer on `1433` |
| 69 | +- OceanBase on `2881` |
| 70 | + |
| 71 | +The observed Hive2/Hive3 and MySQL compose projects came from another checkout |
| 72 | +under `/mnt/disk2/suyiteng/doris`, not from this worktree. PostgreSQL and |
| 73 | +Iceberg compose projects came from this worktree. |
| 74 | + |
| 75 | +## Regression Configuration |
| 76 | + |
| 77 | +The Hive p0 suite directly reads these keys from |
| 78 | +`regression-test/conf/regression-conf.groovy`: |
| 79 | + |
| 80 | +```groovy |
| 81 | +enableHiveTest |
| 82 | +externalEnvIp |
| 83 | +hive2HmsPort |
| 84 | +hive2HdfsPort |
| 85 | +hive3HmsPort |
| 86 | +hive3HdfsPort |
| 87 | +hdfsUser |
| 88 | +``` |
| 89 | + |
| 90 | +It also uses framework helpers backed by: |
| 91 | + |
| 92 | +```groovy |
| 93 | +brokerName |
| 94 | +hdfsPasswd |
| 95 | +hdfsFs |
| 96 | +``` |
| 97 | + |
| 98 | +For local Hive2/Hive3 Docker with default ports, the relevant values are: |
| 99 | + |
| 100 | +```groovy |
| 101 | +enableHiveTest = true |
| 102 | +
|
| 103 | +hive2HmsPort = 9083 |
| 104 | +hive2HdfsPort = 8020 |
| 105 | +hive2ServerPort = 10000 |
| 106 | +hive2PgPort = 5432 |
| 107 | +
|
| 108 | +hive3HmsPort = 9383 |
| 109 | +hive3HdfsPort = 8320 |
| 110 | +hive3ServerPort = 13000 |
| 111 | +hive3PgPort = 5732 |
| 112 | +
|
| 113 | +hdfsUser = "doris-test" |
| 114 | +brokerName = "broker_name" |
| 115 | +hdfsPasswd = "" |
| 116 | +``` |
| 117 | + |
| 118 | +`externalEnvIp` must be the address that Doris BE can use to reach the Hive |
| 119 | +Docker services. For a fully local setup this is usually: |
| 120 | + |
| 121 | +```groovy |
| 122 | +externalEnvIp = "127.0.0.1" |
| 123 | +``` |
| 124 | + |
| 125 | +If BE runs in a context where `127.0.0.1` is not the Docker host, use the host |
| 126 | +IP selected by `run-thirdparties-docker.sh` through `IP_HOST`. |
| 127 | + |
| 128 | +## Config Gaps Noted In regression-conf.groovy.bak |
| 129 | + |
| 130 | +`regression-test/conf/regression-conf.groovy.bak` already contains the core Hive |
| 131 | +keys listed above. The notable follow-ups are: |
| 132 | + |
| 133 | +- `externalEnvIp` was set to `172.20.32.136`; verify this is reachable from BE, |
| 134 | + or replace it with `127.0.0.1` for local host-mode Docker. |
| 135 | +- FE/JDBC addresses in the file used `9033`, `9022`, and `8033`; align these |
| 136 | + with the current Doris worktree cluster ports before running regression. |
| 137 | +- `enableRangerTest` is not present. This is acceptable for the Hive p0 suite |
| 138 | + because the Ranger-specific branch is guarded and skipped unless explicitly |
| 139 | + enabled. |
| 140 | + |
| 141 | +## Thirdparty Script Settings |
| 142 | + |
| 143 | +Before starting Hive through `run-thirdparties-docker.sh`, set a unique |
| 144 | +`CONTAINER_UID` in: |
| 145 | + |
| 146 | +```bash |
| 147 | +docker/thirdparties/custom_settings.env |
| 148 | +``` |
| 149 | + |
| 150 | +The default `CONTAINER_UID="doris--"` is rejected by the startup script. |
| 151 | + |
| 152 | +Hive baseline restore also depends on these settings: |
| 153 | + |
| 154 | +```bash |
| 155 | +s3Endpoint |
| 156 | +s3BucketName |
| 157 | +HIVE_BASELINE_VERSION |
| 158 | +HIVE_BASELINE_TARBALL_CACHE |
| 159 | +``` |
| 160 | + |
| 161 | +If baseline download is unavailable, place the matching tarball manually under |
| 162 | +`HIVE_BASELINE_TARBALL_CACHE`. |
0 commit comments