Skip to content

Commit 1beaab5

Browse files
xiangfu0xiangfu0
authored
upgrade superset to 0.36.0 (#5239)
* upgrade superset to 0.36.0 * Adding superset examples
1 parent 44b0a91 commit 1beaab5

File tree

10 files changed

+2519
-195
lines changed

10 files changed

+2519
-195
lines changed

‎docker/images/pinot-superset/.dockerignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,5 @@
1919

2020
*
2121
!bin/
22+
!examples/
2223
!requirements-db.txt

‎docker/images/pinot-superset/.python-version‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎docker/images/pinot-superset/Dockerfile‎

Lines changed: 102 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -17,96 +17,122 @@
1717
# under the License.
1818
#
1919

20-
ARG NODE_VERSION=latest
21-
ARG PYTHON_VERSION=latest
22-
23-
# --- Build assets with NodeJS
24-
25-
FROM node:${NODE_VERSION} AS build
20+
######################################################################
21+
# PY stage that simply does a pip install on our requirements
22+
######################################################################
23+
ARG PY_VER=3.6.9
24+
FROM python:${PY_VER} AS superset-py
25+
26+
RUN mkdir /app \
27+
&& apt-get update -y \
28+
&& apt-get install -y --no-install-recommends \
29+
build-essential \
30+
default-libmysqlclient-dev \
31+
libpq-dev \
32+
&& rm -rf /var/lib/apt/lists/*
2633

2734
# Superset version to build
2835
ARG SUPERSET_VERSION=master
29-
ENV SUPERSET_HOME=/var/lib/superset/
36+
ENV SUPERSET_SRC=/app/superset-src
3037

3138
# Download source
32-
WORKDIR ${SUPERSET_HOME}
33-
RUN wget -O /tmp/superset.tar.gz https://github.com/apache/incubator-superset/archive/${SUPERSET_VERSION}.tar.gz && \
34-
tar xzf /tmp/superset.tar.gz -C ${SUPERSET_HOME} --strip-components=1
35-
36-
# Build assets
37-
WORKDIR ${SUPERSET_HOME}/superset/assets
38-
RUN npm install && \
39-
npm run build
40-
41-
# --- Build dist package with Python 3
42-
43-
FROM python:${PYTHON_VERSION} AS dist
44-
45-
# Copy prebuilt workspace into stage
46-
ENV SUPERSET_HOME=/var/lib/superset/
47-
WORKDIR ${SUPERSET_HOME}
48-
COPY --from=build ${SUPERSET_HOME} .
49-
COPY requirements-db.txt .
50-
51-
# Create package to install
52-
RUN python setup.py sdist && \
53-
tar czfv /tmp/superset.tar.gz requirements.txt requirements-db.txt dist
54-
55-
# --- Install dist package and finalize app
56-
57-
FROM python:${PYTHON_VERSION} AS final
39+
WORKDIR ${SUPERSET_SRC}
40+
RUN wget -qO /tmp/superset.tar.gz https://github.com/apache/incubator-superset/archive/${SUPERSET_VERSION}.tar.gz \
41+
&& tar xzf /tmp/superset.tar.gz -C ${SUPERSET_SRC} --strip-components=1
42+
43+
# First, we just wanna install requirements, which will allow us to utilize the cache
44+
# in order to only build if and only if requirements change
45+
COPY requirements-db.txt /app/
46+
RUN cp ${SUPERSET_SRC}/requirements.txt /app/ \
47+
&& cd /app \
48+
&& pip install --no-cache -r requirements.txt \
49+
&& pip install --no-cache -r requirements-db.txt
50+
51+
######################################################################
52+
# Node stage to deal with static asset construction
53+
######################################################################
54+
FROM node:10-jessie AS superset-node
55+
56+
ARG NPM_BUILD_CMD="build"
57+
ENV BUILD_CMD=${NPM_BUILD_CMD}
58+
ENV SUPERSET_SRC=/app/superset-src
59+
60+
# NPM ci first, as to NOT invalidate previous steps except for when package.json changes
61+
RUN mkdir -p /app/superset-frontend
62+
RUN mkdir -p /app/superset/assets
63+
64+
COPY --from=superset-py ${SUPERSET_SRC}/docker/frontend-mem-nag.sh /
65+
COPY --from=superset-py ${SUPERSET_SRC}/superset-frontend/package* /app/superset-frontend/
66+
RUN /frontend-mem-nag.sh \
67+
&& cd /app/superset-frontend \
68+
&& npm ci
69+
70+
# Next, copy in the rest and let webpack do its thing
71+
COPY --from=superset-py ${SUPERSET_SRC}/superset-frontend /app/superset-frontend
72+
# This is BY FAR the most expensive step (thanks Terser!)
73+
RUN cd /app/superset-frontend \
74+
&& npm run ${BUILD_CMD} \
75+
&& rm -rf node_modules
76+
77+
78+
######################################################################
79+
# Final lean image...
80+
######################################################################
81+
ARG PY_VER=3.6.9
82+
FROM python:${PY_VER} AS lean
83+
84+
ENV SUPERSET_SRC=/app/superset-src
85+
86+
ENV LANG=C.UTF-8 \
87+
LC_ALL=C.UTF-8 \
88+
FLASK_ENV=production \
89+
FLASK_APP="superset.app:create_app()" \
90+
PYTHONPATH="/app/pythonpath" \
91+
SUPERSET_HOME="/app/superset_home" \
92+
SUPERSET_PORT=8080
5893

5994
# Configure environment
6095
ENV GUNICORN_BIND=0.0.0.0:8088 \
6196
GUNICORN_LIMIT_REQUEST_FIELD_SIZE=0 \
6297
GUNICORN_LIMIT_REQUEST_LINE=0 \
6398
GUNICORN_TIMEOUT=60 \
6499
GUNICORN_WORKERS=3 \
65-
GUNICORN_THREADS=4 \
66-
LANG=C.UTF-8 \
67-
LC_ALL=C.UTF-8 \
68-
PYTHONPATH=/etc/superset:/home/superset:$PYTHONPATH \
69-
SUPERSET_REPO=apache/incubator-superset \
70-
SUPERSET_VERSION=${SUPERSET_VERSION} \
71-
SUPERSET_HOME=/var/lib/superset
100+
GUNICORN_THREADS=4
72101
ENV GUNICORN_CMD_ARGS="--workers ${GUNICORN_WORKERS} --threads ${GUNICORN_THREADS} --timeout ${GUNICORN_TIMEOUT} --bind ${GUNICORN_BIND} --limit-request-line ${GUNICORN_LIMIT_REQUEST_LINE} --limit-request-field_size ${GUNICORN_LIMIT_REQUEST_FIELD_SIZE}"
73102

74-
# Create superset user & install dependencies
75-
WORKDIR /tmp/superset
76-
COPY --from=dist /tmp/superset.tar.gz .
77-
RUN useradd -U -m superset && \
78-
mkdir -p /etc/superset && \
79-
mkdir -p ${SUPERSET_HOME} && \
80-
chown -R superset:superset /etc/superset && \
81-
chown -R superset:superset ${SUPERSET_HOME} && \
82-
apt-get update && \
83-
apt-get install -y \
84-
build-essential \
85-
curl \
86-
default-libmysqlclient-dev \
87-
freetds-bin \
88-
freetds-dev \
89-
libffi-dev \
90-
libldap2-dev \
91-
libpq-dev \
92-
libsasl2-2 \
93-
libsasl2-dev \
94-
libsasl2-modules-gssapi-mit \
95-
libssl1.0 && \
96-
apt-get clean && \
97-
tar xzf superset.tar.gz && \
98-
pip install dist/*.tar.gz -r requirements.txt -r requirements-db.txt && \
99-
rm -rf ./*
100-
101-
# Configure Filesystem
103+
RUN useradd --user-group --no-create-home --no-log-init --shell /bin/bash superset \
104+
&& mkdir -p ${SUPERSET_HOME} ${PYTHONPATH} \
105+
&& apt-get update -y \
106+
&& apt-get install -y --no-install-recommends \
107+
build-essential \
108+
default-libmysqlclient-dev \
109+
libpq-dev \
110+
&& rm -rf /var/lib/apt/lists/*
111+
112+
COPY --from=superset-py /usr/local/lib/python3.6/site-packages/ /usr/local/lib/python3.6/site-packages/
113+
# Copying site-packages doesn't move the CLIs, so let's copy them one by one
114+
COPY --from=superset-py /usr/local/bin/gunicorn /usr/local/bin/celery /usr/local/bin/flask /usr/bin/
115+
COPY --from=superset-node /app/superset/static/assets /app/superset/static/assets
116+
COPY --from=superset-node /app/superset-frontend /app/superset-frontend
117+
118+
## Lastly, let's install superset itself
119+
COPY --from=superset-py ${SUPERSET_SRC}/superset /app/superset
120+
COPY --from=superset-py ${SUPERSET_SRC}/setup.py ${SUPERSET_SRC}/MANIFEST.in ${SUPERSET_SRC}/README.md /app/
121+
RUN cd /app \
122+
&& chown -R superset:superset * \
123+
&& pip install -e .
124+
125+
COPY --from=superset-py ${SUPERSET_SRC}/docker/docker-entrypoint.sh /usr/bin/
126+
102127
COPY bin /usr/local/bin
103-
WORKDIR /home/superset
104-
VOLUME /etc/superset \
105-
/home/superset \
106-
/var/lib/superset
128+
COPY examples /etc/examples/pinot
129+
130+
WORKDIR /app
107131

108-
# Finalize application
109-
EXPOSE 8088
110-
HEALTHCHECK CMD ["curl", "-f", "http://localhost:8088/health"]
111-
CMD ["gunicorn", "superset:app"]
112132
USER superset
133+
134+
HEALTHCHECK CMD ["curl", "-f", "http://localhost:8088/health"]
135+
136+
EXPOSE ${SUPERSET_PORT}
137+
138+
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]

‎docker/images/pinot-superset/Makefile‎

Lines changed: 0 additions & 64 deletions
This file was deleted.

‎docker/images/pinot-superset/README.md‎

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -23,31 +23,21 @@
2323

2424
Docker image for [Superset](https://github.com/ApacheInfra/superset) with Pinot integration.
2525

26-
This docker build project is based on Project [docker-superset](https://github.com/amancevice/docker-superset) and specialized for Pinot.
2726

2827
## How to build
2928

30-
Please modify file `Makefile` to change `image` and `superset_version` accordingly.
29+
Below command will build docker image and tag it as `apachepinot/pinot-superset:0.36.0`.
3130

32-
Below command will build docker image and tag it as `superset_version` and `latest`.
31+
You can also build directly with `docker build` command by setting arguments:
3332

3433
```bash
35-
make latest
34+
docker build --build-arg SUPERSET_VERSION=0.36.0 --tag apachepinot/pinot-superset:0.36.0 .
3635
```
3736

38-
You can also build directly with `docker build` command by setting arguments:
39-
```bash
40-
docker build \
41-
--build-arg NODE_VERSION=latest \
42-
--build-arg PYTHON_VERSION=3.6 \
43-
--build-arg SUPERSET_VERSION=0.34.1 \
44-
--tag apachepinot/pinot-superset:0.34.1 \
45-
--target build .
46-
```
4737
## How to push
4838

4939
```bash
50-
make push
40+
docker push apachepinot/pinot-superset:0.36.0
5141
```
5242

5343
## Configuration
@@ -61,9 +51,9 @@ Place this file in a local directory and mount this directory to `/etc/superset`
6151

6252
The image defines two data volumes: one for mounting configuration into the container, and one for data (logs, SQLite DBs, &c).
6353

64-
The configuration volume is located alternatively at `/etc/superset` or `/home/superset`; either is acceptable. Both of these directories are included in the `PYTHONPATH` of the image. Mount any configuration (specifically the `superset_config.py` file) here to have it read by the app on startup.
54+
The configuration volume is located alternatively at `/etc/superset` or `/app/superset`; either is acceptable. Both of these directories are included in the `PYTHONPATH` of the image. Mount any configuration (specifically the `superset_config.py` file) here to have it read by the app on startup.
6555

66-
The data volume is located at `/var/lib/superset` and it is where you would mount your SQLite file (if you are using that as your backend), or a volume to collect any logs that are routed there. This location is used as the value of the `SUPERSET_HOME` environmental variable.
56+
The data volume is located at `/app/superset_home` and it is where you would mount your SQLite file (if you are using that as your backend), or a volume to collect any logs that are routed there.
6757

6858
## Kubernetes Examples
6959

0 commit comments

Comments
 (0)