Add Docker image for Unity Catalog by pflooky · Pull Request #18 · unitycatalog/unitycatalog

pflooky · 2024-06-14T06:32:50Z

Some small changes are included in this PR to help create a working Docker image for Unity Catalog:

Add Dockerfile
Add script to run build and push for Docker images
Add GitHub action to build and upload Docker image
Add uc-docker script for CLI to work straight away on the Docker image
Allow for server.properties and log4j2.properties file pathways to be set via environment variables or system properties
Fix banner having two v's
Change --format to --output in the documentation

…uild folders to gitignore

…ocated

… 'v'

tdas · 2024-06-15T16:15:29Z

Thank you for this pr. Docker would be great. I am going to play with this thing as soon as I get a chance.

tdas · 2024-06-15T16:17:10Z

README.md

  as well as provide a convenient way to explore the content of any UC server implementation.

 ### Prerequisites
+- Docker


You do have to clone this repo still, so the 2 options has to be under the instructions of cloning the repo.

You don't need to clone the repo if running the docker run ... command. It will pull the image from Dockerhub and start running straight away.

Can you elaborate a bit more about this? what will be the full command of docker run ...
If we are adding a completely new way to run these steps, then we need to write down the exact steps that people can copy paste and run and leave nothing to assumption.

And I am not as familiar with Docker as i want to be. I am guessing that this docker run ... command is going to download from dockerhub? who is going to publish docker hub?

tdas · 2024-06-15T16:18:50Z

docker-action.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+


Could you add some comments in the file on what does this file do?

tdas · 2024-06-15T16:19:59Z

docker-action.sh

@@ -0,0 +1,32 @@
+#!/bin/bash


Does this script have to be in the root of the repo. Keeping th root of the repo uncluttered is better

# Conflicts: # .gitignore # README.md # etc/db/h2db.mv.db

stoffeastrom · 2024-06-17T16:00:10Z

Would love to see a docker image for this as well but I'm wondering how datacatering is affiliated to unitycatalog? Since the image is built in this repo wouldn't it be better if it's published under unitycatalog?

pflooky · 2024-06-18T00:51:16Z

Would love to see a docker image for this as well but I'm wondering how datacatering is affiliated to unitycatalog? Since the image is built in this repo wouldn't it be better if it's published under unitycatalog?

I was using datacatering to test the docker image creation. Happy to change to unitycatalog.

When this PR is merged, to get the GitHub action to work, someone will have to add in DOCKER_HUB_USER and DOCKER_HUB_TOKEN to the GitHub action secrets that has access to upload to unitycatalog.

tdas · 2024-06-20T14:58:23Z

Dockerfile

+
+ENV SERVER_PROPERTIES_FILE=/opt/app/etc/conf/server.properties
+ENV SERVER_JOG4J_CONFIGURATION_FILE=/opt/app/etc/conf/server.log4j2.properties
+ENV CLI_JOG4J_CONFIGURATION_FILE=/opt/app/etc/conf/cli.log4j2.properties


Where are these ENV variables used? What's the point of assigning them?

tdas · 2024-06-20T15:03:21Z

Dockerfile

+WORKDIR /opt/app
+
+COPY server/target/unitycatalog-server-assembly.jar unitycatalog-server.jar
+COPY examples/cli/target/unitycatalog-cli-assembly.jar unitycatalog-cli.jar


whats the purpose of the CLI jar in docker? we are not running the CLI in docker

tdas · 2024-06-20T15:04:02Z

README.md


 ### Run the UC Server
+```shell
+docker run -d -i --name unitycatalog -p 8081:8081 unitycatalog/unitycatalog:0.1.0


this docker run step will only work with published versions of the server? like 0.1?
my goal was for these steps to be runnable by anyone with the current unpublished code as soon as they clone this repo

We could document how to build the container locally, and afterwards also execute it locally.

tdas · 2024-06-20T15:04:23Z

bin/uc-docker

@@ -0,0 +1,26 @@
+#!/bin/bash


what is the point of this file

tdas · 2024-06-20T15:06:51Z

examples/cli/src/main/java/io/unitycatalog/cli/UnityCatalogCli.java

        return apiClient;
    }
+
+    private String getLogPropertiesFilePath() {


why do we need to define these properties. why not just keep it simple and always load the configurations from /etc/conf?

tdas

Overall, thank you for your attempt to dockerize the whole thing. But there are a lot of changes in this PR that I dont understand the purpose of. could you please bear with me, and answer my questions so that we understand what we are merging and what will be needed to maintain this?

Cupprum · 2024-06-24T18:13:41Z

build/docker-action.sh

Is there a reason why this is a separate script, and not part of yaml pipeline file?

Signed-off-by: Peter Flook <[email protected]>

pdbg · 2024-07-21T14:21:50Z

+1

Nice to have dockerization version of uc.

LeonardoSanBenitez · 2024-07-23T09:13:19Z

+1
Thanks for the initiative.
I'm getting started now with UC outside databricks and this docker image is great to have.

# Description of changes For this PR I took an alternative approach to #18 and #22 and created a Dockerfile with build and start scripts that require minimal intervention and interaction with the codebase. From the codebase the only change is the .gitignore where I added the .DS_Store which can be helpful in the future for contributors using Mac OS, PR #22 is a great start but maybe oversimplified. PR #18 Has good thought put into it but I wanted to stay close to the recommended way of running Unity Catalog as outlined in the project's README. I tried not to fiddle directly with the jars and use the provided `/bin/start-uc-server` to run the catalog. With this approach the Dockerfile remains focused on building the environment and any changes to how the environment should run can be made in the future inside the `start-uc-server` script rather than the Dockerfile. # Rationale of the PR This pull request introduces a way to run Unity Catalog using Docker containers. It provides a Dockerfile that builds the necessary environment and separate bash scripts for building and starting the catalog. This simplifies the process for users by requiring minimal interaction with the codebase itself. The included README provides detailed instructions on how to use these scripts to build and run the Unity Catalog container. > [!NOTE] > The `README.md` contains two API calls that create an external and an managed table. > These APIs are not working yet because they are not supported by the catalogue yet. Signed-off-by: Jean Boutros <[email protected]> --------- Signed-off-by: Jean Boutros <[email protected]> Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Denny Lee <[email protected]>

creechy · 2024-08-08T17:04:59Z

This duplicates a PR #116 that included Docker functionality

# Description of changes For this PR I took an alternative approach to unitycatalog#18 and unitycatalog#22 and created a Dockerfile with build and start scripts that require minimal intervention and interaction with the codebase. From the codebase the only change is the .gitignore where I added the .DS_Store which can be helpful in the future for contributors using Mac OS, PR unitycatalog#22 is a great start but maybe oversimplified. PR unitycatalog#18 Has good thought put into it but I wanted to stay close to the recommended way of running Unity Catalog as outlined in the project's README. I tried not to fiddle directly with the jars and use the provided `/bin/start-uc-server` to run the catalog. With this approach the Dockerfile remains focused on building the environment and any changes to how the environment should run can be made in the future inside the `start-uc-server` script rather than the Dockerfile. # Rationale of the PR This pull request introduces a way to run Unity Catalog using Docker containers. It provides a Dockerfile that builds the necessary environment and separate bash scripts for building and starting the catalog. This simplifies the process for users by requiring minimal interaction with the codebase itself. The included README provides detailed instructions on how to use these scripts to build and run the Unity Catalog container. > [!NOTE] > The `README.md` contains two API calls that create an external and an managed table. > These APIs are not working yet because they are not supported by the catalogue yet. Signed-off-by: Jean Boutros <[email protected]> --------- Signed-off-by: Jean Boutros <[email protected]> Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Denny Lee <[email protected]> Signed-off-by: Kevin Wang <[email protected]>

pflooky added 16 commits June 14, 2024 08:06

Add in Dockerfile, add in GitHub action to create Docker image, add b…

7864591

…uild folders to gitignore

Remove SCRIPT_DIR and ROOT_DIR from docker action script

70ecd25

Use sbt assembly to create fat jar for docker image

cbdae7d

Add more memory for sbt assembly

41218a1

Add in etc folder to Dockerfile

b82716a

Add in WORKDIR to Dockerfile

0025f23

Add in hack to cd /opt/app since server relies on relative pathways

2c82fc8

Allow for environment variables to control where properties file is l…

0235254

…ocated

Change to copy etc etc

70ff3dd

Change the name of the assembly jar

d5da7a8

Resolve sbt assembly problem with log4j-core, fix banner having extra…

61794b1

… 'v'

Add in uc-docker to Dockerfile

adef8be

Add in default server URL to automatically connect to uc server

450fb7c

Fix version in banner

0c49cc8

Fix README and tutorial to use --output instead of --format

a7b01db

Re-enable sbt-tests github action

16a8039

tdas reviewed Jun 15, 2024

View reviewed changes

This was referenced Jun 15, 2024

Add a Dockerfile #22

Closed

added Dockerfile and updated tutorials.md on how to use the Dockerfile #42

Closed

EladLeev mentioned this pull request Jun 15, 2024

feat: Base Helm chart #45

Open

4 tasks

pflooky added 3 commits June 17, 2024 08:19

Move docker-action script to build directory, add extra comments

a3cd4a7

Merge remote-tracking branch 'unity/main'

7f4f3e6

# Conflicts: # .gitignore # README.md # etc/db/h2db.mv.db

Fix up relative directories in docker-action script

41d1e29

dennyglee mentioned this pull request Jun 18, 2024

Add Dockerfile for UC #58

Closed

Update to use unitycatalog docker image prefix

d20f24f

tdas reviewed Jun 20, 2024

View reviewed changes

bin/uc-docker

@@ -0,0 +1,26 @@

#!/bin/bash

Copy link

Contributor

tdas Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the point of this file

tdas reviewed Jun 20, 2024

View reviewed changes

tdas requested changes Jun 20, 2024

View reviewed changes

Cupprum reviewed Jun 24, 2024

View reviewed changes

build/docker-action.sh

Copy link

Cupprum Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this is a separate script, and not part of yaml pipeline file?

jeanboutros mentioned this pull request Jun 30, 2024

Run Unity Catalog in a Docker container #116

Merged

Merge branch 'main' into main

d82aed4

Signed-off-by: Peter Flook <[email protected]>

pflooky requested a review from tdas July 2, 2024 02:33

hongbo-miao mentioned this pull request Jul 24, 2024

Data Catalog hongbo-miao/hongbomiao.com#17272

Closed

creechy closed this Aug 8, 2024

dennyglee pushed a commit that referenced this pull request Sep 2, 2024

Expand entities (#18)

87b62c3

tdas pushed a commit that referenced this pull request Sep 5, 2024

[ui] Expand entities (#18)

37cd5c4

rtyler pushed a commit to rtyler/unitycatalog that referenced this pull request Sep 5, 2024

Expand entities (unitycatalog#18)

0c5918f

Conversation

pflooky commented Jun 14, 2024

Uh oh!

tdas commented Jun 15, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stoffeastrom commented Jun 17, 2024

Uh oh!

pflooky commented Jun 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdbg commented Jul 21, 2024

Uh oh!

LeonardoSanBenitez commented Jul 23, 2024

Uh oh!

creechy commented Aug 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants