Skip to content

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

1 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

Iceberg + Nessie + S3

  1. Clone the projectnessie/nessie repo

  2. Navigate to the docker directory and run the following command:

    docker compose -f catalog-auth-s3/docker-compose.yml up
  3. Get a token from the auth service:

    token=$(curl http://127.0.0.1:8080/realms/iceberg/protocol/openid-connect/token --user client1:s3cr3t -d 'grant_type=client_credentials' -d 'scope=profile' | jq -r .access_token)
  4. Use the token to query the API:

    $ curl http://127.0.0.1:19120/api/v2/config -H "Authorization: Bearer $token"
    {
      "defaultBranch" : "main",
      "minSupportedApiVersion" : 1,
      "maxSupportedApiVersion" : 2,
      "actualApiVersion" : 2,
      "specVersion" : "2.2.0",
      "noAncestorHash" : "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
      "repositoryCreationTimestamp" : "2024-09-29T02:08:47.562244091Z",
      "oldestPossibleCommitTimestamp" : "2024-09-29T02:08:47.562244091Z"
    }
  5. Configure $HOME/.pyiceberg.yaml

    catalog:
      rest:
        uri: http://127.0.0.1:19120/iceberg
        token: <token from a previous step>
        # TODO: Confirm if these are truly required here, or we can delegate to Nessie
        # https://github.com/apache/iceberg-python/issues/1028
        # https://github.com/projectnessie/nessie/issues/9318
        s3.endpoint: http://127.0.0.1:9002
        s3.access-key-id: minioadmin
        s3.secret-access-key: minioadmin
  6. Download sample data

    curl https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet -o ./yellow_tripdata_2023-01.parquet
    
  7. Run the script to load data into the catalog:

    uv run python connect.py

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages