Apache Spark Connect Client for C#

Communicating to Spark Connect Server with a thin C# gRPC Client.

· Nuget package · Spark Connect Architecture . Spark Connect Overview · Golang SDK · Rust SDK ·

• Apache Spark Connect Client for C#
  • Current State of the Project
  • Getting started
  • High Level Design
  • Features
  • Contributing
  • Built With

This project houses the experimental client for Spark Connect for Apache Spark written in C#.

Current State of the Project

Currently, the Spark Connect client for C# is highly experimental and should not be used in any production setting.

Getting started

This section explains how run Spark Connect C# locally from scratch.

Step 1: Install Docker Desktop - for running a Spark Server in a Container.

Step 2: Install Dotnet 8 CLI.

💡 Tip: Install Visual Studio Community Edition for a rich debugging experience in C#.

Step 3: Clone this repo via git.

Step 4: Spin up Spark Connect Server in a container:

Powershell:

$GIT_ROOT = git rev-parse --show-toplevel
Set-Location -Path "${GIT_ROOT}\projects\spark-connect-csharp\src\spark.connect.test\Common\SparkHost"

docker-compose -f docker-compose-spark-connect-server.yaml up -d

Bash:

GIT_ROOT=$(git rev-parse --show-toplevel)
cd "${GIT_ROOT}/projects/spark-connect-csharp/src/spark.connect.test/Common/SparkHost"

docker-compose -f docker-compose-spark-connect-server.yaml up -d

Step 5: Run the demo project spark.connect.demo for an end-to-end demonstration.

$GIT_ROOT = git rev-parse --show-toplevel
Set-Location -Path "${GIT_ROOT}\projects\spark-connect-csharp\src\spark.connect.demo"

dotnet run --project spark.connect.demo.csproj --framework net8.0

GIT_ROOT=$(git rev-parse --show-toplevel)
cd "${GIT_ROOT}/projects/spark-connect-csharp/src/spark.connect.demo"

dotnet run --project spark.connect.demo.csproj --framework net8.0

High Level Design

TODO

Features

TODO

Contributing

This project uses a Devcontainer to manage Build Dependencies. The Devcontainer is tested upon every commit, to ensure the development environment is always ready.

See the Devcontainer on how to get a development environment setup using Docker Desktop.

Built With

These are the tools this project is built and maintained with:

• nx - Build management tool
• docker - Containerization
• devcontainer - VS Code dev container