Apache Spark has been a popular choice for large-scale distributed data processing. However, as data teams move to cloud architectures and separate computes from client interfaces, the traditional tightly coupled Spark driver model has begun to reveal its limitations. In this article we will explore the new Spark Connect feature, the future of the remote execution. What is Spark Connect? Spark Connect is a decoupled client-server protocol that lets Spark clients, like Python or Java applications, interact with a Spark driver process over the network. Unlike traditional Spark applications where the client starts and controls the driver, Spark Connect uses a gRPC-based protocol to communicate with a running Spark Connect server. Think of it as Spark as a Service for your data apps and notebooks. Spark Connect Spark Connect decoupled client-server protocol gRPC-based protocol running Spark Connect server Spark as a Service Spark Connect is introduced in Spark 3.4 and further improved in 3.5. It changes how clients connect to and interact with a Spark cluster, providing more flexibility, scalability, and language support. Spark Connect Spark 3.4 3.5 Spark Connect is not a cluster manager. It's a protocol that allows clients to communicate with a Spark driver remotely, while still using traditional cluster modes underneath (like YARN or Kubernetes).
Spark Connect makes client-side development easier and is ideal for integrating Spark into tools like VSCode, Jupyter, or web apps.
Decoupling the client from the Spark cluster makes it easier to upgrade and scale the cluster separately from the client. This approach removes dependency conflicts and offers greater flexibility in language support. Spark Connect is not a cluster manager. It's a protocol that allows clients to communicate with a Spark driver remotely, while still using traditional cluster modes underneath (like YARN or Kubernetes). Spark Connect not a cluster manager protocol remotely YARN Kubernetes Spark Connect makes client-side development easier and is ideal for integrating Spark into tools like VSCode, Jupyter, or web apps. Spark Connect client-side development easier VSCode Jupyter web apps Decoupling the client from the Spark cluster makes it easier to upgrade and scale the cluster separately from the client. This approach removes dependency conflicts and offers greater flexibility in language support. Spark cluster dependency conflicts Why Spark Connect? Before Spark Connect, running a Spark application meant you had to combine the Spark driver with your client logic. This led to long startup times, dependency conflicts, and poor IDE integration. It was also difficult to use interactive notebooks or mobile/web-based interfaces with Spark backend. combine the Spark driver with your client logic. long startup times, dependency conflicts, poor IDE integration. interactive notebooks or mobile/web-based interfaces With Spark Connect, clients are lightweight and only need a compatible client library. You can embed Spark inside VSCode, Jupyter notebooks, web apps, and mobile apps. This setup allows for easier scaling and faster iteration. Spark Connect lightweight VSCode, Jupyter notebooks, web apps, and mobile apps How does Spark Connect Work? A connection is established between the client and the Spark server.
The client converts a DataFrame query into an unresolved logical plan, which describes what the operation should do, not how it should be executed.
The unresolved logical plan is encoded and sent to the Spark server.
The Spark server optimizes and executes the query.
The Spark server sends the results back to the client. A connection is established between the client and the Spark server. The client converts a DataFrame query into an unresolved logical plan, which describes what the operation should do, not how it should be executed. DataFrame unresolved logical plan The unresolved logical plan is encoded and sent to the Spark server. unresolved logical plan encoded The Spark server optimizes and executes the query. optimizes executes The Spark server sends the results back to the client. results Practical example: Using Spark Connect with PySpark Step 1: Start the Spark Connect Server # This launches the Spark Connect endpoint
$ ./bin/spark-connect-server # This launches the Spark Connect endpoint
$ ./bin/spark-connect-server Step 2: Connect from a Python Client from pyspark.sql import SparkSession

# sc:// is the special URI scheme used for Spark Connect
spark = SparkSession.builder.remote("sc://localhost:<PORT>").getOrCreate()

df = spark.read.csv("example.csv", header=True)
df.groupBy("category").count().show() from pyspark.sql import SparkSession

# sc:// is the special URI scheme used for Spark Connect
spark = SparkSession.builder.remote("sc://localhost:<PORT>").getOrCreate()

df = spark.read.csv("example.csv", header=True)
df.groupBy("category").count().show() Best for the following use cases Interactive Data Science: Use Jupyter or VSCode to run Spark jobs remotely
CI/CD Pipelines: Validate jobs in GitHub Actions or GitLab CI
Remote Data Apps: Build APIs and dashboards powered by Spark
Multi-Tenant Platforms: Serve multiple users via a single Spark backend Interactive Data Science: Use Jupyter or VSCode to run Spark jobs remotely Interactive Data Science: CI/CD Pipelines: Validate jobs in GitHub Actions or GitLab CI CI/CD Pipelines: Remote Data Apps: Build APIs and dashboards powered by Spark Remote Data Apps: Multi-Tenant Platforms: Serve multiple users via a single Spark backend Multi-Tenant Platforms: Limitations Spark Connect is still the early stages, so some features like complex UDFs or Streaming might have limited support.
You need to upgrade to at least Spark 3.5+ for a more stable version.
Monitoring and debugging are still developing for Spark Connect. Spark Connect is still the early stages, so some features like complex UDFs or Streaming might have limited support. Spark Connect UDFs Streaming You need to upgrade to at least Spark 3.5+ for a more stable version. Monitoring and debugging are still developing for Spark Connect. Spark Connect alternatives Spark Job Server and Apache Livy are similar projects that expose Spark jobs through REST APIs. It is typically used to manage job submissions from external apps like dashboards and notebooks, enabling remote interaction with Spark. However, it differs fundamentally in design, use cases, and maturity. Spark Job Server Apache Livy REST APIs remote Feature

Spark Connect

Spark Job Server

Apache Livy



Type

Built-in gRPC client-server protocol

External REST API server

REST-based Spark session manager



Official Status

✅ Native to Apache Spark (3.4+)

❌ Community project (not officially maintained)

🟡 Incubating under Apache (inactive since 2021)



Client Language Support

Python, Scala, Java, Go, Rust, Dotnet

REST only, language-agnostic

REST + limited Scala/Python clients



Architecture

Lightweight clients + Spark driver over gRPC

External server + job runners

External service managing Spark sessions



Latency / Interactivity

⚡ Very low latency, interactive (DataFrame API)

High (submit job, poll status)

Medium-high



Streaming Support

❌ Limited (in progress)

❌ No

🟡 Partial (limited with batch-like APIs)



Stateful Sessions

✅ Persistent client-side SparkSession

✅ Yes (Job Server Contexts)

✅ Yes (Livy Sessions)



Authentication / Security

SSL/gRPC auth (evolving)

Manual or custom

Kerberos, Hadoop-compatible



Ease of Deployment

✅ Easy with Spark 3.5+

❌ Complex, often fragile

❌ Tricky to deploy & scale



Use Case Fit

Interactive apps, notebooks, CI/CD

Ad hoc job submission, dashboards

Multi-user notebooks, REST access



Extensibility / Maintenance

✅ Actively developed

❌ Unmaintained / legacy

🟡 Outdated, low activity Feature

Spark Connect

Spark Job Server

Apache Livy



Type

Built-in gRPC client-server protocol

External REST API server

REST-based Spark session manager



Official Status

✅ Native to Apache Spark (3.4+)

❌ Community project (not officially maintained)

🟡 Incubating under Apache (inactive since 2021)



Client Language Support

Python, Scala, Java, Go, Rust, Dotnet

REST only, language-agnostic

REST + limited Scala/Python clients



Architecture

Lightweight clients + Spark driver over gRPC

External server + job runners

External service managing Spark sessions



Latency / Interactivity

⚡ Very low latency, interactive (DataFrame API)

High (submit job, poll status)

Medium-high



Streaming Support

❌ Limited (in progress)

❌ No

🟡 Partial (limited with batch-like APIs)



Stateful Sessions

✅ Persistent client-side SparkSession

✅ Yes (Job Server Contexts)

✅ Yes (Livy Sessions)



Authentication / Security

SSL/gRPC auth (evolving)

Manual or custom

Kerberos, Hadoop-compatible



Ease of Deployment

✅ Easy with Spark 3.5+

❌ Complex, often fragile

❌ Tricky to deploy & scale



Use Case Fit

Interactive apps, notebooks, CI/CD

Ad hoc job submission, dashboards

Multi-user notebooks, REST access



Extensibility / Maintenance

✅ Actively developed

❌ Unmaintained / legacy

🟡 Outdated, low activity Feature

Spark Connect

Spark Job Server

Apache Livy Feature Feature Spark Connect Spark Connect Spark Job Server Spark Job Server Apache Livy Apache Livy Type

Built-in gRPC client-server protocol

External REST API server

REST-based Spark session manager Type Type Built-in gRPC client-server protocol Built-in gRPC client-server protocol External REST API server External REST API server REST-based Spark session manager REST-based Spark session manager Official Status

✅ Native to Apache Spark (3.4+)

❌ Community project (not officially maintained)

🟡 Incubating under Apache (inactive since 2021) Official Status Official Status ✅ Native to Apache Spark (3.4+) ✅ Native to Apache Spark (3.4+) ❌ Community project (not officially maintained) ❌ Community project (not officially maintained) 🟡 Incubating under Apache (inactive since 2021) 🟡 Incubating under Apache (inactive since 2021) Client Language Support

Python, Scala, Java, Go, Rust, Dotnet

REST only, language-agnostic

REST + limited Scala/Python clients Client Language Support Client Language Support Python, Scala, Java, Go, Rust, Dotnet Python, Scala, Java, Go, Rust, Dotnet REST only, language-agnostic REST only, language-agnostic REST + limited Scala/Python clients REST + limited Scala/Python clients Architecture

Lightweight clients + Spark driver over gRPC

External server + job runners

External service managing Spark sessions Architecture Architecture Lightweight clients + Spark driver over gRPC Lightweight clients + Spark driver over gRPC External server + job runners External server + job runners External service managing Spark sessions External service managing Spark sessions Latency / Interactivity

⚡ Very low latency, interactive (DataFrame API)

High (submit job, poll status)

Medium-high Latency / Interactivity Latency / Interactivity ⚡ Very low latency, interactive (DataFrame API) ⚡ Very low latency, interactive (DataFrame API) High (submit job, poll status) High (submit job, poll status) Medium-high Medium-high Streaming Support

❌ Limited (in progress)

❌ No

🟡 Partial (limited with batch-like APIs) Streaming Support Streaming Support ❌ Limited (in progress) ❌ Limited (in progress) ❌ No ❌ No 🟡 Partial (limited with batch-like APIs) 🟡 Partial (limited with batch-like APIs) Stateful Sessions

✅ Persistent client-side SparkSession

✅ Yes (Job Server Contexts)

✅ Yes (Livy Sessions) Stateful Sessions Stateful Sessions ✅ Persistent client-side SparkSession ✅ Persistent client-side SparkSession ✅ Yes (Job Server Contexts) ✅ Yes (Job Server Contexts) ✅ Yes (Livy Sessions) ✅ Yes (Livy Sessions) Authentication / Security

SSL/gRPC auth (evolving)

Manual or custom

Kerberos, Hadoop-compatible Authentication / Security Authentication / Security SSL/gRPC auth (evolving) SSL/gRPC auth (evolving) Manual or custom Manual or custom Kerberos, Hadoop-compatible Kerberos, Hadoop-compatible Ease of Deployment

✅ Easy with Spark 3.5+

❌ Complex, often fragile

❌ Tricky to deploy & scale Ease of Deployment Ease of Deployment ✅ Easy with Spark 3.5+ ✅ Easy with Spark 3.5+ ❌ Complex, often fragile ❌ Complex, often fragile ❌ Tricky to deploy & scale ❌ Tricky to deploy & scale Use Case Fit

Interactive apps, notebooks, CI/CD

Ad hoc job submission, dashboards

Multi-user notebooks, REST access Use Case Fit Use Case Fit Interactive apps, notebooks, CI/CD Interactive apps, notebooks, CI/CD Ad hoc job submission, dashboards Ad hoc job submission, dashboards Multi-user notebooks, REST access Multi-user notebooks, REST access Extensibility / Maintenance

✅ Actively developed

❌ Unmaintained / legacy

🟡 Outdated, low activity Extensibility / Maintenance Extensibility / Maintenance ✅ Actively developed ✅ Actively developed ❌ Unmaintained / legacy ❌ Unmaintained / legacy 🟡 Outdated, low activity 🟡 Outdated, low activity Conclusion Spark Connect is the future of remote Spark native interaction. It's fast and perfect for Developers, Notebooks, and Micro-services.
Livy and Spark Job Server were temporary solutions before Spark had native client-server support. They work well for some REST API-based job orchestration scenarios but are now considered outdated and are not maintained.
If you're starting a new project, go with Spark Connect. If you're maintaining an older system, Livy or Spark Job Server might still be useful for now. Spark Connect is the future of remote Spark native interaction. It's fast and perfect for Developers, Notebooks, and Micro-services. Spark Connect Livy and Spark Job Server were temporary solutions before Spark had native client-server support. They work well for some REST API-based job orchestration scenarios but are now considered outdated and are not maintained. Livy Spark Job Server If you're starting a new project, go with Spark Connect. If you're maintaining an older system, Livy or Spark Job Server might still be useful for now. Spark Connect Livy or Spark Job Server

This story contains new, firsthand information uncovered by the writer.

Apache

Picking the Wrong Spark File Format Could Hurt Your Data Strategy

How to Write Complex Queries in Apache Spark SQL Using CTE (WITH Clause)

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Spark Connect Makes PySpark Play Nice with Notebooks, IDEs, and Web Apps

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Picking the Wrong Spark File Format Could Hurt Your Data Strategy

3 Best Hadoop Alternatives to Consider for Migration

The Noonification: Introduction to Python Debugging with Pdb (8/25/2022)

8 Lessons For Building Data Companies On Solid Ground

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution

Picking the Wrong Spark File Format Could Hurt Your Data Strategy

3 Best Hadoop Alternatives to Consider for Migration

The Noonification: Introduction to Python Debugging with Pdb (8/25/2022)

8 Lessons For Building Data Companies On Solid Ground

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps