Charmed Apache Spark (revision 2)

Jul 29, 2025

We’re excited to introduce Apache Kyuubi as part of our expanded Apache Spark charmed bundle—bringing enhanced multi-tenant support and a simplified SQL interface for big data analytics.

This release includes Apache Kyuubi, the Spark History Server, Spark Integration Hub, Spark Client Snap, and Apache Spark OCI Images to deliver a seamless, production-ready fully open-source data lakehouse experience. The Charmed Apache Spark with Apache Kyuubi is available on charmhub.io at the 3.4/stable and 3.5/stable channels.

Charmhub: 3.4/stable, 3.5/stable | Docs | Deploy guide | System requirements

Features

General

• [DPE-4344;DPE-2863] Bundle implementation and integrated testing
• [DPE-5784] Azure AKS solution validation

Kyuubi

• [DPE-2858] Basic charm functionalities and testing (s3 support)
• [DPE-4324] Support for Azure Object Storage and ADLSv2
• [DPE-4349] Enable Kyuubi server high-availability
• [DPE-4351] Canonical Observability Stack integration
• [DPE-5783] Support for external access with Kubernetes NodePort and LoadBalancer
• [DPE-5805] Enable JDBC endpoint encryption
• [DPE-7087;DPE-4350] In-place upgrades with v3 user-experience
• [DPE-7089] Canonical built artifact to provide bug fixes and security patching
• [DPE-7090] Backup and Restore

Spark History Server

• [DPE-2471] Upgrade to Juju 3
• [DPE-2859] OIDC authentication
• [DPE-3491] Add tls-chain support in the Spark History Server
• [DPE-4324] Support for Azure Object Storage and ADLSv2
• [DPE-4351] Canonical Observability Stack integration
• [DPE-6266] Prepare charm for charmcraft 3

Spark Integration Hub

• [DPE-2861] Basic charm functionalities and testing (s3 support)
• [DPE-3724] Create Integration Hub OCI image
• [DPE-4324] Support for Azure Object Storage and ADLSv2
• [DPE-4351] Canonical Observability Stack integration
• [DPE-5794] Implementation of client <> server charm relation

Spark Client Snap

• [DPE-2164] Expose Spark SQL in Spark Client snap
• [DPE-3552] Add self signed certificate handling to the Spark-Client on spark-submit
• [DPE-4482] Add tests for Azure Object Storage and ADLSv2
• [DPE-7131] Add beeline client to snap

Charmed Spark OCI Images

• [DPE-2164] Add Spark SQL into Charmed Spark Rock image
• [DPE-2858] Add Apache Kyuubi Image
• [DPE-3012] Add Volcano-integration binaries
• [DPE-3104] Create Jupyter notebook image
• [DPE-3194] Integrate Apache Iceberg jars with Rock Image
• [DPE-3514] Add Apache Kyuubi entrypoint to Charmed Spark base rock image
• [DPE-4324] Support for Azure Blob Storage and ADLSv2
• [DPE-4327] Create Spark Image with RAPIDS library for GPU support
• [DPE-4351] Canonical Observability Stack integration

Bug fixes

Spark History Server

• [DPE-5036;DPE-3666] Create S3 bucket if not existing
• [DPE-6861] Invalid proxy base when exposed under a subdomain

Spark Client Snap

• [DPE-3472] Fix error when using S3 with Pyspark or Spark shell
• [DPE-3621] Fix usage of KUBECONFIG env variable

Charmed Spark OCI Images

• [DPE-3017] Patch Pebble issue on shutdown on failure

Breaking changes

Charmed Spark OCI Images

• [DPE-3067] Changes following up entrypoint service refactoring
  (This change removed the python files needed by jupyter service, as the jupyter binaries were moved into a separate image to reduce the security surface of the image)

Other improvements

• [DPE-4325] Security hardening documentation
• [DPE-4326] SSDLC compliance and artifacts
• [DPE-508;DPE-1748;DPE-7089] Artifacts & distributions built from source
• [DPE-5849] Kyuubi usage documentation

Security

The new artifacts have been fixing the following CVEs:

Compatibility