Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 64349 Views
- 171 replies
- 92 kudos
12-09-2025
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 4948 Views
- 4 replies
- 6 kudos
4 weeks ago
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 3779 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 479 Views
- 1 replies
- 2 kudos
Tuesday
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 726 Views
- 12 replies
- 18 kudos
Wednesday
The calendar changed. The conversations didn’t. The year turned somewhere between a solved thread and a new question. No pause. No reset – just learning, continuing right where it left off. Here’s how it unfolded Voices That Moved the Conversatio...
- 204 Views
- 2 replies
- 6 kudos
Tuesday
Community Activity
- 20 Views
- 2 replies
- 0 kudos
checkpoint changes not working on my databricks job
Hi,I do have a job processing kafka stream using kafka.readstream process and due to some issue we changed the checkpoint path to other path and it pulled all the records and later when i changed to the original checkpoint location it is not pulling ...
- 20 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
As I understand, you want to ingest "only new records coming through Kafka via old checkpoint path".I would suggest that you can change .option("startingOffsets", "earliest") to .option("startingOffsets", "latest"). It would start reading from the ne...
- 0 kudos
- 10 Views
- 0 replies
- 0 kudos
Capture magic command
%%capture magic command not only suppresses cell output but also assigns it to a variable. You can later print cell output just by using the standard print() function #databricksRead about 12 magic commands in my blogs:- https://www.sunnydata.ai/blog...
- 10 Views
- 0 replies
- 0 kudos
- 23 Views
- 1 replies
- 1 kudos
Identifiers Everywhere
Last from "everywhere" improvements in Spark 4.1 / Runtime 18 is IDENTIFIER(). Lack of support for IDENTIFIER() in many places is a major pain, especially when creating things like Materialized Views or Dashboard Queries. Of course, we need to wait a...
- 23 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks for sharing @Hubert-Dudek !
- 1 kudos
- 15 Views
- 0 replies
- 0 kudos
https://app.thebricks.com/file/384e226b-2ed0-4b41-8a09-238b8ae5011a/171@123428df-b694-490c-a24d-7eeb
https://app.thebricks.com/file/384e226b-2ed0-4b41-8a09-238b8ae5011a/171@123428df-b694-490c-a24d-7eeb33dda17f:0/visual-board
- 15 Views
- 0 replies
- 0 kudos
- 37 Views
- 1 replies
- 0 kudos
Unable to Connect to Oracle from Databricks UC Cluster (DBR 15.4) – ORA-12170 Timeout Error
Hi all,I’m trying to connect to an Oracle database from my Databricks UC cluster (DBR 15.4) using the ojdbc8.jar driver, which I’ve installed on the cluster. Here’s the code I’m using:df = spark.read.format("jdbc")\ .option("url", jdbc_url)\ ...
- 37 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @SatabrataMuduli ,I'm quite sure this is networking issue. You didn't provide much detalis about your environment , so I'll give you general advise. You cannot reach on premise oracle database unless networking is explicitly configured or your dat...
- 0 kudos
- 82 Views
- 2 replies
- 0 kudos
Resolved! Looking for a Databricks Data Engineer Associate
Hi everyone,I’m currently preparing for a Databricks certification and wanted to check if there are any exam vouchers, discounts, or community programs available. Paying the full exam fee is a bit difficult for me at the moment.If anyone has informat...
- 82 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Siladitya ,You're lucky Currently, there is a Databricks Learning Festival. You can get 50% discount for any exam if you complete one learning path on databricks academy. Self-Paced Learning Festival: 09 January - 30 Janu... - Databricks Communi...
- 0 kudos
- 21 Views
- 0 replies
- 0 kudos
how to create a workspace with community edition?
how to create a workspace with community edition?workspace is not being created after login to community edition.
- 21 Views
- 0 replies
- 0 kudos
- 42 Views
- 1 replies
- 0 kudos
Regarding - Managed vs External volumes and tables
From a creation perspective, the steps for managed and external volumes appear almost identical:Both require a storage credentialBoth require an external locationBoth point to customer-owned S3So what exactly makes a volume “managed” vs “external”?Wh...
- 42 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Managed and external volumes may look the same because both store data in the customer’s S3 and use customer IAM roles. However, the real difference is who controls the data folder.With a managed volume, Databricks creates the folder in S3 and contro...
- 0 kudos
- 957 Views
- 4 replies
- 6 kudos
[PARTNER BLOG] Building an Agent-Native Data Quality Manager with Databricks Apps and DQX
Introduction: The Data Quality ParadoxWhat We BuiltThe Technology StackArchitecture Deep DiveThe Four-Layer ArchitectureLayer 1: UsersLayer 2: Databricks Apps PlatformLayer 3: Compute LayerLayer 4: Data LayerWhy This Design?The User Experience: A Com...
- 957 Views
- 4 replies
- 6 kudos
Latest Reply
- 6 kudos
Hi @Shivika , @akjain , @Mario_D The above provided links should work now. Please let me know, if it doesn't. Best, Nivethan V
- 6 kudos
- 4186 Views
- 4 replies
- 5 kudos
How to use Service Credentials to enable the RBAC on the Key Vaults
How to use Service Credentials in Unity Catalog for Azure Key Vault RBAC Integration Azure Databricks now supports a service credential object in Unity Catalog, enabling secure and governed access to external cloud services like Azure Key Vault. A se...
- 4186 Views
- 4 replies
- 5 kudos
Latest Reply
- 5 kudos
I get same response as @eminescence today 10 Jan 2026. On Serverless. Then [Diagnose Error] says: "The CREATE SERVICE CREDENTIAL statement is not supported in Databricks SQL. To access Azure Key Vault secrets in Databricks, you should use Databricks...
- 5 kudos
- 66 Views
- 2 replies
- 0 kudos
Feature request: Allow to set value as null when not present in schema evolution
I want to raise a feature request as follows.Currently, in the Automatic schema evolution for merge when a column is not present in the source dataset it is not changed in the target dataset. For e.g.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.t...
- 66 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@Dhruv-22 ProblemWhen using MERGE INTO ... WITH SCHEMA EVOLUTION, if a column exists in the target table but is not present in the source dataset, that column is left unchanged on matched rows.Solution ThinkingThis can be emulated by introspecting th...
- 0 kudos
- 37 Views
- 0 replies
- 1 kudos
Secret magic commands
Secret magic commands, there are a lot of them. Check my blogs to see which one can simplify your daily work. First one is %%writefile, which can be used to write a new file, for example, another notebook #databricks more magic commands:- https://dat...
- 37 Views
- 0 replies
- 1 kudos
- 643 Views
- 2 replies
- 2 kudos
FAQ for Databricks Learning Festival (Virtual): 09 January - 30 January 2026
General Q: How can I check whether I have completed the required modules? Login to your Databricks Customer Academy Account and select ‘My Activities’ within the user menu in the top left. Under the Courses tab, you can verify whether all courses wit...
- 643 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Jim, I believe this post might have a typo. "The voucher will be valid for approximately 90 days and will expire before 30 January 2026. You must schedule and sit for the exam before this date. We strongly recommend booking the slots early, since ...
- 2 kudos
- 65 Views
- 3 replies
- 2 kudos
Data validation with df writes using append mode
Hi Team,Recently i came across a situation where I had to write a huge data and it took 6 hrs to complete...later when i checked the target data , I saw 20% of the total records written incorrectly or corrupted because the source data itself was corr...
- 65 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Hey @K_Anudeep , thanks a lot for tagging me into the GitHub issue.. This is exactly what I want " validate and commit" feature and i se you have already raised a PR for the same with a new option called . I will try this out and check if it satisfie...
- 2 kudos
- 76 Views
- 5 replies
- 2 kudos
Updating Job Creator to Service Principal
Regarding data governance best practices: I have jobs created by a user who has left the organization, and I need to change the job creator to a service principal. Currently, it seems the only option is to clone the job and update it. Is this the rec...
- 76 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
I agree with @nayan_wylde , for auditing, creator is important and it should in immutable by nature.
- 2 kudos
Announcements
| User | Count |
|---|---|
| 2234 | |
| 1221 | |
| 1068 | |
| 976 | |
| 950 |
Latest from our Blog
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
[PARTNER BLOG] Building an Agent-Native Data Quality Manager with Databricks Apps and DQX
Introduction: The Data Quality ParadoxWhat We BuiltThe Technology StackArchitecture Deep DiveThe Four-Layer ArchitectureLayer 1: UsersLayer 2: Databricks Apps PlatformLayer 3: Compute LayerLayer 4: Da...
[PARTNER BLOG] Zerobus Ingest on Databricks
Introduction TL;DR ZeroBus Ingest is a serverless, Kafka-free ingestion service in Databricks that allows applications and IoT devices to stream data directly into Delta Lake with low latency and mini...
[PARTNER BLOG] Beyond ADLS Limits: File Arrival Triggers for File Updates
Beyond ADLS Limitations: Making File Arrival Triggers Work for Existing File Updates Using a Flag File MechanismThe Flag File MechanismThe Root Problem: Triggers Only Work on “Create”, Not “Modify” Ev...
