When working with Databricks pipelines and workflows, failures are inevitable. While Databricks provides built-in notifications for job failures, these alerts are often not customizable and may not fit specific reporting or formatting needs. A more flexible and cost-effective approach is to set up custom email notifications that include pipeline details and error messages in a structured format, such as an Excel attachment. Databricks pipelines and workflows not customizable custom email notifications This blog walks through approaches to implement custom notifications using SMTP, SendGrid, Azure Logic Apps, and Microsoft Graph API. SMTP SendGrid Azure Logic Apps Microsoft Graph API Why Custom Notifications? Why Custom Notifications? Flexible formatting: Include pipeline metadata, error messages, and runtime details.
Attachments: Share structured reports (Excel, CSV, etc.) instead of plain text.
Cost efficiency: Avoid additional third-party monitoring solutions.
Integration options: Easily plug into existing email infrastructure. Flexible formatting: Include pipeline metadata, error messages, and runtime details. Flexible formatting Attachments: Share structured reports (Excel, CSV, etc.) instead of plain text. Attachments Cost efficiency: Avoid additional third-party monitoring solutions. Cost efficiency Integration options: Easily plug into existing email infrastructure. Integration options Approach 1: SMTP-Based Notifications Approach 1: SMTP-Based Notifications Using SMTP with Python inside a Databricks notebook, you can generate an Excel report and send it via email whenever a pipeline fails. SMTP Example Implementation Example Implementation import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email.mime.text import MIMEText
from email import encoders
from io import BytesIO
import pandas as pd

#Sample pipeline history
df = spark.createDataFrame([ ('pipeline1', 'success', '7min'), ('pipeline1', 'fail', '3min'), ('pipeline1', 'success', '10min') ], ["PipelineName", "Status", "Duration"])

# Convert DataFrame to Excel
output = BytesIO() 
with pd.ExcelWriter(output, engine='xlsxwriter') as writer: 
df_pd = df.toPandas() 
df_pd.to_excel(writer, index=False, sheet_name='Sheet1') 
workbook = writer.book 
worksheet = writer.sheets['Sheet1'] import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email.mime.text import MIMEText
from email import encoders
from io import BytesIO
import pandas as pd

#Sample pipeline history
df = spark.createDataFrame([ ('pipeline1', 'success', '7min'), ('pipeline1', 'fail', '3min'), ('pipeline1', 'success', '10min') ], ["PipelineName", "Status", "Duration"])

# Convert DataFrame to Excel
output = BytesIO() 
with pd.ExcelWriter(output, engine='xlsxwriter') as writer: 
df_pd = df.toPandas() 
df_pd.to_excel(writer, index=False, sheet_name='Sheet1') 
workbook = writer.book 
worksheet = writer.sheets['Sheet1'] # Apply formatting
header_format = workbook.add_format({
    'bold': True,
    'bg_color': '#FFF00',
    'border': 1,
    'align': 'center',
    'valign': 'vcenter'
})
for col_num, value in enumerate(df_pd.columns):
    worksheet.write(0, col_num, value.upper(), header_format)

cell_format = workbook.add_format({'border': 1})
for row in range(1, len(df_pd) + 1):
    for col in range(len(df_pd.columns)):
        worksheet.write(row, col, df_pd.iloc[row-1, col], cell_format)

for i, col in enumerate(df_pd.columns):
    worksheet.set_column(i, i, 20)
output.seek(0)

# Email configuration
sender = "from@example.com"
receiver = "to@example.com"
subject = "Pipeline Execution Report"
body = """Hello Team,

Please find the attachment of the latest pipeline report.

Thanks,
Pipeline Team"""

msg = MIMEMultipart()
msg['From'] = sender
msg['To'] = receiver
msg['Subject'] = subject
msg.attach(MIMEText(body, 'plain'))

part = MIMEBase('application', 'vnd.openxmlformats-officedocument.spreadsheetml.sheet')
part.set_payload(output.read())
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="pipeline_report.xlsx"')
msg.attach(part)

smtp_server = "smtp.office.com"
smtp_port = 587

with smtplib.SMTP(smtp_server, smtp_port) as server:
    server.starttls()
    server.login(sender, "sender_password")
    server.send_message(msg)

print("Email sent successfully with Excel attachment") # Apply formatting
header_format = workbook.add_format({
    'bold': True,
    'bg_color': '#FFF00',
    'border': 1,
    'align': 'center',
    'valign': 'vcenter'
})
for col_num, value in enumerate(df_pd.columns):
    worksheet.write(0, col_num, value.upper(), header_format)

cell_format = workbook.add_format({'border': 1})
for row in range(1, len(df_pd) + 1):
    for col in range(len(df_pd.columns)):
        worksheet.write(row, col, df_pd.iloc[row-1, col], cell_format)

for i, col in enumerate(df_pd.columns):
    worksheet.set_column(i, i, 20)
output.seek(0)

# Email configuration
sender = "from@example.com"
receiver = "to@example.com"
subject = "Pipeline Execution Report"
body = """Hello Team,

Please find the attachment of the latest pipeline report.

Thanks,
Pipeline Team"""

msg = MIMEMultipart()
msg['From'] = sender
msg['To'] = receiver
msg['Subject'] = subject
msg.attach(MIMEText(body, 'plain'))

part = MIMEBase('application', 'vnd.openxmlformats-officedocument.spreadsheetml.sheet')
part.set_payload(output.read())
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="pipeline_report.xlsx"')
msg.attach(part)

smtp_server = "smtp.office.com"
smtp_port = 587

with smtplib.SMTP(smtp_server, smtp_port) as server:
    server.starttls()
    server.login(sender, "sender_password")
    server.send_message(msg)

print("Email sent successfully with Excel attachment") Scheduling Notifications Scheduling Notifications You can automate the notification trigger by scheduling the notebook: Option 1: Databricks Jobs Option 1: Databricks Jobs Create or edit a Databricks job.
Add a task dependency so the notification script runs only if the previous task fails.
This ensures error details are captured and reported immediately. Create or edit a Databricks job. Add a task dependency so the notification script runs only if the previous task fails. if the previous task fails This ensures error details are captured and reported immediately. Option 2: Azure Logic Apps Option 2: Azure Logic Apps Configure a Logic App that listens for pipeline failures.
Pass pipeline details and attachments via an API call in JSON format.
Logic Apps handle email delivery and retry mechanisms. Configure a Logic App that listens for pipeline failures. Pass pipeline details and attachments via an API call in JSON format. Logic Apps handle email delivery and retry mechanisms. Conclusion Conclusion While Databricks provides basic failure notifications, extending them with custom SMTP or Logic App workflows ensures: custom SMTP or Logic App workflows Rich, formatted reports.
Team visibility with detailed context.
Seamless integration with enterprise communication tools. Rich, formatted reports. Team visibility with detailed context. Seamless integration with enterprise communication tools. This approach is cost-effective, scalable, and easily adaptable for large-scale pipeline monitoring.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Databricks

Seamless

Custom Email Notifications for Databricks Pipeline Failures

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Practical Guide to the Saga Pattern in Spring Boot Microservices

Help! My Azure Site Performance Sucks! — Part 1

Async vs Sync Benchmark (.NET): The Difference Between Asynchronous and Synchronous Methods

Building a CI Pipeline with Databricks dbx Tool and GitLab

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Optimize Your Data Engine With Data as a Service (DaaS) and Multi-Tenancy

A Practical Guide to the Saga Pattern in Spring Boot Microservices

Help! My Azure Site Performance Sucks! — Part 1

Async vs Sync Benchmark (.NET): The Difference Between Asynchronous and Synchronous Methods

Building a CI Pipeline with Databricks dbx Tool and GitLab

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Optimize Your Data Engine With Data as a Service (DaaS) and Multi-Tenancy

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps