Databricks Platform Expert

Databricks Platform Expert

- - - - - - - - - - - -

Key Responsibilities

Databricks Platform Expert

• Manage, configure, and administer Databricks workspaces, Clusters, SQL Warehouses, Serverless, jobs, and workspace objects.

• Implement and manage Unity Catalog, including catalogs, schemas, tables, access controls, and data lineage.

• Optimize cluster policies, auto-scaling strategies, and cost management for Serverless and Classic compute.

• Serve as the SME for Databricks infrastructure, governance, and security best practices.

• Monitor workspace performance, cluster stability, logs, job reliability, and platform health.

• Implement CI/CD pipelines for notebooks, jobs, and Delta Live Tables using Git integration.

• Support user provisioning, access controls (ACLs), secrets management, and workspace SSO.

• Write efficient Spark (PySpark / SQL / Scala) code for ETL, data transformations, and pipeline optimizations.

• Assist data engineering teams with Spark job debugging, performance tuning, and code reviews.

• Build and maintain production-grade pipelines leveraging Delta Lake, Databricks Jobs, and DLT.

• Implement and manage RBAC, SCIM provisioning, AIM, service principals, and cluster access controls.

• Ensure compliance with enterprise data governance, audit, and logging requirements.

• Manage secrets Key Vault and enforce secure credential handling.

• Support audit reports, compliance reviews, and workspace security configuration.

• Monitor job failures, cluster lifecycle performance, and system events using Databricks logs and cloud-native monitoring tools (Azure Monitor).

• Create automated alerts and observability dashboards for platform usage, cost, and performance.

• Troubleshoot Databricks runtime issues, library conflicts, and Spark execution failures.

• Collaborate with cloud and network teams on VNet, peering, and private-link connectivity issues.

• Develop cost governance policies for cluster sizes, job policies, and SQL Warehouse tiers.

• Identify opportunities to reduce cost via autoscaling, spot instances (classic clusters), and job consolidation.

 

Required Qualifications

• 4–6 years of experience working with Databricks as an administrator or data engineer.

• Strong expertise in Apache Spark programming (PySpark preferred; SQL or Scala is a plus).

• Hands-on experience with Databricks Jobs, cluster configuration, SQL Warehouses, and Unity Catalog.

• Deep understanding of Delta Lake, ACID transactions, and lakehouse architecture.

• Experience with Git, CI/CD, and DevOps concepts for data engineering workflows.

• Knowledge of cloud platforms ( Azure).

• Familiarity with IAM, networking basics, monitoring tools, and security patterns