Lumenalta
Posted 6 days agoVerified live 1d ago
Senior Databricks Engineer
Brief overview
Remote
UndergradOr in progress
$80k–$150k/yrStated range
4+ yrsMinimum
DatabricksDatabricks platform architecturePythonPySparkSQLTesting frameworks - PyTestGit-based workflowsCI/CD pipelinesDevOps practicesDelta LakeUnity CatalogDatabricks Asset BundlesLakeflow JobsInfrastructure as Code - TerraformAWSAWS S3AWS IAM roles
About the company
Lumenaltalumenalta.com
Lumenalta is a software and app development company that creates unique digital transformation solutions using innovative technology.
Job description
Summary
Lumenalta is seeking a Senior Data Engineer - Databricks with strong expertise in Databricks to help design and implement modern enterprise data platforms. This role involves building scalable pipelines and designing robust data systems in the cloud while contributing to platform architecture and ensuring best practices across development, testing, deployment, and governance.
Responsibilities
- Design and implement scalable enterprise data platforms using Databricks and AWS
- Architect and oversee end-to-end data platform implementations
- Develop robust data pipelines using Python, PySpark, and SQL
- Apply strong engineering practices including testing, CI/CD, and version control
- Implement and manage Databricks platform components including Delta Lake, Unity Catalog, Databricks Asset Bundles, and Lakeflow Jobs
- Collaborate with engineering teams to ensure maintainable, scalable, and secure data solutions
- Implement Infrastructure as Code practices to support repeatable environments
- Contribute to architectural decisions and guide best practices for the data platform
Skills
- Deep expertise in Databricks, including platform architecture and best practices
- Experience as a Solution Architect or Data Platform Owner designing end-to-end implementations
- Strong programming experience with Python, PySpark, and SQL
- Experience using testing frameworks such as PyTest
- Solid experience with Git-based workflows, CI/CD pipelines, and DevOps practices
- Hands-on experience with Delta Lake, Unity Catalog, Databricks Asset Bundles (DABs), and Lakeflow Jobs
- Experience with Infrastructure as Code using Terraform
- Strong AWS experience, particularly S3 and IAM roles
- Strong communication and collaboration skills in distributed teams
- Experience designing lakehouse architectures at scale
- Experience optimizing distributed data workloads
- Experience working in consulting or client-facing engineering environments
- Experience implementing governance, security, and data lineage frameworks
- Strong hands-on experience with AWS Glue (ETL, jobs, crawlers, workflows)
Qualifications
Must Haves
- Deep expertise in Databricks, including platform architecture and best practices
- Experience as a Solution Architect or Data Platform Owner designing end-to-end implementations
- Strong programming experience with Python, PySpark, and SQL
- Experience using testing frameworks such as PyTest
- Solid experience with Git-based workflows, CI/CD pipelines, and DevOps practices
- Hands-on experience with Delta Lake, Unity Catalog, Databricks Asset Bundles (DABs), and Lakeflow Jobs
- Experience with Infrastructure as Code using Terraform
- Strong AWS experience, particularly S3 and IAM roles
- Strong communication and collaboration skills in distributed teams
Nice to Haves
- Experience designing lakehouse architectures at scale
- Experience optimizing distributed data workloads
- Experience working in consulting or client-facing engineering environments
- Experience implementing governance, security, and data lineage frameworks
- Strong hands-on experience with AWS Glue (ETL, jobs, crawlers, workflows)
More jobs like this
