Databricks Engineer Job at Conflux Systems, Phoenix, AZ

  • Conflux Systems
  • Phoenix, AZ

Job Description

Databricks Engineer

We are seeking a highly skilled Databricks Engineer to lead the design, implementation, and optimization of scalable cloud-based data platforms. This role requires deep expertise in Lakehouse architecture, Databricks, Apache Spark, and Delta Lake, along with proven experience delivering end-to-end enterprise data engineering solutions.

Key Responsibilities

  • Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
  • Lead implementation of:
  • Cloud-based data lakehouse platforms integrating diverse data sources.
  • Real-time data processing pipelines for operational and analytical use cases.
  • Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.
  • Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
  • Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
  • Establish data quality frameworks and governance standards.
  • Implement and manage Unity Catalog, including fine-grained security and access controls.
  • Leverage Databricks components such as:
  • Delta Live Tables
  • Autoloader
  • Structured Streaming
  • Databricks Workflows
  • Integration with orchestration tools (e.g., Apache Airflow)
  • Drive CI/CD automation, deployment strategies, and DevOps best practices.
  • Optimize performance of pipelines, Spark jobs, and compute resources.
  • Provide architectural guidance and technical leadership across cross-functional teams.
  • Engage with stakeholders and clients to translate business requirements into scalable technical solutions.

Deep expertise in:

  • Databricks and cloud-native storage/compute platforms
  • Apache Spark (batch & streaming)
  • Delta Lake & Lakehouse architecture
  • Distributed data processing systems
  • Strong hands-on programming skills in Python, PySpark, Scala, and SQL.

Job Tags

Similar Jobs

PATHWAYS HEALTHCARE, LLC

Nurse Educator Job at PATHWAYS HEALTHCARE, LLC

 ...Healthcare is a leading, physician- and nurse practitioner-led home health and hospice...  ...MAService Area: This position requires travel to our multiple locations in New England,...  ...Responsibilitie s:Assist in creating and implementing education curriculum with Clinical Education... 

Amazon

Sr. Learning & Development Specialist , AWS Senior Talent & Transformation Job at Amazon

 ...incorporating adult learning principles, leadership research, and Amazon's Leadership PrinciplesDrive innovation in program design,...  ...country/region youre applying in isnt listed, please contact your Recruiting Partner.The base salary range for this position is listed below... 

US Postal Service

Postal Clerk - Horse Shoe Job at US Postal Service

NO EXPERIENCE REQUIRED PAID TRAINING PROVIDED JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...

Pike School

Director of Marketing and Communications Job at Pike School

 ...execute multi-channel campaigns (digital ads, print collateral, recruitment drives) that...  .... ~ Digital Media & Advertising: Manage the daily operations and growth of the website...  ...) and social channels (Instagram, Facebook, LinkedIn); collaborate with external partners... 

PSI Services

Test Proctor Job at PSI Services

Test Center Administrator Embark on a thrilling journey as a Test Center Administrator at PSI, a pioneering, diverse, and inclusive company revolutionizing the way exams are conducted. As a Test Center Administrator, you'll play a pivotal role in ensuring exam security...