Senior Data Engineer - Azure Databricks

• Design, build, and optimize scalable end-to-end data pipelines for structured and unstructured data from diverse sources (APIs, databases, files).
• Develop and manage robust ETL/ELT workflows for both batch and real-time data processing, ensuring high data quality and reliability.
• Monitor and optimize pipeline performance using autoscaling, caching, and cost-effective strategies.
• Leverage Databricks and Apache Spark (PySpark, Spark SQL) for distributed data processing and transformation.
• Implement data storage solutions using Medallion Architecture and Delta Lake, ensuring schema evolution and ACID compliance.
• Collaborate with data scientists, analysts, engineers, and business stakeholders to align data solutions with business goals.
• Stay current with emerging technologies in data engineering, especially within the Azure and Databricks ecosystem.

• Bachelor’s degree in Computer Science, Information Systems, or a related field.
• 5–10 years of hands-on experience in data engineering, with a strong track record in building and optimizing data pipelines.
• Proven expertise in Databricks, Apache Spark (PySpark, Spark SQL), Python, and SQL.
• Experience integrating Databricks with Azure services (Data Lake Storage, Data Factory, SQL Database, Event Hubs).
• Familiarity with both batch and real-time data processing tools such as Azure Event Hubs and Kafka.
• Proficient in Git and Azure DevOps for CI/CD and source control, including Databricks Asset Bundles.
• Strong understanding of data quality, monitoring, and performance optimization.
• Excellent documentation, collaboration, and communication skills.
• Mature, independent, and customer-oriented with strong time management and problem-solving abilities.

ConnectedGroup