Positions:2 Full Time
Experience
6 - 10 Years
Senior Data Engineer
About the Role
We are seeking an experienced Senior Data Engineer to join our data engineering team. In this role, you will design, develop, and optimize large-scale data pipelines and architectures using modern cloud technologies. You will work with cross-functional teams to build robust ETL/ELT solutions, manage data lakes and lakehouses, and ensure high-performance data processing systems that support critical business intelligence and analytics initiatives.
Key Responsibilities
- Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks
- Develop and optimize ETL/ELT processes for large-scale distributed data processing
- Build and maintain data lake and lakehouse architectures on Azure cloud platform
- Write efficient SQL and PySpark code for data transformation and analysis
- Implement Lakeflow Declarative Pipelines for streamlined data workflows
- Perform performance tuning and optimization of data processing systems
- Manage version control using Git and implement CI/CD pipelines for data solutions
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements
- Ensure data quality, security, and governance standards are met across all pipelines
- Mentor junior team members and contribute to technical best practices
Required Skills & Experience
Mandatory Requirements
- Azure Data Factory (ADF): Strong hands-on experience in designing and managing data pipelines
- Azure Databricks: Proficiency in building and optimizing distributed data processing solutions
- Apache Spark: In-depth knowledge of Spark architecture and distributed computing concepts
- Lakeflow Declarative Pipelines: Practical experience with declarative pipeline development
- SQL: Expert-level proficiency in writing complex queries and data manipulation
- PySpark: Strong programming skills in PySpark for data transformation
- Azure Cloud Platform: Hands-on experience with Azure data services and cloud infrastructure
- ETL/ELT Concepts: Strong understanding of data integration patterns and methodologies
- Data Warehousing Principles: Knowledge of dimensional modeling, schema design, and best practices
- Distributed Data Processing: Experience working with large-scale data systems and handling big data challenges
- Performance Tuning & Optimization: Ability to identify bottlenecks and optimize data pipeline performance
- Data Lake & Lakehouse Architectures: Understanding of modern data architecture patterns and implementation
- Git & CI/CD: Experience with version control systems and continuous integration/deployment practices
Preferred Skills & Experience
- Experience with Delta Lake and Unity Catalog for data governance
- Knowledge of streaming frameworks and real-time data processing
- Exposure to Azure Synapse Analytics or Microsoft Fabric
- Working experience in Agile/Scrum development environments
- Familiarity with data governance frameworks and security best practices
Qualifications
- Bachelor's degree in Computer Science, Engineering, Mathematics, or related field (or equivalent professional experience)
- Minimum 5+ years of experience as a Data Engineer or similar role
- Proven track record of delivering production-grade data solutions
- Strong problem-solving and analytical skills
- Excellent communication and collaboration abilities
- Ability to work independently and as part of a team
What We Offer
- Competitive salary and benefits package
- Opportunity to work with cutting-edge data technologies
- Professional development and training opportunities
- Collaborative and inclusive work environment
- Flexible working arrangements
- Career growth and advancement opportunities
How to Apply
Interested candidates are encouraged to submit their resume, portfolio, and a brief cover letter highlighting relevant experience with the required technologies. Please include examples of data engineering projects you have worked on.
PIef36931c3661-37437-40780962
