GCP, Biq Query, Data Flow, Dataproc 88003

Cephas Consultancy Services Private Limited • Chennai, Tamil Nādu, India • 2h ago

Positions:2 Full Time
Experience
7 - 11 Years

Job Description: Senior Data Engineer

Position Overview

We are seeking a talented and experienced Senior Data Engineer (Engineer 3) to join our organization. In this role, you will be responsible for designing, building, and maintaining comprehensive data solutions including data infrastructure, pipelines, and platforms for collecting, storing, processing, and analyzing large volumes of data efficiently and accurately. You will collaborate with business and technology stakeholders to deliver scalable, reliable, and cost-effective data solutions that support analytics, machine learning, and business intelligence initiatives.

Key Responsibilities

Collaborate with business and technology stakeholders to understand current and future data requirements and translate them into actionable data solutions
Design, build, and maintain reliable, efficient, and scalable data infrastructure for data collection, storage, transformation, and analysis
Plan, design, build, and maintain scalable data solutions including data pipelines, data models, and applications for efficient and reliable data workflow
Design, implement, and maintain existing and future data platforms such as data warehouses, data lakes, and data lakehouses for both structured and unstructured data
Design and develop analytical tools, algorithms, and programs to support data engineering activities including writing scripts and automating tasks
Ensure optimum performance of data systems and identify improvement opportunities through monitoring and optimization
Design and implement data architecture with specialized knowledge in various data transformation mechanisms
Leverage big data tools and programming frameworks that meet organizational requirements for data storage, processing, analytics, and data science while considering long-term strategy, tool selection, and cost-effectiveness
Deliver appropriate analytical data modeling techniques based on data usage needs for Data Science, Machine Learning, Visualization, and other business requirements
Develop scalable and maintainable data solutions that support organizational growth
Incorporate CI/CD pipelines to improve efficiency and quality of development and deployment processes, accelerate release cycles, and enhance overall product stability
Establish benchmarks and adopt proactive monitoring and reliability techniques to ensure data system performance and availability
Adopt and implement appropriate data governance processes to ensure compliance with data protection and data safeguard requirements at both regional and organizational levels
Own and execute initiatives and projects within timelines, mitigate risks, and ensure delivery of high-quality solutions on schedule
Participate in periodic retrospectives to identify opportunities for continuous improvement
Mentor junior team members and contribute to team excellence by proactively identifying and resolving roadblocks
Productionize components and models at scale, working effectively with large data volumes
Stay current with emerging technologies and introduce innovative solutions and best practices to the organization
Maintain strong communication with cross-functional teams and stakeholders through clear oral, written, and interpersonal communication

Required Qualifications

Experience: 7+ years of professional Data Engineering work experience
Google Cloud Platform (GCP) Expertise: Hands-on experience with BigQuery, Dataflow, Dataproc, Data Fusion, and Cloud SQL
Infrastructure & Orchestration Tools: Proficiency in Terraform, Tekton, and Apache Airflow
Data Processing & Streaming: Strong experience with Kafka, Pub/Sub, and PySpark
Programming Languages: Expert-level proficiency in Python and API development
Database Technologies: Solid experience with PostgreSQL and relational database design
Data Architecture: Demonstrated expertise in designing and implementing scalable data solutions and data models
Big Data Technologies: Strong knowledge of big data tools and frameworks for large-scale data processing
CI/CD & DevOps: Understanding of continuous integration and continuous deployment practices
Problem-Solving: Strong analytical and troubleshooting skills with ability to optimize data systems
Communication: Excellent oral, written, and interpersonal communication skills with ability to work across teams

Preferred Qualifications

Experience with Jira for project tracking and collaboration
Knowledge of Data Governance frameworks and best practices
Experience with machine learning platforms and model deployment
Familiarity with data visualization tools and business intelligence platforms
Background in cloud data architecture and multi-cloud environments
Experience with containerization technologies and Kubernetes

Key Competencies

Technical Excellence: Deep expertise in data engineering technologies and best practices
Scalability & Performance: Ability to design systems that handle large data volumes and optimize for performance
Architecture Design: Capability to design comprehensive data solutions aligned with business requirements
Data Governance & Compliance: Understanding of data protection regulations and implementation of governance frameworks
Ownership & Accountability: Takes ownership of projects and delivers results within timelines
Collaboration: Works effectively with cross-functional teams including data scientists, analysts, and business stakeholders
Innovation & Learning: Proactive in learning new technologies and introducing innovative solutions
Mentorship: Ability to mentor and guide junior team members
Reliability & Monitoring: Establishes monitoring practices and ensures system reliability and uptime
Self-Motivation: Self-driven professional who takes initiative and contributes to team success

What We're Looking For

We seek an experienced Data Engineer who is:

Inquisitive and proactive with a genuine interest in learning new tools and techniques
Capable of working independently while being a strong team player
Familiar with productionizing components and models at scale
Committed to staying current with technology trends and industry best practices
Able to communicate complex technical concepts clearly to both technical and non-technical audiences
Driven to deliver high-quality solutions that meet business objectives

Role Level: Engineer 3 (Senior)

This is a senior-level position requiring 7+ years of experience. The successful candidate will demonstrate leadership qualities, technical depth, and the ability to influence data engineering practices across the organization.