GCP Data Engineer

Exp- 5 to 10 years

Notice- Immediate

Location-Pan India.

5 to 10 years of experience GCP environment.

Expertise in Apache beam, Big-Query, Dataflow, Python, SQL

Experience in DataProc, Composer, Airflow, Pyspark

Job Summary:

We are seeking a skilled GCP Data Engineer to design, build, and maintain scalable data pipelines and infrastructure on Google Cloud Platform (GCP). The ideal candidate will be responsible for managing large datasets, transforming raw data into usable formats, and enabling data-driven decision-making across the organization. As a GCP Data Engineer, you will collaborate with data scientists, analysts, and software engineers to ensure reliable data flow, optimize performance, and support analytics and machine learning initiatives. You should have deep expertise in cloud technologies, big data tools, and programming, especially within the GCP ecosystem.

Key Responsibilities:

1. Data Pipeline Design and Development

• Architect, build, and maintain robust ETL/ELT pipelines on GCP using services such as Dataflow, Dataproc, Cloud Composer, and Cloud Functions.
• Ingest, transform, and process large volumes of structured and unstructured data from multiple sources (databases, APIs, streaming data).
• Develop automated workflows for data ingestion, validation, and enrichment.

2. Data Storage and Management

• Design and implement data storage solutions using BigQuery, Cloud Storage, Cloud SQL, and other GCP data storage services.
• Optimize data storage schemas and partitioning strategies for performance and cost-efficiency.
• Manage data lifecycle, retention policies, and data governance practices.

3. Collaboration with Analytics and Data Science Teams

• Work closely with data analysts and data scientists to understand data requirements and provide clean, structured datasets.
• Support machine learning model development by preparing training datasets and implementing data pipelines for model deployment.
• Implement data quality checks and monitor data integrity.

4. Cloud Infrastructure and Automation

• Use Infrastructure as Code (IaC) tools like Terraform or Deployment Manager to automate GCP resource provisioning.
• Monitor and troubleshoot pipeline failures and cloud resource utilization.
• Implement security best practices, including IAM policies, encryption, and access controls.

5. Performance Optimization and Cost Management

• Monitor query performance in BigQuery and optimize SQL queries for speed and cost.
• Analyze and optimize data pipeline performance using GCP monitoring tools such as Stackdriver.
• Identify opportunities to reduce cloud costs while maintaining system reliability.

6. Documentation and Reporting

• Document data architecture, pipeline designs, and operational procedures.
• Provide regular reports on data pipeline status, data quality metrics, and infrastructure usage.
• Maintain knowledge base for best practices and lessons learned.