Job Summary
The Lead AWS Data Engineer provides technical leadership and hands on execution for enterprise data platforms hosted on Amazon Web Services (AWS). This role leads the design migration modernization and operation of cloud native data architectures supporting mission critical financial advisor payout mobility and corporate analytics platforms.
Responsibilities
Role Summary
The Lead AWS Data Engineer provides technical leadership and hands on execution for enterprise data platforms hosted on Amazon Web Services (AWS). This role leads the design migration modernization and operation of cloud native data architectures supporting mission critical financial advisor payout mobility and corporate analytics platforms.
The Lead AWS Data Engineer owns end to end technical decision making for AWS data platforms including architecture security orchestration and production readiness. The role requires deep expertise in AWS data services Python and PySpark development workflow orchestration data lake design infrastructure as code and operational excellence along with the ability to mentor engineers and partner effectively with infrastructure IAM and database teams.
________________________________________
Key Responsibilities
Technical Leadership & Platform Ownership
Serve as the technical lead and design authority for AWS data engineering initiatives across multiple enterprise platforms.
Own architectural decisions related to scalability reliability security and cost optimization of AWS data platforms.
Define and enforce engineering standards coding patterns and operational best practices for cloud data pipelines.
Provide hands on technical guidance design reviews and code reviews for data engineers.
________________________________________
AWS Data Platform Engineering
Lead the design development and support of cloud native data pipelines using Amazon S3 AWS Glue (PySpark) MWAA (Apache Airflow) and AWS Step Functions.
Drive on premises to AWS data platform migrations including reverse engineering of legacy ETL workflows and re implementation using AWS native services.
Re architect legacy Oracle Data Integrator (ODI) based ETL processes into scalable PySpark based Glue jobs.
Optimize Spark workloads for performance memory usage and cost efficiency in AWS Glue environments.
________________________________________
Data Lake Iceberg & Architecture Design
Architect and implement enterprise AWS data lakes using Medallion architecture (Bronze Silver Gold).
Design and manage Apache Iceberg tables to support incremental processing schema evolution and efficient data lake operations.
Establish standardized ingestion transformation and consumption patterns across financial mobility and corporate datasets.
Ensure data quality reconciliation lineage and auditability across all layers of the data platform.
________________________________________
Workflow Orchestration & Automation
Lead orchestration strategy using MWAA (Managed Workflows for Apache Airflow).
Design and implement Airflow DAGs in Python to orchestrate end to end workflows including Glue jobs validations and downstream dependencies.
Implement scheduling retry logic monitoring and failure handling to ensure resilient and scalable pipelines.
Integrate orchestration workflows with AWS services such as S3 Glue Athena Iceberg based data lakes and downstream systems.
________________________________________
Security Infrastructure & AWS Networking
Drive implementation of AWS security best practices including IAM role design least privilege access encryption using AWS KMS and secrets management.
Lead configuration of AWS networking components such as VPC Endpoints (VPCE) to enable secure service to service communication.
Manage infrastructure provisioning using Terraform ensuring repeatable and auditable deployments across DEV QA and PROD environments.
Coordinate with IAM network DevOps and DBA teams to resolve access firewall and Oracle database connectivity challenges.
________________________________________
Production Readiness Operations & Support
Own production readiness for AWS data platforms including configuration secrets access controls and deployment planning.
Act as the escalation point for complex production issues performing root cause analysis and permanent fixes.
Implement logging metrics and alerting using Amazon CloudWatch to meet enterprise SLAs and availability targets.
Support parallel run and hybrid architectures during migration phases to ensure business continuity.
________________________________________
Automation Compliance & Regulatory Enablement
Design and oversee Python based automation solutions supporting operational efficiency and compliance initiatives (e.g. file retention and document processing).
Ensure pipeline designs meet regulatory audit and enterprise governance requirements including traceability and controlled data handling.
________________________________________
Collaboration & Stakeholder Engagement
Partner closely with data architects DevOps teams infrastructure teams and business stakeholders to deliver AWS data solutions aligned with enterprise strategy.
Translate business and platform requirements into scalable technical designs and execution plans.
Produce technical documentation and support knowledge transfer to enable long term platform sustainability.
________________________________________
Required Qualifications
Bachelor degree in Computer Science Engineering or related field (or equivalent practical experience).
Extensive hands on experience designing and leading AWS data engineering solutions.
Proven experience leading or owning on premises to AWS data platform migrations.
Advanced proficiency in Python and PySpark for data processing automation and orchestration.
Strong experience with AWS Glue Amazon S3 MWAA (Airflow) Step Functions CloudWatch and related services.
Experience with Infrastructure as Code preferably Terraform.
Strong understanding of data engineering principles including ETL design data modeling and pipeline optimization.
Demonstrated experience supporting enterprise production grade data platforms.
________________________________________
Preferred / Nice to Have Qualifications (Not Mandatory)
Experience with Apache Iceberg or similar data lake table formats.
Exposure to analytics or BI platforms (e.g. ThoughtSpot Tableau).
Exposure to Oracle databases or legacy ETL tools (e.g. ODI).
Experience in financial services or regulated enterprise environments.
Familiarity with CI/CD practices for data engineering workloads.
________________________________________
About Cognizant:
Cognizant (Nasdaq: CTSH) is an AI Builder and technology services provider, bridging the gap between AI investment and enterprise value by building full-stack AI solutions for our clients. Our deep industry, process and engineering expertise enables us to build an organization’s unique context into technology systems that amplify human potential, drive tangible outcomes and keep global enterprises ahead in a fast-changing world. See how at cognizant.ai or @cognizant.
Additional employment information
Compensation information is accurate as of the date of this posting. Cognizant reserves the right to modify this information at any time, subject to applicable law.
Language requirements vary depending on roles, but we ask that all candidates have basic English proficiency for company-wide communications purposes. For roles based in Quebec, professional English proficiency is required, as you’ll deliver services to and collaborate with stakeholders outside the province who may not speak French.
Cognizant is an equal opportunity employer. Your application and candidacy will not be considered based on race, color, sex, religion, creed, sexual orientation, gender identity, national origin, disability, genetic information, pregnancy, veteran status or any other characteristic protected by federal, provincial or local laws.
If you have a disability that requires reasonable accommodation to search for a job opening or submit an application, please email [email protected] with your request and contact information.
