As a Databricks Developer, you will design and build the enterprise data pipelines that power analytics, reporting and AI initiatives for a leading company in the energy sector. Join a fully remote data engineering team working hands-on with cutting-edge Lakehouse technology.
HIGH-IMPACT DATA PROJECTS | LATEST LAKEHOUSE TECH | FULLY REMOTE | LEARNING & GROWTH |
Don't tick every box? If you meet around 70% of the requirements above, we'd still encourage you to apply.
ABOUT THE ROLE
We are looking for a highly skilled Data Engineer with 5+ years of experience to design, build and optimize enterprise data pipelines on the Databricks Lakehouse platform for a leading energy sector company. In this role, you will be the hands-on technical driver responsible for transforming raw data into high-quality, actionable datasets. You will build and maintain a Medallion architecture, optimize Spark workloads, and ensure the data infrastructure seamlessly supports advanced analytics, BI dashboards and emerging Generative AI applications.
KEY RESPONSIBILITIES
Data Pipeline Engineering
● Design, build and maintain scalable, robust ETL/ELT pipelines using Python, SQL and Apache Spark within the Databricks environment.
● Implement and manage a robust Medallion architecture (Bronze, Silver, Gold layers) to process and refine data from diverse sources.
● Develop and maintain the Gold semantic layer specifically optimized for high-performance consumption by BI tools (e.g., Power BI).
Platform Optimization & Architecture
● Optimize Databricks workloads, cluster configurations and Spark queries to ensure high performance and cost efficiency.
● Work extensively with open table formats, specifically Delta Lake and Apache Iceberg, to ensure ACID compliance, time travel and efficient data storage.
● Execute complex data migrations, including transitioning legacy workloads from traditional cloud data warehouses (e.g., AWS Redshift) into the Databricks Lakehouse.
Data Governance & Automation
● Implement data governance and access control policies at the table, row and column levels using Databricks Unity Catalog.
● Automate deployment processes and pipeline orchestration using Databricks Workflows, CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) and tools like Terraform.
● Embed data quality checks and monitoring directly into pipelines to ensure strict Master Data Management (MDM) standards are upheld.
AI & Advanced Analytics Support
● Collaborate closely with Data Scientists and AI Engineers to provision clean, structured data for machine learning model training and inference.
● Support the data foundations required for GenAI frameworks, autonomous agents and AI observability platforms.
REQUIRED SKILLS & EXPERIENCE
● 5+ years of dedicated data engineering experience in an enterprise environment.
● Expert-level proficiency in Python and SQL.
● Extensive hands-on experience with Databricks, Apache Spark and Delta Lake.
● Strong understanding of distributed systems, big data architecture and data modeling techniques (e.g., Kimball, Data Vault).
● Deep familiarity with cloud-native data services (AWS, Azure or GCP), specifically cloud storage (S3/ADLS) and compute provisioning.
● Proven experience with version control (Git), CI/CD methodologies and agile software development life cycles.
NICE TO HAVE
● Experience evaluating and working with Apache Iceberg alongside Delta Lake.
● Familiarity with streaming data architectures (e.g., Structured Streaming, Kafka).
● Experience building backend frameworks or internal tools using lightweight libraries like Streamlit.
EDUCATION
● Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering or a related field.
PREFERRED CERTIFICATIONS
● Databricks Certified Data Engineer Associate or Professional.
● AWS, Azure or GCP data/cloud certifications.
WORKING MODEL
Fully remote position.
关于高知特 (Cognizant)
高知特(Cognizant)（纳斯达克代码：CTSH）作为一家AI Builder和相关技术服务提供商，致力于通过打造全栈AI解决方案，帮助企业将人工智能投资转化为实际价值。公司凭借深厚的行业经验、流程优化和工程技术专长，将企业独特的业务场景融入科技系统，赋能组织释放人才潜能，推动切实成果，并帮助全球企业在瞬息万变的环境中保持领先。如需了解更多详情，敬请访问 cognizant.ai 或关注@cognizant。
补充雇佣信息
薪酬信息截至本职位发布之日为准。Cognizant 保留在适用法律允许的范围内随时修改该信息的权利。
申请人可能需要通过现场面试或视频会议的方式参加面试。此外，候选人在每次面试时可能需要出示其当前所在州或政府签发的有效身份证件。
Cognizant 是一家提供平等就业机会的雇主。在招聘过程中，您的申请和候选资格不会因种族、肤色、性别、宗教、信仰、性取向、性别认同、国籍、残疾、遗传信息、怀孕、退伍军人身份或任何其他受联邦、州或地方法律保护的特征而受到影响。
