Data Pipeline Engineer

New

Skills

Python Spark

Job Overview

As a Data Pipeline Engineer, you will be responsible for designing, developing, and operating large-scale data pipelines to improve internal processes. Your role will involve integrating data sources, writing robust code, and ensuring the efficiency of data warehousing operations.

Responsibilities
  • Design, develop, and operate large-scale data pipelines
  • Improve and automate internal processes
  • Integrate data sources to meet business needs
  • Write robust, maintainable, well-documented code
  • Monitor jobs and pipelines and alert the team
Requirements & Qualifications
  • 2-4 years of Data Eng and data warehousing experience
  • Strong Python, Parquet, Spark, Azure Databricks, Delta Lake skills
  • Experience with Informatica ETL and Sigma analytics environments
  • SQL development: procedures, triggers, indexing, partitioning
  • ETL/ELT and data-warehousing best practices knowledge
  • Experience with CI/CD tools and Azure cloud

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: 12 Months

Share this job:

Similar Jobs

Staff Software Engineer

New

Design complex systems with teams

Build and manage backend services in AWS

Ai AWS C++ Cloud Services

Senior Software Engineer

New

Design and maintain cloud-native infrastructure

Modernize identity infrastructure with Terraform

AWS Gcp Go Iam

Backend Engineer for Fleet Ops

New

Design complex backend systems

Build and extend features for fleet ops

AWS C++ Golang gRPC

Staff Platform Engineer

New

Architect, deploy, and maintain a scalable multi-tenant SaaS environment in AWS

Optimize AWS services for cost, performance, and reliability

AWS CircleCI Cloudformation Github Actions

Senior Solutions Architect - Communications

New

Deliver technical and business value to clients

Engage clients through demos and integrations

Client Relationship Management Cloud Integrations Databricks Java

Senior Solutions Engineer

New

Research customer problems and craft solutions

Own the technical win and drive the deal

Ai Apis Automation LLMs

Lead Tools Engineer

New

Lead design and development of internal tools

Implement and maintain build systems and CI/CD pipelines

AWS Bash Bash Scripting C#

Data Platform Engineer

New

Build scalable and reliable data systems

Collaborate with cross-functional teams

BigQuery Docker Go Kubernetes

ML Engineer - Core Automation

New

Apply ML techniques effectively

Define strategic roadmaps

Data Mining Deep Learning Git Jira

Software Engineer, Account Abuse

New

Implementing defenses against abuse

Collaborating with data scientists

Communication Skills Data Analysis Python Sql

Full Stack Engineer

New

Architect and develop external-facing APIs for developers.

Build and enhance a no-code platform for non-technical users.

Agile Git Python React

AI Financial Software Internship

New

Build financial software using AI technology

Work on impactful products with a mentor and team

Ai Collaboration Communication Creativity

Full Stack Software Engineer II

New

Build and maintain software products

Design and develop new product ideas

Go Javascript Node.js Postgres

Senior ML Scientist

New

Develop and implement advanced ML algorithms

Design and refine ML models

CI/CD Deep Learning Git Machine Learning

VP Product Pricing Actuary

New

Enterprise ownership and governance of modeling ecosystem

Leading model development for financial aspects

C++ Financial Modeling Python Sql

AI Engineer (Backend Services)

New

Design backend services for intelligent agent workflows

Develop APIs and orchestration layers in Python and TypeScript

Aws bedrock Aws Cdk Python Terraform

Cyber Security Engineer

New

Plan, execute, and deliver security projects with minimal guidance.

Manage and audit Identity Provider solutions (Entra, LDAP).

Ansible Bash Go Java

Lead, Advanced Analytics

New

Conduct advanced data analysis and modeling

Design and monitor metrics

Collaboration Mentoring Modeling Process Automation

Staff Software Engineer

New

Designing complex software systems

Leveraging AI tools for efficiency

Ai AWS C++ Golang

Lead Data Scientist - Recommendations

New

Lead and develop recommendation algorithms

Collaborate with cross-functional teams

Algorithm Development Cross-functional Collaboration Data Analysis Data Mining

Remote Data Analyst at MailerLite

New

Hiring a remote Data Analyst for full-time position.

No geographical restrictions for remote work.

Collaboration Communication Skills Dashboard creation Data Analysis

Senior QA Automation Engineer

New

Collaborate with cross-functional teams to refine testing processes for mobile applications

Lead mobile test automation projects focusing on UI automation

Android Appium Javascript/typescript Kotlin

Lead Security Engineer

New

Lead Digital Forensics and Incident Response team at Dave

Establish core forensics workflows and evolve SIEM

Cloud Security Digital Forensics Python Terraform

Senior Software Engineer

New

Analyze threat actor behavior and abuse patterns.

Implement new techniques for abuse detection.

AWS Cloud infrastructure Cross-functional Collaboration Javascript

AI Architect

New

Architect and build next-generation AI products

Lead and mentor engineering teams

Ai Tools Cloud Platforms Data Analytics Full-stack Development

AI Model Serving Engineer

New

Develop backend solutions using Python

Deploy applications on GCP and other cloud providers

AWS Azure Backend Development Docker

AI Research Collaboration Engineer

New

Improve AI systems through code validation and refinement

Validate algorithms and software concepts for accuracy

C++ Debugging Java Javascript

Software Engineering Intern

New

Contribute to software development on the Airship project

Design and develop software in collaboration with other systems

C C++ Documentation Python

Staff Software Engineer, Architecture

New

Design and implement cloud-native, distributed systems

Lead cross-functional collaboration for aligning architecture with business strategy

AWS Azure Distributed systems Ember.js

Senior Software Engineer - Infrastructure

New

Design and architect scalable infrastructure systems for ML inference platform

Optimize Kubernetes deployments for cost-effective model serving

AWS Gcp Go Kubernetes

Electrical Engineer, Robotics

New

Design, verify, and transition electronics to production

Own robotics stages, motor drives, and controls

Packaging Python Rust

Senior TAM - Cloud Value Maximizer

New

Maximize cloud value for key business customers

Lead technical engagements and architecture design

Ansible AWS Azure Docker

Data Scientist, Marketing Innovation

New

Define success metrics for marketing systems

Design measurement frameworks for always-on marketing systems

Causal Inference Collaboration Communication Data Translation

Product Security Engineer for Starshield

New

Design and build security infrastructure for Starshield

Develop security features for various systems

C++ Golang Network Security Prototyping

Senior QA Engineer

New

Develop and maintain automated test strategies

Build scalable and maintainable automation frameworks

CI/CD Github Actions Java Javascript

Data Scientist/AI Trainer

New

Develop and maintain Python code for data analysis, model evaluation, and AI workflow automation.

Design and refine prompts for LLMs to optimize conversational performance.

Conversational AI Data Analysis Data Science Machine Learning

Data Engineer III

New

Design and optimize ETL/ELT pipelines

Build and maintain data transformation infrastructure

Docker Kubernetes Postgresql Python

Quality Engineering Manager

New

Lead and mentor team members in quality engineering

Implement AI-powered automation strategy

Automated Testing C# Ci/cd Pipelines Cypress

Data Engineering Lead at Battle Road

New

Lead and build Battle Road's data platform

Architect AWS-based data solutions

AWS Azure Data Warehousing Gcp

Software Engineer - Card Integration

New

Develop card network transactions at scale

Create Digital Wallet Tokens via Apple Pay and Google Pay

Java Monitoring Python

Senior I/O Psychologist

New

Design and implement skills-based assessment solutions

Lead job analyses and validation studies

Cross-functional Collaboration Python R

Senior SRE Engineer

New

Design, implement, and maintain reliable production systems

Lead incident response and perform post mortem analysis

AWS Azure Gcp Go

Senior QA Engineer

New

Develop and implement automated test strategies for web applications

Design and execute various automated tests

Automated Testing CI/CD Java Javascript

Infrastructure Software Engineer

New

Architect multi-cloud systems for Scale AI's platform.

Collaborate with platform, product teams, and customers to scale infra.

AWS Azure Gcp Javascript

Azure Automation Engineer

New

Design and implement DB-driven workflows

Translate designs into production systems

C# CI/CD Javascript Python

Lead Software Engineer

New

Lead complex technical projects

Design and build scalable features

Gcp Javascript MERN Stack Node.js

Senior Software Engineer II

New

Lead projects for R&D and analytics

Collaborate with cross-functional teams

Azure Cloud Services Containerization Gcp

AI Insurance Prototypes Fullstack Engineer

New

Prototype and iterate on AI-driven insurance products

Deliver end-to-end prototypes and MVPs

Ai Tools AWS CI/CD Fullstack Development

Customer Success Engineer - Database

New

Provide technical support for database and cloud infrastructure

Architect enterprise-level DB solutions for large-scale workloads

Bash Go Kafka Linux

Senior AI Security Engineer

New

Secure Autonomous Agent Development

Threat Modeling and Analysis

API Security Go LangChain Memory management
overtime