AI Engineer, Quality

New

Skills

Postgres Python React Typescript Vector Databases

Job Overview

Design and build a unified evaluation platform serving as the single source of truth for agentic systems and audit workflows. Opportunities to work remotely from anywhere.

Responsibilities
  • Own the evaluation infrastructure stack, including integration with LangSmith and LangGraph
  • Develop automated pipelines to evaluate new models within hours of release
  • Design evaluation harnesses and frameworks to measure effectiveness, latency, and cost
  • Design prompts, retrieval pipelines, and agent orchestration to scale reliably
  • Implement evaluation frameworks for model outputs and agent behaviors
Requirements & Qualifications
  • Multiple years of experience shipping production software in complex, real-world systems
  • Experience with TypeScript, React, Python, and Postgres
  • Built and deployed LLM-powered features serving production traffic
  • Designed observability/tracing infrastructure for AI/ML systems
  • Experience with evaluation platforms (LangSmith, Langfuse, or similar)

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: 12 Months

Share this job:

Similar Jobs

Data Ecosystem Engineer II

New

Build self-service data experiences for analytics and reporting

Create fault-tolerant data pipelines across microservices and third-party systems

Ai Airflow Django Javascript

Senior Backend Engineer

New

Design, build, and maintain distributed backend systems

Work closely with product and engineering teams

AWS Docker Golang Grafana

Senior Software Engineer

New

Build next-gen crypto-forward products and features

Solve large-scale, complex problems across web-scale apps and blockchain

Docker Ethereum Go Mongodb

Engineering Manager

New

Develop and execute technical roadmap for US banking product expansion

Act as technical authority for the banking product

Java Kafka Postgres RabbitMQ

Senior Backend Engineer

New

Design, build, and maintain scalable distributed backend systems

Work closely with product and engineering peers

AWS Docker Golang Grafana

Senior Backend Engineer

Posted 3 days ago

Design, build, and maintain scalable distributed backend systems

Work closely with product and engineering peers to shape product decisions

AWS Docker Golang Kubernetes

Senior Backend Engineer

Posted 4 days ago

Design, build, and maintain scalable distributed backend systems

Work closely with product and engineering peers

AWS Docker Golang Grafana

Senior Software Engineer, Accounting

Posted 5 days ago

Design and architect Accounting Platform systems

Mentor engineers and ensure quality standards

AWS Css Html Javascript

Senior Data Engineer

Posted 5 days ago

Design and build scalable data pipelines for clinical RWD

Implement Python/SQL ETL/ELT with AWS services

Agile Airflow CI/CD Git

Senior Backend Engineer

Posted 5 days ago

Design, build, and maintain scalable distributed backend systems

Own backend services end-to-end

AWS Docker Golang Kubernetes

Staff Backend Engineer - WHOOP

Posted 6 days ago

Drive backend engineering

Lead design and implementation of new features

Agile Development AWS Backend Development Java

Senior Backend Engineer

Posted 6 days ago

Design, build, and maintain scalable distributed backend systems

Shape product decisions and contribute to technical discussions

AWS Docker Golang Grafana

Customer Support Representative

Posted 7 days ago

Triage tickets and assess urgency.

Interact with cross-functional teams.

Github Jira Json Mysql

Senior Backend Engineer

Posted 8 days ago

Design, build, and maintain scalable distributed backend systems.

Collaborate with product and engineering teams to shape product decisions.

AWS Docker Golang Grafana

Senior Software Engineer, Global Contractor

Posted 8 days ago

Seeking experienced software engineers with problem-solving abilities

Emphasis on collaboration and communication skills

Ai Clean code Github Google Cloud

Senior Backend Engineer

Posted 8 days ago

Design, build, and maintain scalable distributed backend systems

Work closely with product and engineering peers to shape product decisions

AWS Docker Golang Grafana

Senior Software Engineer at First Advantage

Posted 10 days ago

Seeking an experienced Senior Software Engineer to work on AI-driven solutions and data pipelines

Building and optimizing solutions using modern cloud and data technologies

AWS Devops Engineer Front end

Senior Backend Engineer - Auth Infrastructure

Posted 10 days ago

Build and maintain core authentication infrastructure at scale.

Optimize performance and reliability of authentication systems.

Architecture Engineer Go Golang

Senior Data Engineer

Posted 11 days ago

Design and implement Python-based automation frameworks for database lifecycle management

Collaborate with infrastructure teams to integrate systems via APIs

AWS Devops Engineer Kubernetes

Senior Software Engineer - Ruby on Rails

Posted 12 days ago

Develop and enhance web applications using modern technologies

Mentor junior team members and participate in team discussions

AWS C# Css Engineer

Senior Software Engineer - Blackbaud

Posted 12 days ago

Develop and enhance web applications using Ruby on Rails, Elixir, and Phoenix.

Evaluate and improve security and performance of critical application components.

AWS C# Css Engineer

Senior Backend Engineer

Posted 15 days ago

Design and build scalable distributed backend systems.

Collaborate with product and engineering teams to shape product decisions.

AWS Docker Golang Grafana

Principal Software Engineer - Ruby on Rails

Posted 15 days ago

Develop and enhance web applications using Ruby on Rails, Elixir, and Phoenix.

Evaluate and improve security and performance of critical application components.

AWS C# Css Docker

Software Engineer at Sticker Mule

Posted 33 days ago

Help innovate and develop Sticker Mule's commerce platform

Work on developing the Stores platform and related tools

Docker Go Golang Google Cloud Platform

Full-Stack NodeJS Engineer

Posted 37 days ago

Hiring a Full-Stack NodeJS Software Engineer for a full-time role in India

Working on a cloud-based software platform for mentoring and networking

Engineer Javascript Kubernetes NodeJS

Senior Database Automation Engineer

Posted 43 days ago

Design and implement Python-based automation frameworks for database management

Collaborate with infrastructure teams to integrate systems via APIs

Automation AWS Devops Engineer

Senior Backend Engineer - Auth Infrastructure

Posted 43 days ago

Build critical infrastructure for authentication and authorization

Optimize performance and reliability of authentication and authorization systems

Architecture Engineer Go gRPC

Senior Software Engineer, Global Contractor

Posted 50 days ago

Empower financial advisors and clients through AI technology

Develop smarter workflows and enterprise-grade security solutions

Ai Ai technology ChatGPT Engineer

Backend Developer Opportunity

Posted 60 days ago

Seeking mid-to-senior Backend Developer with Node.js expertise

Emphasizing strong coding skills and ability to work independently

AWS Docker Elasticsearch Gitlab

Senior Backend Engineer

Posted 75 days ago

Developing core authentication infrastructure for GitLab.

Optimizing authentication and authorization performance.

Architecture Engineer Go Golang

Software Engineer at Sticker Mule

Posted 84 days ago

Develop the new Stores platform for online selling

Collaborate with tools like Give, Ship, and Notify to support sellers

Docker Go Google Cloud Platform GraphQL

Full Stack Engineer

Posted 96 days ago

Develop and maintain various customer-facing UI and server-side components.

Ensure optimal performance of databases and responsiveness to front-end requests.

AWS Engineer Full stack GraphQL

Node.js Backend Developer

Posted 124 days ago

Empower engineers to work independently and proactively

Deliver scalable backend solutions using Node.js

AWS Docker Elasticsearch Github

Database Automation Engineer Role

Posted 139 days ago

Automate database lifecycle management

Ensure zero-downtime migrations and replication

Automation AWS Devops Engineer

Senior Backend Engineer Authentication

Posted 140 days ago

Design and implement scalable authentication infrastructure

Advance GitLab’s zero-trust and microservices architecture

Architecture Cloud Go Golang

Senior Software Engineer Remote

Posted 142 days ago

Build healthcare-focused software solutions

Integrate and manage complex data sources

AWS Docker Elasticsearch Git

Podcast Transcription Automation

Posted 154 days ago

Automate podcast audio download and transcription

Extract and summarize key insights from transcripts

Gemini Postgres Python Rust

Remote Software Engineer Role

Posted 155 days ago

Accelerate development of the Stores platform

Build scalable and maintainable commerce tools

Docker Engineer Go Golang

Database Automation Engineer

Posted 171 days ago

Automate database lifecycle management

Optimize database performance and reliability

Devops Engineer Kubernetes Mongodb

Global Remote Software Engineer

Posted 176 days ago

Empower financial advisors with AI

Hire skilled global software engineers

Ai Ai Tools Api Integration Engineer

Node.js Backend Developer

Posted 187 days ago

Hire experienced Node.js backend developer

Foster independent and proactive work culture

Automated Testing AWS Docker Elasticsearch

Backend Software Engineer

Posted 193 days ago

Develop scalable backend solutions

Enhance application performance and reliability

Docker Engineer Java Kafka

Sr. Software Engineer Role

Posted 196 days ago

Develop and scale the Stores commerce platform

Implement and integrate supporting tools (Give, Ship, Notify)

Computer Science Degree Docker Engineer Go

Senior Data Engineer VRChat

Posted 226 days ago

Enhance and scale data infrastructure

Build and optimize data platforms

AWS Engineer Etl Pipeline Development Mongodb

Backend Software Engineer Role

Posted 238 days ago

Develop scalable backend services for healthcare solutions

Collaborate with cross-functional and distributed teams

Architecture AWS Engineer Mongodb

Senior Technical Support Engineer

Posted 240 days ago

Deliver expert technical support for Clari's platform

Resolve customer issues efficiently and empathetically

Customer Service Customer Support Engineer Json

Senior QA Engineer Role

Posted 240 days ago

Lead QA processes for multiple products

Develop and execute manual and automated tests

Javascript Jira Mysql Postgres

Remote Data Engineer VRChat

Posted 240 days ago

Design and maintain scalable data infrastructure

Develop and optimize ETL pipelines

AWS Engineer Mongodb Mysql

Fresher NodeJS Angular Developer

Posted 240 days ago

Hire entry-level full stack developers

Develop scalable e-commerce/FinTech platforms

AngularJS Css Html Kafka

TypeScript React Postgres Engineer

Posted 275 days ago

Maintain and enhance a TypeScript/React/Postgres application

Resolve bugs, feature requests, and technical debt

Google Analytics GraphQL Next.js Postgres
overtime