AI Agent Testing Specialist

New

Skills

Computer science Data Analytics Data Annotation Data Science Machine Learning Natural Language Processing QA Software Engineering

At Mindrift, we are seeking an AI Agent Testing Specialist to design realistic evaluation scenarios for LLM-based agents. You will create test cases, define gold-standard behavior, and work with developers to ensure clarity and accuracy in agent actions.

Key Responsibilities
  • Design structured test scenarios based on real-world tasks
  • Define gold path and acceptable agent behavior
  • Annotate task steps, expected outputs, and edge cases
  • Work with devs to test scenarios and improve clarity
  • Review agent outputs and adapt tests accordingly
Required Skills & Qualifications
  • Bachelor's and/or Master's Degree in relevant fields
  • Background in QA, software testing, data analysis, or NLP annotation
  • Good understanding of test design principles
  • Strong written communication skills in English
  • Comfortable with structured formats like JSON/YAML
  • Basic experience with Python and JS
  • Curious and open to working with AI-generated content
  • Ready to learn new methods and work remotely

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: 12 Months

Share this job:

Similar Jobs

AI Offense-Defense Dynamics Lead Researcher

New

Lead research to decode offense-defense dynamics of AI systems

Develop frameworks for analyzing AI capabilities

Ai Systems Communication Computer science Cybersecurity

Senior iOS Engineer at Goodnotes

New

Build mission-critical services for millions of users

Architect and design scalable solutions

CircleCI Computer science Engineer Github

Remote Design Engineer

Posted 3 days ago

Hiring a Senior Design Engineer for a remote, full-time position

Opportunity to work on GitHub's software development platform

Computer science Cross-functional Collaboration Css Development

Staff Backend Engineer

Posted 4 days ago

Lead technical direction and complex initiatives

Architect and build scalable systems

Ai Tools Architecture Computer science Engineer

Content Optimization Engineer

Posted 4 days ago

Optimizing and enhancing technical course content within various domains

Improving learner engagement and satisfaction through hands-on learning experiences

Computer science Css Docker Engineer

Senior Software Engineer

Posted 5 days ago

Design and develop cloud-native API first platform for patented data and AI-powered Security Knowledge Platform

Build and maintain integrations connecting platform with customer systems, tools, and more

Agile Ai Tools Architecture Computer science

Senior Software Engineering Manager

Posted 5 days ago

Lead and manage a highly skilled engineering team

Drive architectural evolution towards a high-performance ecosystem

Agile Ai Tools API Backend Development

Senior MacOS Engineer

Posted 6 days ago

Build the best Mac desktop experience for notes capture and presentation

Collaborate with cross-functional teams within Goodnotes

CircleCI Computer science Engineer Github

Remote Staff GTM Engineer

Posted 7 days ago

Develop go-to-market strategies for Netlify products.

Collaborate with cross-functional teams for successful product launches.

Communication Skills Computer science Cross-functional Collaboration Engineer

Rust Engineering Lead

Posted 9 days ago

Drive Rust adoption in Canonical and upstream projects

Design and implement software in Rust for Linux systems

C C++ Cloud Computer science

Senior Software Engineer

Posted 16 days ago

Revolutionize financial access for underserved customers in Africa and India

Strengthen existing products and develop new ones

Agile Development Android API Design Backend Development

Backend Engineer - Data

Posted 17 days ago

Revolutionize financial access in Africa and India

Build innovative financial products

Agile Agile Methodologies Ai Tools Api Development

Software Engineer II

Posted 23 days ago

Developing a cloud native API first platform

Ensuring high reliability and scalability of the platform

Agile Cloud Computer science Engineer

Software Engineer II

Posted 23 days ago

Design and develop a cloud-native API first platform

Drive modernization efforts for a reliable service-oriented platform

Agile Cloud-native architecture Computer science Cybersecurity

Senior Software Engineer

Posted 23 days ago

Design and develop a cloud-native API-first platform for a Security Knowledge Platform™

Ensure high reliability and service patterns for third-party integrations

Cloud Cloud-native architecture Computer science Decision-making

Senior Manager, Software Engineering

Posted 23 days ago

Lead and grow a skilled full-stack engineering team.

Drive architectural initiatives for high-performance ecosystem.

Agile Ai Tools API Cloud Native

Platform iOS Engineer

Posted 27 days ago

Develop cutting-edge solutions for networked operations

Enable seamless data synchronization and collaboration for users

CircleCI Computer science Engineer Ios Development

Remote Data Manager

Posted 28 days ago

Hiring a remote Data Manager for a full-time position.

Supporting climate change initiatives through data analysis.

Communication Skills Computer science Data Data Analysis

AI Offense-Defense Dynamics Researcher

Posted 34 days ago

Decode offense-defense dynamics of AI technologies

Develop models to predict AI system impacts

Communication Computer science Researcher Risk Management

Senior Statistical Programmer

Posted 36 days ago

Seeking a Senior Statistical Programmer with advanced SAS programming skills

Join a supportive team in a data-focused clinical research organization

Clinical Research Computer science Ich Guidelines Sas

Frontend Software Engineer (React)

Posted 36 days ago

Hiring experienced frontend engineers for high-impact AI collaborations.

Developing and validating coding benchmarks in React, TypeScript, or JavaScript.

Computer science Debugging Engineer Integration Testing

Senior MacOS Engineer

Posted 38 days ago

Build the best Mac desktop experience for notes capture and presentation

Collaborate with cross-functional teams to achieve the mission

CircleCI Computer science Engineer Github

AI Training Specialist - Back End Engineer

Posted 39 days ago

Train and improve AI models in Back End Engineering

Evaluate AI-generated outputs for accuracy and clarity

Ai training Analytical Thinking Apis Back end engineering

Staff Software Engineer - GTM Systems

Posted 39 days ago

Hiring a remote Staff Software Engineer

Work on GTM Systems

Collaboration Computer science Engineer Remote Work

Rust Engineering Lead

Posted 41 days ago

Drive Rust adoption in Canonical and upstream projects

Design and implement high-quality software in Rust

C C++ Cloud Computer science

Software Engineer - App Stores

Posted 41 days ago

Develop clean web service APIs using Python and optionally Golang

Design and implement new features from spec to production

Api Development Architecture Computer science Engineer

Staff Backend Engineer

Posted 41 days ago

Lead technical direction and execution of high-impact initiatives

Architect and build scalable systems for mission-critical workflows

Ai Tools Architecture Computer science Engineer

Staff Security Assurance Engineer - Third Party Risk Management

Posted 48 days ago

Manage and mature third-party risk management program

Evaluate security controls and documentation of third parties

Computer science Engineer ISO 27001 Jira

Senior Software Engineer at Branch

Posted 48 days ago

Revolutionize financial access for underserved banking customers

Design and maintain multiple technologies and systems

Agile Methodologies Android development API Design Computer science

Backend Engineer - Data

Posted 48 days ago

Revolutionizing financial access for underserved customers across Africa and India.

Building out APIs and backend systems for new financial products.

Agile Agile Methodologies Ai Tools Api Development

AI Chemistry Expert Training

Posted 48 days ago

Train and improve AI models using Chemistry and Python expertise

Review and evaluate AI-generated code and content

Ai Chemical Engineering Chemistry Computer science

Poland Software Engineering Intern

Posted 49 days ago

Offer an internship opportunity in software engineering at Dropbox

Allow remote work from anywhere in Poland

C++ Computer science Engineer Independent Work

Sr. Software Engineer (AI)

Posted 53 days ago

Hiring a Sr. Software Engineer specialized in AI.

Full-time remote position in the United States.

Ai Ai Algorithms Ai solutions Computer science

Junior Data Analyst Position

Posted 53 days ago

Support statewide efforts to end veteran homelessness

Collect and synthesize data to identify trends

Communication Skills Computer science Data Analysis Data Science

Staff Frontend Engineer Role

Posted 63 days ago

Develop and maintain a secure, scalable orchestration platform.

Design frontend components and APIs for multi-tenant environments.

Api Development AWS CI/CD Computer science

Education Services Developer

Posted 63 days ago

Build and maintain virtual training environments

Support and troubleshoot sandbox systems for trainings

Angular Azure C# Communication

Staff Backend Engineer Role

Posted 73 days ago

Architect and scale backend platforms

Lead and mentor engineering teams

Ai Tools Architecture Computer science Distributed systems

Ubuntu Server Packaging Engineer

Posted 73 days ago

Maintain and optimize Ubuntu Server packages

Foster collaboration in a global distributed team

Cloud Computer science Containerization Debian packaging

Rust Engineering Lead Role

Posted 73 days ago

Drive Rust adoption across Canonical products

Develop and maintain Rust-based Linux software

C C++ Cloud Computer science

Senior Graphics Engineer Role

Posted 75 days ago

Develop and maintain advanced graphics engines

Lead feature innovation using modern technologies

Architecture Computer science Engineer Ios Development

Core Zero-Knowledge Engineer

Posted 75 days ago

Advance zk-EVM scalability and performance

Design and implement zero-knowledge cryptographic solutions

C++ Computer science Engineer Go

Workday Integrations Specialist

Posted 75 days ago

Manage Workday integrations from design to support

Ensure data security and compliance in all integrations

Computer science Documentation Hris Security

Senior Backend Engineer Role

Posted 80 days ago

Revolutionize financial access for underserved markets

Design and scale secure backend financial systems

Android API Design Backend Development Computer science

Third-Party Security Assurance

Posted 82 days ago

Mature and operate third-party risk management program

Evaluate and audit security of external vendors

Cissp Certification Computer science Engineer Jira

Third-Party Security Assurance

Posted 82 days ago

Manage and mature third-party risk program

Conduct vendor security assessments and audits

Cissp Certification Computer science Engineer ISO 27001

Salesforce Solution Architect Role

Posted 87 days ago

Design scalable Salesforce architectures

Translate business needs into technical solutions

Apex Architecture Cloud Computer science

InnoDB Team Lead Role

Posted 91 days ago

Lead the InnoDB software development team.

Design and implement robust database solutions.

Agile Methodologies Algorithms C++ Computer science

Senior Application Security Engineer

Posted 95 days ago

Conduct security assessments and reviews

Develop secure software practices

Architecture Cloud Code Review Computer science

AI Offense-Defense Dynamics Lead

Posted 98 days ago

Analyze offense-defense dynamics of AI systems

Develop frameworks and models for AI risk prediction

Communication Computer science Researcher Risk Management

Senior Generative AI Engineer

Posted 99 days ago

Develop and optimize generative AI models for enterprise clients

Lead and enhance large language model training and fine-tuning

Computer science Engineer Generative AI Machine Learning
overtime