As an SDET at JedAI, you will focus on ensuring the reliability, safety, and performance of our GenAI platform. This includes developing robust automated tests, evaluating AI outputs for hallucinations, and implementing guardrails to uphold AI safety standards. You will collaborate with cross-functional teams to drive quality across the entire AI lifecycle, leveraging state-of-the-art tools and methodologies.
This role sits in the Gen AI organization within Technology & Digital for Disney Experiences. It works closely with Application teams from across the company. The Lead Gen AI QA/SDET, Guardrails, & Evaluations will report to the Delivery Lead.
Key Responsibilities:
- Design, build, and maintain automated testing frameworks for both backend and frontend components for Gen AI platforms and technologies.
- Develop and execute comprehensive test strategies to validate AI models, agents, and knowledge base integrations.
- Evaluate generative AI outputs for hallucinations using tools such as Arize and other evaluation frameworks.
- Implement and test AI guardrails to ensure safe and responsible model behavior, including red teaming exercises and adversarial testing.
- Collaborate closely with developers, product managers, and stakeholders to refine requirements and enhance software resilience.
- Collaborate with existing System Integration Testing QA team and Performance Engineering team for Load Testing.
- Conduct functional, regression, and performance testing to ensure optimal system behavior.
- Participate in code reviews, providing feedback on testability and quality.
- Maintain thorough testing documentation, including test plans, cases, and technical specifications.
- Research and integrate industry best practices for AI safety, automation, and continuous delivery.
- Testing for AI hallucinations and factual accuracy using leading evaluation tools
- Building and validating guardrails to prevent unsafe or biased outputs.
- Red teaming and adversarial testing to proactively identify vulnerabilities.
- Ongoing research into AI safety and responsible deployment practices.
- Automation for continuous process improvement and reduction of product defects
- Lead the documentation of defects, build and execute test plans, help debug and diagnose difficult to complex code issues in selected environments
- Influence the Software Development Life Cycle process through collaboration with Infrastructure, Development Operations and development teams
- Promote and maintain solid technical relationships with QA Engineers and developers
- Develop & Identify new metrics per applicative for gen ai use case.
Responsibilities:
- Validate and help define acceptance criteria in BDD/Gherkin.
- Design, develop, and implement test-related code to create tools, unit tests, test scripts, test harnesses, and other infrastructure for validating Windows and/or mobile applications, libraries, drivers, and interfaces
- Take ownership over test automation, including designing or customizing automation tools as needed
- Develop feature code and support other engineering efforts when not working on test-based deliverables
- Quickly translate technical requirements and product goals into appropriate code and automated tests
- Design test plans and support testing tools and processes
- Help manage continuous integration tools like Jenkins, Xcode Server, and VS Build Server to help manage test targets, create reports, and support other users of the tool
- Participate in creating technical specifications and performing code reviews with other engineers
- Participate in agile planning and scope estimation for team projects
- Generate and maintain documentation as needed.
- iOS Development or QE experience a must
Required Skills & Experience:
- Demonstrated experience as a SDET with machine learning / AI experience or Machine Learning / AI Engineer.
- Proficiency in programming languages such as JavaScript, Python, and Node.js.
- Proficiency in GCP, Azure, and/or AWS.
- Deep understanding of modern testing methodologies and best practices for software and AI systems.
- Experience with automation frameworks and tools.
- Hands-on experience with CI/CD pipelines and DevOps practices.
- Familiarity with RESTful API testing using tools like Postman.
- Strong analytical and problem-solving abilities.
- Excellent communication skills for cross-functional collaboration.
- Experience with database testing and SQL.
- Exposure to AI safety, hallucination detection, and red teaming is highly desirable
- Willingness and motivated to learn new technology as required.
