We have partnered with our client in their search for a
Responsibilities
Job Summary:
As an QA Engineer within Acentra Health’s AI & Analytics team, you will ensure the accuracy, reliability, and validity of our AI, ML, and Generative AI (GenAI) products and services. Leveraging your expertise in quality assurance methodologies, you will perform Functional, UI, and API testing to effectively validate products such as AI chatbots, ML/GenAI-generated clinical documents, microservices, and other software solutions. Your role will tackle unique challenges in GenAI environments, including output quality assurance and hallucination detection, ensuring stringent standards are consistently met. Aligned with this, you will support and contribute to our maturing AI Inter-Rater Reliability (AI-IRR) process and services.
Collaborating closely with data scientists, software developers, engineers, and stakeholders, you will design and execute comprehensive test plans and strategies specific to our AI and GenAI platforms. Responsibilities include creating and refining detailed test cases, verifying prompt-response accuracy, and ensuring the ethical and responsible use of AI outputs, ultimately contributing to improved patient outcomes.
Responsibilities:
- Test Planning & Execution: Develop and implement test strategies, detailed test plans, and test cases tailored to AI, ML, and GenAI solutions
- Quality Assurance & UAT: Conduct thorough Functional, UI, and API testing, including GenAI output quality, hallucination detection, and user acceptance testing to ensure optimal application performance
- Collaboration: Work closely with data scientists, software developers, engineers, and project managers to validate algorithms and integrate QA best practices into development processes
- Automation & Optimization: Implement automation testing frameworks to optimize testing efforts, including automated GenAI evaluation workflows
- Data Integrity & Bias Detection: Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training
- AI Inter-Rater Reliability: Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines
- Documentation & Reporting: Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement
- Compliance & Best Practices: Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage
- Continuous Improvement: Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies
- Read, understand, and adhere to all corporate policies including policies related to HIPAA and its Privacy and Security Rules
The list of accountabilities is not intended to be all-inclusive and may be expanded to include other education- and experience-related duties that management may deem necessary from time to time.
Qualifications
Required Qualifications
Education:
- Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent work experience will be considered
Experience:
- 3+ years of core quality assurance experience (preferably at the enterprise level, managing large scale projects), with a strong background in traditional QA and UAT methodologies
- Exposure to AI/ML: some experience in AI or Machine Learning (ML)
- Proficiency in RESTful APIs and testing tools such as Postman, specifically for testing GenAI model APIs like OpenAI and Hugging Face
- Experience verifying training data integrity, bias detection, and prompt-response validation for AI models
- Familiarity with data annotation tools, and quality control processes
- Demonstrated experience in testing complex software applications
- Proven experience designing, documenting, and executing comprehensive test plans for AI-driven products
- Strong analytical thinking, attention to detail, excellent communication skills, and the ability to thrive in a fast-paced, cross-functional team environment
Technical Skills:
- Strong experience with QA tools and frameworks like PyTest, Selenium, or JUnit
- Solid knowledge of software development life cycle (SDLC) and agile methodologies
- Understanding of AI and ML technologies
- Solid knowledge of Python, with experience or knowledge in Gen AI libraries like LangChain and LlamaIndex (preferred)
- Familiarity with data annotation tools and quality control processes for GenAI-specific tasks
- Knowledge in testing AI, ML, and Generative AI outputs, including Functional, UI, and API testing
- Knowledge of GenAI-specific annotation for RAG and LLM fine-tuning
Soft Skills:
- Excellent analytical and problem-solving skills with keen attention to detail
- Strong communication and collaboration skills, with the ability to explain complex technical issues to diverse audiences
- Ability to work effectively in a fast-paced, cross-functional team environment
Preferred Qualifications
- Experience with GenAI model evaluation metrics such as perplexity and BLEU
- Expereince with Gen AI evaluation frameworks like RAGAS, and GenAI concepts, algorithms, and output validation techniques
- Experience in healthcare technology or data analytics environments
- Familiarity with regulatory and ethical standards related to AI and healthcare solutions
- Bullet One
- Bullet Two
- Bullet Three
Skills Required
- Bullet One
- Bullet Two
- Bullet Three
Education & Work Experience
Title QA Engineer
Location McLean, VA
Client Industry Healthcare
Compensation $60-70/hr
Ref ID 1681715
