Senior QA Analyst

Location: This position is remote within Colombia

We are seeking a Senior QA Analyst & AI Evaluation Strategist to plan, design, and execute test strategies for our rapidly expanding suite of agentic and AI-powered product features. This is a unique evolution of the traditional QA role. Instead of focusing solely on predictable, binary (pass/fail) code paths, you will specialize in evaluating probabilistic software. You won’t be building or training AI models from scratch; instead, your focus will be on the integrity, orchestration, and evaluation of the system. You will be responsible for defining how we test multi-agent workflows, curating test data, and validating that our analytical dashboards, APIs, and data lakes process non-deterministic data accurately and safely. If you have an analytical, system-thinking QA mindset and are eager to strategize the future of AI product quality, we encourage you to apply.

What you'll do

• Strategic QA Planning: Formulate comprehensive test plans, timelines, and execution strategies optimized for multi-agent architectures and AI-driven data pipelines.
• Evaluation Framework Design: Collaborate with data scientists and product managers to define and standardize statistical evaluation rubrics (e.g., tracking Share of Model, Net Sentiment, and Citation Authority).
• System Integrity & Governance: Establish and supervise risk-adaptive checkpoints to ensure data privacy, compliance, and accurate reasoning across user journeys.
• Cross-Functional Alignment: Act as the primary bridge between Engineering, Data Science, and Product to translate complex business objectives into clearly defined, structured QA requirements.
• Ecosystem Verification: Ensure end-to-end platform consistency, verifying everything from raw data ingestion and API gateways to final user-facing analytical surfaces.
• Orchestrate Testing Frameworks: Design and run manual and automated functional validation processes for core application layers, API endpoints, and multi-step agentic workflows.
• Curate "Golden" Test Sets: Oversee the compilation and maintenance of baseline validation datasets used to test edge cases, user intent, and potential model anomalies.
• Validate Analytics and Metrics: Audit the calculations and automated workflows behind complex composite data scores, ensuring reporting values match underlying metrics reliably.
• Execute Adversarial Scenarios: Coordinate functional "red-teaming" or boundary testing to evaluate how gracefully features handle prompt variations, ambiguous inputs, or conversational drift.
• Biases & Safety Auditing: Perform targeted validation checks to ensure compliance with strict internal data policies, verifying proper PII data security and content filter safety.
• Defect Profiling & Distribution: Document and classify standard functional bugs alongside statistical behavioral defects in JIRA, mapping out trends to help developers find root causes efficiently.
• Monitor Platform Metrics: Utilize modern cloud logging and observability dashboards to keep tabs on execution latency, data drift, and overall user feedback patterns.

What we're looking for

• Education: Bachelor’s degree in Computer Science, Business Analytics, Information Technology, or a related field (or equivalent practical experience).
• Experience: 3+ years of proven experience in QA Engineering, Systems Analysis, or Software Testing, with a track record of leading test planning for complex digital platforms.
• Analytical & Spectrum Thinking: A strong data-driven mindset with the ability to shift away from purely binary testing toward evaluating data trends, thresholds, and statistical outcomes.
• Core QA Competencies: Mastery of traditional software testing methodologies, regression testing, API testing tools, Agile environments, and project tracking ecosystems (such as JIRA).
• Basic Scripting Familiarity: Ability to work with basic programming/scripting (ideally Python or JavaScript) to trigger automated test collections, query databases, or execute pipelines.
• Conceptual AI Familiarity: General understanding of how generative AI features function, including an operational awareness of typical behavior challenges (e.g., hallucinations, context limits, and instructions adherence).
• Hands-on experience with agentic QA workflows and AI-assisted testing tools (Claude Code, Codex, Antigravity, or similar) — this is the core of the role.
• Solid automation experience with Playwright, Cypress, or another JS framework — enough to design a suite from scratch and keep it healthy.
• Must have B2/C1 English Level

Preferred Skills and Qualifications

• AI Tooling & Observability: Exposure to AI evaluation tools, prompt engineering playgrounds, or data lake reporting architectures (e.g., BigQuery, GA4 platforms).
• Advanced Automation: Experience utilizing automated test frameworks to validate non-static content or multi-tier API environments.
• Integration Knowledge: Familiarity with cloud ecosystem mapping (like Google Cloud Platform components) or headless endpoint integrations.
• Data Verification: Comfort using basic SQL queries to audit background schema structures and perform data verification checks.
• Direct experience with CMS platforms such as Sitecore, Contentstack, Brightspot, or AEM.
• A foundation in WCAG 2.1 AA accessibility testing

If the selected candidate holds a degree in Engineering or a related profession, they must present their professional license. To verify your degree requires this license, please visit www.copnia.gov.co and https://www.consejoprofesional.org.co/. 

About Huge

Huge is a design and technology company. We create products and experiences that grow the world’s most ambitious brands. We do this by designing experiences for people, not users, and uncovering new sources of growth by leveraging our creative talent, our proprietary platform LIVE and unlocking the advantages brought to us by emerging technologies. We believe all experiences should be intelligent, shoppable and unique to every brand.

Huge’s nearly 1,000 thinkers, tinkerers, makers and creators, have been problem-solving across North America, Europe, and Latin America for over 25 years. Interested? You’ll find more information at www.hugeinc.com.

Huge is committed to creating an inclusive employee experience for all. Regardless of race, gender, religion, sexual orientation, age, disability, or if you’re parenting the next generation of innovators, we firmly believe that our work is at its best when everyone feels free to be their most authentic self.

Huge is an equal opportunity employer (EOE). We strongly support diversity in the workforce. We are committed to an inclusive, barrier-free recruitment and selection process and work environment. If you are contacted for a job opportunity, please advise us of any accommodation needed to ensure you have access to a fair and equitable process. Any information received relating to accommodation will be addressed confidentially.

Workers shall not be required to pay employers’ or agents’ recruitment fees or other related fees for their employment. If any such fees are found to have been paid by workers, such fees shall be repaid to the worker.

#LI-POST #LI-Remote