Senior AI Data Annotation Jobs Montreal

Senior AI data annotation roles in Montreal focused on training data quality for LLM training pipelines on Rex.zone. You will lead data labeling workflows, RLHF and prompt evaluation, QA evaluation, and annotation guidelines compliance across NLP and computer vision annotation. This full-time remote role supports model performance improvement through high-precision tasks like named entity recognition, content safety labeling, and structured evaluation rubrics used by AI labs, tech startups, and annotation vendors.

Job Image

Senior AI Data Annotation Jobs Montreal

LinkedIn Job Metadata: Title: Senior AI Data Annotation Jobs Montreal; Date: 25-02-2026; Company: Rex.zone; Country: US; Remote Type: Remote; Employment Type: FULL_TIME; Experience Level: Mid-Senior; Industry: Technology; Job Function: Engineering; Skills: AI data annotation, data labeling, RLHF, prompt evaluation, QA evaluation, annotation guidelines, NLP, named entity recognition, computer vision annotation, content safety labeling, LLM training pipelines, training data quality; Salary Currency: USD; Salary Min: 63360; Salary Max: 126720; Pay Period: YEAR

About the Role

As a senior AI data annotation specialist on Rex.zone, you will own end-to-end labeling and evaluation workflows that improve large language model evaluation and downstream model performance. You will translate product and research goals into annotation guidelines, calibrate labeler agreement, run QA evaluation, and deliver high-signal datasets for supervised fine-tuning and RLHF. Work includes NLP labeling (intent classification, sentiment, named entity recognition), prompt evaluation (helpfulness, factuality, instruction following), content safety labeling, and computer vision annotation where needed.

What You Will Do

You will: lead training data quality programs with measurable acceptance criteria; design and maintain annotation guidelines compliance checks; run rubric-based LLM evaluation and prompt evaluation with adjudication; perform RLHF preference labeling calibration and reviewer training; execute QA evaluation sampling plans, error taxonomies, and root-cause analysis; coordinate with engineering and research on labeling schema, dataset specs, and versioning; optimize throughput while preserving precision/recall targets; document decisions to support audits and reproducibility across AI/ML pipelines.

Core Workstreams

Workstreams include: RLHF preference data creation and evaluation rubrics; large language model evaluation for safety, policy compliance, and factuality; NLP labeling such as named entity recognition, classification, and span annotations; content safety labeling for toxicity, self-harm, and policy categories; computer vision annotation (bounding boxes, polygons, keypoints) when projects require multimodal support; training data quality monitoring using inter-annotator agreement, disagreement resolution, and gold set maintenance.

Requirements

Requirements include: 3+ years in AI data annotation, data labeling, or QA evaluation for ML datasets; hands-on experience with LLM training pipelines, RLHF concepts, or prompt evaluation; strong understanding of annotation guidelines and adjudication processes; ability to analyze label quality with agreement metrics and error analysis; clear technical writing for guidelines and decision logs; comfort working remotely with cross-functional engineering and research teams.

Preferred Qualifications

Preferred: experience with NLP tasks (named entity recognition, text classification) and computer vision annotation; familiarity with content safety labeling and policy/risk frameworks; experience building gold datasets, running calibration sessions, and coaching reviewers; exposure to dataset tooling, version control practices, and structured sampling for QA evaluation.

Tools and Workflow

You will work with annotation platforms and internal tooling to manage queues, review flows, and gold sets. You will apply rubric-driven evaluation, maintain structured label schemas, track training data quality metrics, and collaborate asynchronously in a remote full-time environment. Outputs include dataset specs, guideline documents, QA reports, and issue triage artifacts used to drive model performance improvement.

Why Rex.zone

Rex.zone connects skilled annotators and evaluators with full-time remote opportunities supporting AI labs, tech startups, BPOs, and annotation vendors. This role emphasizes rigorous evaluation, reproducible labeling operations, and measurable improvements to training data quality for modern LLM systems.

How to Apply

Apply through Rex.zone with a brief summary of your AI data annotation experience, domains (NLP, computer vision annotation, content safety labeling), and examples of QA evaluation or guideline ownership. Highlight any RLHF, prompt evaluation, or large language model evaluation work and your approach to annotation guidelines compliance and model performance improvement.

Frequently Asked Questions

  • Q: Is this a remote full-time role?

    Yes. The position is marked Remote and is FULL_TIME.

  • Q: What does “Senior AI data annotation” mean in practice?

    It means owning high-impact labeling and evaluation workflows: writing annotation guidelines, leading calibration, running QA evaluation, adjudicating edge cases, and producing training data quality improvements for LLM training pipelines and model performance improvement.

  • Q: What types of tasks are included?

    Tasks commonly include data labeling for NLP (named entity recognition, classification, span labeling), RLHF preference labeling, prompt evaluation for LLM quality, QA evaluation and auditing, content safety labeling, and occasional computer vision annotation depending on project needs.

  • Q: Do I need prior RLHF experience?

    It is strongly preferred. If you have not done RLHF directly, comparable experience with large language model evaluation, rubric design, and adjudication is relevant.

  • Q: Why does the page mention Montreal if the job is US/Remote?

    The posting targets the Montreal job-search intent while keeping the role Remote and Country set to US per the required defaults, so candidates searching Montreal-based opportunities can still find and apply via Rex.zone.

  • Q: What skills should I emphasize to be competitive?

    Emphasize training data quality, annotation guidelines compliance, QA evaluation methods, prompt evaluation and rubric design, RLHF familiarity, NLP labeling (including named entity recognition), and content safety labeling experience.

230+Domains Covered
120K+PhD, Specialist, Experts Onboarded
50+Countries Represented

Industry-Leading Compensation

We believe exceptional intelligence deserves exceptional pay. Our platform consistently offers rates above the industry average, rewarding experts for their true value and real impact on frontier AI. Here, your expertise isn't just appreciated - it's properly compensated.

Work Remotely, Work Freely

No office. No commute. No constraints. Our fully remote workflow gives experts complete flexibility to work at their own pace, from any country, any time zone. You focus on meaningful tasks - we handle the rest.

Respect at the Core of Everything

AI trainers are the heart of our company. We treat every expert with trust, humanity, and genuine appreciation. From personalized support to transparent communication, we build long-term relationships rooted in respect and care.

Ready to Shape the Future of AI Data Operations?

Apply Now.