Remote Data Annotation Jobs Madrid

Remote Data Annotation Jobs Madrid on Rex.zone focus on training data creation for AI/ML systems, including data labeling, RLHF, prompt evaluation, and QA evaluation for large language models and computer vision models. You will follow annotation guidelines compliance to improve training data quality, support model performance improvement, and help build reliable LLM training pipelines for AI labs, tech startups, and annotation vendors. Explore Rex.zone to apply to full-time remote roles and work on NLP tasks like named entity recognition, content safety labeling, and multilingual evaluation workflows aligned to production-grade AI deployment.

Job Image

Remote Data Annotation Jobs Madrid

Date: 25-02-2026 | Company: Rex.zone | Country: US | Remote Type: Remote | Employment Type: FULL_TIME | Experience Level: Mid-Senior | Industry: Technology | Job Function: Engineering | Skills: Remote data annotation, data labeling, RLHF, prompt evaluation, QA evaluation, annotation guidelines compliance, training data quality, named entity recognition, computer vision annotation, content safety labeling, LLM evaluation, model performance improvement | Salary Currency: USD | Salary Min: 63360 | Salary Max: 126720 | Pay Period: YEAR

About the Role

In this full-time remote role, you will deliver high-quality labeled datasets and evaluation signals used in AI/ML training workflows. Your work will support LLM evaluation, RLHF feedback collection, prompt evaluation, and multimodal annotation (text, image, and mixed content) while maintaining consistent annotation guidelines compliance. You will collaborate with operations and engineering stakeholders to resolve edge cases, calibrate labeling quality, and improve training data quality for model performance improvement across NLP, computer vision, and content safety domains.

Key Responsibilities

You will execute production annotation tasks (classification, ranking, span labeling, NER, summarization evaluation, and safety policy labeling), perform QA evaluation through self-checks and peer reviews, and document edge-case decisions for guideline alignment. You will contribute to RLHF workflows by providing preference judgments, rationale writing, and comparative model response evaluation. You will support dataset health by tracking error patterns, proposing guideline clarifications, and escalating ambiguous examples to maintain stable training data quality and consistent LLM training pipelines.

Projects and Task Types

Common projects include named entity recognition for multilingual text, intent and topic labeling, factuality and groundedness checks for LLM outputs, prompt evaluation for instruction-following behavior, and content safety labeling against policy taxonomies. Depending on program needs, you may also handle computer vision annotation such as bounding boxes, segmentation, image-text alignment, and quality scoring for multimodal models. The goal is to generate reliable supervision signals that drive model performance improvement in production environments.

Required Qualifications

You have experience with structured labeling work, evaluation frameworks, or quality assurance processes in data operations, ML support, or content moderation environments. You can interpret complex guidelines, apply consistent judgment, and communicate edge cases clearly. You are comfortable working with annotation tools, task queues, and rubric-based scoring for prompt evaluation and QA evaluation. You have strong written English for rationale-based RLHF comparisons and can maintain throughput without sacrificing training data quality.

Preferred Qualifications

Experience with RLHF pipelines, LLM evaluation rubrics, or prompt evaluation programs is preferred. Familiarity with NLP concepts (token spans, named entity recognition, intent classification), computer vision annotation standards, and content safety labeling taxonomies is a plus. You may have exposure to inter-annotator agreement, calibration sessions, and error analysis that improves annotation guidelines compliance and reduces variance across annotators.

Quality and Performance Expectations

Success is measured through accuracy against gold standards, consistency with annotation guidelines compliance, and contribution to training data quality. You will meet SLAs for throughput, maintain low rework rates, and participate in calibration to align judgments across the team. You will help identify guideline gaps, propose fixes, and support model performance improvement by delivering clean, well-documented supervision signals for LLM training pipelines.

Tools and Workflow

You will work in web-based annotation environments and use task management systems for assignment tracking and QA evaluation. You will follow structured rubrics for prompt evaluation and LLM evaluation, record rationales for RLHF preference data, and participate in periodic audits. Workflow includes reading guideline updates, completing calibration sets, running self-QA checks, and responding to feedback to maintain stable training data quality.

Who This Role Is For

This role fits candidates seeking remote data annotation jobs Madrid who want full-time work supporting AI labs, tech startups, BPOs, and annotation vendors. If you enjoy careful rubric-based evaluation, consistent labeling, and improving datasets that power NLP, computer vision, and content safety systems, you will match the day-to-day responsibilities. Rex.zone provides a platform to find and apply to these remote opportunities while building specialization in LLM evaluation and RLHF workflows.

How to Apply on Rex.zone

Apply through Rex.zone by completing your profile, highlighting annotation, QA evaluation, and prompt evaluation experience, and listing domain strengths such as NLP, computer vision annotation, or content safety labeling. Be ready to pass guideline comprehension checks and short calibration tasks designed to verify annotation guidelines compliance and consistent decision-making. Candidates with strong documentation habits and training data quality focus tend to progress quickly.

Frequently Asked Questions

  • Q: What are remote data annotation jobs Madrid?

    They are remote roles where you label and evaluate data used to train AI/ML systems, including tasks like data labeling, RLHF preference judgments, prompt evaluation, QA evaluation, named entity recognition, computer vision annotation, and content safety labeling. The output supports training data quality and model performance improvement in LLM training pipelines.

  • Q: Is this role fully remote and full-time?

    Yes. The role is marked Remote and FULL_TIME, with structured production workflows, QA evaluation checkpoints, and scheduled calibration to maintain annotation guidelines compliance.

  • Q: Do I need machine learning engineering experience?

    You do not need to be an ML engineer, but you should be comfortable with rigorous rubrics, consistent labeling decisions, and documentation. Experience with QA evaluation, guideline-based work, or RLHF and LLM evaluation is helpful.

  • Q: What kinds of datasets will I work on?

    You may work on NLP datasets (named entity recognition, classification, ranking), LLM evaluation datasets (prompt evaluation, helpfulness and factuality scoring), RLHF preference datasets (pairwise comparisons with rationales), computer vision annotation (bounding boxes or segmentation), and content safety labeling aligned to policy taxonomies.

  • Q: How is quality measured?

    Quality is measured through accuracy against gold standards, consistency and annotation guidelines compliance, inter-review agreement trends, audit results from QA evaluation, and stable performance across calibration sets that protect training data quality.

  • Q: What employers typically hire for these roles?

    These roles are commonly offered by AI labs, technology companies, tech startups, BPOs, and annotation vendors building or evaluating NLP, computer vision, and content safety systems for production deployments.

  • Q: How do I apply via Rex.zone?

    Use Rex.zone to submit your application, include keyword-aligned experience in remote data annotation, data labeling, RLHF, prompt evaluation, and QA evaluation, and complete any screening or calibration tasks to demonstrate annotation guidelines compliance.

230+Domains Covered
120K+PhD, Specialist, Experts Onboarded
50+Countries Represented

Industry-Leading Compensation

We believe exceptional intelligence deserves exceptional pay. Our platform consistently offers rates above the industry average, rewarding experts for their true value and real impact on frontier AI. Here, your expertise isn't just appreciated - it's properly compensated.

Work Remotely, Work Freely

No office. No commute. No constraints. Our fully remote workflow gives experts complete flexibility to work at their own pace, from any country, any time zone. You focus on meaningful tasks - we handle the rest.

Respect at the Core of Everything

AI trainers are the heart of our company. We treat every expert with trust, humanity, and genuine appreciation. From personalized support to transparent communication, we build long-term relationships rooted in respect and care.

Ready to Shape the Future of AI Data Operations?

Apply Now.