Remote Data Annotator Jobs Madrid

Remote data annotator jobs in Madrid on Rex.zone focus on data labeling and AI/ML evaluation that power LLM training pipelines, RLHF, and training data quality. As a remote data annotator, you will follow annotation guidelines compliance to tag text, images, and prompts for model performance improvement, including QA evaluation, prompt evaluation, named entity recognition, and content safety labeling. These roles support AI labs, tech startups, and annotation vendors building NLP and computer vision systems. Explore full-time remote opportunities, understand project workflows and quality metrics, and apply through Rex.zone to help improve large language model evaluation and production AI reliability.

Remote Data Annotator Jobs Madrid

About the Role

You will perform end-to-end data annotation and evaluation for AI/ML systems, producing high-quality labeled datasets for NLP and computer vision. Work includes RLHF-style preference ranking, prompt evaluation for LLM behavior, QA evaluation against gold standards, and adherence to detailed annotation guidelines. You will document edge cases, propose clarifications to guidelines, and support training data quality initiatives that reduce noise and improve model performance.

What You Will Do

Core tasks include creating and validating labels for text and image data; performing named entity recognition and taxonomy tagging; running content safety labeling (policy-based classification for sensitive content); conducting prompt evaluation and response grading; executing RLHF ranking and pairwise comparisons; completing QA checks, audits, and inter-annotator agreement improvements; writing clear rationales for complex judgments; and collaborating with project leads to improve annotation workflows and dataset consistency.

Workflows and Domains You May Support

Projects may include large language model evaluation, instruction-following and helpfulness/harmlessness grading, multilingual NLP labeling, retrieval-augmented generation evaluation, toxicity and safety policy labeling, computer vision annotation (bounding boxes, polygons, keypoints), and dataset curation for model training pipelines. You may work with AI labs, tech startups, BPOs, and annotation vendors delivering production-grade datasets.

Required Qualifications

Mid-to-senior experience in data annotation, data labeling, or QA evaluation; strong written reasoning and ability to follow annotation guidelines compliance; familiarity with LLM concepts, prompt evaluation, and RLHF-style ranking; comfort using labeling tools and spreadsheets; ability to maintain high accuracy under productivity targets; and experience communicating edge cases and proposing guideline updates.

Preferred Qualifications

Experience with named entity recognition, ontology/taxonomy design, or linguistic annotation; computer vision annotation experience (boxes, polygons, segmentation); content safety labeling experience using policy frameworks; experience measuring training data quality via sampling and audit plans; exposure to inter-annotator agreement methods; and familiarity with AI/ML evaluation metrics and model behavior testing.

Quality Standards and Success Metrics

Success is measured by training data quality, guideline adherence, low rework rates, high audit scores, strong inter-annotator agreement, clear rationales for subjective evaluations, and consistent throughput. You will help maintain QA evaluation processes, improve label consistency, and contribute to model performance improvement by reducing ambiguity and drift across batches.

Remote Work and Collaboration

This is a Remote, FULL_TIME role. You will coordinate asynchronously with distributed teams, follow secure data handling practices, and deliver labeled outputs on schedule. Collaboration includes calibration sessions, feedback cycles, and regular updates to ensure consistent interpretation of annotation guidelines.

How to Apply on Rex.zone

Apply through Rex.zone by submitting your profile, relevant annotation experience, and domain strengths (NLP, computer vision, content safety, LLM training pipelines). Be prepared for a short guideline comprehension test and a QA evaluation sample to confirm accuracy, reasoning quality, and consistency.

Frequently Asked Questions

Q: What are remote data annotator jobs in Madrid?
They are remote roles focused on data labeling and AI/ML evaluation tasks that support training datasets for NLP, computer vision, and large language model evaluation. Work often includes prompt evaluation, RLHF ranking, content safety labeling, and QA evaluation against annotation guidelines.
Q: Is this role remote and full-time?
Yes. The posting is marked Remote with Employment Type set to FULL_TIME, and work is completed remotely while collaborating with distributed teams.
Q: What skills are most important for this job?
Key skills include data annotation, data labeling, RLHF evaluation, prompt evaluation, QA evaluation, named entity recognition, computer vision annotation, content safety labeling, and understanding how outputs fit into LLM training pipelines.
Q: What types of projects might I work on?
You may support LLM training pipelines (instruction-following and safety), RLHF preference ranking, multilingual NLP labeling, named entity recognition, content safety labeling, and computer vision annotation such as bounding boxes or segmentation.
Q: How is quality measured in data annotation work?
Common metrics include audit scores, guideline adherence, inter-annotator agreement, consistency across batches, clear rationales for decisions, and low rework rates that improve training data quality and model performance improvement.
Q: Do I need prior RLHF or prompt evaluation experience?
It is strongly preferred for LLM-focused work. If you have general data labeling experience, you can often ramp up through calibration, guideline training, and supervised QA evaluation.
Q: Who hires for these roles via Rex.zone?
Employers can include AI labs, tech startups, BPOs, and annotation vendors needing reliable data labeling and evaluation capacity for production AI systems.
Q: Are contract or freelance options available?
This specific posting is FULL_TIME, but the remote data annotation market also includes contract and freelance engagements depending on project needs.

230+Domains Covered

120K+PhD, Specialist, Experts Onboarded

50+Countries Represented

Industry-Leading Compensation

We believe exceptional intelligence deserves exceptional pay. Our platform consistently offers rates above the industry average, rewarding experts for their true value and real impact on frontier AI. Here, your expertise isn't just appreciated - it's properly compensated.

Work Remotely, Work Freely

No office. No commute. No constraints. Our fully remote workflow gives experts complete flexibility to work at their own pace, from any country, any time zone. You focus on meaningful tasks - we handle the rest.

Respect at the Core of Everything

AI trainers are the heart of our company. We treat every expert with trust, humanity, and genuine appreciation. From personalized support to transparent communication, we build long-term relationships rooted in respect and care.

Ready to Shape the Future of AI Data Operations?

Apply Now.