Remote AI Data Annotation Jobs in New York

Remote AI Data Annotation Jobs in New York at Rex.zone focus on training data quality for modern AI systems, including data labeling, RLHF evaluation, prompt evaluation, and QA review across NLP and computer vision workflows. You will follow annotation guidelines compliance, apply rubric-based judgments, and help drive model performance improvement for large language model evaluation and content safety labeling. These roles support LLM training pipelines used by AI labs, tech startups, and annotation vendors, with tasks like named entity recognition, image and video annotation, and policy-aligned safety classification. Explore full-time remote opportunities designed for consistent throughput, high accuracy, and measurable quality metrics.

Remote AI Data Annotation Jobs in New York

About the Role

You will perform remote AI data annotation supporting production AI/ML training workflows. Your work will include data labeling for NLP and computer vision, RLHF-style preference ranking, prompt evaluation for instruction-following behavior, and QA evaluation using calibrated scoring rubrics. You will interpret and apply detailed annotation guidelines, document edge cases, and resolve ambiguity through consistent rationale. You will also contribute to training data quality initiatives by analyzing disagreement patterns, reducing label noise, and proposing improvements that support model performance improvement and reliable offline evaluation.

Key Responsibilities

Produce high-accuracy labels for text, image, and multimodal datasets aligned to project schemas. Execute RLHF tasks such as pairwise preference judgments, ranking, and rationale writing when required. Complete prompt evaluation and response grading for helpfulness, correctness, and policy alignment. Perform QA evaluation, including peer review, spot checks, and adjudication with clear decision logs. Apply named entity recognition and taxonomy tagging for NLP datasets, including span selection and attribute assignment. Support computer vision annotation such as bounding boxes, polygons, keypoints, and segmentation masks per spec. Conduct content safety labeling across categories (harassment, self-harm, sexual content, hate, violence) using policy definitions and threshold rules. Track throughput and quality metrics, follow escalation paths, and maintain annotation guidelines compliance across batches.

What Success Looks Like

Consistently meets accuracy targets on gold sets and maintains stable inter-annotator agreement. Demonstrates strong calibration on ambiguous cases and applies rubrics with minimal variance. Produces clear rationales that improve reviewer alignment and reduce rework. Identifies recurring dataset issues (schema gaps, unclear instructions, class overlap) and proposes actionable fixes. Delivers reliable throughput while maintaining training data quality and auditability for downstream large language model evaluation.

Required Qualifications

Experience in AI data annotation, data labeling, QA evaluation, or related data operations. Demonstrated ability to follow annotation guidelines compliance standards and work within rubric-based evaluation. Familiarity with NLP concepts such as named entity recognition, intent classification, and text categorization. Ability to write concise rationales, handle sensitive content, and maintain confidentiality on proprietary datasets. Strong attention to detail, comfort with repetitive tasks, and ability to meet productivity expectations in a remote environment.

Preferred Qualifications

Hands-on experience with RLHF, preference ranking, prompt evaluation, or large language model evaluation. Exposure to computer vision annotation tasks such as bounding boxes, polygons, and segmentation. Experience supporting content safety labeling and policy-driven moderation datasets. Prior work with AI labs, tech startups, BPOs, or annotation vendors in production labeling pipelines. Familiarity with quality programs such as gold data creation, calibration sessions, and disagreement analysis.

Tools and Workflow

Work is completed in web-based labeling and evaluation tools with task queues, guideline wikis, and QA workflows. You will use structured rubrics, gold checks, and reviewer feedback loops to improve training data quality. Projects may span NLP classification, named entity recognition, computer vision annotation, RLHF preference ranking, and content safety labeling. Documentation and communication happen asynchronously with clear handoffs, versioned guidelines, and audit-ready decision notes.

Work Conditions and Location

Remote role based in the US, aligned with New York job search intent. You must be able to perform focused labeling and evaluation work, handle occasional sensitive content for content safety labeling, and maintain consistent availability during agreed working hours. This is a full-time remote position supporting ongoing AI/ML training pipelines.

How to Apply on Rex.zone

Apply via Rex.zone by submitting your profile, selecting Remote AI Data Annotation Jobs in New York, and completing any required skills checks. Keep your work samples focused on data labeling, QA evaluation, RLHF-style judgments, or prompt evaluation when applicable. Qualified candidates may receive calibration tasks to confirm annotation guidelines compliance and training data quality standards.

Frequently Asked Questions

Q: Are these roles fully remote?
Yes. The Remote Type is Remote, and the work is completed online using labeling and evaluation tools.
Q: What kinds of annotation tasks are included?
Common tasks include data labeling for NLP and computer vision, named entity recognition, content safety labeling, prompt evaluation, RLHF preference ranking, and QA evaluation for training data quality.
Q: Is this job entry-level or senior?
This posting targets Mid-Senior experience, emphasizing consistent quality, calibration, and ownership of guideline adherence.
Q: What does RLHF mean in day-to-day work?
RLHF work typically involves ranking or comparing model outputs, scoring responses with rubrics, writing rationales, and helping align model behavior for large language model evaluation.
Q: Do I need a computer vision background?
Not always, but experience with computer vision annotation (bounding boxes, polygons, segmentation) is a plus because projects may span multimodal datasets.
Q: Will I review sensitive content?
Some projects involve content safety labeling. If assigned, you will follow defined policies and escalation paths to label content consistently and safely.
Q: What makes candidates successful in AI data annotation?
Success comes from strong attention to detail, annotation guidelines compliance, stable rubric calibration, clear rationale writing, and a focus on training data quality that supports model performance improvement.
Q: How does Rex.zone fit into the application process?
Rex.zone is the platform where you explore the role, apply, and complete any assessments or calibration tasks required for production AI/ML labeling workflows.

230+Domains Covered

120K+PhD, Specialist, Experts Onboarded

50+Countries Represented

Industry-Leading Compensation

We believe exceptional intelligence deserves exceptional pay. Our platform consistently offers rates above the industry average, rewarding experts for their true value and real impact on frontier AI. Here, your expertise isn't just appreciated - it's properly compensated.

Work Remotely, Work Freely

No office. No commute. No constraints. Our fully remote workflow gives experts complete flexibility to work at their own pace, from any country, any time zone. You focus on meaningful tasks - we handle the rest.

Respect at the Core of Everything

AI trainers are the heart of our company. We treat every expert with trust, humanity, and genuine appreciation. From personalized support to transparent communication, we build long-term relationships rooted in respect and care.

Ready to Shape the Future of AI Data Operations?

Apply Now.