Remote Data Labeling Jobs in New York

Remote data labeling jobs in New York on Rex.zone focus on creating high-quality training data for AI systems used in NLP and computer vision. You will label text, images, and conversations to support LLM training pipelines, RLHF ranking, prompt evaluation, and content safety labeling, while following annotation guidelines compliance and improving training data quality. This role connects day-to-day annotation work to model performance improvement through QA evaluation, error analysis, and consistent taxonomy decisions across datasets for AI labs, tech startups, and annotation vendors.

Job Image

Remote Data Labeling Jobs in New York — LinkedIn Job Metadata

Title: Remote Data Labeling Specialist (New York) Date: 25-02-2026 Company: Rex.zone Country: US Remote Type: Remote Employment Type: FULL_TIME Experience Level: Mid-Senior Industry: Technology Job Function: Engineering Skills: Data labeling, Data annotation, RLHF evaluation, Prompt evaluation, QA evaluation, Training data quality, Annotation guidelines compliance, Named entity recognition, Computer vision annotation, Content safety labeling Salary Currency: USD Salary Min: 63360 Salary Max: 126720 Pay Period: YEAR

About the Role

You will support AI/ML training workflows by labeling and evaluating multimodal datasets (text, image, and conversational data). Daily work includes applying annotation taxonomies, performing RLHF-style preference ranking, running prompt evaluation and response grading, and documenting edge cases to improve guideline clarity. You will collaborate with data operations, ML engineers, and QA reviewers to ensure dataset consistency, reduce label noise, and improve downstream model performance.

What You Will Do

Key responsibilities include: (1) Perform data labeling and data annotation for NLP and computer vision tasks using provided tools and schemas. (2) Execute QA evaluation workflows such as spot checks, inter-annotator agreement reviews, and systematic error tagging. (3) Complete RLHF evaluation tasks including preference ranking, pairwise comparisons, and rubric-based scoring for LLM responses. (4) Conduct prompt evaluation to validate instruction-following, reasoning quality, and safety behaviors. (5) Maintain annotation guidelines compliance by escalating ambiguous cases, proposing clarifications, and following versioned instructions. (6) Contribute to training data quality improvements through feedback loops, consistency audits, and taxonomy refinement.

Core Workstreams You May Support

Depending on project needs, you may work on: (1) Named entity recognition and entity linking for searchable knowledge extraction. (2) Computer vision annotation including bounding boxes, polygons, keypoints, and attribute tagging. (3) Content safety labeling for policy categories such as harassment, self-harm, hate, violence, and adult content. (4) LLM evaluation including helpfulness, factuality, refusal quality, and conversational coherence. (5) Data curation tasks such as deduplication flags, language identification, and domain tagging for dataset stratification.

Requirements

To succeed in these remote data labeling jobs based in New York, you should have: (1) Professional experience in data labeling, annotation operations, or evaluation workflows. (2) Strong attention to detail with consistent application of rubrics and taxonomies. (3) Familiarity with QA evaluation concepts such as agreement checks, sampling, and review queues. (4) Comfort working with ambiguous examples and documenting rationales for decisions. (5) Ability to follow annotation guidelines compliance standards and meet throughput and quality targets.

Preferred Qualifications

Nice to have: (1) Experience with RLHF evaluation, preference ranking, or LLM response grading. (2) Exposure to NLP tasks such as named entity recognition, sentiment, intent, or classification. (3) Exposure to computer vision annotation (boxes, polygons, segmentation). (4) Understanding of content safety labeling and policy-driven decisioning. (5) Experience working with AI labs, tech startups, BPOs, or annotation vendors in production labeling pipelines.

Tools and Workflow

You will use web-based labeling platforms and QA tools to complete tasks, track throughput, and manage review feedback. Work typically includes calibration sessions, rubric updates, golden set checks, and periodic quality audits. You will be expected to communicate clearly in written form when escalating edge cases and summarizing evaluation findings.

Compensation and Employment Details

This is a full-time remote role for candidates in the US, aligned to New York job search intent. Compensation is listed in USD and paid on a yearly basis within the stated range, depending on project scope, demonstrated quality metrics, and relevant annotation/evaluation experience.

How to Apply on Rex.zone

Apply through Rex.zone to be considered for current and upcoming remote data labeling jobs in New York. Your application should highlight relevant annotation experience, QA evaluation exposure, RLHF or prompt evaluation familiarity, and examples of guideline-driven decision making that improves training data quality and model performance improvement.

Frequently Asked Questions

  • Q: What are remote data labeling jobs in New York on Rex.zone?

    They are full-time remote roles where you create and evaluate training data for AI systems by labeling text, images, and conversations, including tasks like RLHF evaluation, prompt evaluation, QA evaluation, named entity recognition, computer vision annotation, and content safety labeling.

  • Q: Is this role truly remote and available to candidates in New York?

    Yes. The remote type is Remote and the posting is optimized for New York-based job seekers while remaining US-based and remote eligible.

  • Q: What is RLHF evaluation and why is it included in data labeling work?

    RLHF evaluation (Reinforcement Learning from Human Feedback) involves preference ranking and rubric-based scoring of model responses. It is used to improve large language model behavior and is a common extension of modern data labeling and QA evaluation workflows.

  • Q: What does training data quality mean in practice?

    Training data quality typically refers to accurate labels, consistent taxonomy application, strong annotation guidelines compliance, low label noise, and thorough QA evaluation so models train on reliable examples that drive model performance improvement.

  • Q: What domains will I label for (NLP, computer vision, content safety)?

    Projects may include NLP tasks like named entity recognition, computer vision annotation like bounding boxes or segmentation, and content safety labeling for policy categories. Assignment depends on current employer demand across AI labs, tech startups, BPOs, and annotation vendors.

  • Q: Is this contract or freelance work, and are other levels available?

    This posting is FULL_TIME and Mid-Senior. Rex.zone may also list contract, freelance, entry-level, and senior roles depending on project openings, but this page targets a full-time remote data labeling job intent.

  • Q: What skills should I emphasize to be competitive?

    Emphasize data labeling, data annotation, training data quality practices, annotation guidelines compliance, QA evaluation, RLHF evaluation, prompt evaluation, and domain skills such as named entity recognition, computer vision annotation, and content safety labeling.

230+Domains Covered
120K+PhD, Specialist, Experts Onboarded
50+Countries Represented

Industry-Leading Compensation

We believe exceptional intelligence deserves exceptional pay. Our platform consistently offers rates above the industry average, rewarding experts for their true value and real impact on frontier AI. Here, your expertise isn't just appreciated - it's properly compensated.

Work Remotely, Work Freely

No office. No commute. No constraints. Our fully remote workflow gives experts complete flexibility to work at their own pace, from any country, any time zone. You focus on meaningful tasks - we handle the rest.

Respect at the Core of Everything

AI trainers are the heart of our company. We treat every expert with trust, humanity, and genuine appreciation. From personalized support to transparent communication, we build long-term relationships rooted in respect and care.

Ready to Shape the Future of AI Data Operations?

Apply Now.