[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"job-github-repo-code-annotator-remote":3},{"Slug":4,"Header":5,"Ques":36,"job_category":69},"github-repo-code-annotator-remote",{"title":6,"desc":7,"content":8},"Repo-based Code Annotator","Remote, repo-focused role building reproducible Docker test environments, reviewing unit tests, validating SWE-Bench\u002FTerminal-Bench workflows, and writing clear task documentation. Compensation: USD $80–$120 per day, subject to skills and experience.",[9,12,15,18,21,24,27,30,33],{"h2":10,"desc":11},"About the Role","This role focuses on making software evaluation reproducible and trustworthy. You will:\n\n- Build standardized, reproducible test environments using Docker images to replicate known issues and validate expected outputs.\n- Review and strengthen unit test coverage to assess correctness and stability of target repositories.\n- Validate task and test set completeness for workflows aligned with SWE-Bench and Terminal-Bench.\n- Produce high-quality task documentation (e.g., task.yaml and README) that emphasizes reproducibility and standardized processes.",{"h2":13,"desc":14},"Key Responsibilities","- Author Dockerfiles and compose minimal, efficient images that reliably reproduce target repo states.\n- Analyze existing test suites; add or refine pytest-based tests to increase signal without introducing flakiness.\n- Implement and verify controls for randomness and external I\u002FO (mocking, seeding, fixtures).\n- Ensure Git\u002FGitHub workflows produce clean, reproducible pull requests with clear diffs and instructions.\n- Maintain task harnesses and automation scripts in Python (CLI tools, fixtures, adapters).\n- Validate benchmark-aligned workflows (SWE-Bench\u002FTerminal-Bench) for repeatability and fairness.\n- Document assumptions, environment variables, entrypoints, and verification steps for each task.",{"h2":16,"desc":17},"Required Skills","- Proficient in Linux command line and Shell scripting; comfortable with grep, sed, awk, curl, jq.\n- Strong Python programming skills for harnesses, tests, and automation.\n- Proficient with Docker: writing Dockerfiles, building images, and ensuring reproducible builds.\n- Familiar with pytest and unit testing practices, including mocking and controlling randomness.\n- Comfortable with Git\u002FGitHub workflows; able to submit clean, reproducible pull requests.",{"h2":19,"desc":20},"Professional Background","- Background in Computer Science, Software Engineering, Artificial Intelligence, or related fields; or experience in Software Development, Test Engineering, DevOps, or Data Engineering.\n- Preference for contributors to open-source projects, especially in automated testing, CI\u002FCD, or containerization.",{"h2":22,"desc":23},"Bonus Points","- Experience with high-performance languages (Go or Rust).\n- Familiarity with sandbox and orchestration tech (Docker Compose, Podman).\n- Ability to design datasets\u002Ftasks that discourage “task cheating.”\n- Understanding of benchmark design principles: fairness, repeatability, and scalability.\n- Experience with CI\u002FCD or automated testing systems and cross-disciplinary work.",{"h2":25,"desc":26},"Compensation","USD $80–$120 per day, dependent on demonstrated skills, experience, and the complexity of assigned tasks.",{"h2":28,"desc":29},"Work Arrangement","- Remote opportunity.\n- Collaborate asynchronously via GitHub issues\u002FPRs and documented workflows.\n- Outcome-oriented: emphasis on reproducibility, clarity, and measurable test improvements.",{"h2":31,"desc":32},"Tools & Technologies","- Linux, Shell (bash\u002Fsh), grep\u002Fsed\u002Fawk\u002Fcurl\u002Fjq\n- Python, pytest\n- Docker, Dockerfiles\n- Git, GitHub\n- Optional: Go, Rust, Docker Compose, Podman",{"h2":34,"desc":35},"Quality & Standards","- Reproducibility: deterministic environments with documented versions and seeds.\n- Clarity: task.yaml\u002FREADME must include setup, usage, and validation steps.\n- Maintainability: modular harnesses, minimal external dependencies, clear test fixtures.\n- Integrity: benchmark-aligned tasks that are resistant to shortcuts and ensure fair evaluation.",{"title":37,"content":38},"Repo-based Code Annotator — FAQ",[39,42,45,48,51,54,57,60,63,66],{"Q":40,"A":41},"What does “repo-based code annotation” entail in this role?","You will prepare repositories for reliable automated evaluation by building Dockerized environments, refining unit tests, and writing documentation that makes tasks repeatable and verifiable. The focus is on correctness, coverage, and clear instructions.",{"Q":43,"A":44},"How do SWE-Bench and Terminal-Bench relate to day-to-day work?","You will align workflows and test sets with the requirements of SWE-Bench and Terminal-Bench, ensuring tasks are complete, fair, and reproducible, with deterministic behavior and well-defined validation steps.",{"Q":46,"A":47},"Which programming and tooling skills are essential?","Python (for harnesses\u002Ftests), Linux shell tools (grep\u002Fsed\u002Fawk\u002Fcurl\u002Fjq), Docker (Dockerfiles, reproducible builds), pytest (fixtures, mocking, seeding), and Git\u002FGitHub for clean PRs.",{"Q":49,"A":50},"Is experience with Go or Rust required?","No. Go and Rust are a plus for performance-oriented tasks, but they are not mandatory for success in this role.",{"Q":52,"A":53},"How is success measured?","By reproducible environments, improved test coverage and stability, clear documentation, and workflows that meet benchmark standards without flakiness or ambiguity.",{"Q":55,"A":56},"Is the position fully remote and asynchronous?","Yes. Work is remote and collaboration occurs via GitHub and documented processes, with an emphasis on asynchronous communication.",{"Q":58,"A":59},"What level of seniority is expected?","A practitioner comfortable with Python testing, Docker, and Linux tooling. You should be able to independently produce reproducible environments and clear documentation without requiring advanced research-level expertise.",{"Q":61,"A":62},"What compensation can I expect?","The daily rate is USD $80–$120, dependent on your proven skills, relevant experience, and the complexity of work assigned.",{"Q":64,"A":65},"Do I need open-source contributions to be considered?","Open-source contributions are preferred—especially in testing, CI\u002FCD, or containerization—but they are not strictly required.",{"Q":67,"A":68},"What kind of documentation is expected?","Task.yaml and README files that state environment setup, dependencies, execution steps, validation procedures, and any assumptions or constraints. The goal is repeatability and clarity.","Code Annotation"]