LILT is seeking a Native Language Specialist in Serbian to design, build, and validate high-signal evaluation tasks for large language models, focusing on multilingual software challenges in terminal workflows. This role emphasizes creating authentic non-English coding environments to rigorously test AI robustness across language, encoding, and locale edge cases.
What You'll Do
- Design, build, and validate Terminal-Bench tasks that test the limits of large language models on multilingual software challenges
- Evaluate coding agents through realistic task engineering
- Create realistic task environments using datasets and files in the native language, ensuring content remains in the target language
- Identify failure points where AI does not work by prompting and translating in the native language
- Support development of robust reference implementations for tasks
- Write highly reliable, deterministic verifier scripts using rubric-based judging only when strictly necessary
- Analyze execution logs to calibrate task difficulty across standard Terminal-Bench configurations
- Calibrate task difficulty levels (Easy to Very Hard) against various model tiers (Haiku, Sonnet, Opus)
- Participate in a 4-layer human quality control process: creation, human review, calibration review, and audit
- Ensure fairness, grammatical accuracy, and benchmark integrity through both human and automated LLM-based checks
What We're Looking For
- 5+ years of industry experience in software engineering
- Proven track record at leading technology companies or graduation from top-tier engineering universities
- Native or near-native fluency in Serbian with deep understanding of grammar, register, and phrasing rules
- High English proficiency
- Strong proficiency in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Working familiarity with coding agents
- Deep technical understanding of multilingual text processing pitfalls
- Expertise in encoding/decoding robustness and Unicode normalization
- Knowledge of locale-dependent conventions including collation, casing, and non-Gregorian dates
- Experience with text I/O, toolchain interoperability, and safe string operations
Nice to Have
- Experience with bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts for Serbian language
Technical Stack
- Python
- Shell scripting
- Data processing
- Terminal/CLI workflows
- Unicode normalization
- Encoding/decoding systems
- Locale-aware text processing
Team & Environment
- Global community of linguists, subject matter experts, and language professionals
- Distributed network of independent contractors collaborating on AI benchmark projects
- Reporting structure not specified
Benefits & Compensation
- Remote freelance opportunity with full schedule autonomy
- Work when you want, as much or as little as you want
- No fixed hours, no check-ins, no micromanaging
- Competitive rates with prompt payments
- No chasing invoices
- Work on AI and language technology projects
- Contribute to shaping how humans and machines communicate
- Join a global community advancing human knowledge
- Access to diverse, innovative projects that expand your portfolio
- Opportunities to sharpen skills across industries and domains
- Supportive professional network and community
- Streamlined application process tailored to expertise
- Work on diverse projects from anywhere, any time
Compensation: Competitive rates, prompt payments
Work Mode
- Work from anywhere, anytime; full autonomy over schedule and workload
LILT is an equal opportunity employer. We extend equal opportunity to all individuals without regard to race, religion, color, national origin, ancestry, sex, sexual orientation, gender identity, age, physical or mental disability, medical condition, genetic characteristics, veteran or marital status, pregnancy, or any other classification protected by applicable laws. We are committed to fair employment and eliminating discriminatory practices.
