Dev.Pro is hiring a Senior LLM Systems Engineer - OPS00041

Dev.Pro is hiring a Senior LLM Systems Engineer to architect, implement, and optimize multi-node LangGraph pipelines for a civic intelligence startup building a data-backed Trust Score for elected officials. This role focuses on graph-based orchestration, data integration, evaluation pipelines, and latency optimization.

What You'll Do

• Build multi-node LangGraph workflows with dynamic routing, shared state, and conditional edges.
• Define and manage the graph schema with nodes as active processors and edges as routing logic.
• Implement LLM-driven intent classification to route user input across nodes or mark it out-of-scope.
• Manage state synchronization so nodes only update relevant data.
• Connect nodes to multiple external JSON-returning endpoints and normalize data structures.
• Design intermediate transformation layers to unify inconsistent API responses.
• Use mock endpoints and seamlessly transition them to live APIs without breaking architecture.
• Handle asynchronous and parallel API calls to minimize latency.
• Implement prompt and query-level guardrails to block unsafe inputs and prevent data leakage.
• Design state-aware routing logic that enforces isolation and privacy constraints between nodes.
• Build internal validators for payloads, ensuring all node inputs/outputs conform to strict schemas.
• Develop synthetic JSON-based ground-truth datasets to test system output deterministically.
• Build automated evaluation scripts to calculate F1, precision, recall, and exact-match scores.
• Compare multiple LLMs (Claude, GPT, etc.) by node and metric to determine best performers.
• Automate regression testing for new model versions or prompt updates.
• Use async execution and parallel node evaluation to reduce latency.
• Stream partial responses to improve perceived performance.
• Profile system components to locate and fix slow or sequential bottlenecks.
• Implement caching or smart pre-fetching for frequently used data sources.
• Build logging, tracing, and metric dashboards for every node and edge in the graph.
• Define error-handling strategies for malformed API responses or timeouts.
• Maintain test coverage across orchestration logic, node isolation, and evaluation functions.
• Implement CI/CD hooks to automatically re-evaluate the system before deployment.

What We're Looking For

• Advanced Python skills including async I/O, FastAPI, type hinting, logging, and pytest.
• Expert-level LangGraph orchestration knowledge of state machines, conditional edges, and node composition.
• Deep experience with LLM APIs (OpenAI, Anthropic, etc.) including structured prompting, JSON mode, and token usage optimization.
• Data Engineering background in JSON schema normalization, API integration, and validation layers.
• Proven ability to build Evaluation Systems with F1, precision/recall metrics, dataset design, and automated scoring.
• Expertise in Asynchronous & Parallel Processing using asyncio, concurrent futures, and non-blocking execution.
• Experience implementing Security / Guardrails such as prompt validation, regex filters, payload sanitization, and sandboxing.
• DevOps / Tooling proficiency with Docker, CI/CD, logging, and observability (Grafana, Prometheus, OpenTelemetry).

Nice to Have

• Experience with LangChain, LlamaIndex, or orchestration frameworks beyond LangGraph.
• Familiarity with vector databases (pgvector, Pinecone, Weaviate).
• Background in deterministic or safety-critical LLM applications (finance, legal, etc.).
• Comfort with multi-model evaluation pipelines and A/B testing at node level.

Technical Stack

• Python, FastAPI, LangGraph
• OpenAI, Anthropic
• Docker, Grafana, Prometheus, OpenTelemetry

Work Mode

This role supports a global work mode.

Dev.Pro operates on values of democracy, human rights, and state sovereignty, and offers challenging projects with world-leading clients, a modern technology stack, and career opportunities.