Diligent Corporation is hiring a Technical Director, Data Engineering

Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms. This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation. You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job. The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity. What You’ll Do - Design and build scalable data platforms and distributed processing systems - Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data - Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows - Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware - Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams - Improve performance, reliability, observability, and cost efficiency of data systems - Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities - Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity - Mentor engineers technically through design reviews, architecture guidance, and pair debugging - Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence What We’re Looking For Strong Hands-On Engineering Experience Candidates should have significant real-world experience building and operating production systems using many of the following: Data & Distributed Systems - Airflow - Elasticsearch / OpenSearch - Vector databases and semantic retrieval systems - MongoDB, PostgreSQL, DynamoDB, or similar platforms Cloud & Infrastructure - AWS - AWS CDK - Serverless architectures - Distributed observability and monitoring stacks AI / Search / Modern Data Applications - LLM integration patterns - RAG architectures - Embeddings and vector search - MCP servers and AI orchestration frameworks - LangChain, LlamaIndex, DSPy, or similar ecosystems - AI evaluation, tracing, and observability tooling - Search relevance and ranking systems Backend Engineering - Python strongly preferred - Experience with Java, Go, or TypeScript is a plus - API design and distributed service architectures - Event-driven and asynchronous systems The Right Candidate - Still enjoys building and debugging systems directly - Has strong technical depth, not just architectural vocabulary - Comfortable operating in ambiguity and fast-moving environments - Understands how to simplify systems instead of endlessly abstracting them - Has experience modernizing legacy platforms and evolving architectures incrementally - Can distinguish between engineering fundamentals and hype cycles - Values shipping working systems over theoretical perfection What Success Looks Like - Engineering teams can move faster because the underlying platforms are reliable and scalable - AI and data systems become production-grade rather than experimental prototypes - Infrastructure costs and operational complexity are reduced through better architecture - Search, ingestion, and retrieval systems improve significantly in performance and relevance - Teams adopt consistent, maintainable technical patterns without excessive bureaucracy Nice to Have - Experience with large-scale news, document, or regulatory data pipelines - Experience with search relevancy tuning and semantic retrieval - Exposure to governance, compliance, risk, or enterprise SaaS platforms - Familiarity with modern AI engineering workflows and agentic systems - Experience supporting both startup-speed execution and enterprise-scale operations