Responsibilities
- Design and construct production data pipelines using Lakeflow Declarative Pipelines, Autoloader, and Structured Streaming, taking full ownership from ingestion to transformation, data quality, and CI/CD deployment via Declarative Automation Bundles.
- Architect and deploy Lakehouse solutions on Databricks, including medallion architecture, Delta Lake, and Unity Catalog, customized to meet client analytics, AI, and application requirements.
- Develop and maintain Databricks transformation layers using DLT pipelines, PySpark notebooks, and dbt, incorporating data quality constraints and service level agreements.
- Design and uphold data and AI foundations such as Unity Catalog, Feature Store, MLflow, and Model Serving to support production machine learning, agent workflows, and AI-enhanced digital products.
- Collaborate with product and backend engineers to design data models, APIs, and application data contracts, ensuring the platform effectively serves product needs beyond just data warehousing.
- Consult with clients to comprehend their data challenges, formulate data strategies, and implement durable solutions.
- Adjust approach based on project demands, sometimes leading data architecture discussions with clients, other times providing specialized data expertise to internal teams.
- Operate within multi-cloud environments, primarily AWS and Azure, recommending data platforms centered on Databricks when aligned with client architecture and objectives.
- Advocate for data governance through Unity Catalog, integrating access control, lineage, data quality policies, and compliance as essential components of every engagement.
- Design data-to-application architectures, including Lakebase-backed services and Databricks Apps, linking governed data to AI workflows, digital products, and user experiences.
- Contribute to building the Databricks practice by developing accelerators, supporting internal enablement, pursuing certifications, and assisting with partner go-to-market materials alongside delivery tasks.
Benefits
- Collaborate with enthusiastic and skilled colleagues who continuously seek improvements.
- Work in an environment where respect, mutual trust, and egoless collaboration are top priorities.
- Join colleagues who are serious about their work but maintain a lighthearted attitude and enjoy having fun.
- Be part of a team known for excellence that contributes to the community through education, mentoring, and sponsorship.
- Work on products and accounts with significant impact and broad reach.
- Help establish a data practice specialization from scratch, influencing market strategies, accelerator development, and the definition of this work at a digital product company.
