Abacus Insights is looking for a Senior Data Engineer to join our Tech Ops division. In this role, you will collaborate with customers, data vendors, and internal teams to integrate data into our state-of-the-art cloud platform and drive the successful adoption of core data management solutions.
What You'll Do
- Partner with clients and implementation teams to understand data distribution requirements.
- Design and develop data pipelines integrating with Databricks and Snowflake, ensuring accuracy and integrity.
- Lead architecture and implementation of solutions for health plan clients, optimizing cloud-based technologies.
- Translate business requirements into technical solutions in collaboration with cross-functional teams.
- Implement robust data security measures and ensure compliance with healthcare regulations such as HIPAA.
- Provide technical leadership, mentor junior engineers, and stay current with industry trends.
- Optimize data processes for efficiency, review architectures regularly, and maintain detailed documentation.
- Conduct training sessions for team members and clients to enhance understanding and utilization of data solutions.
What We're Looking For
- Bachelor’s degree in Computer Science, Computer Engineering, or a related IT field.
- 5+ years of experience in data engineering.
- Basic knowledge of U.S. healthcare data (e.g., medical claims, member eligibility).
- Strong communication skills with both technical and non-technical stakeholders.
- Proficiency in Python and SQL.
- Experience building and managing ETL pipelines using Databricks workflows or similar orchestration frameworks.
- Familiarity with structured and semi-structured data, and processing using PySpark.
- Working knowledge of AWS services (S3, SQS) or equivalent cloud services.
- Experience with Terraform and GitLab CI/CD.
- Skills in query tuning and performance optimization in SQL and/or SparkSQL.
- Familiarity with data warehousing solutions such as Snowflake or similar platforms.
Nice to Have
- Experience working for a cloud-based data services provider serving large healthcare clients.
- Knowledge of DBT and Kafka.
Technical Stack
- Languages: Python, SQL
- Platforms/Tools: Databricks, Snowflake, PySpark, AWS (S3, SQS), Terraform, GitLab CI/CD, SparkSQL
Team & Environment
You will be joining the Tech Ops division, working in a collaborative, cross-functional environment.
Benefits & Compensation
- Unlimited paid time off
- Work from anywhere flexibility
- Comprehensive health coverage with multiple plan options
- Equity for every employee
- Growth-focused environment
- Home office setup allowance
- Monthly cell phone allowance
Work Mode
This is a remote position with a global team.
Abacus Insights is an Equal Opportunity Employer dedicated to building diverse teams and providing equal employment opportunities to all applicants, prohibiting discrimination and harassment.
