MediaRadar seeks a Senior Data Engineer to be responsible for architecting, building, and scaling robust data pipelines that ensure flawless quality for millions of daily data points. You will lead a team, ensure data integrity, and collaborate cross-functionally on a modern cloud data stack.
What You'll Do
- Design, implement, and optimize scalable and reliable ELT/ETL pipelines using Databricks, Spark, Python, and Store Procedures.
- Model complex data and build solutions for both batch and streaming workloads.
- Guide and support a team of data engineers, providing technical direction and resolving challenges.
- Elevate the team's skills through mentorship and constructive code reviews.
- Develop and implement comprehensive testing frameworks, data validation rules, and QA plans.
- Proactively monitor system performance, tune complex SQL queries, and troubleshoot production issues.
- Perform root-cause analysis and implement lasting solutions to improve system health and reliability.
- Actively participate in an Agile environment (Scrum, sprints, backlog grooming) and collaborate with cross-functional teams.
- Help evaluate and introduce new technologies and best practices to continuously improve our data platform.
What We're Looking For
- A bachelor’s or master’s degree in computer science, engineering, or a related field (or equivalent practical experience).
- 7+ years of hands-on experience in data engineering.
- Strong focus on Databricks, Apache Spark, and 1 RDBMS solution for processing large-scale datasets.
- Expert-level proficiency in Python (including PySpark).
- Strong understanding of data structures, algorithms, and OOP principles.
- Deep expertise in SQL and Spark SQL, with proven experience writing and optimizing complex analytical queries.
- Hands-on experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, and Data frame APIs.
Nice to Have
- Experience building and maintaining CI/CD pipelines using tools like Azure DevOps.
- Previous experience migrating a legacy data system to a modern, unified data platform.
- Familiarity with containerization (Docker, Kubernetes).
- Experience working in a fast-paced, product-driven environment.
Technical Stack
- Azure
- Databricks
- Spark
- Python
- SQL
- Delta Lake
- Unity Catalog
Benefits & Compensation
- Medical, Dental & Vision Insurance
- 401k with Company Match
- Flexible PTO
- Commuter Benefits
- Gym Discounts
- Summer Fridays
- Compensation: $130,000 - $160,000
Work Mode
This role is open to candidates located in the United States.
MediaRadar is committed to creating an inclusive and accessible workplace where everyone can thrive. We believe that diversity of backgrounds, perspectives, and experiences makes us stronger and more innovative.
We are proud to be an Equal Opportunity Employer and make employment decisions without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, or any other legally protected status.
