Spear AI is looking for a Senior Data Engineer to build the next-generation data management and AI platform for maritime domain awareness. You will work on real-world projects that directly impact warfighter capabilities by processing data from sonobuoy sensors and U.S. Navy SONAR data.
What You'll Do
- Implement real-time data pipelines with MQTT and Redpanda for stream processing.
- Implement offline data pipelines using Dagster for batch processing.
- Parse and process binary message formats from various data sources.
- Build data warehouses using Postgres, Apache Iceberg, Parquet, and S3.
- Design data models that allow for high-performance queries.
- Validate and normalize data sources.
- Improve local development and CI/CD using modern tooling and GitHub Actions.
What We're Looking For
- Expertise in time-series data processing and analysis.
- Proficiency in Python and Rust for data engineering workflows.
- Experience with binary message parsing.
- Experience with row-based and columnar-based data formats.
- Experience with OLTP and OLAP databases.
- Knowledge of distributed systems, streaming architectures, and batch processing patterns.
- Hands-on experience with a batch orchestrator such as Dagster or Airflow.
- Hands-on experience with a streaming platform such as Redpanda or Kafka.
- Hands-on experience with binary message formats such as Protobuf.
Nice to Have
- Experience with IoT devices and sensors.
- Digital signal processing experience.
- Geospatial analysis and GIS experience.
- Familiar with working in monorepos.
Technical Stack
- Languages: Python, Rust
- Streaming: MQTT, Redpanda, Kafka
- Orchestration: Dagster, Airflow
- Databases & Storage: Postgres, Apache Iceberg, Parquet, S3
- Formats & Tools: Protobuf, GitHub Actions
Team & Environment
You'll join a small team with a flat organizational structure and collaborate directly with leadership.
Benefits & Compensation
- Unlimited PTO and dedicated sick time.
- Comprehensive health, dental, and vision insurance.
- 11 paid holidays annually.
- Professional development opportunities and certification reimbursement.
- 401(k) with company match.
- Flexible, hybrid, or fully remote work arrangements.
- Relocation assistance.
- Referral and performance bonuses.
- Life insurance and disability coverage.
- Technology home office setup stipend.
Work Mode
This is a remote position.
Spear AI is an equal opportunity employer.
