About the Role
The developer will be responsible for building and maintaining scalable stream processing solutions using Apache Flink, working closely with data engineers and platform teams to implement robust data pipelines.
Responsibilities
- Design and implement real-time data processing workflows using Apache Flink
- Optimize Flink jobs for performance, scalability, and fault tolerance
- Collaborate with data engineers to integrate streaming pipelines with data sources and sinks
- Monitor and troubleshoot running Flink applications in production environments
- Ensure data consistency and accuracy across streaming systems
- Develop custom connectors and user-defined functions for Flink
- Work with large-scale distributed data platforms and messaging systems
- Participate in code reviews and system design discussions
- Maintain documentation for data pipelines and processing logic
- Support incident response and root cause analysis for data stream issues
- Contribute to improving data quality and processing reliability
- Implement event-time processing and windowing logic in Flink
- Integrate Flink applications with storage systems such as Kafka, S3, and databases
- Apply best practices in configuration management and deployment automation
- Assist in capacity planning and resource optimization for cluster operations
- Stay current with developments in Apache Flink and stream processing technologies
- Write unit and integration tests for streaming components
- Collaborate with platform teams on infrastructure improvements
- Ensure compliance with data governance and security policies
- Provide technical guidance to junior team members
- Participate in agile development cycles and sprint planning
- Evaluate new tools and frameworks for potential integration
- Support deployment of Flink applications across multiple environments
- Troubleshoot data skew and backpressure issues in streaming jobs
- Contribute to internal knowledge sharing and training initiatives
Nice to Have
- Contributions to open-source stream processing projects
- Experience with other stream processing frameworks like Spark Streaming or Kafka Streams
- Knowledge of machine learning pipelines in streaming contexts
- Experience with large-scale data platforms in production
- Background in real-time analytics or event processing systems
- Familiarity with cloud providers such as AWS, GCP, or Azure
- Understanding of data privacy and compliance requirements
- Experience with schema evolution and data versioning
- Exposure to high-throughput, low-latency systems
- Participation in incident management and on-call rotations
Compensation
Competitive salary based on experience
Work Arrangement
Remote with flexible hours
Team
Collaborative engineering team focused on real-time data processing
Technology Stack
- Primary use of Apache Flink for stream processing
- Integration with Apache Kafka for messaging
- Deployment on Kubernetes clusters
- Use of cloud storage including S3 and managed databases
- Monitoring via Prometheus and alerting with Grafana
Team Culture
- Emphasis on technical excellence and continuous learning
- Regular knowledge-sharing sessions and team retrospectives
- Supportive environment for innovation and experimentation
- Focus on sustainable work practices and work-life balance
- Open communication and transparent decision-making
Available for qualified candidates
