Data Software Engineer
Israel, Tel Aviv
Responsibilities
- Use your skills and experience to design and develop our innovative big data platform
- Use big data frameworks such as Spark, Kafka, Redshift, Airflow, Flink, k8s, Hudi and so on to develop large scale data pipelines
- Lead, deliver and own high scale projects end to end
- Work closely with the data science team
- Lead technological initiatives and improvements while instilling development of “best practices”
- Develop processes and tools to monitor, analyze, maintain and improve data operation, performance, and usability
Requirements
- At least 2 years experience in Scala/Java or equivalent language
- Proven experience in building, optimizing, and maintaining large scale big data pipelines using popular open-source frameworks (Kafka, Spark, Hive, Presto, Airflow, etc)
- Strong SQL abilities and hands on experience with SQL and NoSql DBs
- Experience with AWS cloud services such as EMR, Redshift, S3, Athena, Glue – advantage
- Experience in Python
- Experience in stream processing frameworks such as Flink
- Experience with scientific computing packages (numpy, scipy, pandas etc.) – advantage
