Senior Data Engineer Full Time

We’re looking for a Senior Data Engineer for a business that is a top player within the aerial robotics space. A direct competitor to DJI, this business is bringing drones to the mainstream with its menu of affordable consumer & commercial drones.

They also offer services across different areas such as emergency response, security, surveying & mapping to name a few, creating many revenue streams for the business.

Job responsibilities:

• Create and maintain optimal data pipeline architecture, identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading data from a wide variety of data sources.
• Build analytics tools that utilize the data pipeline to provide actionable insights.
• Build ML data Pipeline for deep learning and ETL for logs.

Must-haves:

• Master’s degree or above, in Computer Science, Math, EE, Robotics, or equivalent
• degrees
• 3+ years as software engineer in developing big data applications
• Experienced in developing large scale batch data processing (ETL) and real time streaming data process
• In-depth understanding in Apache Spark (Preferred); Familiar with Spark API,
• Spark SQL, Spark data frame. Had experience in using spark for large scale ETL applications. Knowledge of spark internals is preferred.
• Familiar with optimization skills in big data processing. Familiar with big data storage formats such as Parquet, ORC, Hudi, Iceberg, Deltalake, etc.
• Familiar with Time Series Engine such as Influx DB or TDEngine is preferred
• Familiar with cloud native workflow engine such as Argo or Airflow is preferred
• Familiar with cloud native skills such as K8S or Docker