Mission
Key Responsibilities:
Design, develop, and implement efficient ELT/ETL processes for large datasets.Build and optimize data processing workflows using Apache Spark.Utilize Python for data manipulation, transformation, and analysis.Develop and manage data pipelines using Apache Airflow.Write and optimize SQL queries for data extraction, transformation, and loading.Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions.Work within an on-premise computing environment for data processing and storage.Ensure data quality, integrity, and performance throughout the data lifecycle.Participate in the implementation and maintenance of CI/CD pipelines for data processes.Utilize Git for version control and collaborative development.Troubleshoot and resolve issues related to data pipelines and infrastructure.Contribute to the documentation of data processes and systems. Profile
Minimum 2 years of professional experience as a programmer working with large datasets.Experience in at least 1 project involving the processing of large datasets.Experience in at least 1 project programming with Python.Experience in at least 1 project within an on-premise computing environment.Proven experience programming with Apache Spark.Proven experience programming with Python.Proven experience programming with Apache Airflow.Proven experience programming with SQL.Familiarity with Hadoop concepts.Proven experience in programming ELT/ETL processes.Understanding of CI/CD principles and practices.Proficiency in using a version control system (Git).Strong self-organization skills and a goal-oriented approach.Excellent interpersonal and organizational skills, including planning.Strong communication, creativity, independence, professionalism, stress resistance, and inquisitiveness.Adaptability and flexibility, with an openness to continuous learning and development. Work is conducted in a hybrid system: 2 times a month from one of our office - Warsaw, Katowice, Poznan, Rzeszow, Lodz or Lublin.