Senior Data Engineer

Senior Data Engineer

Ciklum

Remote, Bulgaria

About the role:

As a Senior Data Engineer, become a part of a cross-functional development team engineering experiences of tomorrow. Together, we will work on the project to build and operate a serverless data platform powering analytics on AWS. Our cutting-edge environment focuses on scalable pipelines, Iceberg-based lakehouse architecture, and modern serverless processing models.

Thus, a commitment to collaborative problem solving, sophisticated design, and product quality is essential.

Responsibilities:

• Design and build high-performance data pipelines, including Python AWS Lambda functions, Step Functions workflows, and Spark jobs to process events and batch workloads;
• Own the data lake by modeling, evolving, and optimizing Apache Iceberg tables backed by S3 and Glue;
• Query and serve data with Trino and Athena, manage SQL templates using Jinja2, and ensure reliable execution;
• Ensure performance and reliability using PyArrow for fast in-memory operations;
• Provision AWS resources using Terraform and maintain strong Infrastructure as Code practices;
• Ship well tested code with pytest and moto, ensuring high quality and reliability;
• Collaborate with analytics, product, and infrastructure teams to prioritize work, design schemas, and uphold data contracts.

Requirements:

• Deep Python expertise in building production data services and libraries;
• Hands-on experience with PySpark, PyIceberg, PyArrow, GraphFrames, Lambda, Step Functions, S3, Glue, IAM, and serverless or event-driven patterns;
• Knowledge of data lakehouse architecture, Iceberg table design, partitioning, schema evolution, compaction, and cost or performance tuning;
• Experience with distributed processing using PySpark on EMR or similar frameworks, with understanding of shuffles, joins, and optimization;
• Strong SQL proficiency across Trino or Athena, with ability to template queries and maintain standards;
• Practical skills in Terraform workflows, container build pipelines, and EMR image customization;
• Testing mindset including unit and integration testing, AWS mocking, and CI automation;
• Comfort using AI-powered developer tools such as Cursor.