Expert Data Engineer

Expert Data Engineer

Ciklum

Remote, Bulgaria

About the role:

As a Expert Data Engineer, become a part of a cross-functional development team engineering experiences of tomorrow. Build and operate our serverless data platform powering analytics on AWS.

Responsibilities:

• Design and build high-performance data pipelines: Build Python AWS Lambda functions, Step Functions workflows, and Spark jobs to process events and batch workloads;
• Own the data lake: Model, evolve, and optimize Apache Iceberg tables backed by S3 and Glue;
• Query & serve data: Enable analytics via Trino and Athena; manage SQL templates with Jinja2 and ensure reliable execution;
• Performance & reliability: Use PyArrow for fast in-memory ops;
• Infrastructure as Code: Provision AWS resources with Terraform;
• Quality & testing: Ship well tested code with pytest and moto;
• Collaboration: Partner with analytics, product, and infra to prioritize, design schemas, and uphold data contracts.

Requirements:

• Deep Python expertise: building production data services and libraries;
• Hands-on experience: PySpark, PyIceberg, PyArrow and GraphFrames, Lambda, Step Functions, S3, Glue, IAM; comfort with event-driven and serverless patterns;
• Data lakehouse know how: Iceberg table design, partitioning, schema evolution, compaction, and cost/perf tuning;
• Distributed processing: PySpark on EMR or similar; understanding of shuffles, joins, and optimization;
• SQL proficiency: across Trino/Athena; ability to template queries and enforce standards;
• IaC discipline: Terraform workflows, Container build pipelines and EMR image customization;
• Testing mindset: Unit/integration tests, AWS mocking, and CI automation;
• AI tooling: Comfortable using AI-powered developer tools (e.g., Cursor).