Sr. Data Engineer - PySpark, Python & Cloudera (CDP)
We are looking for an experienced Senior Data Engineer with strong expertise in PySpark, Python, and Cloudera Data Platform (CDP) to join a high-performing Data Engineering team supporting enterprise-scale Digital Products & Transaction Banking initiatives.
The ideal candidate will have extensive experience designing, building, and optimizing large-scale data pipelines within modern Big Data ecosystems. This role requires strong technical expertise in distributed data processing, cloud-native data platforms, data quality, and enterprise data engineering best practices.
The successful candidate will work closely with Data Architects, Data Scientists, Analytics teams, Product Owners, and Business Stakeholders to deliver scalable, secure, and high-performance data solutions that power business intelligence, analytics, and machine learning initiatives.
Key Responsibilities
- Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
- Build robust batch and distributed data processing solutions using Cloudera Data Platform (CDP).
- Develop and optimize ETL/ELT pipelines for structured and unstructured enterprise datasets.
- Design scalable data ingestion frameworks from multiple enterprise data sources.
- Ensure high data quality, integrity, governance, and availability across enterprise platforms.
- Perform data profiling, cleansing, transformation, and validation activities.
- Optimize Spark jobs for performance, scalability, and resource utilization.
- Work closely with Data Scientists to prepare datasets for analytics and machine learning use cases.
- Collaborate with Product Owners, Business Analysts, Architects, and cross-functional engineering teams.
- Monitor, troubleshoot, and resolve production data pipeline issues.
- Participate in code reviews and implement engineering best practices.
- Create technical documentation and maintain data engineering standards.
- Support continuous improvement of enterprise data platforms and engineering processes.
- Participate in Agile/Scrum ceremonies including sprint planning, backlog grooming, stand-ups, and retrospectives.
Required Technical Skills
- 8–10+ years of experience in Data Engineering.
- Strong hands-on expertise in Python.
- Strong hands-on expertise in PySpark.
- Extensive experience with Cloudera Data Platform (CDP).
- Strong understanding of distributed data processing frameworks.
- Experience building enterprise-scale ETL/ELT pipelines.
- Strong knowledge of Big Data technologies.
- Experience with Hadoop ecosystem technologies.
- Strong SQL programming and query optimization skills.
- Experience with data modelling and data transformation techniques.
- Knowledge of data quality, validation, and governance principles.
- Experience working with Git and version control systems.
- Understanding of CI/CD practices for data engineering.
- Strong understanding of REST APIs and data integration patterns.
- Experience working with structured, semi-structured, and unstructured datasets.
Nice to Have
- Experience working with cloud-native data platforms.
- Exposure to machine learning data pipelines.
- Knowledge of feature engineering and data preparation for AI/ML workloads.
- Experience with workflow orchestration tools.
- Exposure to containerization technologies such as Docker and Kubernetes.
- Experience with DevOps practices for Data Engineering.
Required Competencies
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management skills.
- Ability to work in fast-paced Agile delivery environments.
- Strong ownership mindset with focus on quality and delivery.
- Ability to collaborate effectively with business and technical stakeholders.
- Strong debugging and performance optimization capabilities.
- Ability to manage multiple priorities and deliver within tight timelines.
Preferred Domain Experience
- Banking
- Financial Services
- Digital Products
- Transaction Banking
- Enterprise Data Platforms
Similar roles
Relationship Manager - Malyali/Turkish/Kurdish/Italian/Chinese
Meraki Group·Dubai, United Arab Emirates
Generate and qualify leads through various sales channels. Conduct property presentations and site visits for clients. Understand customer requirements and recommend suitable properties. Build and maintain strong relationships with prospective and existing clients. Follow up on inquiries and convert leads into successful sales. Achieve monthly sales targets and contribute to business growth. Coordinate with internal teams to ensure a smooth…
- Full-time
- Green Visa — self-sponsored, no tie
AI Engineer
Intelligen Group·Sydney, Australia
At Intelligen, we’re building something different. Over the past three years, we’ve evolved from a bold startup to a trusted partner for some of Australia’s most complex data transformations. We’ve helped organisations move from legacy to modern platforms, embedded governance into decision-making, and are bringing AI into the hands of the business, responsibly and at speed. Now, as we scale…
- Hybrid
- Full-time
- Skilled Independent 189 — PR, no sponsor
AI Engineer
Intelligen Group·Melbourne, Australia
At Intelligen, we’re building something different. Over the past three years, we’ve evolved from a bold startup to a trusted partner for some of Australia’s most complex data transformations. We’ve helped organisations move from legacy to modern platforms, embedded governance into decision-making, and are bringing AI into the hands of the business, responsibly and at speed. Now, as we scale…
- Hybrid
- Full-time
- Skilled Independent 189 — PR, no sponsor
Sr. Finacle Techno-Functional Consultant
GSSTech Group·Dubai, United Arab Emirates
We are seeking an experienced Senior Finacle Techno-Functional Consultant with strong expertise in Finacle Core Banking, Finacle Scripting, and Oracle PL/SQL to support enterprise Core Banking transformation initiatives. The ideal candidate will possess deep techno-functional knowledge of Finacle modules, strong customization and integration experience, and the ability to bridge business requirements with technical implementation across large-scale banking projects. This role…
- Full-time
- Green Visa — self-sponsored, no tie
Senior BI Engineer - (Tableau Developer)
GSSTech Group·Dubai, United Arab Emirates
We are seeking an experienced Senior BI Engineer – Tableau Developer to join our Data Engineering chapter supporting Digital Products & Data & Analytics initiatives. The ideal candidate will have extensive hands-on experience in designing, developing, and maintaining enterprise-grade Tableau solutions that transform complex business data into intuitive, scalable, and actionable dashboards. This role requires strong expertise in Tableau development,…
- Full-time
- Green Visa — self-sponsored, no tie
Sr. Solution Architect - Security & Cloud Architecture (Banking)
GSSTech Group·Dubai, United Arab Emirates
We are looking for an experienced Senior Solution Architect with strong expertise in Enterprise Solution Architecture, Security Architecture, Cloud-Native Technologies, and Modern Application Design to support large-scale digital transformation initiatives within the banking sector. The ideal candidate should possess extensive experience designing secure, scalable, resilient, and high-performing enterprise solutions across web, mobile, APIs, microservices, cloud platforms, and network security architectures.…
- Full-time
- Green Visa — self-sponsored, no tie
Call Center Executive
Meraki Group·Dubai, United Arab Emirates
Meraki Education is looking for enthusiastic and customer-oriented individuals to join our call center team. The role involves handling inbound and/or outbound calls, assisting families interested in admissions at our school, with their queries, resolving issues, and ensuring parent satisfaction while following company policies and procedures. Key Responsibilities: * Answer inbound calls from families promptly and professionally. * Make outbound…
- Full-time
- Green Visa — self-sponsored, no tie
Manager Performance & Attribution Reporting
Emirates Investment Authority·Abu Dhabi, United Arab Emirates
• The Manager – Performance & Attribution Reporting is responsible for managing the day-to-day delivery of investment performance measurement, attribution reporting, and the portfolio analytics framework across EIA's full investment portfolio. • The role ensures accurate, timely, and transparent performance reporting across all asset classes and investment portfolios, supporting investment decision-making, portfolio oversight, and governance. • Operating within a multi-asset,…
- Full-time
- Green Visa — self-sponsored, no tie
