Sr. Data Engineer - (RealTime Streaming)
We are looking for an experienced Senior Data Engineer with strong expertise in real-time streaming technologies and large-scale data engineering solutions. The ideal candidate will have hands-on experience designing, building, and managing highly scalable and fault-tolerant streaming data platforms using Kafka, Flink, Java, and PySpark.
The candidate will be responsible for developing high-throughput, low-latency data pipelines, managing Kafka clusters, implementing cloud-native data solutions, and ensuring system reliability, security, and scalability in enterprise environments.
Key Responsibilities
- Design, develop, implement, and manage Kafka-based real-time streaming architectures capable of handling high-volume and low-latency workloads.
- Build and maintain scalable streaming data pipelines using Kafka ecosystem components, including:
- Kafka Connect
- ksqlDB
- Schema Registry
- Develop and optimize real-time data processing applications using:
- Apache Flink
- Java
- PySpark
- Perform Kafka cluster setup, administration, configuration, tuning, monitoring, and performance optimization.
- Ensure Kafka clusters are highly available and implement disaster recovery strategies and best practices.
- Design and implement fault-tolerant, scalable, and resilient streaming systems for enterprise-grade applications.
- Work with cloud-native applications and modern data engineering frameworks to build scalable solutions.
- Automate infrastructure provisioning and deployment using Infrastructure-as-Code (IaC) tools such as Terraform.
- Implement and follow GitOps practices for deployment automation and infrastructure management.
- Apply robust security standards and best practices, including:
- SSL / mTLS
- SASL authentication
- ACL management
- Collaborate with cross-functional teams to deliver real-time data engineering solutions aligned with business and analytics requirements.
Required Skills & Technologies
Mandatory Skills
- Apache Kafka
- Apache Flink
- Java
- PySpark
- Real-Time Streaming Architecture
- Kafka Cluster Management
- Kafka Connect
- ksqlDB
- Schema Registry
- Streaming Data Pipelines
- Cloud-Native Applications
- Terraform
- Infrastructure-as-Code (IaC)
- GitOps
- High Availability & Disaster Recovery
- Performance Tuning & Optimization
- Fault Tolerance & Scalability
- SSL / mTLS
- SASL
- ACLs
Preferred Experience
- Strong experience in building enterprise-grade real-time data platforms.
- Hands-on experience working in high-throughput and low-latency environments.
- Experience with large-scale distributed data processing systems.
- Exposure to cloud platforms and modern DevOps practices will be an added advantage.
Similar roles
Accountant / Assistant Accountant (CA)
SAURABH SOFTSOURCING PRIVATE LIMITED·Ahmedabad, India
The ideal candidate will be involved with preparing financial reports and statements, bank reconciliations, and conducting cyclical audits. Moreover, the candidate must have strong interpersonal skills and possess a strong business acumen. Responsibilities Create ad-hoc reports for various business needs Prepare tax documents Compile and analyze financial statements Manage budgeting and forecasting Qualifications Bachelor's degree in Accounting or related field…
- Full-time
Copywriter
Grapes Worldwide·Delhi, India
About us Grapes, India’s leading Integrated communications agency nurtures digital strategy and marketing approach across paid, earned, and owned platforms. With both brand and business impact in the forefront, Grapes offers full services in Digital and Communication Solutions – Strategy Wonks at head and Creative at heart, we are strong in setting KPIs, goals and executing innovative-creative campaigns. We combine…
- Full-time
Senior / Lead SAP Signavio Architect
BlackStone eIT·India
Role Summary We are seeking an experienced SAP Signavio Architect to lead business process transformation, process architecture, and process optimization initiatives using the SAP Signavio Business Transformation Suite. The successful candidate will work closely with business stakeholders, enterprise architects, SAP functional teams, and transformation leaders to establish process architecture, identify improvement opportunities, and support SAP S/4HANA transformation programs. The role…
- Remote
- Full-time
Data Scientist, Growth
Lovable·London, Canada
TL;DR — The first data scientist in our new London growth team. You own the growth funnel end to end, from acquisition to activation to retention to expansion, find the levers, and work directly with growth engineering to ship experiments into the product. Not decision support: you drive the numbers. WHY LOVABLE? Lovable lets anyone and everyone build software with…
- On-site
- Full-time
- Express Entry — PR day one, no employer
Data Analyst Administrator
Reed·United Kingdom
We are recruiting for a Data Analyst Administrator to join a high-profile public-sector organisation on an interim contract. This role is responsible for providing effective and efficient administration of all driver, operator and vehicle licensing administration functions within the London taxi and private hire directorate. Key Information: Location: Stratford (hybrid - 3 days per week in the office, 2 days…
- On-site
- Full-time
- £15–£15 / yr
Data Analyst
VIQU IT Recruitment·Solihull, United Kingdom
Data Analyst Solihull (5 days per week) 35,000 per annum VIQU has partnered with a global organisation looking to expand their dat capabilities. This role will be the sole Data Analyst position in the UK, working closely with the head of Technology, and the wider data team to transform business data into insights. You'll be responsible for analysing and manipulating…
- On-site
- Full-time
- £30k–£35k / yr
Alts Data Ops Analyst
Addepar1·Edinburgh, United Kingdom
<div class="content-intro"><p><span style="text-decoration: underline;"><strong>Who We Are</strong></span></p> <p>Addepar is a global data and AI platform empowering investment professionals to turn complex financial information into actionable intelligence. Addepar unifies portfolio, market and client data in a total portfolio view and delivers AI-powered insights within investment and client workflows. More than 1,400 firms in nearly 60 countries use Addepar to manage and advise…
- On-site
- Full-time
Project Manager
Nirman Consultants·New Delhi, India
Company Description: Nirman Consultants is an ISO 9001:2015 certified, award-winning full-service architecture consultancy established in 1990. The firm operates as a total project management organization, offering single-window consulting across architecture, engineering, sustainability, landscape, and urban planning. Nirman Consultants has successfully delivered projects across India and internationally, spanning industrial, commercial, institutional, residential, and mixed-use developments. …
- Full-time
