Senior Data Platform Engineer(Microsoft Fabric, ML Ops)

Description 

Coretek is seeking a Senior Engineer, Machine Learning and Data to build and operationalize Microsoft Fabric-first data science and MLOps platforms for enterprise clients. This is a platform engineering role rather than a model development role. You will build the governed environments, pipelines, CI/CD automation, data quality gates, and observability that allow client data science teams to move batch prediction and forecasting workloads from a notebook into scheduled, monitored production execution. 

You will work as part of a distributed delivery team alongside US-based architects and project managers, implementing against an agreed architecture and owning significant technical workstreams end to end. Success in this role means the platform runs unattended, reproducibly, and under service identity, with clear runbooks that let the client operate it after handoff.

Key Responsibilities 

• Build Microsoft Fabric workspace structures spanning Sandbox, Dev/Staging, and Production, including environment isolation, workspace naming and ownership, and Fabric RBAC patterns mapped to Entra ID groups. 

• Implement governed read access to enterprise data warehouse sources and controlled write-back into data science owned schemas, covering feature tables, model version metadata, model artifact references, prediction outputs, and experiment structures. 

• Develop reusable batch prediction pipeline templates covering data extraction, feature preparation, data quality validation, model execution, output validation, table write-back, and alerting. 

• Build forecasting pipeline patterns where they diverge from standard batch scoring, including time-series inputs, rolling forecasts, and horizon-based outputs. 

• Implement CI/CD for notebooks and platform assets: Git integration for Fabric, branching and pull request standards, automated unit and integration tests, Fabric deployment pipelines, Azure DevOps pipelines where Fabric-native capability falls short, and a single manual approval gate before production promotion. 

• Configure orchestration and scheduling across time-based, trigger-based, and manual execution, with DAG-style visibility that surfaces the failed stage. 

• Implement data quality gates covering schema validation, null and missing value thresholds, value range checks, and basic distributional anomaly detection, wired to block downstream model execution and raise an alert on failure. 

• Configure all unattended execution to run under managed identities or service principals with secrets held in Azure Key Vault. No scheduled or production process may depend on individual user credentials or interactive sessions. 

• Establish model, code, environment, and package versioning standards so any production run is traceable to a versioned combination of code, configuration, environment definition, and data reference, with a demonstrable rollback path. 

• Build monitoring and observability: compute and job health, ETL and pipeline execution status, data quality alerts, model drift detection and health-check notebooks, alert thresholds, and routing to client-designated channels. 

• Define package installation and pinning standards for Python and R, and enforce that only reviewed and approved dependencies reach production environments. 

• Validate that data science owned output tables are consumable by Power BI, and document the access pattern. 

• Produce operational runbooks, handoff documentation, and knowledge transfer material for client IT and data science teams. 

Required Skills and Experience 

• 5+ years in data engineering, machine learning engineering, or MLOps, with production delivery on Microsoft Azure. 

• Hands-on Microsoft Fabric experience: workspaces, OneLake, Lakehouse and Warehouse structures, Fabric Notebooks, Fabric RBAC, and deployment pipelines. Deep Synapse or Databricks background with demonstrable Fabric work will be considered. 

• Strong Python and SQL, with production-grade code structured for reuse, testing, and scheduled execution rather than exploratory notebooks alone. 

• Demonstrated MLOps practice: model versioning, reproducibility, artifact and metadata management, environment pinning, and promotion across environments. 

• Experience operationalizing batch scoring or forecasting workloads on a schedule, including retry logic, failure handling, and output persistence. 

• CI/CD applied to data and notebook assets using Azure DevOps or GitHub Actions, including automated unit and integration testing. 

• Orchestration and scheduling experience with dependency management and pipeline-level failure visibility, using Fabric Data Pipelines, Azure Data Factory, Airflow, or equivalent. 

• Working knowledge of a data quality framework such as Great Expectations, Soda, or an equivalent rules-based validation approach. 

• Solid grasp of Azure identity and security: Entra ID, service principals, managed identities, Azure Key Vault, and RBAC. Specifically, experience making scheduled workloads run headless with no user-bound authentication. 

• Understanding of Power BI consumption patterns against lakehouse and warehouse tables, sufficient to validate and document downstream access. 

• Clear written English and the ability to produce runbooks and design documentation that a client operations team can follow without the author present. 

• Minimum four hours of daily overlap with US Eastern business hours for standups, design reviews, and milestone demonstrations. 

• Bachelor's degree in Computer Science, Data Science, Engineering, or a related field. 

Preferred 

• Working knowledge of R in a platform context: R kernels in notebooks, renv for dependency pinning, and wrapping client-provided R workloads for scheduled execution. Client data science teams frequently write in R even when the platform is built in Python. 

• Time-series forecasting exposure, including rolling origin evaluation and horizon-based output structures. 

• Experimentation platform patterns: experiment configuration, metrics, treatment assignment, matched datasets, and result storage. 

• Model drift detection and baseline statistics monitoring in production. 

• Infrastructure as Code with Bicep or Terraform. 

• Certifications such as Fabric Data Engineer Associate (DP-700), Fabric Analytics Engineer Associate (DP-600), Azure Data Scientist Associate (DP-100), Azure Data Engineer Associate (DP-203), or DevOps Engineer Expert (AZ-400). 

• Consulting or professional services delivery background, working to fixed scope and milestone acceptance. 

• Experience in a regulated or security-reviewed environment where third-party and open-source packages require formal approval before production use.