Back to data projects

Clinical + wearable medallion pipeline

KafkaDelta LakeSparkAzure MonitorLog AnalyticsRBAC

Motivation

Healthcare analytics needed trustworthy progression from raw events to governed metrics while satisfying compliance requirements for sensitive data.

Thinking model

  • Separate reliability concerns by layer: raw capture, cleaned data, and decision-ready models.
  • Attach governance controls where data changes state, not only at final dashboards.
  • Make quality checks part of promotion criteria between layers.

Architecture

Ingest

Clinical + wearable sources

Storage

Bronze layer

Process

Silver layer

Serve

Gold layer metrics

Ops

RBAC + lineage + audit

Flow edges

raw events: Clinical + wearable sources → Bronze layercleansed + normalized: Bronze layer → Silver layerbusiness models: Silver layer → Gold layer metricsgovernance hooks: Bronze layer → RBAC + lineage + auditlineage checkpoints: Silver layer → RBAC + lineage + auditaccess controls: Gold layer metrics → RBAC + lineage + audit
  • Layering strategy reduces downstream ambiguity and supports reusable quality checks.
  • Governance controls are embedded throughout transformation boundaries.

Build

Core components

  • Implemented Bronze/Silver/Gold data lifecycle for healthcare analytics workflows.
  • Integrated governance controls (RBAC, lineage, and auditability) into data flows.
  • Mapped app backend workflows to platform datasets so operational and analytics views stayed consistent.

Quality controls

  • Layer-specific quality checks applied before model promotion.
  • Audit-friendly change visibility for sensitive datasets.

Observability

  • Pipeline and data-service monitoring via Azure Monitor + Log Analytics.
  • Alerting focused on layer freshness and service continuity risks.

Outcomes

Data trust model

Medallion architecture established as the standard for sensitive analytics datasets.

Compliance posture

Governance controls aligned to HIPAA/GDPR-oriented operating requirements.

Analytics reliability

Consistent promotion path from raw inputs to decision-ready views.

Tradeoffs

  • Introduced extra transformation stages to improve trust and governability.
  • Accepted additional modeling overhead in exchange for stronger data contracts.

Confidentiality note

  • Sensitive healthcare entity mappings are omitted while implementation approach is retained.

Work with me

Building a data platform like this?

I work with teams building data systems that need to be reliable, governed, and fast to iterate.

Start a project