Data platform architecture · regulated environments
How I architect.
I design data platforms as solutions to business problems — then prove the platform actually matches the design.
Middle East market focus. Vendor-agnostic on tooling, deep on Snowflake and the AWS data stack. I select technology against constraints, not preference — and in this region, regional product availability and in-Kingdom residency are constraints that eliminate most of the shortlist before anyone gets to compare features.
01 Engineer → architect
Not a title — four things changed. I still write SQL and Python; the difference is the code is now downstream of a documented decision, and the decision is downstream of a business requirement someone signed.
| As an engineer | As an architect | |
|---|---|---|
| Problem I solve | Technical: this pipeline is slow, this model is wrong | Business: leadership can't get event numbers they trust in time to act |
| What I optimise | Correctness, runtime | Cost, scalability, security, privacy — as designed properties |
| What I produce | Working pipelines and models | Solution architecture, decision records, risk matrix, acceptance criteria |
| Who I talk to | Analysts, BI developers, engineers | Executive leadership, cybersecurity, infrastructure, delivery vendors — plus the engineers |
02 How I work — three phases
Assessment
Interview stakeholders on both sides — business (what decision are you trying to make, what's stopping you) and technical (platform, infrastructure, cybersecurity, governance). Analyse the real landscape, not the diagram of it. Name risks, assess maturity honestly.
Output Current-state architecture, findings & a risk register — the artefact most people skip, and the one that makes the target state defensible.
Design
Architecture pattern & style first (warehouse / lake / lakehouse; centralised or federated; batch, streaming, or both). Then technology selection against criteria. Then low-level design as a set of views, so each stakeholder reads the one they own.
Output Target-state architecture plus a decision log explaining every non-obvious choice and what was traded away.
Reviewal
Walk the design through the people who actually sign — cybersecurity, infrastructure, the platform owner, executive leadership. Their questions are the design review. If it can't clear that room, it isn't a design — it's a preference.
Output Signed-off architecture, with any forced change captured as a decision rather than a silent edit.
03 Requirements discipline
Most platform failures I've seen trace back to a requirement nobody wrote down. I separate three kinds:
What it must do
Absent, the business problem isn't solved. Serve historical ticketing & attendance for reporting; ingest competition results on cadence; support natural-language querying.
How it must perform
Absent, it's solved but with ongoing pain. Encryption at rest & in transit; freshness SLA per source; BI-tool integration; auditable access.
What eliminates options
Absent, you redesign. These are what I hunt for in the first week.
In-Kingdom data residency under Saudi PDPL & NDMO. That single requirement decided the warehouse region, constrained the cloud-provider choice, ruled out several otherwise-strong governance and AI tools outright, and set the standard every later decision was measured against. Not a checkbox at the end — the first line of the design.
04 How I select technology
I don't start from a product. I start from criteria, score the candidates, and write down what I gave up.
- Requirements fit — does it serve the functional requirements and the analytics / AI roadmap, or only today's reports?
- Regional availability — is it actually available in-region, with the features I need there? In the Gulf this is the first filter, not the last — it routinely kills the technically best option.
- Enterprise readiness — production-grade for a regulated organisation, or a promising tool I'd be maintaining myself?
- Low-code vs code, matched to the actual user — code is right for engineers, wrong for a governance steward. The user persona is a selection criterion of equal weight to the feature matrix.
- Skillset available — what the team can still run twelve months from now, after the implementation partner has gone.
- Budget & total cost — licence cost is what people compare; TCO — add-ons, per-connector pricing, the storage line nobody modelled — is what hurts.
- Operational overhead — SaaS vs PaaS vs self-managed, measured in headcount, not features.
- Security & shared responsibility — where the boundary sits, and whether the controls I'm promised are controls I can verify.
05 Selected architecture decisions
Written as decision records, because that's the deliverable. Each names the constraint that drove it and the trade-off I accepted. Click to expand.
Warehouse platform & region
Decision Snowflake, deployed in a Saudi cloud region.
Driving constraint In-Kingdom residency (PDPL / NDMO) — an ASR, not a preference.
Warehouse platform & region
Decision Snowflake, deployed in a Saudi cloud region.
Driving constraint In-Kingdom residency (PDPL / NDMO) — an ASR, not a preference.
Considered. Multiple cloud-native warehouses across providers.
Why. Regional availability with the required feature set was the binding filter. Once residency was applied, the shortlist collapsed to a small set; from there, elasticity, separation of storage and compute, and native governance (RBAC, masking, account-level auditing) decided it.
Trade-off accepted. Consumption pricing needs active cost governance to stay predictable — which is why cost attribution was designed in, not bolted on. Regional feature parity is checked per release, not assumed.
Transformation layer
Decision Code-based transformation (dbt) over a low-code ETL tool.
Driving constraint Model correctness must be testable & reviewable; changes traceable to an author and a reason.
Transformation layer
Decision Code-based transformation (dbt) over a low-code ETL tool.
Driving constraint Model correctness must be testable & reviewable; changes traceable to an author and a reason.
Why. Version control, code review, automated tests, and lineage generated from the code rather than maintained beside it. The team's core skill is SQL, so the tool matches the skillset criterion.
Trade-off accepted. Non-engineers can't change transformation logic — deliberate. Business users are served through the governed semantic layer and self-service marts, not by editing models.
Orchestration
Decision Managed Airflow over self-hosted.
Driving constraint A small team supporting a live global event can't also be a platform-operations team.
Orchestration
Decision Managed Airflow over self-hosted.
Driving constraint A small team supporting a live global event can't also be a platform-operations team.
Why. Operational overhead and time-to-market dominated. A managed service removes upgrade, scaling, and availability work from a team whose time is better spent on data.
Trade-off accepted. Less control over runtime versions and plugin freedom; some patterns have to be adapted to the managed environment.
Governance & data-quality tooling
Decision Split the stack by persona — code-based tests for engineers in-pipeline; a point-and-click catalogue & quality UI for stewards. Open-source first, validated by PoC before any commercial procurement.
Driving constraint Governance isn't an engineering function — the people accountable for ownership don't write code; residency rules out several commercial platforms.
Governance & data-quality tooling
Decision Split the stack by persona — code-based tests for engineers in-pipeline; a point-and-click catalogue & quality UI for stewards. Open-source first, validated by PoC before any commercial procurement.
Driving constraint Governance isn't an engineering function — the people accountable for ownership don't write code; residency rules out several commercial platforms.
Why. A single tool optimised for engineers produces a governance programme on paper and none in practice. Running the PoC first turns observed gaps into evidence-backed procurement requirements, instead of buying a licence against a feature list.
Trade-off accepted. Self-hosted open source carries upgrade/ops overhead, and the PoC has to be honest about where it falls short — that finding is the input to the commercial case, not a failure.
Natural-language access to the warehouse
Decision Ship it — with residency and zero data retention as the deciding criteria, not model quality.
Driving constraint Regulatory review had to be clearable before capability mattered. An AI feature that can't pass cybersecurity review is not a feature.
Natural-language access to the warehouse
Decision Ship it — with residency and zero data retention as the deciding criteria, not model quality.
Driving constraint Regulatory review had to be clearable before capability mattered. An AI feature that can't pass cybersecurity review is not a feature.
Why. Least-privilege read-only role scoped to approved marts only, on a dedicated warehouse so AI consumption is separately attributable. Grounded in documented metric definitions so answers come from the modelled layer, not from the model guessing. Access behind a data-owner approval workflow, audited on the same terms as human access.
Trade-off accepted. A narrower scope than an unconstrained agent, and real work to maintain the semantic context. Both are cheap next to an unauditable path to production data.
Cost as a designed property
Decision Treat cost as a first-class quality attribute with its own architecture, not a monthly surprise.
Driving constraint Consumption pricing punishes waste silently unless spend is attributable.
Cost as a designed property
Decision Treat cost as a first-class quality attribute with its own architecture, not a monthly surprise.
Driving constraint Consumption pricing punishes waste silently unless spend is attributable.
Why. Attribution first — metering joined to query history, tagged at source so every statement traces to a model, a job, and an identity. Once spend is attributable, waste is visible: idle compute, row-by-row loads, uniform retention on rebuildable layers, orphaned clones. Then the levers: auto-suspend & right-sizing, bulk staged loading, transient tables for reproducible intermediates, retention set per layer by recovery value.
Outcome. Recurring compute on the refactored ingestion pipelines dropped by an order of magnitude, and storage growth went from compounding to predictable with a documented rule per layer.
Trade-off accepted. Shorter retention on rebuildable layers means recovery is a rebuild, not a rollback — documented and accepted deliberately.
06 Architectural views I produce
I don't hand one diagram to everyone. Each stakeholder gets the view they're accountable for, and each has to stand on its own.
Conceptual
The solution as an answer to the business problem — no product names required. For leadership.
Technology
Components and how they fit together.
Data
Source → landing → medallion layers → governed marts → consumption, with modelling & schema decisions attached.
Network
Connectivity, boundaries, private paths, egress control.
Security
Identity, RBAC hierarchy, encryption, masking & classification, audit logging, exfiltration controls.
Deployment
Environments, promotion path, CI/CD, what runs where and in which region.
Governance
Ownership, catalogue, lineage, quality gates, approval workflows.
Logging & monitoring
Task state and data condition: freshness, row deltas, null rates on critical columns, alerts to a named owner.
07 Quality attributes I design against
Named up front, each with the mechanism that delivers it — so the design can be reviewed against them, not just admired.
08 Architecture governance & delivery acceptance
The part of architecture that isn't design — and where I add the most value. When a platform is delivered by an implementation partner, the architect's job ends not at the target state, but at verified acceptance.
- Acceptance criteria written before delivery, per phase, with explicit environment-by-environment sign-off gates. Specific criteria get met; ambiguous ones get argued about later.
- Verification against live platform state, not documentation. Every finding is backed by a query or artefact anyone can re-run. Design documents describe intent; the platform is the fact.
- Design-to-build delta tracking. Where the build diverges from the approved design, I record it, judge variation vs defect, and route it — amend the design through a decision record, or correct the build. Silent divergence is the failure mode.
- Controls verified, not asserted. If a design says masking, MFA, or network restriction is in place, I confirm it exists before signing anything that depends on it.
- Findings sequenced by risk — security & compliance first, then structural model defects, then operational discipline. A prioritised remediation plan beats a flat list of complaints.
- An observational register. "I've observed X — can you confirm the intended behaviour" opens a technical conversation; "fix X" opens a contractual one. The first resolves faster.
Deliverables from this work
Acceptance-criteria checklists · design-to-build delta registers · annotated design reviews · architecture decision records · risk matrices — the documents that let a non-technical decision-maker act with confidence.
09 What a client or employer receives
→ Let's talk architecture
If you're hiring for data platform architecture in a regulated environment, I'd like to hear about it.