Practices – Data Pipelines

By

Build for the right latency

Break transformations into small tasks

Start with a steel thread

Utilise cloud – define your pipelines with infrastructure-as-code

Data models are important and should be addressed iteratively

Apply master data / reference data pragmatically to support merging

Use orchestration and workflow tools