Data Engineering Roadmap
Data Engineering Roadmap
Data Products
Create core pipelines as end-trend products instead of a mishmash of storages, pipelines, ML, visualizations, KPIs etc.
Branch out to Product Features, System capabilities and User Experience.
Data Product 1
-
System Capabilities
-
User Experience
-
Features
Data Product 2
Others…
Bespoke Data Pipelines
These are pipelines that are not supported as product yet because of very limited use-cases.
-
Maintain Existing
-
Introduce New
Raw Data Tracking
All data products eventually depend on raw data. Branch out based on core classifiers (e. Front-End vs Back-End but could be anything else too).
Front-end
-
Coverage
-
Upgrade Existing
-
Introduce New
-
-
Tooling
-
Introduce New
-
Evolution of existing
-
-
Data quality
-
Processes
-
Systems
-
Backend
-
Coverage
-
Tooling
-
Data Quality
3rd Party
-
Coverage
-
Tooling
-
Data quality
Data Governance
High level management of data
Definitions
-
Entities
-
Definitions
-
Modeling
-
-
Meta-data mgmt
-
Process
-
Systems
-
Tooling
-
Discovery
Data Quality Management
Data Protection
-
GDPR
-
PII Management
- Ex. anonymization
-
GDPR Requests Mgmt
- Ex. Automation
-
Reporting/Audit
-
-
Data Security
-
Anonymization
-
Access Control
-
Auditing
-
Data-Ops
-
Self-servability
-
Tooling
-
Data Modeling (aggregations, cubbies)
-
People
Growth
-
Identify Leadership potential
-
Team Size
-
Hire
-
Downsize
-
Mobility
-
-
Reorg
Upskill
-
Analytics/DS
-
Tech
-
Domain
Visibility
-
Publications
-
Conferences
Data Infrastructure
Maintain, Upgrade or Deprecate
-
Storage
-
Visualization
-
Pipelines/Processing
-
Tracking
Introduce new or Migrate
-
Storage
-
Visualization
-
Pipelines/Processing
-
Tracking
Cost Management
-
Measure
-
Reduce
Processes
-
Development
-
CI/CD
-
Observability