How I built a dual-pipeline architecture that gives enterprise reporting teams depth, and operational teams speed — from the same source of truth.
Introduction:
Most organizations are sitting on a data goldmine they cannot see
Jira is everywhere. Over 65,000+ organizations use it daily. And yet, most treat it as nothing more than a ticket tracker — a digital to-do list that the project manager checks and the CTO occasionally glances at.
What if I told you that Jira contains some of the richest, most honest operational data in your entire enterprise? Every sprint velocity, every blocked ticket, every reopened issue, every overdue task — all of it is a signal. The challenge is not collecting the data. The challenge is making it useful to two completely different audiences at once: the executive who wants strategic reporting, and the delivery lead who needs to know what is on fire right now.
That is exactly the problem I set out to solve. And the solution required building not one, but two distinct data pipelines from a single Jira source — each optimized for a different purpose, a different audience, and a different tempo.
The question I kept asking myself was: what does each stakeholder actually need — and would a single pipeline ever serve both well? The answer shaped everything that followed.
The Big Picture — Architecture at a Glance
A tale of two pipelines
Before diving into the detail, here is the architecture I designed. It is deliberately split — not because I could not merge it, but because merging it would have forced painful tradeoffs on both sides.
Two pipelines. One source. Each earning its existence by serving a distinct need — not by replicating effort.
The challenge I faced
The problem with treating every stakeholder the same
When I first approached this project, the tempting path was obvious: pull Jira data, put it in Power BI, done. But within the first week of stakeholder conversations, it became clear that "done" looked very different depending on who you asked.
Stakeholder A – Executive / Reporting Team
Needs cross-system data — Jira joined with HR, finance, and release calendars. Wants a polished, governed report that is trusted and auditable. Latency of a nightly refresh is acceptable.
Stakeholder B – Delivery Lead / Scrum Master
Needs to see what is happening right now — open blockers, sprint burndown, overdue tickets. Does not care about HR data. Needs fast refresh, no data model complexity.
The deeper challenge was Jira's data model itself. Jira stores custom fields as opaque key-value pairs — field IDs like customfield_10042 that mean nothing outside their project context. Across a multi-project environment, the same concept might live in five different field IDs. Without a proper mapping layer, every downstream consumer has to figure this out themselves.
The moment I realized we needed two separate pipelines was when a delivery lead said: "I just want to see what is blocked." And the reporting manager said: "I need this joined to the headcount data." The same Jira pull could never serve both without compromise.
There was also a governance concern. Direct Jira connectors to Power BI are fast to set up but difficult to govern at scale. Every transformation baked into a Power BI measure is a transformation that lives outside version control, outside peer review, and invisible to anyone who is not a Power BI developer.
How I overcame it
Designing with separation of concerns in mind
The architecture answer was clean once the problem was framed correctly: Oracle handles the complexity, Power BI handles the presentation. The two pipelines do not compete — they specialize.
For Pipeline 1, I extracted Jira issues, sprints, epics, and custom fields via the REST API and landed them in a raw ODS staging schema in Oracle — immutable, with full audit metadata. From there, a transformation layer standardized field names, applied business rules, built surrogate keys, and modelled slowly changing dimensions. The integrated layer created conformed dimensional models and KPI-ready fact tables. Finally, the semantic layer exposed curated views and data marts to Power BI — the reporting team never touches a raw table.
Pipeline 1 — key design decision:Oracle as the contract
Curated views in Oracle become the stable interface between engineering and analysis. Any schema change in Jira is absorbed in the transformation layer, not propagated to reports.
Pipeline 2 — key design decision: JQL as the filter
Targeted JQL queries ensure the direct connector only pulls what the operational dashboard needs — active sprint issues in specific projects. No over-fetching, no bottlenecks.
Where AI genuinely earned its place in this project
I want to be specific here — not vague claims about "using AI to go faster." Here are the exact points where AI tooling made a measurable difference, and the tools I would particularly recommend for this type of work.
After establishing a centralized data foundation with Jira, Oracle, and Power BI, the next challenge emerged: making insights accessible to everyone, not just Power BI experts.
Traditional dashboards require users to:
- Navigate multiple reports
- Apply filters
- Understand metrics and KPIs
- Drill through various pages
- Interpret trends manually
- While dashboards provide visibility, they still demand effort from users to find answers.
This is where AI genuinely earned its place in the architecture.
AI Copilot for Business Intelligence
Instead of asking users to navigate dashboards, the AI Copilot allows them to interact with data using natural language.
Users can ask questions such as:
- "What is the health of the current sprint?"
- "Why has velocity decreased over the last three sprints?"
- "Which projects are at risk of delay?"
- "Show teams with the highest defect rates."
- "Summarize release blockers for this month."
The AI interprets the question, analyzes the underlying Jira and Oracle data, and provides an instant business-friendly response.
The AI does not replace Power BI
Instead, it sits on top of the semantic model and acts as an intelligent guide that helps users discover insights faster.
Organisational Value
What began as a Jira reporting solution evolved into an enterprise decision intelligence platform. By unifying data, establishing a trusted analytics foundation, and introducing AI-driven insights, the architecture enables leaders and teams to understand performance, identify risks early, and make informed decisions with confidence. Teates measurable value through efficiency, transparency, and smarter business outcomes.
Final Thought
The best data architecture is not the most elegant one. It is the one that correctly identifies who needs what, by when, and with what guarantees — then builds the simplest possible thing that delivers that. Sometimes that means two pipelines where one would feel cleaner. Serve the stakeholder, not the diagram.
If you are facing a similar challenge — Jira as a source, multiple audiences with competing needs, and pressure to do it with one pipeline — my advice is this: resist the temptation to merge. Spend the time to understand each audience's real requirements. The architecture will follow naturally, and it will be far more maintainable than a single pipeline trying to be everything to everyone.
Raj Kanango
Professional Services Delivery- Global Competency Centre
HPE Operations – Services Experience Delivery
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
