Agile Data Warehousing: Do You Scrum?

When IT professionals speak of “agile development,” they could be referring to any of countless overlapping schools of thought. It’s best to tread lightly and keep an agile mind to find some hybrid or innovative approach that specifically meets your needs.

By most accounts, this term “agile” refers to any approach that straddles the razor’s edge between traditional top-down development and sheer adhocracy. Agile approaches attempt to speed the development process while enabling rapid shifts in development priorities to meet changing user requirements.

Agile approaches involve incremental, iterative, collaborative development among cross-functional teams consisting of IT professionals and business users. Under agile development, self-organizing teams refine requirements and craft modular solution components as they go. Teams hold regular checkpoint status meetings to review work in progress and reprioritize tasking. At every point in an agile program, the team develops useful prototypes that can stand alone as production-grade business technology systems, or can function as building blocks for larger, more complex, multifunctional systems. Documentation plays catchup to the work being performed, rather than constraining it through imposition of fixed, top-down specifications.

All of this contrasts with the traditional top-down “waterfall” development methodology, in which IT professionals define a complete master plan, requirements definition, technical specification, work breakdown structure, schedule, milestones, and deliverables in advance of the work. Increasingly, IT is exploring agile development approaches in the context of data warehousing (DW) projects. For starters, I recommend that you read my colleague Boris Evelson’s recent report, “Agile BI Out Of The Box,” in which he discusses the role of metadata-driven modeling tools in accelerating the development of ETL scripts, SQL queries, OLAP cubes, report definitions, and other analytics project deliverables.

To the extent that self-organizing teams of IT and business professionals can use these tools to brainstorm and catalyze an emerging consensus on end-to-end analytics system requirements, designs, and deliverables, they become a key facilitator of agile development. Likewise, it would be very useful to have an underlying DW ecosystem that has the elasticity to be rapidly refactored and redeployed into any agile topology you define.

However, agility-enabling tooling and platforms are not enough. You can't get agile simply from implementing some fancy software product. Without development process discipline an agile DW initiative can quickly degenerate into anarchy, as everybody brings their own preferred tool into the effort, nobody talks to anybody else, no one reuses anybody else’s work, and nothing interoperates. To keep agile development from evolving into an ongoing farce, DW professionals must also explore agile project management approaches for organizing cross-functional teams and focusing their joint efforts on on key business requirements.

With agile development, the key concern is keeping communications channels open, vibrant, and productive across complex teams that usually have no paramount leader. One useful agile discipline (though by no means the only) is “scrum,” a rugby term that suggests the rough-and-tumble of a complex, fast-moving peer group driving toward a common goal. The key roles in this approach are the “Scrum Master,” who neutralizes distractions that might derail the team from achieving maximum productivity, and one or more “Product Owners,” who make sure the team is driving at a clearly defined business goal.

In a DW context, the Scrum Master might be someone who makes sure that the diverse data stewards achieve steady progress on crafting a common glossary of business terms that span diverse subject areas, while the technical specialties—such as ETL developers, data architects, and report designers—incorporate those definitions into the underlying metadata and modeling layer that they, hopefully, share. The Product Owner, in turn, might be the chief business analyst in the chief functional domains—customer, financial, HR, or what have you—who will be using the analytic applications being developed through the scrum approach.

Obviously, the agility of any particular scrum initiative depends on the chemistry among all the players, but especially on the rapport between the Scrum Master (the “IT-facing” facilitator, as it were) and the Product Owner (the “customer-facing” facilitator). Also, this approach depends critically on those individuals’ “conversational” skills in helping the team to quickly prioritize their efforts, remedy common problems, explore alternative approaches, identify possible gotchas, and learn from past mistakes. Furthermore, the team as a whole will flounder in wasted work products if they don’t have at least a common, albeit-unfinished mental picture of some long-range target enterprise DW architecture that they are, block by building block, constructing. No one wants to be the unwitting builder of a new Tower of Babel.

That’s where it’s essential for agile DW development teams to at least have a common architectural reference framework. Agile DW devotees should enroll in one or another architectural school before they put on their cleats and try out for the scrum team. Is your agile DW program building conformed independent data marts, centralized multi-domain DW, hub-and-spoke DW, or federated DWs—or pursuing some innovative hybrid of those approaches? Is your agile DW program also attempting to deliver a “single version of the truth” through a comprehensive data quality, data governance, and master data management program? Is your agile DW program also attempting to build flexible hooks into the infrastructure so that it can be easily extended to support in-database analytics, complex event processing, process analytics, unstructured content, and other emerging requirements? Of course, we at Forrester have published guidance on such issues, including this report on topologies, this on federation, this on in-database analytics, and this on DW scaling. Our agile minds are always available to customers who wish to scrum their ideas at a critical juncture in any DW project.

Yes, speed is of the essence on agile development, and you don’t need to have  worked out detailed strategies on any of these approaches before you launch your next agile DW project. But you at least need to know whether these are potential future requirements that must be kept as options, and for which you may need to invite the stakeholders into your scrum’s ongoing discussions.

Whether you scrum or conform to any other school of agile project coordination, conversation is key. Everybody important—from the IT and business sides—must be continually looped into all evolving discussions. That way, when you collectively decide to swerve your agile DW squad toward some new goal, nobody is straggling, kicking anybody else’s shins, or losing sight of the ball.