Service-Oriented Analytics: Tapping into the Predictive Smarts of Your Entire Organization

By James Kobielus

Advanced analytics is a key competitive weapon of companies everywhere. Visionary organizations are those that take a future-facing, analysis-driven perspective on new challenges. They do this by grounding management forecasts in solid historical information sets, leveraging and extending companies’ existing investments in data mining and predictive modeling.

To become truly future-focused, organizations must build out their predictive muscles through deepening commitment to these and other advanced analytics technologies, which also include interactive visualization, multivariate statistical analysis, text analytics, and massively parallel enterprise data warehousing. However, enterprises must be careful not to adhere to the common practice of implementing advanced analytics tools in tactical, application-specific silos. One of the downsides of this traditional practice is that diverse predictive modeling teams can find it difficult to share their deep domain expertise, best statistical approaches, and most powerful data exploration and visualization features.

How can companies realize the transformative potential of predictive analytics for business success? For starters, you must get rid of siloes that fragment your data mining initiatives into separate camps. You must also build a bridge between your data mining operations and the teams that manage your text analytics, business intelligence, complex event processing, and business process management efforts. And the key approach for silo-smashing is service-oriented architecture (SOA).

At first glance, SOA may seem like a foreign topic to many analytics professionals, but it shouldn’t be. In the broadest perspective, SOA refers to best practices for encouraging greater reuse, sharing, and cross-platform interoperability among key business resources. Typically, one associates SOA with reuse of one specific type of resource: application functionality that is distributed across heterogeneous, networked platforms. Nevertheless, key SOA principles—such as standards-based service virtualization, reuse, brokering, and governance—are as applicable to predictive models as to any other resource that lives online.

Predictive models empower your product managers, marketing specialists, risk managers, process analysts, senior executives, and other personnel with access to sophisticated forecasting, time-series analysis, and scenario-testing tools. Each model is a statistical encapsulation of your business’ current view of the future in some specific application, subject, or decision-support area. Incorporating the expertise of subject-matter experts, these models allow organizations to gauge the potential impact of future projects, campaigns, and other initiatives, and also to adjust execution of these initiatives in mid-stream.

Predictive analytics need not be a purely blue-sky planning tool. This technology can sit at the core of your SOA strategy, and leading-edge enterprises are in fact doing that. Companies in such verticals as financial and telecommunications are embedding predictive logic deeply into data warehouses, business process management (BPM) platforms, complex event processing (CEP) streams, and operational applications.

What will it take for your company to fully align your advanced analytics efforts with your SOA strategy? Most important, Service-Oriented Analytics requires an executive-level commitment to becoming a predictive enterprise on all levels. From the perspective of your existing predictive modeling teams, this will require an ongoing focus on collaboration across business, function, and subject domains. You should create a culture and offer incentives that encourage modeling professionals to reuse each other’s expertise on problems that cross multiple domains.

Reuse is everything. Here are some high-level guidelines for establishing a reuse-friendly Service-Oriented Analytics practice in your company:

• Reuse modeling best practices:  Start by consulting Forrester’s report on setting up a Business Intelligence Solution Center (BISC), which we defined as “an institutional steward, protector, and forum for BI best practices.” Given that predictive analytics is a key segment of BI, you will find it necessary to incorporate this technology into your BISC’s scope.
• Reuse modelers: Cultivate a professional cadre of predictive modeling experts who are more than just wizards in advanced statistics and mathematics. Encourage subject-matter experts in all business areas to undertake predictive modeling projects and to team with modeling experts in other projects, applications, and business units. Provide incentives for modelers to regularly move between business units and subject domains, thereby spreading their expertise throughout the enterprise.
• Reuse models: You should begin to investigate predictive-model governance tools, which support version control, check-in/check-out, and other controls over models created in and imported from diverse tools. Model governance tools—from vendors such as SAS, SPSS, and KXEN--facilitate reuse, consolidation, combination, and cross-synthesis among disparate models. And you should also investigate options for embedding predictive models within BI, BPM, and other operational applications, thereby leveraging your growing analytic asset into new deployments.

As you deploy predictive models into operational applications, you should provide other applications with SOA-based access to them through Web services, Web 2.0, and other standardized interfaces. In that way, you will be creating a critical bridge to your application development teams and be creating a thoroughly Service-Oriented Analytics environment.

Your first steps down this road are clear. Forrester will be happy to assist you in developing a detailed enterprise roadmap for Service-Oriented Analytics. In so doing, you’ll be better able to tap into the future-facing analytic expertise that exists throughout your business.

Is BAM Relevant In The Age of Lean Processes?

By James Kobielus

In any dynamic business environment, the last thing you want is to indulge in navel-gazing. If you’re finely attuned to monitoring internal processes, but are not adept at responding to breaking events or anticipating the future, you will find yourself marginalized in the new global economy.

In a turbulent world, the most successful companies are those that stay lean--in other words, trimmed of excess overhead, fit for new challenges, and flexible enough to evolve rapidly. Forrester advises customers to make their business processes leaner, through such measures as reducing the number of steps, routing paths, participants, decision points, milestones, and revisions. As you do, you will find concomitant improvements in speed, cost, quality, and customer satisfaction.

Clearly, you can’t do that well if you don’t have, at the very least, the means to monitor a wide range of business processes in unified dashboards, reports, and other applications. That’s where traditional business activity monitoring (BAM) tools, which are a key component of most business process management (BPM) environments, play a critical role. However, critical as BAM may be, it is often deployed in a predominantly navel-gazing process-tracking capacity (remember what that “M” stands for) and, as such, is as much part of the problem as the solution.

Why? Well, for starters, most BAM environments are deployed in conjunction with specific vendors’ BPM platforms--hence can only track the subset of business processes that run on those platforms. But, even more limiting, BAM dashboards are often the stand-alone province of process administrators. Only in a handful of  BAM-using organizations do alerts and metrics escalate beyond the process administrator level and trigger the necessary human and automated workflows necessary to respond to breaking issues--or nip them in the bud before they can become issues.

To stay relevant, BAM needs to evolve in the age of lean processes so that it can become a tool for business agility at all levels. Next week, Forrester BPM analyst Clay Richardson and I will do a podcast on this very topic. In  that podcast, and in a joint presentation that Natalie Petouhoff and I will do at Forrester Business and Technology Forum in Chicago in October, I’ll sketch out a vision for the next-generation of BAM (which I half-jokingly call “uber-BAM”).

Here is a rough sketch of the maturity levels of BAM's evolution to realize this vision:

• BAM Level 1: Business activity monitoring: This is today’s BAM baseline. Essentially, it combines BPM with business intelligence (BI), interactive dashboarding, and complex event processing to deliver process metrics to consoles used solely by human process administrators.
• BAM Level 2: Business action mobilization: This is the BAM level that many forward-looking users and vendors have implemented. It combines all Level 1 functionality with business rules engines (BRE) to drive sophisticated human and automated workflows in response to breaking events. It puts the “immediate action loop” into the BAM equation.
• BAM Level 3: Business adaptability management: This is the promise of truly lean uber-BAM, and is found only in a vanguard of visionary  business environments. It combines all Level 2 functionality with embedded predictive-analytics models, which are developed and maintained by business process architects and/or subject-matter process experts. Continuously mining historical and real-time metrics, these in-process analytics drive automated “recommendation engines” that adapt coordinate response to changing conditions. Recommendation engines enable human and automated decision agents to seize opportunities and neutralize threats well ahead of the curve.

Going forward, this new vision of BAM’s evolution and promise will inform Forrester’s coverage of process agility and optimization.

IBM Goes Deeply Predictive, Announces Acquisition of SPSS

By James Kobielus

IBM dropped a big bombshell at the start of any already action-packed day for the analyst community. At this moment, I’m sitting, along with several dozen of my peers from Forrester and other firms, at the IBM Smart Analytics System launch event in Hawthorne NY. I’ll blog on IBM’s other announcements in a separate items.

The bombshell was IBM’s announcement that it’s acquiring SPSS, a long-established, leading provider of predictive analytics (PA), data mining (DM), statistical analysis, and text mining tools. The acquisition, subject to the usual shareholder approvals and regulatory reviews, is expected to close later this year. But, even in advance of that near-certain consummation, IBM’s bold move has already sent shockwaves throughout the analytics market.

Most important, IBM has acquired the second largest vendor of PA/DM solutions, dwarfed only by privately held SAS Institute. In this segment, IBM’s proposed acquisition is having the same impact that its Cognos buy had on the business intelligence (BI) market two years ago. In many discussions with Forrester customers, SPSS is often mentioned as a key solution provider for predictive modeling and statistical analysis against structured, semi-structured, and unstructured content.

For IBM’s competitive standing in the data management market, this acquisition represents one of the last missing pieces of its Information On Demand (IOD) portfolio. By acquiring SPSS, IBM has acquired a substantial PA/DM brand with a very loyal set of longtime customers who have build their customer churn, supply chain optimization, and other predictive models on its best-of-breed platform. SPSS recently underlined its feature-comprehensive value proposition through re-branding around the “Predictive Analytics Software” (PASW) family name. But longtime customers didn’t need to be reminded, of course, that what used to be known as “Clementine” defines a functional high-bar in this solution segment.

IBM would probably be the first to admit that it took its focus off the PA/DM market over the past several years as it build out the BI, data warehousing (DW), and other pieces of its IOD portfolio. IBM had never really exited the PA/DM market, but casual observers might have thought otherwise. However, the vendor three years ago chose to de-emphasize its Intelligent Miner tools--which support mining of structured data--as stand-alone offerings. It essentially buried these solutions, moving them into its InfoSphere Balanced Warehouse family, where they are now offered as features of its Enterprise Edition DW software, rather than as stand-alone tools that would be enhanced and evolved independently.

IBM and SPSS’s respective customer bases should rest assured that overlaps among their respective product portfolios are not extensive. Once the acquisition closes, IBM is almost certain to build out its SPSS brand and, over the coming 1-2 years, phase out the Intelligent Miner technology within its InfoSphere portfolio. One tricky issue is which text analytics solution family--SPSS’ or IBM’s OmniFind solutions--will prevail as the parent converges these offerings in its IOD portfolio. Another issue is how IBM will integrate the SPSS offerings into its still-evolving in-database analytics roadmap for InfoSphere Balanced Warehouse. Hopefully, IBM will maintain and extend SPSS’ already extensive in-database analytics integration with a broad range of vendor DWs, including such Big Blue rivals as Oracle, Microsoft, Sybase, and Teradata.

Who loses from IBM’s acquisition of SPSS? Fundamentally, one can’t help think that SAP missed the boat by not seizing the opportunity to acquire partner SPSS, whose Clementine technology it OEMs, has integrated with its BI technology, and sells as SAP BusinessObjects Predictive Workbench. PA/DM is an increasingly key component of a full-fledged BI solution stack. However, the remaining field of vendors with stand-alone, horizontally applicable PA/DM vendors consists primarily of vendors who are large but proudly and stubbornly independent (especially, SAS Institute); high-quality but much less widely adopted (e.g., KXEN, ThinkAnalytics); or specialized on customer, financial, scientific, or other specialized analytics (e.g., Unica, Fair Isaac, Accelrys).

Some IBM rivals in the BI space already have strong PA/DM tools, most notably Oracle (Oracle Data Miner) and TIBCO/Spotfire (the Insightful tools). Among BI vendors, Microsoft, MicroStrategy, and Information Builders have PA/DM capabilities, but they are not to a SAS or SPSS level of sophistication. These and other BI vendors should also be scouting for strategic acquisitions.

What do you think? Will IBM’s acquisition of SPSS lead to further merger and acquisition activity in this space as other leading BI players strengthen their PA/DM solutions?

By James Kobielus

IBM dropped a big bombshell at the start of any already action-packed day for the analyst community. At this moment, I’m sitting, along with several dozen of my peers from Forrester and other firms, at the IBM Smart Analytics System launch event in Hawthorne NY. I’ll blog on IBM’s other announcements in a separate items.

The bombshell was IBM’s announcement that it’s acquiring SPSS, a long-established, leading provider of predictive analytics (PA), data mining (DM), statistical analysis, and text mining tools. The acquisition, subject to the usual shareholder approvals and regulatory reviews, is expected to close later this year. But, even in advance of that near-certain consummation, IBM’s bold move has already sent shockwaves throughout the analytics market.

Most important, IBM has acquired the second largest vendor of PA/DM solutions, dwarfed only by privately held SAS Institute. In this segment, IBM’s proposed acquisition is having the same impact that its Cognos buy had on the business intelligence (BI) market two years ago. In many discussions with Forrester customers, SPSS is often mentioned as a key solution provider for predictive modeling and statistical analysis against structured, semi-structured, and unstructured content.

For IBM’s competitive standing in the data management market, this acquisition represents one of the last missing pieces of its Information On Demand (IOD) portfolio. By acquiring SPSS, IBM has acquired a substantial PA/DM brand with a very loyal set of longtime customers who have build their customer churn, supply chain optimization, and other predictive models on its best-of-breed platform. SPSS recently underlined its feature-comprehensive value proposition through re-branding around the “Predictive Analytics Software” (PASW) family name. But longtime customers didn’t need to be reminded, of course, that what used to be known as “Clementine” defines a functional high-bar in this solution segment.

IBM would probably be the first to admit that it took its focus off the PA/DM market over the past several years as it build out the BI, data warehousing (DW), and other pieces of its IOD portfolio. IBM had never really exited the PA/DM market, but casual observers might have thought otherwise. However, the vendor three years ago chose to de-emphasize its Intelligent Miner tools--which support mining of structured data--as stand-alone offerings. It essentially buried these solutions, moving them into its InfoSphere Balanced Warehouse family, where they are now offered as features of its Enterprise Edition DW software, rather than as stand-alone tools that would be enhanced and evolved independently.

IBM and SPSS’s respective customer bases should rest assured that overlaps among their respective product portfolios are not extensive. Once the acquisition closes, IBM is almost certain to build out its SPSS brand and, over the coming 1-2 years, phase out the Intelligent Miner technology within its InfoSphere portfolio. One tricky issue is which text analytics solution family--SPSS’ or IBM’s OmniFind solutions--will prevail as the parent converges these offerings in its IOD portfolio. Another issue is how IBM will integrate the SPSS offerings into its still-evolving in-database analytics roadmap for InfoSphere Balanced Warehouse. Hopefully, IBM will maintain and extend SPSS’ already extensive in-database analytics integration with a broad range of vendor DWs, including such Big Blue rivals as Oracle, Microsoft, Sybase, and Teradata.

Who loses from IBM’s acquisition of SPSS? Fundamentally, one can’t help think that SAP missed the boat by not seizing the opportunity to acquire partner SPSS, whose Clementine technology it OEMs, has integrated with its BI technology, and sells as SAP BusinessObjects Predictive Workbench. PA/DM is an increasingly key component of a full-fledged BI solution stack. However, the remaining field of vendors with stand-alone, horizontally applicable PA/DM vendors consists primarily of vendors who are large but proudly and stubbornly independent (especially, SAS Institute); high-quality but much less widely adopted (e.g., KXEN, ThinkAnalytics); or specialized on customer, financial, scientific, or other specialized analytics (e.g., Unica, Fair Isaac, Accelrys).

Some IBM rivals in the BI space already have strong PA/DM tools, most notably Oracle (Oracle Data Miner) and TIBCO/Spotfire (the Insightful tools). Among BI vendors, Microsoft, MicroStrategy, and Information Builders have PA/DM capabilities, but they are not to a SAS or SPSS level of sophistication. These and other BI vendors should also be scouting for strategic acquisitions.

What do you think? Will IBM’s acquisition of SPSS lead to further merger and acquisition activity in this space as other leading BI players strengthen their PA/DM solutions?

BI, Analytics, And CEP: Some Fruitful Potential Follow-Ons From Software AG’s Acquisition Of IDS Scheer

 by James Kobielus

Yes, of course, Software AG is buying IDS Scheer primarily for the latter’s ARIS family of business process management (BPM) tools. I’ll leave it to my Forrester colleagues who focus on BPM--on both the IT and TI sides of the house--to call out the ramifications for Software AG’s positioning in that market.

But, believe it or not, this deal will also launch Software AG into the growing markets for business intelligence (BI), analytics, and complex event processing (CEP) solutions. We bet you didn’t realize that IDS Scheer has ARIS solutions in these fast growing markets, but in fact they do--and they’re continue to evolve those offerings.

It’s no surprise that IDS Scheer’s BI, analytics, and CEP offerings supplement and extend its BPM portfolio. Its CEP solution, ARIS Process Event Monitor, supports business activity monitoring (BAM). Its analytics offerings, ARIS Process Performance Management and ARIS Performance Dashboard, support visualization, dashboarding, scorecarding, drilldown, and alerting on process key performance indicators (KPIs), both historical and real-time. And its forthcoming BI offering, ARIS MashZone, will support self-service user development of reports, dashboards, and other views of process and business metrics.

IDS Scheer has little market share in these non-core segments. And the vendor is no immediate threat, by itself or under its future corporate parent, to the leaders in the BI, analytics, and CEP segments. Indeed, its forthcoming mashup-oriented BI offering only provides a subset of the features available from market leaders such as SAP Business Objects, IBM Cognos, and MicroStrategy. But the fact that Software AG will soon be able to provide its own offerings in those segments, rather than rely wholly on partners, represents an important step in its attempt to field a full service oriented architecture (SOA) solution stack.

As noted in a blog entry a year and a half ago, BI is the crown jewel in any comprehensive SOA solution portfolio. SOA suites cannot be considered feature-complete unless they incorporate a comprehensive range of BI features. This acquisition continues the ongoing SOA solution build-out strategy that motivated Software AG to acquire webMethods in 2007.

But it’s not clear yet whether Software AG plans to flesh out its BI, analytics, and CEP strategies going forward and thereby confront SAP, Oracle, IBM, Microsoft, and other SOA full-stack vendors head-on in these segments. It is also unclear how much effort or expense Software AG would incur in extricating the IDS Scheer offerings from the larger ARIS portfolio in order to make them more general-purpose and less BPM-centric. Nevertheless, Software AG will at the very least have a strong set of enabling technologies to support any such strategy in the near future.

What’s most exciting, and potentially differentiating, about the Software AG/IDS Scheer BI portfolio is the combination of CEP with mashup and an in-memory architecture to support truly real-time, interactive analytics. In other words, Software AG/IDS Scheer could take a page out of the book of another SOA full-stack vendor: TIBCO and its Spotfire product group. In doing so, Software AG/IDS Scheer would also be well-positioned to duke it out with SAP, IBM, Microsoft, and Oracle, all of which are beginning to emphasize in-memory CEP-enabled BI strategies. As we noted in a report from late 2008, in-memory architectures are coming to dominate the BI arena. Likewise, Forrester has called attention in to the growing adoption of CEP for truly real-time BI.

Whether Software AG capitalizes on the opportunity to expand its SOA solution stack into BI remains to be seen. Considering that it took Oracle more than a year to publicly declare how it will position BEA’s CEP and data federation technologies within its own SOA stack, we may have to wait a while before Software AG and IDS Scheer craft an equivalent roadmap--if they ever do.

But if they wait too long, the newly merging vendors may find that the dynamic SOA, BI, and CEP markets have passed them by.

BI Mashup Maturity Model? Oxymoron? Au Contraire Mon Frère!

By James Kobielus

In one of my recent tweets, I commented that Forrester has developed a maturity model for enterprise adoption of mashup-style, self-service development of business intelligence (BI) applications. Indeed, we have, and it will appear in my forthcoming Forrester report, “Mighty Mashups: Do-It-Yourself Business Intelligence for the New Economy.”

Another tweeter--an astute, but sadly, non-Forrester BI analyst--scoffed that “BI mashup maturity model” is an oxymoron. Respectfully, I must disagree. Enterprises are adopting self-service BI approaches for many reasons--principally, to cut costs in a tight economy, to unclog the development backlog, and to speed delivery of actionable, targeted intelligence to decision makers. Also, companies are providing users with BI tools to do interactive, deeply dimensional exploration of information pulled from enterprise data warehouses (EDW), marts, cubes, transactional applications, and other systems. Furthermore, organizations everywhere have adopted browser-oriented BI environments that leverage the full Web 2.0 interactivity and collaboration.

Sitting at the convergence of those trends is BI mashup, which Forrester sees as the new paradigm for truly pervasive decision-support systems. What throws off some people is the term “mashup,” which sometimes gets pigeonholed as simply referring to using, say, Google Maps to display geocoded performance metrics and sundry Internet-sourced data in a browser-based dashboard. Yes, BI mashup encompasses that approach to presenting and integrating diverse data, but its application is much broader.

Just as important, BI mashup is not bleeding-edge. Rather, BI mashup leverages the in-memory BI clients, semantic virtualization layers, data federation middleware, automated data discovery, and other next-generation BI tools and platforms.

No one vendor or user has yet put together an end-to-end BI environment that is entirely focused on mashup-style self-service development. However, Forrester sees the BI industry converging toward as mashup-oriented architecture over the coming 2-3 years. With that in mind, we sketched out a BI maturity model that encompasses the following four levels (the first 3 of which are represented in case studies in the upcoming report):

• Level 1: Lightweight presentation mashup against transactional applications: This basic maturity level is for companies that have no prior BI or EDW; have little in-house BI expertise; and are comfortable with allowing casual users to use their browsers to customize parameterized reports from data from packaged business applications.                                                                
• Level 2: Deep presentation mashup against EDW: This level is for organization that do have prior BI and centralized EDWs, but have an understaffed BI development group and/or  power users and data modelers urgently require the ability to mashup and explore historical and current data within sophisticated BI workspaces.
• Level 3: Full BI mashup in federated environment: This level is for organizations that have decentralized, dynamic data management environments, and have the expertise to design reusable, composite data services to seamlessly mashup internal and external information.
• Level 4: Full collaborative mashup with IT governance: This level is for organizations that want to encourage subject  matter experts and operational users to collaborate on analytics created through mashup, but who are also concerned that all mashups be controlled, governed, and monitored in accordance with enterprise policies and best practices.

As I said, it will take a few years before we see a substantial number of enterprise case studies that implement the pinnacle of collaborative mashup with tight governance. Nevertheless, when you follow the evolution of next-generation solution portfolios from leading BI vendors such as SAP, IBM, Microsoft, and others, it’s clear that self-service user-centric mashup, to varying degrees, is a core theme.

BI mashup has such a strong business case that we’re confident it’s more than simply a “down economy” theme. It will almost certainly grow in importance for information and knowledge management professionals as the economy improves.

Database Religions Dissolve Into The Big Billowing Virtual Data Cloud

By James Kobielus

Virtualization is a venerable old computing concept that has achieved new life in recent years.

Virtualization brings to life a new world of more flexible service provisioning while cleverly emulating the old world that is being replaced. Virtualization refers to any approach that abstracts the external interface from the internal implementation of some service, functionality, or other resource.

The promise of virtualization is that, no matter how scattered and diverse, all pooled resources behave as if they were a single unified resource, both for usage and administration. In a sense, this is the practical magic that Arthur C. Clarke identified with advanced technology. The external interface may conceal various facts about the implementations of the underlying resources. The virtualized resources may:

•    run on diverse operating and application platforms;
•    have been deployed on nodes in diverse locations;
•    have been aggregated across diverse hosting platforms (or partitioned within a single hosting platform, either through virtual machine software, separate CPUs, or separate blade servers); and have been provisioned dynamically in response to a client request.

When Noel Yuhanna and I presented on enterprise database virtualization last week at Forrester IT Forum, we took pains to point out that is not a radically new paradigm. In fact, database administrators (DBAs) have been doing virtualization for a long time and not realizing it. We’re all familiar with such database virtualization approaches as policy-based server clustering, massive parallel processing database grids, and enterprise information integration. In these environments, you can identify the virtualization layer as “single system image,” “semantic abstraction,” or some other approach.

What all these approaches share is that they make two or more repositories behave as if they were a single database for unified access, query, reporting, predictive analytics, and other applications. If you wish, I could drill down further into the layers of database virtualization—data virtualization, transaction virtualization, and platform virtualization—but that would be too much for a mere blog post.

One twist that I didn’t have time to explore in depth last week is the notion that the traditional hub-and-spoke enterprise data warehousing (EDW) architecture is itself a form of database virtualization. The hub-and-spoke model transforms analytic data to a common “spoke-side” semantic access model, such as star schema or columnar. As such, this approach abstracts from the data models (usually 3NF relational) implemented at the EDW hub tier, the staging tier (perhaps file-based), and OLTP sources (perhaps hierarchical, XML, or what have you).

When you realize that each data-persistence approach has its optimal deployment sphere, you’re thinking database virtualization. At that point, you start to realize that the various database religions—relational is supreme, columnar is king, and so forth—are not absolute truths. They’re simply sectarian texts in a tradition of longer vintage: the evolution of truly all-encompassing data virtualization clouds.

Yes, I’m using “cloud” in this context because it best describes this new paradigm. Cloud-based virtualization is beginning to seep into analytic infrastructures. To support flexible mixed-workload analytics, the EDW, over the coming five to 10 years, will evolve into a virtualized, cloud-based, and supremely scalable distributed platform.

What are the outlines of this new paradigm? The virtualized EDW will allow data to be transparently persisted in diverse physical and logical formats to an abstract, seamless grid of interconnected memory and disk resources and to be delivered with sub-second delay to consuming applications. EDW application service levels will be ensured through an end-to-end, policy-driven, latency-agile, distributed-caching and dynamic query-optimization memory grid, within an information-as-a-service (IaaS) environment. Analytic applications will migrate to the EDW platform and leverage its full parallel-processing, partitioning, scalability, and optimization functionality. At the same time, DBAs will need to make sure that cloud-based DW offerings meet their organizations’ most stringent security, performance, availability, and other service-level requirements.

I won’t opine here and now on how much enterprise data will be persisted in public clouds vs. private environments that incorporate many of the same platform virtualization technologies. I’ll save that discussion for the upcoming Forrester reports that Noel and I are developing in virtualization of transactional and analytic databases, respectively.

Expect those in Q3 or thereabouts. Thanks everybody who attended our preso last week in Vegas!

Information Post-Discovery - Latest BI Trend

By Boris Evelson

I just came back from an exciting week in Orlando, FL, shuttling between SAP SAPPHIRE and IBM Cognos Forum conferences. Thank you, my friends at SAP and IBM for putting the two conferences right next to each other (time- and location-wise), and for saving me an extra trip!

Both conferences showed new and exciting products and both vendors are making great progress towards my vision of “next generation BI”: automated, pervasive, unified and limitless.  I track about 20 different trends under these four categories, but there’s a particular one that is especially catching my attention these days. It went largely under covers at both conferences, and I was struggling with how to verbalize it, until my good friend and peer, Mark Albala, of http://www.info-sight-partners.com, put it in excellent terms for me in an email earlier today: it’s all about “pre-discovery” vs. “post-discovery” of data.

We can debate endlessly the pros and cons of traditional row oriented RDBMS vs. newer DBMS architectures specifically designed for BI and OLAP (like columnar – Sybase IQ, Vertica, Paraccel,  inverted index – Microsoft FAST Search, Endeca, Attivio, tokenized – illuminate, and in-memory analytical DBMS – TIBCO Spotfire, QlikTech) – and I had lots of fun doing that on the DM Radio show last Thursday!  One thing, however, remains undisputed: traditional RDBMS and OLAP architectures require pre-discovery of data, aka data integration and data modeling. No matter how much flexibility and richness we think we built into out relational or multidimensional data models, they are still only as good as our initial design. If we did not anticipate the types of questions that would be asked of our application in the future, no fixed relational or multidimensional data models will be able to help us.

But the world moves way too fast on us. For example, the methodology behind economic capital calculation in the financial services industry, according to Basel II requirements, may change on a weekly, sometimes even on a daily, basis due to regulatory and competitive pressures. No traditional data models and BI tools can keep up with such furiously quickly changing requirements. As a result, one of our recent surveys found that more than half of the respondents did not have most of the information they were looking for in their BI applications, and close to two thirds relied on IT for new BI requests.

What’s the answer? There are many, but one partial answer is post-discovery, rather than pre-discovery of data. For example, an inverted index DBMS from Attivio, or a tokenized data store from illuminate, or in-memory models from TIBCO Spotfire and QlikTech just need you to index or loading data "as is", not really requiring any modeling up front. And because all of these technologies can indeed cross reference every attribute with every other attribute (it's an index!), a virtual data model is created on the fly simply by virtue of asking it a question. Gone are the days of having to analyze your requirements, document your requests, work with IT to make it happen - a process that often takes weeks or months.

Sounds good? It does, but this is obviously not a BI panacea. Yet. I do not think we will see a mass conversion to these analytical engines that allow for post-discovery of information and data models in the short term. Why?

• Many challenges still remain with these technologies such as lack of operational BI (you typically need to reload entire model or rebuild entire index to make updates), administration (partitioning, and modular backup and restore are not easy tasks with index DBMS today), and other mission critical, production features of large enterprise DBMS.
• All these new technologies still rely on someone else doing all of the upfront leg work to integrate, reconcile and cleanse the data. There is no magical work around the hard work of planning, designing and implementing data quality and master data management processes and applications.

How’s all this related to SAP and IBM, you ask? Simple.

• SAP is heavily promoting its guided searchengine – Explorer – previously known as Polestar. It’s a similar index to Endeca and Microsoft FAST Search, capable of allowing BI users discovering answers to previously unplanned questions. Unlike Attivio, though, Explorer still requires an underlying data model, either in the form of Business Objects Universe or SAP BW, but because under the covers it uses the SAP TRex index engine, the right technology is there, and it’s a huge step in the right direction for SAP.
• IBM is also leading the market here. While its Cognos GoSearch product allows for some guided exploration too, it’s the acquisition of Exeros a couple of weeks ago that gives IBM unique capabilities (with some competition from Composite Software and Sypherlink) to post-discoverdata, data relationships and continuously update data models and data content based on the newly discovered information. I can't wait till IBM comes up with a fully automated way to update the data model, Cognos Framework Manager, and Cognos reports and dashboards based on a source system change!

This is yet another proof – and I’ll never get tired of saying this – that BI market is as vibrant, exciting and far from commoditization as ever!

Self-Service Business Intelligence Depends on Automated Data Discovery

 By James Kobielus

If you tuned into my Forrester teleconference yesterday, you heard me discuss the end-to-end infrastructure necessary to fully support mashup-style self-service business intelligence (BI).

One of the key features for BI mashup is automated source-data discovery, which spares information workers from having to find new data sources or fresh updates from existing sources. Instead, the user simply relies on the BI and back-end data virtualization infrastructure to perform these critical activities as ongoing background tasks. Once new sources and feeds are discovered, transformed to a common semantic model, and published to a BI-mashup registry, all the user needs to do is drag and drop them visually into their mashed-up reports, dashboards, and other analytics.

Automated discovery is not only key to BI mashup, but to trustworthy data as well, because it helps detect and remediate anomalies across disparate data sources. Only a few vendors on the market today provide strong features for automated source discovery. One of them is Composite Software, which recently released an appliance that performs these functions. Another is Exeros, which is the closest thing to an automated-data-discovery pure-play in the market today.

Or, rather, was the closest thing, until IBM announced this morning that it is acquiring Exeros. I’ve been following Exeros for several years and have long considered them a strong candidate for acquisition by a leading BI, data warehousing (DW), data integration (DI), or data quality (DQ) vendor. On IBM’s part, this acquisition makes great sense as a complement to its InfoSphere and Optim portfolios on the data management and governance side of the house.

It will also fit nicely with IBM’s Cognos portfolio as a key enabler, potentially, for BI self-service mashup. As I stated on my teleconference, some vendors are further ahead on putting together a completely mashup-enabling end-to-end BI solution, and Cognos is among them. You can download the teleconference slides from Forrester’s website, listen to my streaming audio, and/or wait for my forthcoming report for more in-depth thoughts on this topic.

Now the ball’s in IBM’s rivals’ courts regarding whether, when, and how they plan to add automated source discovery to their BI portfolios.

Free BI Is Still No Free Lunch

By Boris Evelson

In my recent BI Belt Tightening For Tough Economic Times document I explored a few low-cost alternatives to traditional, mainstream, and typically relatively expensive Business Intelligence (BI) tools. While some of these alternatives indeed were a fraction of a cost of a characteristic large enterprise BI software license, there were even fewer truly zero cost options. But there were some. For example, you can:

• Leverage and use no-cost bundled BI software already in-house.Small departments and workgroups may be able to leverage BI software that comes bundled at no additional cost with BI appliances, database management systems (DBMSes), and application licenses. You can consider using these few free licenses from Actuate, IBM Cognos, Information Builders, Jaspersoft, Microsoft, MicroStrategy, Panorama, Pentaho, and SAP Business Objects for additional functions such as testing, QA, and prototyping. While these few free licenses are just a drop in the bucket in a typical large enterprise BI license requirements, do look around and don’t waste money on BI products you may already have.

• Use free Open Source BI components if you are a developer, since the source code is freely and easily available. Access to source code that makes the software easy to embed in other software solutions, so Open Source (OS) is also popular with independent software vendors (ISVs). However, OS BI as an enterprise grade BI solution, is not truly free. Even though individual reporting, OLAP, and even ETL components are available for free, most open source vendors sell add-on products that tie all components into a more fully functional BI application, such as specific dbms and ERP application connectors, GUI administration utilities and others

Now MicroStrategy is taking the concept of free BI one step further. Today MicroStrategy announced that it is making its Release 9software available for up to 100 users/consumers of info (1 CPU, 2 developers, 2 power users, online support only) at no cost! Now that’s a whole different story. Now we are talking about not just individual components, but a complete BI solution. We are also not talking about a small use case for just testing groups or QA or prototyping. With 100 users you can roll a BI application out for free to a whole department in a large enterprise or to a small business. Is this the beginning of a trend? Between OS BI and MicroStrategy new direction, I can’t help but to ponder: is BI (and other enterprise applications) going to go the way of printers or cell phones that are being given away almost at cost just to be able to hook and reel in the customers for supplies and service?

And is it really free? Unfortunately not - it won’t be that easy for a while. As you saw recently in my 80/20 rule blog, typical BI costs involve so much services and so many indirect costs, that I do not see a truly free BI lunch in the near future.

Dislocation Intelligence In A Brutal Economy

By James Kobielus

It's painful to see the auto, newspaper, construction, financial services, and so many other formerly vibrant sectors of the world economy go down the proverbial tubes. One of the most nauseating realities is when millions of people lose their jobs, homes, and communities in a seeming blink.

I grew up in the perpetually recessionary Detroit area. I'm attuned to the regional dislocations that come from depending too much on an industry that has seen better days. Abandoned storefronts, dilapidated housing, vacant lots, tumbleweed-quiet city streets--all of it evidence of a growing ghost town, telltale signs of a marginal economy that depends on government programs, private charity, and low-wage service jobs. During my college years, I was a policy analyst with an urban coalition in downtown Detroit, and I could see that the slide was long-term and nigh irreversible.

Even in relatively well-off areas such as Washington DC, where I've spent close to a quarter-century, we’re not immune to serious economic dislocations. The National Capital Region, so reliant on federal spending, is likely to feel the brunt of whatever cuts Obama will almost certainly make to close this massive deficit. And even here you can’t escape dislocations in sectors that we all depend on, such as retailing, as evidenced by, for example, the ex-Circuit City big boxes that seem even bigger and boxier now that they’re totally empty.

In recent years, corporations have adopted location intelligence solutions to support their market-entry strategies, such as adding new stores and waging marketing campaigns. They also use these tools--essentially, geographic information systems coupled with predictive modeling--to optimize their footprint in existing markets. And to a lesser extent, they also use location intelligence to plan their exit strategy of closings and retrenchments. But when departures are hasty--such as any Chapter 11 proceeding--all we’re left with are vast tracts of vacant real estate, plus many formerly employed people who must find a way to survive amid ruins. Imagine how a General Motors or Chrysler bankruptcy will hit Detroit--it will be a liquidation to rival Hurricane Katrina in its devastation of a major city.

Dislocation intelligence is something that the leaders of the auto companies--and the Obama administration--should exercise when making the tough decisions to restructure this critical industry. People’s pain should be factored into decisions to close and relocate plants, so that, for example, whole cities--such as Flint, Toledo, and Janesville--can make a smooth exit from over-reliance on this one industry. As part of that effort, urban planners should consider the infrastructure that remains behind, so that, for example, whole regions of a city are not suddenly deprived of hospitals, grocery stores, and other basic amenities. Imagine you’re on welfare and have to somehow go 10 miles to buy a loaf of bread.

Mapping tools are just tools, not salvation. Detroit long ago passed a point of no return. The US auto industry will never recover the manufacturing jobs that attracted people from all over the world to southern Michigan, northern Ohio, and other regions.

But we as a country should try to smooth over these economic dislocations so that they don’t completely wipe some places off the map.