Efficiency, Sustainability, And Digital Natives Accelerate Paper’s Path To Digital

  By Sheri McLeish

As people spend more time consuming information digitally at home and at work, reliance on paper continues to decrease. But how far are we across the Digital Divide? In 1975, George E. Pake, then head of Xerox Corp.’s Palo Alto Research Center, predicted that in 1995 his office would be completely different: “There will be a TV-display terminal with keyboard sitting on his desk. I’ll be able to call up documents from my files on the screen, or by pressing a button. I can get my mail or any messages. I don’t know how much hard copy I’ll want in this world.”

Thirty-five years later Pake’s vision largely came true for iWorkers, who today toggle between laptops, smart phones, and other devices to create, consume, and manage information. But an interesting thing happened along the way. With the initial rise in popularity of email and the Internet in the late 1990s, there was actually a bump in paper usage, not a decrease. People printed out their emails, receipts for e-purchases, documents, presentations – you get the idea. In fact, the World Wildlife Fund reported that some 10,000 sheets of paper are used each year by the average office worker in Western industrialized countries.

The other challenge faced by businesses has been the explosion of content types. Firms today need to support and coordinate multiple inbound and outbound channels. For processes like customer onboarding, paper persists due to lack of integration or reliance on paper by customers, partners, and staff. Today only a quarter of enterprises report that 25-50% of their customers are fully paperless, according to a recent Forrester Information Capture survey. Eighty-seven percent of firms have, are in the process of, or are planning to implement or upgrade information capture technology in their companies. The No. 1 investment area? No surprise here: integrating with enterprise apps.

This ongoing effort to remove paper from processes is driven by good, solid business reasons: For each sheet of paper used there are costs for purchasing, storage, copying, printing, labor, postage, disposal, and recycling. The ROI for capture investments is one of the most straight-forward for a technology investment. Driven by cost savings, increased productivity, and improved access and retention of electronic documents, companies continue to invest steadily in information capture to support business processes like registration, invoicing, and contract management.

Most Content Now Born Digital
But there’s another other huge driver of paper reduction in the enterprise: the iWorker. I know I still have a few piles of paper lingering at the office and more at home, but I haven’t looked at that stuff in years. I use my smart phone for a shopping list; I seem to have a hard time locating pens because I so rarely use them.

And I’m not unique. Forrester believes that well over two-thirds of content created today originates digitally. Consider:

• In our Forrester Workforce Analysis Survey, Q2 2009, 79% responded that they use word processing tools, and 60% report using word processing daily.
• Spreadsheets have similar but slightly lower use, with about half of information workers using spreadsheets daily.
• From the same study iWorkers are shown to be pretty happy with word processing tools – 82% indicate high satisfaction

This high level of satisfaction and ongoing use of tools to work with content in digital form supports the notion that most of the content being handled by iWorkers today is in electronic formats. And this will accelerate as Millennials who were “born digital” make up greater proportions of the workforce.

It’s Getting Easier To Be Green
Paper reduction also fits into the Green mindset. Along with embracing digital authoring tools, iWorkers indicate a strong desire to reduce paper in the workplace (75% strongly agree) and recognize the benefits of digital content such as facilitating quicker distribution and ability to reuse content. Paper-based or whiteboard content from collaborative work sessions increasingly are moving to the digital realm. During a tour of EMC’s storage manufacturing facility last week several of the process efficiencies touted were tied to digitizing information collection, including a widescreen digital whiteboard on the shop floor to replicate a physical whiteboard and an application that provides electronic access to the state of components as they are being tested, a task that previously required the technicians to walk through the test rooms and write down the information to then be inputted.

Custom development to automate and digitize manual processes remains core to ongoing process improvement. But for more informal content development, eventually iWorkers will take for granted originating this content digitally. Today adoption of wikis and blogs is still nascent, with under 10% of information workers using these tools. But as collaborative environments become more accessible via Web-based platforms like the Google Wave, Novell’s Pulse, IBM/Lotus Live, and Microsoft’s SharePoint and Office 2010, it will become even easier to live fully in a digital world of content.

Holdouts Hindered By Real Hurdles
There will be exceptions to the digital rule, however. Content captured via paper forms or that isn’t well supported for electronic input, such as math formulas or diagrams, will persist until better user interfaces make even these uses obsolete on paper. Most iWorkers today are not provisioned a stylus and tablet PC, so doing a first pass on paper or on a whiteboard remains faster even if eventually these drawings need to be captured digitally.

And there will be other holdouts as well: In an interview with NPR earlier this year, one of our era’s most prolific novelists James Patterson admitted he does all of his writing in longhand before turning it over to an assistant to be typeset. But in college classrooms, often the best barometer for what’s to come, more and more professors are allowing laptops and electronic note taking tools to be used in class. And as we look even further ahead, consider that penmanship has taken a backseat to keyboarding in the elementary curriculum at many schools. So for the class of 2025, Forrester predicts that their offices may not be just sans paper, but without a pen too. I guess that puts me ahead of the curve.

What about you? Are you farther along the paperless trail at home or at work? What’s driving paper reduction in your firm? What’s holding you back?

Is BAM Relevant In The Age of Lean Processes?

By James Kobielus

In any dynamic business environment, the last thing you want is to indulge in navel-gazing. If you’re finely attuned to monitoring internal processes, but are not adept at responding to breaking events or anticipating the future, you will find yourself marginalized in the new global economy.

In a turbulent world, the most successful companies are those that stay lean--in other words, trimmed of excess overhead, fit for new challenges, and flexible enough to evolve rapidly. Forrester advises customers to make their business processes leaner, through such measures as reducing the number of steps, routing paths, participants, decision points, milestones, and revisions. As you do, you will find concomitant improvements in speed, cost, quality, and customer satisfaction.

Clearly, you can’t do that well if you don’t have, at the very least, the means to monitor a wide range of business processes in unified dashboards, reports, and other applications. That’s where traditional business activity monitoring (BAM) tools, which are a key component of most business process management (BPM) environments, play a critical role. However, critical as BAM may be, it is often deployed in a predominantly navel-gazing process-tracking capacity (remember what that “M” stands for) and, as such, is as much part of the problem as the solution.

Why? Well, for starters, most BAM environments are deployed in conjunction with specific vendors’ BPM platforms--hence can only track the subset of business processes that run on those platforms. But, even more limiting, BAM dashboards are often the stand-alone province of process administrators. Only in a handful of  BAM-using organizations do alerts and metrics escalate beyond the process administrator level and trigger the necessary human and automated workflows necessary to respond to breaking issues--or nip them in the bud before they can become issues.

To stay relevant, BAM needs to evolve in the age of lean processes so that it can become a tool for business agility at all levels. Next week, Forrester BPM analyst Clay Richardson and I will do a podcast on this very topic. In  that podcast, and in a joint presentation that Natalie Petouhoff and I will do at Forrester Business and Technology Forum in Chicago in October, I’ll sketch out a vision for the next-generation of BAM (which I half-jokingly call “uber-BAM”).

Here is a rough sketch of the maturity levels of BAM's evolution to realize this vision:

• BAM Level 1: Business activity monitoring: This is today’s BAM baseline. Essentially, it combines BPM with business intelligence (BI), interactive dashboarding, and complex event processing to deliver process metrics to consoles used solely by human process administrators.
• BAM Level 2: Business action mobilization: This is the BAM level that many forward-looking users and vendors have implemented. It combines all Level 1 functionality with business rules engines (BRE) to drive sophisticated human and automated workflows in response to breaking events. It puts the “immediate action loop” into the BAM equation.
• BAM Level 3: Business adaptability management: This is the promise of truly lean uber-BAM, and is found only in a vanguard of visionary  business environments. It combines all Level 2 functionality with embedded predictive-analytics models, which are developed and maintained by business process architects and/or subject-matter process experts. Continuously mining historical and real-time metrics, these in-process analytics drive automated “recommendation engines” that adapt coordinate response to changing conditions. Recommendation engines enable human and automated decision agents to seize opportunities and neutralize threats well ahead of the curve.

Going forward, this new vision of BAM’s evolution and promise will inform Forrester’s coverage of process agility and optimization.

IBM Goes Deeply Predictive, Announces Acquisition of SPSS

By James Kobielus

IBM dropped a big bombshell at the start of any already action-packed day for the analyst community. At this moment, I’m sitting, along with several dozen of my peers from Forrester and other firms, at the IBM Smart Analytics System launch event in Hawthorne NY. I’ll blog on IBM’s other announcements in a separate items.

The bombshell was IBM’s announcement that it’s acquiring SPSS, a long-established, leading provider of predictive analytics (PA), data mining (DM), statistical analysis, and text mining tools. The acquisition, subject to the usual shareholder approvals and regulatory reviews, is expected to close later this year. But, even in advance of that near-certain consummation, IBM’s bold move has already sent shockwaves throughout the analytics market.

Most important, IBM has acquired the second largest vendor of PA/DM solutions, dwarfed only by privately held SAS Institute. In this segment, IBM’s proposed acquisition is having the same impact that its Cognos buy had on the business intelligence (BI) market two years ago. In many discussions with Forrester customers, SPSS is often mentioned as a key solution provider for predictive modeling and statistical analysis against structured, semi-structured, and unstructured content.

For IBM’s competitive standing in the data management market, this acquisition represents one of the last missing pieces of its Information On Demand (IOD) portfolio. By acquiring SPSS, IBM has acquired a substantial PA/DM brand with a very loyal set of longtime customers who have build their customer churn, supply chain optimization, and other predictive models on its best-of-breed platform. SPSS recently underlined its feature-comprehensive value proposition through re-branding around the “Predictive Analytics Software” (PASW) family name. But longtime customers didn’t need to be reminded, of course, that what used to be known as “Clementine” defines a functional high-bar in this solution segment.

IBM would probably be the first to admit that it took its focus off the PA/DM market over the past several years as it build out the BI, data warehousing (DW), and other pieces of its IOD portfolio. IBM had never really exited the PA/DM market, but casual observers might have thought otherwise. However, the vendor three years ago chose to de-emphasize its Intelligent Miner tools--which support mining of structured data--as stand-alone offerings. It essentially buried these solutions, moving them into its InfoSphere Balanced Warehouse family, where they are now offered as features of its Enterprise Edition DW software, rather than as stand-alone tools that would be enhanced and evolved independently.

IBM and SPSS’s respective customer bases should rest assured that overlaps among their respective product portfolios are not extensive. Once the acquisition closes, IBM is almost certain to build out its SPSS brand and, over the coming 1-2 years, phase out the Intelligent Miner technology within its InfoSphere portfolio. One tricky issue is which text analytics solution family--SPSS’ or IBM’s OmniFind solutions--will prevail as the parent converges these offerings in its IOD portfolio. Another issue is how IBM will integrate the SPSS offerings into its still-evolving in-database analytics roadmap for InfoSphere Balanced Warehouse. Hopefully, IBM will maintain and extend SPSS’ already extensive in-database analytics integration with a broad range of vendor DWs, including such Big Blue rivals as Oracle, Microsoft, Sybase, and Teradata.

Who loses from IBM’s acquisition of SPSS? Fundamentally, one can’t help think that SAP missed the boat by not seizing the opportunity to acquire partner SPSS, whose Clementine technology it OEMs, has integrated with its BI technology, and sells as SAP BusinessObjects Predictive Workbench. PA/DM is an increasingly key component of a full-fledged BI solution stack. However, the remaining field of vendors with stand-alone, horizontally applicable PA/DM vendors consists primarily of vendors who are large but proudly and stubbornly independent (especially, SAS Institute); high-quality but much less widely adopted (e.g., KXEN, ThinkAnalytics); or specialized on customer, financial, scientific, or other specialized analytics (e.g., Unica, Fair Isaac, Accelrys).

Some IBM rivals in the BI space already have strong PA/DM tools, most notably Oracle (Oracle Data Miner) and TIBCO/Spotfire (the Insightful tools). Among BI vendors, Microsoft, MicroStrategy, and Information Builders have PA/DM capabilities, but they are not to a SAS or SPSS level of sophistication. These and other BI vendors should also be scouting for strategic acquisitions.

What do you think? Will IBM’s acquisition of SPSS lead to further merger and acquisition activity in this space as other leading BI players strengthen their PA/DM solutions?

By James Kobielus

IBM dropped a big bombshell at the start of any already action-packed day for the analyst community. At this moment, I’m sitting, along with several dozen of my peers from Forrester and other firms, at the IBM Smart Analytics System launch event in Hawthorne NY. I’ll blog on IBM’s other announcements in a separate items.

The bombshell was IBM’s announcement that it’s acquiring SPSS, a long-established, leading provider of predictive analytics (PA), data mining (DM), statistical analysis, and text mining tools. The acquisition, subject to the usual shareholder approvals and regulatory reviews, is expected to close later this year. But, even in advance of that near-certain consummation, IBM’s bold move has already sent shockwaves throughout the analytics market.

Most important, IBM has acquired the second largest vendor of PA/DM solutions, dwarfed only by privately held SAS Institute. In this segment, IBM’s proposed acquisition is having the same impact that its Cognos buy had on the business intelligence (BI) market two years ago. In many discussions with Forrester customers, SPSS is often mentioned as a key solution provider for predictive modeling and statistical analysis against structured, semi-structured, and unstructured content.

For IBM’s competitive standing in the data management market, this acquisition represents one of the last missing pieces of its Information On Demand (IOD) portfolio. By acquiring SPSS, IBM has acquired a substantial PA/DM brand with a very loyal set of longtime customers who have build their customer churn, supply chain optimization, and other predictive models on its best-of-breed platform. SPSS recently underlined its feature-comprehensive value proposition through re-branding around the “Predictive Analytics Software” (PASW) family name. But longtime customers didn’t need to be reminded, of course, that what used to be known as “Clementine” defines a functional high-bar in this solution segment.

IBM would probably be the first to admit that it took its focus off the PA/DM market over the past several years as it build out the BI, data warehousing (DW), and other pieces of its IOD portfolio. IBM had never really exited the PA/DM market, but casual observers might have thought otherwise. However, the vendor three years ago chose to de-emphasize its Intelligent Miner tools--which support mining of structured data--as stand-alone offerings. It essentially buried these solutions, moving them into its InfoSphere Balanced Warehouse family, where they are now offered as features of its Enterprise Edition DW software, rather than as stand-alone tools that would be enhanced and evolved independently.

IBM and SPSS’s respective customer bases should rest assured that overlaps among their respective product portfolios are not extensive. Once the acquisition closes, IBM is almost certain to build out its SPSS brand and, over the coming 1-2 years, phase out the Intelligent Miner technology within its InfoSphere portfolio. One tricky issue is which text analytics solution family--SPSS’ or IBM’s OmniFind solutions--will prevail as the parent converges these offerings in its IOD portfolio. Another issue is how IBM will integrate the SPSS offerings into its still-evolving in-database analytics roadmap for InfoSphere Balanced Warehouse. Hopefully, IBM will maintain and extend SPSS’ already extensive in-database analytics integration with a broad range of vendor DWs, including such Big Blue rivals as Oracle, Microsoft, Sybase, and Teradata.

Who loses from IBM’s acquisition of SPSS? Fundamentally, one can’t help think that SAP missed the boat by not seizing the opportunity to acquire partner SPSS, whose Clementine technology it OEMs, has integrated with its BI technology, and sells as SAP BusinessObjects Predictive Workbench. PA/DM is an increasingly key component of a full-fledged BI solution stack. However, the remaining field of vendors with stand-alone, horizontally applicable PA/DM vendors consists primarily of vendors who are large but proudly and stubbornly independent (especially, SAS Institute); high-quality but much less widely adopted (e.g., KXEN, ThinkAnalytics); or specialized on customer, financial, scientific, or other specialized analytics (e.g., Unica, Fair Isaac, Accelrys).

Some IBM rivals in the BI space already have strong PA/DM tools, most notably Oracle (Oracle Data Miner) and TIBCO/Spotfire (the Insightful tools). Among BI vendors, Microsoft, MicroStrategy, and Information Builders have PA/DM capabilities, but they are not to a SAS or SPSS level of sophistication. These and other BI vendors should also be scouting for strategic acquisitions.

What do you think? Will IBM’s acquisition of SPSS lead to further merger and acquisition activity in this space as other leading BI players strengthen their PA/DM solutions?

Microsoft Office 2010: The Odyssey Continues

by Sheri McLeish

Sequels never can match the thrill of the original. But the good ones offer a compelling story of their own, develop familiar characters, and introduce something new and exciting. Last week Microsoft gave developers a backstage pass to preview Office 2010, due out in the first half of next year. The drama unfolds with Microsoft and Google waging a multi-front war with each other in search, browsers, productivity tools, and, soon, operating systems. Glimpses of the fourteenth iteration of Office reveal Web-based lightweight apps along with capabilities geared at improving collaboration, multimedia content development, and email management.

Can Office 2010 save the franchise? Or will a simpler, better customer experience from Google draw in a bigger audience before next summer? And what does it mean for the bit players, independents, and sleepers like the Open Office suites from IBM/Lotus, Novell, and Sun, or for Adobe, Zoho, Thinkfree, Corel … that’s a lot of competition for a sluggish software market that Forrester sees as being down by 5% or more for the year.  The glimmer of hope for software vendors will likely come from subscriptions revenue for software-as-a-service (SaaS) products in 2010, which Forrester projects to grow by 7.5%.

Microsoft wants to lead that growing SaaS market. For enterprises, what’s notable about the Office 2010 story is that Office Web apps can be on-premise or hosted. The lightweight Web browser versions of Word, PowerPoint, Excel and OneNote will be free for consumers through Windows Live. Pow! But the bigger deal for enterprises is the option to host Office Web apps on-premises as annuity customers as well as via a hosted subscription through Microsoft Online Services. This option isn’t offered by Google today and, for the moment, may be what makes Office Web apps a hit in the enterprise.

To date, Microsoft’s dominance in large businesses remains mostly intact, with 57% of firms running Office 2007 and 80% supporting some version of Office.  Combined, the alternatives make up less than 8% of the enterprise market, according to Forrester’s March 2009 North America, Europe, And Asia Pacific Desktop Innovation Online Survey. And of those Forrester surveyed, 78% said they have no plans to look for an alternative to Microsoft Office. Real barriers remain for alternatives, from concerns about content control and security, sunk license costs, and online/offline issues for Web-based tools to fear of rejection by business users. Like it, love it, or not, people have a comfortable, familiar relationship with the Office apps. And that’s a critical edge Microsoft must maintain.

Technology Populism is fueling the collaboration and mobile collaboration markets and blurring the lines between work/life boundaries.  The influence of consumer experience can be equally powerful if harnessed by Google for email and productivity. Most enterprise IT departments rely on the feedback of their business users to measure the value of their productivity tools. Forrester data also shows upgrades generally driven by business demands (34%), because current tools are no longer supported (24%), are no longer compatible (16%), or because the culture demands it (15%). By promoting free access to Web-based tools, Microsoft seeks the sway of the public. Office 2000 ends support this month; Microsoft needs to get those firms on board with 2010 somehow. What will your firm do? What are your barriers to upgrading Office or moving to an alternative? Now is a good time to clarify your firm’s strategy, because 2010 looks like it could be a blockbuster year for buyers prepared to negotiate.

BI, Analytics, And CEP: Some Fruitful Potential Follow-Ons From Software AG’s Acquisition Of IDS Scheer

 by James Kobielus

Yes, of course, Software AG is buying IDS Scheer primarily for the latter’s ARIS family of business process management (BPM) tools. I’ll leave it to my Forrester colleagues who focus on BPM--on both the IT and TI sides of the house--to call out the ramifications for Software AG’s positioning in that market.

But, believe it or not, this deal will also launch Software AG into the growing markets for business intelligence (BI), analytics, and complex event processing (CEP) solutions. We bet you didn’t realize that IDS Scheer has ARIS solutions in these fast growing markets, but in fact they do--and they’re continue to evolve those offerings.

It’s no surprise that IDS Scheer’s BI, analytics, and CEP offerings supplement and extend its BPM portfolio. Its CEP solution, ARIS Process Event Monitor, supports business activity monitoring (BAM). Its analytics offerings, ARIS Process Performance Management and ARIS Performance Dashboard, support visualization, dashboarding, scorecarding, drilldown, and alerting on process key performance indicators (KPIs), both historical and real-time. And its forthcoming BI offering, ARIS MashZone, will support self-service user development of reports, dashboards, and other views of process and business metrics.

IDS Scheer has little market share in these non-core segments. And the vendor is no immediate threat, by itself or under its future corporate parent, to the leaders in the BI, analytics, and CEP segments. Indeed, its forthcoming mashup-oriented BI offering only provides a subset of the features available from market leaders such as SAP Business Objects, IBM Cognos, and MicroStrategy. But the fact that Software AG will soon be able to provide its own offerings in those segments, rather than rely wholly on partners, represents an important step in its attempt to field a full service oriented architecture (SOA) solution stack.

As noted in a blog entry a year and a half ago, BI is the crown jewel in any comprehensive SOA solution portfolio. SOA suites cannot be considered feature-complete unless they incorporate a comprehensive range of BI features. This acquisition continues the ongoing SOA solution build-out strategy that motivated Software AG to acquire webMethods in 2007.

But it’s not clear yet whether Software AG plans to flesh out its BI, analytics, and CEP strategies going forward and thereby confront SAP, Oracle, IBM, Microsoft, and other SOA full-stack vendors head-on in these segments. It is also unclear how much effort or expense Software AG would incur in extricating the IDS Scheer offerings from the larger ARIS portfolio in order to make them more general-purpose and less BPM-centric. Nevertheless, Software AG will at the very least have a strong set of enabling technologies to support any such strategy in the near future.

What’s most exciting, and potentially differentiating, about the Software AG/IDS Scheer BI portfolio is the combination of CEP with mashup and an in-memory architecture to support truly real-time, interactive analytics. In other words, Software AG/IDS Scheer could take a page out of the book of another SOA full-stack vendor: TIBCO and its Spotfire product group. In doing so, Software AG/IDS Scheer would also be well-positioned to duke it out with SAP, IBM, Microsoft, and Oracle, all of which are beginning to emphasize in-memory CEP-enabled BI strategies. As we noted in a report from late 2008, in-memory architectures are coming to dominate the BI arena. Likewise, Forrester has called attention in to the growing adoption of CEP for truly real-time BI.

Whether Software AG capitalizes on the opportunity to expand its SOA solution stack into BI remains to be seen. Considering that it took Oracle more than a year to publicly declare how it will position BEA’s CEP and data federation technologies within its own SOA stack, we may have to wait a while before Software AG and IDS Scheer craft an equivalent roadmap--if they ever do.

But if they wait too long, the newly merging vendors may find that the dynamic SOA, BI, and CEP markets have passed them by.

BI Mashup Maturity Model? Oxymoron? Au Contraire Mon Frère!

By James Kobielus

In one of my recent tweets, I commented that Forrester has developed a maturity model for enterprise adoption of mashup-style, self-service development of business intelligence (BI) applications. Indeed, we have, and it will appear in my forthcoming Forrester report, “Mighty Mashups: Do-It-Yourself Business Intelligence for the New Economy.”

Another tweeter--an astute, but sadly, non-Forrester BI analyst--scoffed that “BI mashup maturity model” is an oxymoron. Respectfully, I must disagree. Enterprises are adopting self-service BI approaches for many reasons--principally, to cut costs in a tight economy, to unclog the development backlog, and to speed delivery of actionable, targeted intelligence to decision makers. Also, companies are providing users with BI tools to do interactive, deeply dimensional exploration of information pulled from enterprise data warehouses (EDW), marts, cubes, transactional applications, and other systems. Furthermore, organizations everywhere have adopted browser-oriented BI environments that leverage the full Web 2.0 interactivity and collaboration.

Sitting at the convergence of those trends is BI mashup, which Forrester sees as the new paradigm for truly pervasive decision-support systems. What throws off some people is the term “mashup,” which sometimes gets pigeonholed as simply referring to using, say, Google Maps to display geocoded performance metrics and sundry Internet-sourced data in a browser-based dashboard. Yes, BI mashup encompasses that approach to presenting and integrating diverse data, but its application is much broader.

Just as important, BI mashup is not bleeding-edge. Rather, BI mashup leverages the in-memory BI clients, semantic virtualization layers, data federation middleware, automated data discovery, and other next-generation BI tools and platforms.

No one vendor or user has yet put together an end-to-end BI environment that is entirely focused on mashup-style self-service development. However, Forrester sees the BI industry converging toward as mashup-oriented architecture over the coming 2-3 years. With that in mind, we sketched out a BI maturity model that encompasses the following four levels (the first 3 of which are represented in case studies in the upcoming report):

• Level 1: Lightweight presentation mashup against transactional applications: This basic maturity level is for companies that have no prior BI or EDW; have little in-house BI expertise; and are comfortable with allowing casual users to use their browsers to customize parameterized reports from data from packaged business applications.                                                                
• Level 2: Deep presentation mashup against EDW: This level is for organization that do have prior BI and centralized EDWs, but have an understaffed BI development group and/or  power users and data modelers urgently require the ability to mashup and explore historical and current data within sophisticated BI workspaces.
• Level 3: Full BI mashup in federated environment: This level is for organizations that have decentralized, dynamic data management environments, and have the expertise to design reusable, composite data services to seamlessly mashup internal and external information.
• Level 4: Full collaborative mashup with IT governance: This level is for organizations that want to encourage subject  matter experts and operational users to collaborate on analytics created through mashup, but who are also concerned that all mashups be controlled, governed, and monitored in accordance with enterprise policies and best practices.

As I said, it will take a few years before we see a substantial number of enterprise case studies that implement the pinnacle of collaborative mashup with tight governance. Nevertheless, when you follow the evolution of next-generation solution portfolios from leading BI vendors such as SAP, IBM, Microsoft, and others, it’s clear that self-service user-centric mashup, to varying degrees, is a core theme.

BI mashup has such a strong business case that we’re confident it’s more than simply a “down economy” theme. It will almost certainly grow in importance for information and knowledge management professionals as the economy improves.

AddressDoctor A Smart DQ Acquisition For Informatica, Still Moving Closer To MDM

by Rob Karel

Informatica today announced its planned acquisition of AddressDoctor GmbH, a premier global postal address data quality software vendor. AddressDoctor provides postal address cleansing and verification for over 240 countries through batch processing or real-time Web services, supporting 40 different character sets and can transliterate non-Latin characters including Greek, Cyrillic, Hebrew, Japanese, Chinese and Korean.

While AddressDoctor may not be a household name for many data management professions, data quality (DQ) platform and master data management (MDM) vendors certainly know it. Vendors such as Capscan, DataFlux, Datanomic, IBM, Initiate Systems and Siperian either OEM or resell AddressDoctor technology to provide address cleansing capabilities within their broader DQ and/or MDM solutions. Even perceived competitors in the niche postal address verification market like Experian’s QAS and Melissa Data have relied on AddressDoctor to expand their global reach for postal address data.

Informatica also had a prior OEM relationship with AddressDoctor, so the ability to provide global address verification is nothing new. Instead, Informatica says the acquisition will support their “Pervasive DQ” strategy and will allow them to more easily embed address validation within their Profiling, DQ Scorecarding and Monitoring solutions, in addition to its Informatica On-demand “cloud” services.

But I also see this acquisition playing additional strategic roles for Informatica. First, by acquiring what amounts to a market-hub of global address verification technology, Informatica has the full attention of many of its DQ and data integration competitors like DataFlux and IBM. While Informatica has of course announced that they intend to support these prior partnerships, the reality is DataFlux and IBM should be shopping around for another channel to provide these capabilities – too risky to rely on the good graces of a tough competitor.

But for me, the most interesting outcome of this acquisition is that Informatica further entangles themselves with already strong partnerships with the two leading pure play MDM vendors, Initiate Systems and Siperian. Last year, Iblogged on Informatica’s acquisition of identity resolution (IR) vendor Identity Systems, and it seemed obvious to me then that Informatica was looking to do more than just to enter the IR market and enhance its DQ offering. I suggested at that time, and I believe even more strongly today, that these acquisitions are just the appetizers for Informatica as they determine how they can most successfully enter the ever-growing MDM market - not just as yet another new entrant in a market full of “me too’s”, but as a legitimate contender to derail MDM market share leaders IBM, Oracle, and SAP.

Here’s how:

Identity Systems, now branded as Informatica Identity Resolution, is certainly a great addition for Informatica’s data quality platform with its strong matching technology, but Identity Systems also remains the core match engine for Siperian’s MDM solution (among other MDM partnerships including D&B’s Purisma and Oracle). And now with the acquisition of AddressDoctor, Informatica also owns one of Siperian’s core data quality components. Add to that the fact that over 50% of Siperian’s customers use Informatica’s PowerCenter data integration platform to load data into the Siperian MDM Hub, it seems that Informatica is already directly influencing some pretty core parts of Siperian’s MDM success.

Informatica also has a solid partnership with Initiate Systems. In fact, Informatica even invested in Initiate, having participating in its last rounding of funding back in June 2008. It’s certainly feasible that Informatica could choose to acquire Initiate’s solid MDM platform instead, but the existing synergies between Informatica and Siperian still have me leaning towards that as the acquisition that can legitimately unseat IBM, Oracle, and SAP.

Of course all of this could be moot if Oracle decides to acquire Informatica to 1) resolve it’s noticeably absent data quality portfolio (beyond partnerships and OEMs), 2) give up trying to rationalize its Oracle-product-centric data integration portfolio (Oracle Data Integrator and Oracle Warehouse Builder) in exchange for a market-leading heterogeneous platform, and 3) eliminate the risk of Informatica entering the MDM market as a competitor. These rumors have been floating around for a while now, and I’m not convinced - but big pockets like Oracle have been known to do the most interesting things…

Database Religions Dissolve Into The Big Billowing Virtual Data Cloud

By James Kobielus

Virtualization is a venerable old computing concept that has achieved new life in recent years.

Virtualization brings to life a new world of more flexible service provisioning while cleverly emulating the old world that is being replaced. Virtualization refers to any approach that abstracts the external interface from the internal implementation of some service, functionality, or other resource.

The promise of virtualization is that, no matter how scattered and diverse, all pooled resources behave as if they were a single unified resource, both for usage and administration. In a sense, this is the practical magic that Arthur C. Clarke identified with advanced technology. The external interface may conceal various facts about the implementations of the underlying resources. The virtualized resources may:

•    run on diverse operating and application platforms;
•    have been deployed on nodes in diverse locations;
•    have been aggregated across diverse hosting platforms (or partitioned within a single hosting platform, either through virtual machine software, separate CPUs, or separate blade servers); and have been provisioned dynamically in response to a client request.

When Noel Yuhanna and I presented on enterprise database virtualization last week at Forrester IT Forum, we took pains to point out that is not a radically new paradigm. In fact, database administrators (DBAs) have been doing virtualization for a long time and not realizing it. We’re all familiar with such database virtualization approaches as policy-based server clustering, massive parallel processing database grids, and enterprise information integration. In these environments, you can identify the virtualization layer as “single system image,” “semantic abstraction,” or some other approach.

What all these approaches share is that they make two or more repositories behave as if they were a single database for unified access, query, reporting, predictive analytics, and other applications. If you wish, I could drill down further into the layers of database virtualization—data virtualization, transaction virtualization, and platform virtualization—but that would be too much for a mere blog post.

One twist that I didn’t have time to explore in depth last week is the notion that the traditional hub-and-spoke enterprise data warehousing (EDW) architecture is itself a form of database virtualization. The hub-and-spoke model transforms analytic data to a common “spoke-side” semantic access model, such as star schema or columnar. As such, this approach abstracts from the data models (usually 3NF relational) implemented at the EDW hub tier, the staging tier (perhaps file-based), and OLTP sources (perhaps hierarchical, XML, or what have you).

When you realize that each data-persistence approach has its optimal deployment sphere, you’re thinking database virtualization. At that point, you start to realize that the various database religions—relational is supreme, columnar is king, and so forth—are not absolute truths. They’re simply sectarian texts in a tradition of longer vintage: the evolution of truly all-encompassing data virtualization clouds.

Yes, I’m using “cloud” in this context because it best describes this new paradigm. Cloud-based virtualization is beginning to seep into analytic infrastructures. To support flexible mixed-workload analytics, the EDW, over the coming five to 10 years, will evolve into a virtualized, cloud-based, and supremely scalable distributed platform.

What are the outlines of this new paradigm? The virtualized EDW will allow data to be transparently persisted in diverse physical and logical formats to an abstract, seamless grid of interconnected memory and disk resources and to be delivered with sub-second delay to consuming applications. EDW application service levels will be ensured through an end-to-end, policy-driven, latency-agile, distributed-caching and dynamic query-optimization memory grid, within an information-as-a-service (IaaS) environment. Analytic applications will migrate to the EDW platform and leverage its full parallel-processing, partitioning, scalability, and optimization functionality. At the same time, DBAs will need to make sure that cloud-based DW offerings meet their organizations’ most stringent security, performance, availability, and other service-level requirements.

I won’t opine here and now on how much enterprise data will be persisted in public clouds vs. private environments that incorporate many of the same platform virtualization technologies. I’ll save that discussion for the upcoming Forrester reports that Noel and I are developing in virtualization of transactional and analytic databases, respectively.

Expect those in Q3 or thereabouts. Thanks everybody who attended our preso last week in Vegas!

Information Post-Discovery - Latest BI Trend

By Boris Evelson

I just came back from an exciting week in Orlando, FL, shuttling between SAP SAPPHIRE and IBM Cognos Forum conferences. Thank you, my friends at SAP and IBM for putting the two conferences right next to each other (time- and location-wise), and for saving me an extra trip!

Both conferences showed new and exciting products and both vendors are making great progress towards my vision of “next generation BI”: automated, pervasive, unified and limitless.  I track about 20 different trends under these four categories, but there’s a particular one that is especially catching my attention these days. It went largely under covers at both conferences, and I was struggling with how to verbalize it, until my good friend and peer, Mark Albala, of http://www.info-sight-partners.com, put it in excellent terms for me in an email earlier today: it’s all about “pre-discovery” vs. “post-discovery” of data.

We can debate endlessly the pros and cons of traditional row oriented RDBMS vs. newer DBMS architectures specifically designed for BI and OLAP (like columnar – Sybase IQ, Vertica, Paraccel,  inverted index – Microsoft FAST Search, Endeca, Attivio, tokenized – illuminate, and in-memory analytical DBMS – TIBCO Spotfire, QlikTech) – and I had lots of fun doing that on the DM Radio show last Thursday!  One thing, however, remains undisputed: traditional RDBMS and OLAP architectures require pre-discovery of data, aka data integration and data modeling. No matter how much flexibility and richness we think we built into out relational or multidimensional data models, they are still only as good as our initial design. If we did not anticipate the types of questions that would be asked of our application in the future, no fixed relational or multidimensional data models will be able to help us.

But the world moves way too fast on us. For example, the methodology behind economic capital calculation in the financial services industry, according to Basel II requirements, may change on a weekly, sometimes even on a daily, basis due to regulatory and competitive pressures. No traditional data models and BI tools can keep up with such furiously quickly changing requirements. As a result, one of our recent surveys found that more than half of the respondents did not have most of the information they were looking for in their BI applications, and close to two thirds relied on IT for new BI requests.

What’s the answer? There are many, but one partial answer is post-discovery, rather than pre-discovery of data. For example, an inverted index DBMS from Attivio, or a tokenized data store from illuminate, or in-memory models from TIBCO Spotfire and QlikTech just need you to index or loading data "as is", not really requiring any modeling up front. And because all of these technologies can indeed cross reference every attribute with every other attribute (it's an index!), a virtual data model is created on the fly simply by virtue of asking it a question. Gone are the days of having to analyze your requirements, document your requests, work with IT to make it happen - a process that often takes weeks or months.

Sounds good? It does, but this is obviously not a BI panacea. Yet. I do not think we will see a mass conversion to these analytical engines that allow for post-discovery of information and data models in the short term. Why?

• Many challenges still remain with these technologies such as lack of operational BI (you typically need to reload entire model or rebuild entire index to make updates), administration (partitioning, and modular backup and restore are not easy tasks with index DBMS today), and other mission critical, production features of large enterprise DBMS.
• All these new technologies still rely on someone else doing all of the upfront leg work to integrate, reconcile and cleanse the data. There is no magical work around the hard work of planning, designing and implementing data quality and master data management processes and applications.

How’s all this related to SAP and IBM, you ask? Simple.

• SAP is heavily promoting its guided searchengine – Explorer – previously known as Polestar. It’s a similar index to Endeca and Microsoft FAST Search, capable of allowing BI users discovering answers to previously unplanned questions. Unlike Attivio, though, Explorer still requires an underlying data model, either in the form of Business Objects Universe or SAP BW, but because under the covers it uses the SAP TRex index engine, the right technology is there, and it’s a huge step in the right direction for SAP.
• IBM is also leading the market here. While its Cognos GoSearch product allows for some guided exploration too, it’s the acquisition of Exeros a couple of weeks ago that gives IBM unique capabilities (with some competition from Composite Software and Sypherlink) to post-discoverdata, data relationships and continuously update data models and data content based on the newly discovered information. I can't wait till IBM comes up with a fully automated way to update the data model, Cognos Framework Manager, and Cognos reports and dashboards based on a source system change!

This is yet another proof – and I’ll never get tired of saying this – that BI market is as vibrant, exciting and far from commoditization as ever!

Oracle’s Sun Acquisition Accelerates Push Into Data Warehousing Appliances

By James Kobielus

Last fall, Oracle CEO Larry Ellison announced that his company was getting into the hardware business, but I think he misspoke. At that time, he was referring to the new HP Oracle Database Machine with Exadata Storage, a high-end data warehousing (DW) appliance that incorporated hardware from his partner, as well as intelligent storage software technology from that partner--and even had the partner’s name first in the product name. If that was the criterion for “getting into the hardware business”--i.e., running on someone else’s hardware--then every software vendor on earth is in the hardware business, by my reckoning.

But today’s Oracle announcement is the real deal. Oracle is acquiring longtime partner Sun Microsystems, putting the software powerhouse fully into the hardware business--and hitting the DW industry like an earthquake. I’ll let my Forrester colleagues blog on the other implications of this deal--for the open source, Java, middleware, SOA, and other markets that Sun is in--and give you a few quick thoughts on the deal’s implications for the DW market.

For starters, this deal will give Oracle the ability to engineer a completely integrated DW appliance composed of all Oracle components, including hardware and software. Now Oracle will be able to take on Teradata and IBM--both of which have long offered their own integrated solutions--more aggressively with high-performance DW offerings. Just as important, Oracle will be able to leverage Sun’s manufacturing scale economies to bring its all-Oracle DW appliances below the $25K-per-terabyte threshold needed for penetration into the midmarket.

Also, Oracle will now have another widely adopted transactional database, the open-source MySQL, that it can--and should--consider tweaking and packaging on an DW appliance. To the extent that Oracle gives customers a choice of DBMSs on a DW appliance platform, it can gain a differentiator that Teradata, IBM, Microsoft, Sybase, and Netezza lack (you have to go to a startup such as Dataupia for multi-DBMS choice on an appliance). Many information managers prefer to stick with their existing DBMSs when building a DW, and prefer to implement that DW on an appliance to take advantage of its out-of-box balanced configuration of CPU, memory, storage, and I/O.

Furthermore, Oracle is acquiring a hardware and operating system vendor that has long been one of the primary platforms on which its own DW/DBMSs, middleware, and tools have been deployed. This acquisition can only be welcome news for joint Oracle-Sun DW customers who have worried about Sun’s solvency for some time now and began to sweat serious bullets when IBM failed to emerge as a white knight. For many Sun customers, an Oracle-powered DW platform will now look like a safer bet than ever.

Of course, there are clear risks in this pending acquisition.

First, a combined Oracle/Sun sows uncertainty among the DW appliance vendors--such as Greenplum and ParAccel--who have partnered with Sun and now find themselves in earnest “co-opetition” with full-competitor (and then some) Oracle.

Second, Oracle’s other DW appliance hardware partners--including HP, IBM, and EMC/Dell--must be concerned that Oracle will now shift focus away from their respective appliance products in favor of those it builds with its own Sun hardware group.

And finally, Oracle’s acquisition of Sun--and possible future development of a MySQL DW appliance--may discourage customers from considering third-party DW appliances, such as from Kickfire--that build on MySQL. If that happens, and a market for non-Oracle-branded MySQL DW appliances never takes root, Oracle will be denying its MySQL customers the choice that Oracle Database customers already enjoy. Currently, Oracle Optimzed Warehouse customers can deploy that enterprise DBMS as a DW on their choice of Sun, HP, IBM, and EMC/Dell platforms.

Let’s hope that Oracle makes the most of its pending Sun acquisition. Ellison either misspoke last fall, or was speaking prophecy. Like most DW vendors, Oracle’s destiny is to grow ever more hardware-dependent for its long-term scalability, performance, and optimization story.