Instrumenting Your Enterprise for Maximum Predictive Power

By James Kobielus

Business is all about placing bets and knowing if the odds are in your favor.

As I noted in my most recent Forrester report, business success depends on your company being able to visualize likely futures and take appropriate actions as soon as possible. You must be able to predict future scenarios well enough to prepare plans and deploy resources so that you can seize opportunities, neutralize threats, and mitigate risks.

Clearly, predictive analytics can play a pivotal role in the day-to-day operation of your business. It can help you focus strategy and continually tweak plans based on actual performance and likely future scenarios. And, as I noted in a recent Forrester blog post, the technology can sit at the core of your service-oriented architecture (SOA) strategy as you embed predictive logic deeply into data warehouses, business process management platforms, complex event processing streams, and operational applications.

The grand promise of predictive analytics—still largely unrealized in most companies—is that it will become ubiquitous, guiding all decisions, transactions, and applications. For the technology to rise to that challenge, organizations must move toward a comprehensive advanced analytics strategy that integrates data mining, content analytics, and in-database analytics. Already, we’ve sketched out a vision of “Service-Oriented Analytics,” under which you break down silos among data mining and content analytics initiatives and leverage these pooled resources across all business processes.

You may agree that this is the right vision but have doubt about whether there is a practical, incremental roadmap for taking your company in that direction.  In fact there is, and it starts with re-assessing the core of most companies’ predictive analytics capability: your data mining tools. As you plan your predictive analytics initiatives, you should avoid the traditional approach of focusing on tactical, bottom-up project-specific requirements. You should also try not to shoehorn your requirements into the limited feature set of whatever modeling tool you currently happen to use.

To become a fully predictive enterprise, you will need to take both a top-down and bottom-up approach to your data mining initiatives. From the top-down, it’s all about building and integrating alternate models of how your business environment is likely to evolve internally and externally. In our recent report on advanced analytics, Boris Evelson, Leslie Owens, and I sketched out the many business processes that can be enriched by predictive analytics.

So how do you instrument your company to become more predictive? For starters, assess whether your analytics tools support the following capabilities for developing, validating, and deploying predictive models:

• Model multiple business scenarios: You should be able to build complex models of multiple, linked business scenarios across different business, process, and subject-area domains, using such key features as strategy maps, ensemble modeling , and champion-challenger modeling.
• Incorporate multiple information types into models: You should be able to develop models against multiple information types, including unstructured content and real-time event streams, while leveraging state-of-the-art algorithm in sentiment analysis and social network analysis.
• Leverage multiple statistical algorithms and approaches in models: You should be able to develop models using the widest, most sophisticated range of statistical and mathematical algorithms and approaches, including regression, constraint-based optimization, neural networks, genetic algorithms, and support vector machines.
• Apply multiple metrics of model quality and fitness: You should be able to score and validate model quality using multiple metrics and approaches, including quality scores, lift charts, goodness-of-fit charts, comparative model evaluation, and auto best-model selection.
• Employ multiple variable discovery and assessment approaches: You should be able to build and validate models using various approaches for variable discovery, profiling, and selection, including decision trees, feature selection, clustering, association rules, affinity analysis, and outlier analysis.

How is this different from predictive analytics as usual? Traditionally, most predictive modeling specialists focus on the latter three capabilities: statistical algorithms and approaches, model quality and fitness, and variable discovery assessment. Most models are built in narrowly scoped business or subject domains—such as customer analytics for marketing campaign management—and only against structured data sources (such as relational tables). Traditionally, few predictive analytics projects have entailed modeling of multiple business scenarios across diverse domains--such as sales, marketing, customer service, manufacturing, and supply chain-- though in the real world these business processes are often quite interconnected. Also, many data mining initiatives fail to incorporate information from unstructured sources—such as text in call-center logs—though this content may be as important as what comes relational databases and other structured sources.

It’s very important to build multi-scenario predictive models against complex information sets, but becoming a fully predictive enterprise demands much more. To instrument your organization for maximum predictive power, you should also tool your advanced analytics to support the following capabilities:

·         DW-integrated data preparation: To speed up and standardize the most time-consuming predictive modeling project tasks, you should be able to leverage your existing data warehouse, extract transform load, data quality, and metadata tools to support a full range of data preparation features. These features include the ability to discover, acquire, capture, profile, sample, collect, collate, aggregate, deduplicate, transform, correct, augment, and load analytical data sets.

·         Deep application and middleware integration: To deliver models deeply into whatever heterogeneous SOA-enabled platform you happen to use, your predictive analytics tool should deploy on and/or integrate with a wide range of enterprise applications, middleware, operating platforms, and hardware substrate. You should be able to deploy models seamlessly into your data warehouse, business intelligence, online analytical processing, data integration, complex event processing, data quality, master data management, and business process management environments. And to play well in your SOA, your predictive modeling tools should support application programming interfaces, languages, tools, and approaches such as Web services, Java, C++, and Visual Studio, as well as emerging languages such as SQL-MapReduce and R.

·         Consistent cross-domain model governance: To avoid fostering an unmanageable glut of myriad models, your predictive analytics solution should support a wide range of tools, features, and interfaces to support life-cycle governance of models created in diverse tools. At the very least, your tools should enable model check in/check-out, change tracking, version control, and collaborative development and validation of models. To realize this promise, it should support a full range of tools, standards, and interfaces for import and embedding of models from other tools, as well as export and sharing of models to other environments.

·         Flexible model deployment: To execute modeling functions--such as data preparation, regression, and scoring—on the widest range of data warehouses and other platforms, your tools should support in-database or embedded analytics. And to scale to the max, your predictive analytics tools should deploy models to massively parallel data warehouses, software-as-a-service environments, and cloud computing fabrics.  Your advanced analytics tools should also support development of application logic in open frameworks—such as MapReduce and Hadoop—to enable convergence of data mining and content analytics in the cloud.

·         Rich interactive visualization: To deliver their precious payload—actionable intelligence—your advanced analytics tools should support interactive visualization of models, data, and results. Ideally, you should be able to visualize all of this in your preferred business intelligence tool, or in the predictive modeling vendor’s integrated visualization layer. Of course, you have every right to expect the full range of visualization techniques--histograms, box plots, heat maps, etc.—regardless of who provides the visualization layer.

As you can see, this goes well beyond data mining as usual. Forrester has a slightly different perspective on the development of the predictive analytics market than you’re likely to get from other sources. We see a robust, flexible, SOA-enabled data mining tools as the centerpiece of advanced analytics for fully predictive enterprises. The competitive stakes are too great for businesses to take the traditional silo-mired approach when implementing this mission-critical technology.

What do you think?

Service-Oriented Analytics: Tapping into the Predictive Smarts of Your Entire Organization

By James Kobielus

Advanced analytics is a key competitive weapon of companies everywhere. Visionary organizations are those that take a future-facing, analysis-driven perspective on new challenges. They do this by grounding management forecasts in solid historical information sets, leveraging and extending companies’ existing investments in data mining and predictive modeling.

To become truly future-focused, organizations must build out their predictive muscles through deepening commitment to these and other advanced analytics technologies, which also include interactive visualization, multivariate statistical analysis, text analytics, and massively parallel enterprise data warehousing. However, enterprises must be careful not to adhere to the common practice of implementing advanced analytics tools in tactical, application-specific silos. One of the downsides of this traditional practice is that diverse predictive modeling teams can find it difficult to share their deep domain expertise, best statistical approaches, and most powerful data exploration and visualization features.

How can companies realize the transformative potential of predictive analytics for business success? For starters, you must get rid of siloes that fragment your data mining initiatives into separate camps. You must also build a bridge between your data mining operations and the teams that manage your text analytics, business intelligence, complex event processing, and business process management efforts. And the key approach for silo-smashing is service-oriented architecture (SOA).

At first glance, SOA may seem like a foreign topic to many analytics professionals, but it shouldn’t be. In the broadest perspective, SOA refers to best practices for encouraging greater reuse, sharing, and cross-platform interoperability among key business resources. Typically, one associates SOA with reuse of one specific type of resource: application functionality that is distributed across heterogeneous, networked platforms. Nevertheless, key SOA principles—such as standards-based service virtualization, reuse, brokering, and governance—are as applicable to predictive models as to any other resource that lives online.

Predictive models empower your product managers, marketing specialists, risk managers, process analysts, senior executives, and other personnel with access to sophisticated forecasting, time-series analysis, and scenario-testing tools. Each model is a statistical encapsulation of your business’ current view of the future in some specific application, subject, or decision-support area. Incorporating the expertise of subject-matter experts, these models allow organizations to gauge the potential impact of future projects, campaigns, and other initiatives, and also to adjust execution of these initiatives in mid-stream.

Predictive analytics need not be a purely blue-sky planning tool. This technology can sit at the core of your SOA strategy, and leading-edge enterprises are in fact doing that. Companies in such verticals as financial and telecommunications are embedding predictive logic deeply into data warehouses, business process management (BPM) platforms, complex event processing (CEP) streams, and operational applications.

What will it take for your company to fully align your advanced analytics efforts with your SOA strategy? Most important, Service-Oriented Analytics requires an executive-level commitment to becoming a predictive enterprise on all levels. From the perspective of your existing predictive modeling teams, this will require an ongoing focus on collaboration across business, function, and subject domains. You should create a culture and offer incentives that encourage modeling professionals to reuse each other’s expertise on problems that cross multiple domains.

Reuse is everything. Here are some high-level guidelines for establishing a reuse-friendly Service-Oriented Analytics practice in your company:

• Reuse modeling best practices:  Start by consulting Forrester’s report on setting up a Business Intelligence Solution Center (BISC), which we defined as “an institutional steward, protector, and forum for BI best practices.” Given that predictive analytics is a key segment of BI, you will find it necessary to incorporate this technology into your BISC’s scope.
• Reuse modelers: Cultivate a professional cadre of predictive modeling experts who are more than just wizards in advanced statistics and mathematics. Encourage subject-matter experts in all business areas to undertake predictive modeling projects and to team with modeling experts in other projects, applications, and business units. Provide incentives for modelers to regularly move between business units and subject domains, thereby spreading their expertise throughout the enterprise.
• Reuse models: You should begin to investigate predictive-model governance tools, which support version control, check-in/check-out, and other controls over models created in and imported from diverse tools. Model governance tools—from vendors such as SAS, SPSS, and KXEN--facilitate reuse, consolidation, combination, and cross-synthesis among disparate models. And you should also investigate options for embedding predictive models within BI, BPM, and other operational applications, thereby leveraging your growing analytic asset into new deployments.

As you deploy predictive models into operational applications, you should provide other applications with SOA-based access to them through Web services, Web 2.0, and other standardized interfaces. In that way, you will be creating a critical bridge to your application development teams and be creating a thoroughly Service-Oriented Analytics environment.

Your first steps down this road are clear. Forrester will be happy to assist you in developing a detailed enterprise roadmap for Service-Oriented Analytics. In so doing, you’ll be better able to tap into the future-facing analytic expertise that exists throughout your business.

Is BAM Relevant In The Age of Lean Processes?

By James Kobielus

In any dynamic business environment, the last thing you want is to indulge in navel-gazing. If you’re finely attuned to monitoring internal processes, but are not adept at responding to breaking events or anticipating the future, you will find yourself marginalized in the new global economy.

In a turbulent world, the most successful companies are those that stay lean--in other words, trimmed of excess overhead, fit for new challenges, and flexible enough to evolve rapidly. Forrester advises customers to make their business processes leaner, through such measures as reducing the number of steps, routing paths, participants, decision points, milestones, and revisions. As you do, you will find concomitant improvements in speed, cost, quality, and customer satisfaction.

Clearly, you can’t do that well if you don’t have, at the very least, the means to monitor a wide range of business processes in unified dashboards, reports, and other applications. That’s where traditional business activity monitoring (BAM) tools, which are a key component of most business process management (BPM) environments, play a critical role. However, critical as BAM may be, it is often deployed in a predominantly navel-gazing process-tracking capacity (remember what that “M” stands for) and, as such, is as much part of the problem as the solution.

Why? Well, for starters, most BAM environments are deployed in conjunction with specific vendors’ BPM platforms--hence can only track the subset of business processes that run on those platforms. But, even more limiting, BAM dashboards are often the stand-alone province of process administrators. Only in a handful of  BAM-using organizations do alerts and metrics escalate beyond the process administrator level and trigger the necessary human and automated workflows necessary to respond to breaking issues--or nip them in the bud before they can become issues.

To stay relevant, BAM needs to evolve in the age of lean processes so that it can become a tool for business agility at all levels. Next week, Forrester BPM analyst Clay Richardson and I will do a podcast on this very topic. In  that podcast, and in a joint presentation that Natalie Petouhoff and I will do at Forrester Business and Technology Forum in Chicago in October, I’ll sketch out a vision for the next-generation of BAM (which I half-jokingly call “uber-BAM”).

Here is a rough sketch of the maturity levels of BAM's evolution to realize this vision:

• BAM Level 1: Business activity monitoring: This is today’s BAM baseline. Essentially, it combines BPM with business intelligence (BI), interactive dashboarding, and complex event processing to deliver process metrics to consoles used solely by human process administrators.
• BAM Level 2: Business action mobilization: This is the BAM level that many forward-looking users and vendors have implemented. It combines all Level 1 functionality with business rules engines (BRE) to drive sophisticated human and automated workflows in response to breaking events. It puts the “immediate action loop” into the BAM equation.
• BAM Level 3: Business adaptability management: This is the promise of truly lean uber-BAM, and is found only in a vanguard of visionary  business environments. It combines all Level 2 functionality with embedded predictive-analytics models, which are developed and maintained by business process architects and/or subject-matter process experts. Continuously mining historical and real-time metrics, these in-process analytics drive automated “recommendation engines” that adapt coordinate response to changing conditions. Recommendation engines enable human and automated decision agents to seize opportunities and neutralize threats well ahead of the curve.

Going forward, this new vision of BAM’s evolution and promise will inform Forrester’s coverage of process agility and optimization.

BI, Analytics, And CEP: Some Fruitful Potential Follow-Ons From Software AG’s Acquisition Of IDS Scheer

 by James Kobielus

Yes, of course, Software AG is buying IDS Scheer primarily for the latter’s ARIS family of business process management (BPM) tools. I’ll leave it to my Forrester colleagues who focus on BPM--on both the IT and TI sides of the house--to call out the ramifications for Software AG’s positioning in that market.

But, believe it or not, this deal will also launch Software AG into the growing markets for business intelligence (BI), analytics, and complex event processing (CEP) solutions. We bet you didn’t realize that IDS Scheer has ARIS solutions in these fast growing markets, but in fact they do--and they’re continue to evolve those offerings.

It’s no surprise that IDS Scheer’s BI, analytics, and CEP offerings supplement and extend its BPM portfolio. Its CEP solution, ARIS Process Event Monitor, supports business activity monitoring (BAM). Its analytics offerings, ARIS Process Performance Management and ARIS Performance Dashboard, support visualization, dashboarding, scorecarding, drilldown, and alerting on process key performance indicators (KPIs), both historical and real-time. And its forthcoming BI offering, ARIS MashZone, will support self-service user development of reports, dashboards, and other views of process and business metrics.

IDS Scheer has little market share in these non-core segments. And the vendor is no immediate threat, by itself or under its future corporate parent, to the leaders in the BI, analytics, and CEP segments. Indeed, its forthcoming mashup-oriented BI offering only provides a subset of the features available from market leaders such as SAP Business Objects, IBM Cognos, and MicroStrategy. But the fact that Software AG will soon be able to provide its own offerings in those segments, rather than rely wholly on partners, represents an important step in its attempt to field a full service oriented architecture (SOA) solution stack.

As noted in a blog entry a year and a half ago, BI is the crown jewel in any comprehensive SOA solution portfolio. SOA suites cannot be considered feature-complete unless they incorporate a comprehensive range of BI features. This acquisition continues the ongoing SOA solution build-out strategy that motivated Software AG to acquire webMethods in 2007.

But it’s not clear yet whether Software AG plans to flesh out its BI, analytics, and CEP strategies going forward and thereby confront SAP, Oracle, IBM, Microsoft, and other SOA full-stack vendors head-on in these segments. It is also unclear how much effort or expense Software AG would incur in extricating the IDS Scheer offerings from the larger ARIS portfolio in order to make them more general-purpose and less BPM-centric. Nevertheless, Software AG will at the very least have a strong set of enabling technologies to support any such strategy in the near future.

What’s most exciting, and potentially differentiating, about the Software AG/IDS Scheer BI portfolio is the combination of CEP with mashup and an in-memory architecture to support truly real-time, interactive analytics. In other words, Software AG/IDS Scheer could take a page out of the book of another SOA full-stack vendor: TIBCO and its Spotfire product group. In doing so, Software AG/IDS Scheer would also be well-positioned to duke it out with SAP, IBM, Microsoft, and Oracle, all of which are beginning to emphasize in-memory CEP-enabled BI strategies. As we noted in a report from late 2008, in-memory architectures are coming to dominate the BI arena. Likewise, Forrester has called attention in to the growing adoption of CEP for truly real-time BI.

Whether Software AG capitalizes on the opportunity to expand its SOA solution stack into BI remains to be seen. Considering that it took Oracle more than a year to publicly declare how it will position BEA’s CEP and data federation technologies within its own SOA stack, we may have to wait a while before Software AG and IDS Scheer craft an equivalent roadmap--if they ever do.

But if they wait too long, the newly merging vendors may find that the dynamic SOA, BI, and CEP markets have passed them by.

After So Many Years Of Ballyhoo, Semantic Web Still Searching For Killer App

By James Kobielus

Cynics might call Semantic Web a technology looking for a solution. And they might have a point.

Semantic Web refers to a long-running World Wide Web Consortium (W3C) initiative that is working toward an ambitious--some might say hopelessly Utopian--goal. At heart, it is a vision for how the World Wide Web should evolve to realize its full interoperability potential.

People vary widely in how they interpret the scope of the Semantic Web initiative. The tech industries are swarming with a wide range of projects, products, and tools that implement different variants of this vision. What vision is that? In the broadest sense, Semantic Web refers to an all-encompassing metadata, description, and policy layer that can, potentially, support universal, automatic, comprehensive end-to-end interoperability across every macro or micro entity—including data, components, services, applications, and services—on every conceivable level.

Whew!!! If that’s not the working definition of “pie in the sky” or “boil the ocean” (pick your metaphor), I don’t know what is. In fact, I’m hard-pressed to refer to Semantic Web as a definable market or solution segment. However, it’s not entirely vacuous.

For starters, organizations can implement W3C-developed semantic description standards—such as Resource Description Framework (RDF) and Web Ontology Language (OWL)--to make the meaning of content unambiguously comprehensible to services, applications, bots, and other automated components. Second, there is a reasonably robust market for “ontology” tools to support RDF/OWL-based modeling of application semantics. Finally, there is some incremental adoption of these tools and concepts in established IT segments, such as:

• Enterprise content management (ECM): Semantic approaches can support more powerful discovery, indexing, search, classification, commentary, and navigation across heterogeneous stores of unstructured and semi-structured content. Semantic search—driven by concepts, not mere text strings--is regarded by some as a primary Semantic Web application. Indeed, many Semantic Web vendors are primarily implementing the technology in search engines that leverage ontology-based concepts to improve search accuracy and reduce spurious hits.

• Enterprise information integration (EII):Semantic approaches enable consolidated viewing, query, and update of structured data that has been retrieved from diverse sources. Indeed, most commercial EII environments present an abstract semantic layer that mediates access to heterogeneous data, such as enterprise resource planning and customer relationship management applications, converging it all to a common presentation-side schema. A handful of those EII vendors have begun to support Semantic Web standards, primarily through third-party software plug-ins

• Enterprise service bus (ESB):Semantic approaches can facilitate multilayered application, process, and service interoperability across disparate environments. To date, there has been little production implementation of Semantic Web standards in the ESB arena, though some vendors have adopted semantics, ontologies, and RDF to describe the conceptual models implemented by application endpoints, agents, and intermediary nodes within ESB-like middleware approaches such as event stream processing.

But Semantic Web approaches are still on the periphery of these markets. 10+ years into its inception, Semantic Web still has no clear killer app. It’s not clear if or when that app will emerge.

Meet One-On-One With Forrester Analysts At Our Business & Technology Leadership Forum 2008

Consistently rated as one of the most popular features of Forrester Events, one-on-one meetings give you the opportunity to discuss the unique technology issues facing your organization with Forrester analysts. Business & Technology Leadership Forum attendees may schedule up to two 20-minute one-on-one meetings with the Forrester analysts of their choice, depending on availability. Registered attendees will be able to schedule one-on-one meetings starting on Monday September 15, 2008. Book early!

• a
• b
• c
• d
• e
• f
• g
• h
• i
• j
• k
• l
• m
• n
• o
• p
• q
• r
• s
• t
• u
• v
• w
• x
• y
• z

B

William Band
Research coverage for Business Process & Applications professionals
Customer relationship management applications, customer experience management, stakeholder alignment, enterprise CRM
See all »

Matthew Brown
Research coverage for Information & Knowledge Management professionals
Marketing and advertising, enterprise portals, intranets and extranets, information and knowledge management
See all »

Peter Burris
Research coverage for Technology Product Management & Marketing professionals
Enterprise marketing platforms, marketing automation, high-tech, application development
See all »

Back to top »

C

Bobby Cameron
Research coverage for CIOs
IT governance, risk, and compliance; the marketing of IT; serving the business; security and risk
See all »

Marc Cecere
Research coverage for CIOs
Designing IT organizations, changing the culture of an IT organization, IT strategic planning
See all »

Patrick M. Connaughton
Research coverage for Business Process & Applications professionals
Supply chain management services, supply chain management applications, enterprise mobility, RFID
See all »

Alex Cullen
Research coverage for CIOs
IT organization; IT strategy, planning, and governance; organizational design and change management, IT management
See all »

Back to top »

E

Boris Evelson
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, business intelligence, OLAP, data warehousing
See all »

Back to top »

G

Chip Gliedman
Research coverage for Business Process & Applications professionals
Customer relationship management, help desk/service desk, customer service and support, packaged applications
See all »

Back to top »

H

Paul D. Hamerman
Research coverage for Business Process & Applications professionals
ERP, human capital management, financial management, business performance solutions
See all »

Brian W. Hill
Research coverage for Information & Knowledge Management professionals
eDiscovery, archiving, records and retention management, enterprise content management (ECM)
See all »

Bradford J. Holmes
Research coverage for Vendor Strategy professionals
Tech marketing tools and best practices; government, high-tech, tech marketing strategies
See all »

Back to top »

K

Rob Karel
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, integration technologies, metadata management, extract
See all »

Rob Koplowitz
Research coverage for Information & Knowledge Management professionals
Information Workplace, collaboration strategy, collaborative platforms, SharePoint
See all »

Back to top »

L

George Lawrie
Research coverage for Business Process & Applications professionals
Retail information technology; consumer goods supply chain; pricing, promotions, and revenue optimization; collaborative processes such as trade promotions management and sales; and operations planning
See all »

Sharyn Leaver
Research coverage for Business Process & Applications professionals
Packaged applications, business process management, ERP, application strategy and selection
See all »

Craig Le Clair
Research coverage for Information & Knowledge Management professionals
ECM, BPM, output management, document processing services
See all »

Back to top »

M

Pete Marston
Research coverage for Business Process & Applications professionals
Customer relationship management, sales force management, software-as-a-service, outsourcing
See all »

Kyle McNabb
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, document imaging, eForms and information capture, enterprise content management
See all »

Thomas Mendel, Ph.D.
Research coverage for Vendor Strategy professionals
Product portfolio strategies, mobile services, business service management, data center management
See all »

Connie Moore
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, business process optimization, IT organization, enterprise content management
See all »

Back to top »

O

Leslie Owens
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, taxonomy and classification, enterprise search platforms, text mining and analytics
See all »

Back to top »

P

Natalie L. Petouhoff, Ph.D.
Research coverage for Business Process & Applications professionals
Customer service and support, customer experience, customer experience management, business strategy for customer experience
See all »

Lisa Pierce
Research coverage for IT Infrastructure & Operations professionals
Voice services, telecommunications services by region, remote access infrastructure, networking
See all »

Tom Pohlmann
Research coverage for CIOs
Business models, high-tech, corporate strategy, tech sector economics, product and solutions strategies
See all »

Stephen Powers
Research coverage for Information & Knowledge Management professionals
Information and knowledge management, digital asset management, enterprise content management, Web content management
See all »

Back to top »

R

Stefan Ried, Ph.D.
Research coverage for Vendor Strategy professionals
Enterprise architecture, Service-oriented architecture, application platforms and programming strategy; application development
See all »

Back to top »

S

Ted Schadler
Research coverage for Information & Knowledge Management professionals
Real-time collaboration tools (instant messaging, presence, document sharing, etc.), cloud-based collaboration and email, mobile collaboration tools and applications, virtual worlds for the enterprise
See all »

Claire Schooley
Research coverage for Information & Knowledge Management professionals
eLearning, information and knowledge management, videoconferencing, Web conferencing, enterprise collaboration, new workforce, retiring workforce
See all »

Back to top »

T

Scott Tiazkun
Research coverage for Business Process & Applications professionals
Financial management; governance, risk, and compliance; financial management applications; security and risk
See all »

Zach Thomas
Research coverage for Business Process & Applications professionals
Human resources management applications, compensation, recruitment strategies, packaged applications
See all »

Back to top »

W

Tim Walters, Ph.D.
Research coverage for Information & Knowledge Management professionals
Web content management, enterprise content management, digital asset management, information and knowledge management
See all »

R "Ray" Wang
Research coverage for Business Process & Applications professionals
Enterprise apps and ERP, software contract negotiations, software partnerships and ecosystems, customer data integration
See all »

Doug Washburn
Research coverage for IT Infrastructure & Operations professionals
Green IT, IT organization, IT infrastructure and operations, IT management
See all »

Back to top »

Y

Gil Yehuda
Research coverage for Information & Knowledge Management professionals
Enterprise Web 2.0 and Social Computing; collaboration strategy, tools, and culture; virtual communities of practice; virtual team collaboration
See all »

Back to top »

CEP For Real-Time BI: Vendor Announcement Events Come In Threes, Apparently

By James Kobielus

As is clear from the general pattern of my Forrester I&KM blog posts, I'm closely tracking the penetration of complex event processing (CEP) into the world of real-time BI and data warehousing (DW). I'm working on a report now providing best practices for real-time DW, which will be available later in Q2. I'll follow up with a trends document on CEP for real-time BI in Q3.

One thing I've noticed is that the BI industry doesn't fully realize how deeply and quickly CEP is beginning to permeate reporting, dashboarding, analytics, and other decision-support apps. Case in point: I was interviewed yesterday by Mary Hayes Weier of InformationWeek, a reporter who definitely knows the BI market. She asked me about areas where BEA might complement Oracle's BI portfolio. I mentioned operational BI (leveraging the semantic federation/virtualization/abstraction layer in BEA AquaLogic Data Services Platform) and CEP (leveraging BEA WebLogic Event Server). She understood the operational BI side of it right away, but stopped me mid-sentence with the question, "What's CEP? I've never heard that acronym before."

Not to single out Mary — who, as I stressed, is definitely one of the best tech reporters in the BI arena — but I've also encountered this same lack of CEP familiarity among BI vendors and users. What I say to everybody is that CEP, primarily associated with real-time SOA apps such as business activity monitoring (BAM), is an enabler for truly real-time (i.e., end-to-end streaming subsecond-latency) BI — as a complement and/or alternative to traditional historical/thru-the-EDW BI. The fact that CEP familiarity is low in the BI space right now is because few of the leading BI vendors have much of a BI-integrated CEP capability — yet.

Interestingly, right after I got off the phone with Mary, I checked the day's news releases and encountered three product announcements ("events"!) from vendors who have staked their BI strategy purely on CEP, both on the middleware side and in the front-end presentation, interaction, and analytic layer. I'm speaking of these vendors/announcements (the titles of which I've edited down to avoid propagating gratuitous marketing boilerplate in this, the Forrester blog):

• Altosoft Introduces Altosoft Insight 3.0: This announcement concerns enhancements to Altosoft Insight. First, the new version 3.0 includes enhancements for continuous event-based business activity monitoring (BAM) and rule-driven workflow-exception alerting across diverse operational systems — i.e., one of CEP's traditional core markets. Second, it includes new functionality for user-driven, code-free, AJAX-enabled visual development of continually refreshable browser-based reports — i.e, for CEP-infused low-latency BI. Third, it beefs up the low-latency event-streaming middleware features of the product, with tweaks to its high-speed, 64-bit in-memory calculation engine for real-time data aggregation, transformation, and optimization, plus cache acceleration for complex operations like multi-source outer joins with intelligent calculation and compression for optimized memory management, while minimizing loading on operational event/data sources. Note that Altosoft can also do dashboarding, reporting, and analytics on data retrieved from third-party DWs, and join/analyze this data alongside continuously refreshed-via-CEP operational BI data from many sources.

• Truviso Delivers Next-Generation Analytics Solution: This announcement concerns enhancements in the new Truviso v2.5. This version adds functionality for high availability; event-based real-time integration with enterprise data sources; event-driven integration with user interfaces (e.g, browsers, portals); and interactive visualization of continuously refreshed CEP/BI analytics. It also supports development of models for customized event-based data flows, including stream-staged data-integration and changed data propagation/visibility. Clearly, changed data capture and low-latency end-to-end propagation is a middleware area where CEP/BI overlaps with best practices for near-real-time/"active" DW.

• TIBCO Spotfire Delivers Visual, Interactive Business Mashups: This announcement concerns Spotfire 2.1, the latest set of user-driven development enhancements to the browser-oriented CEP/BI solution family that TIBCO acquired almost a year ago. This version continues to enhance Spotfire's impressive capabilities for interactive visualization of myriad live event-data feeds in BI/analytics apps, such as reports, dashboards, trending graphs, and comparison charts, which display inside the TIBCO Spotfire Web Player. Spotfire 2.1 implements a Web 2.0 mashup-style development approach.

Of course, this is just a convenience sampling of CEP/BI product announcements on one day. There are of course other vendors going deep on CEP, for BAM and/or BI (re BI, Agent Logic, Aleri, Coral8, GemStone Systems, and Syndera also spring to mind). I will cover them in greater depth in my forthcoming report in Q3. As I noted above, the leading BI vendors are conspicuous, so far, by their absence from this hot new niche. But I suspect that, by the time I publish that report, one or more of the bigger BI vendors will have their own CEP roadmaps mapped out to a greater degree.

For I&KM professionals in the here and now, though, if you want truly real-time BI, you should seriously explore the solutions from these and other CEP/BI pioneers. You should consider these offerings either as an alternative to real-time DW, or as an adjunct to the traditional historical/static EDW for real-time operational BI.

In my upcoming real-time DW report, I'll provide pointers for identifying deployment scenarios for going with real-time DW vs. CEP for real-time/operational BI. The distinctions between these approaches are rapidly blurring. I wouldn't be surprised to see the established EDW vendors add CEP-like deterministic-subsecond-latency features to their offerings.

Oh... I pointed Mary to these CEP/BI vendor announcements. Whether she follows up by speaking with them is up to her. But three such announcements in rapid succession on a single morning tends to catch my attention.

Important, but not earth-shattering, events in a steady and growing stream.

Jim

Competitive Business Intelligence, Harnessed Through Collaboration And CEP, Harvested Across The Cloud

By James Kobielus

Sometimes ideas for blog posts flow out of everyday conversations with colleagues. I want to thank Leslie Owens and Matt Brown for stimulating the following thought train.

The external competitive environment is the cloud where opportunities and threats hang, sometimes latent, sometimes looming. So it only makes sense that enterprises will outsource more of the competitive surveillance to the cloud of external resources, such as analyst firms, third-party market intelligence subscription feeds, social networking, Web 2.0, etc.

Of course, enterprises realize they don't dare outsource the competitive intelligence function entirely. That explains why they maintain research staff, tools, portals, and informational resources in-house. Mostly, these competitive intelligence teams monitor the prevailing market conditions that impact on their companies' core businesses. But, to a great degree, they also serve as an early-warning system helping their organization respond to specific breaking events -- i.e., the "disrupters" -- that threaten to capsize the corporate boat.

Recognizing this perennial "disrupter pre-emption" requirement, enterprises are concerned with best practices for setting up event-driven competitive intelligence operations. These best practices should help them survey the external horizon more comprehensively and proactively. Best practices should also help them foster, harness, and harvest internal collaboration among competitive-intelligence subject matter experts.

Essentially, competitive intelligence operations of this sort practice CEP in the following senses that I described in a previous post:

• Each event may be quite complex in its own right, standing for a linked set of data updates, application state transitions, and process status changes (NEW NOTE: a "disrupter" is any extremely complex, perhaps way too vague, but still undeniably important "event" -- disrupters aren't "tagged" as such -- and they may not be easily identified before, during, or after the fact -- maybe, hey Leslie, on-the-fly social tagging is the best way to approach this squishiness).
• Each event-reliant decision agent (e.g., end user) may access, interact with, and/or consume events through a complex interface (dashboards, analytics, semantic layer, etc.), across multiple devices (desktop, laptop, Blackberry, etc.) and have a complex event-enriched streaming  "experience" (NEW NOTE: competitive intelligence groups make use of the full range of portals, e-mail, IM, search engines, social networking, wikis, blogs, podcasts, workflow, alerts/notifications, etc.).
• Each event-reliant decision agent may be a complex creature in its own right with its own complex, convoluted, squishy decision-making methodology -- i.e., an individual human being with their own habits and cognitive/psychological dispositions; or a group making decisions collectively and collaboratively through workflow, or social networking; or a half-human/half-automated workflow behaving in the herky-jerky manner one would expect from a split-personality decision agent; or a completely automated orchestration of applications triggered by rules engines, etc.) (NEW NOTE: competitive intelligence teams are very human teaming environments, at heart -- everybody is a "sentinel" on the "lookout" for critical events while others are "sleeping" or attending to something else).

Tying in another observation from that earlier post, I expect that CEP for I&KM (i.e., real-time, event-triggered, under-deadline, continuously-refreshing dashboard-sharing collaboration applications) will play a key role in event-driven competitive intelligence everywhere. CEP, hence BI, will be used to beef up organizations' in-house competitive intelligence/surveillance function, supplementing (hopefully not replacing) the outsourced competitive intelligence/surveillance they get from analyst firms such as Forrester.

In such an environment, the self-service, in-house research portal's the chief presentation layer, and BI operates as an intelligence source and/or target accessible via the portal, with real-time/near-real-time data integration approaches (e.g., ESP, CDC, MOM, trickle-feed ETL, etc.) providing the low-latency plumbing to deliver those feeds to the DW/BI/portal/preso front-end.

The BI vendors are laying the foundation for this emerging best practice. If you look at what vendors such as Business Objects are doing, they're making more external, commercial, competitive intelligence feeds accessible, via partnerships with content aggregators/publishers, from their platforms (e.g., http://www.businessobjects.com/news/press_release.asp?id=20070521_006524). They're also providing text mining/analytics-integrated tools (e.g., http://www.businessobjects.com/news/press_release.asp?id=20070924_006494) for searching across internal and external, unstructured/semi-structured data sources. And they're expanding the social networking and other collaborative features, and mashup offerings, for bringing together real-time feeds of internal/external data/events (e.g., http://www.businessobjects.com/news/press_release.asp?id=20080313_00001).

Business Objects is a bit ahead of the industry curve on all these things. But it's clear that, as market leader, they've laid down the chief challenge for all BI vendors, to make their offerings more pervasive in competitive intelligence use cases, and also to harvest the informational resources of the Web 2.0 cloud to the max.

Complex Decisions Driven, But Not Overtaken, By Events

By James Kobielus

As I may have mentioned before, I cover complex event processing (CEP) as it intersects with information and knowledge management (I&KM). Or, more specifically, as it supports real-time business intelligence (BI). Or, perhaps more pedantically, as it enables decision support systems (DSS) to facilitate business agility in response to dynamic conditions.

DSS is the term by which BI was known from the late 70s until the early 90s (fun fact: my first position as an IT industry analyst was in the late 80s, under Peter G.W. Keen, who, along with Michael Scott Morton, published a seminal work on DSS in 1978). DSS is a useful framework for thinking about BI, because it places your focus on the decision "agent," on the flow of that agent's decision-making process, and on the collection of static and dynamic evidence and events that fed that process.

In looking at CEP for I&KM/BI, I'm often compelled to take a DSS point of view. As an analyst, that perspective helps me model the phenomena in the proper context -- i.e.., the many layers of complexity:

• Each event may be quite complex in its own right, standing for a linked set of data updates, application state transitions, and process status changes.
• Each event stream is a complex, linked, ordered, transactional sequence of time-stamped      objects.
• Each event-processing engine may consolidate, cleanse, correlate, filter, and persist objects across a complex set of concurrent event streams.
• Each event-reliant decision agent (e.g., end user) may access, interact with, and/or consume events through a complex interface (dashboards, analytics, semantic layer, etc.), across multiple devices (desktop, laptop, Blackberry, etc.) and have a complex event-enriched streaming "experience" (see previous post authored by myself).
• Each event-reliant decision agent may be a complex creature in its own right with its own complex, convoluted, squishy decision-making methodology -- i.e., an individual human being with their own habits and cognitive/psychological dispositions; or a group making decisions collectively and collaboratively through workflow, or social networking; or a half-human/half-automated workflow behaving in the herky-jerky manner one would expect from a split-personality decision agent; or a completely automated orchestration of applications triggered by rules engines, etc.).

In fact, the organization/team/group/collective is the principal uber-decision agent in most businesses. The BI market is beginning to understand that basic fact, and to address it in their solutions. As you all may recall from a previous post of mine, BI solutions are growing more collaboration-oriented, in terms of vendors adding more workflow, instant messaging, social networking, and Web 2.0 functionality to their offerings.

As CEP increasingly permeates vendors' real-time BI/DW portfolios, I fully expect that real-time, event-triggered, under-deadline, continuously refreshing dashboard-sharing collaboration applications will come to the fore. Maybe we'll rely on social networking to make it happen on the fly (per Rob Koplowitz' great coverage of all this). Maybe we'll lean on our online avatars (per Erica Driver’s work in this area) to model various collective responses before we go "live" with it all.

Clearly, collaboration adds another level of complexity to an organization's ability to respond to events. Ask any news organization trying to coordinate the real-time efforts of diverse reporters, editors, photographers, and others on a breaking story. Or any analyst organization trying to whip together a collaborative blogpost laying out their collective position on some fresh industry development. Under deadline, events must be absorbed, responsibilities divvied up, and actions taken -- and pronto, by whoever's holding the baton. Decisions must be made under time- and resource-stress.

Where real-time event-triggered DSS is concerned, collaboration could be a showstopper, if we don't watch out. Collaboration adds complexity, hence the potential for more decision latency. But collaboration also builds buy-in, and, as such, can be worth the muss and fuss of versions, reviews, revisions, and provisional decisions.

Indeed, having your organization fully behind whatever action was ultimately taken is the best support system any decision could possibly have.

Complex Events, Simple Experiences

By James Kobielus

Before the week is out, a few more thoughts inspired by various things I experienced at and around Forrester's Enterprise Architecture (EA) Forum last week in beautiful (but surprisingly chilly) Coronado, California.

What stuck with me was the presentation and demo by Kevin Lynch, CTO of Adobe. Though I was already quite familiar with Adobe’s Flex technology for rich Internet applications (RIA) technology, I liked the fact that Lynch presented Flex this time around as a business tool--in other words, as an interactive visualization technology for business intelligence (BI), business performance optimization, and event-rich analytics. All of which made perfect sense in a forum for enterprise architects.

As I was watching the demo, I began to think that RIA can be a double-edge sword, where BI is concerned, in terms of RIA’s potential to deliver meaningful, actionable intelligence for decision support.

On the one hand, as Lynch demonstrated, BI application developers can easily--if they don’t restrain themselves--craft any arbitrarily complex configuration of dashboards, scorecards, graphs, reports, query boxes, tickers, widgets, pop-up windows, and other visual (and audio) UI elements to deliver every imaginable form of intelligence to the user through every device or application at their disposal. And, in fact, as more BI environments begin to deliver more real-time event-driven refreshes direct-from-the-source, the more the potential for overburdening the poor end user’s overfull cranium. Where CEP for I&KM is concerned, this can easily degenerate into a nightmare scenario of information overload.

Or the BI/CEP application developers can deliver simplicity. They can use back-end event engines, business rules engines, and other infrastructure to conceal distracting complexities. Earlier that week, I was meeting with a vendor that provides real-time DW, plus in-database analytics, plus rules engines that can deliver, triggered by streaming data, simple recommendations through an Information Worker’s BI environment. For example, the DW/CEP/BRE/analytics environment might provide account reps with crisp, time-sensitive, context-sensitive recommendations to respond to an irate customer while they’re still on the phone--to keep them from bolting to the competition.

All of which brings me to a closing thought re this particular post. The problem with the term “complex event processing” is that it seems to imply a complex UI--hence, that CEP's potential user base is limited to rocket scientists, Wall Street quants, IT industry analysts (gasp!), and other folks who are professionally obliged to handle (indeed, embrace) complexity.

But complexity is part of the problem, not the solution, where CEP for I&KM is concerned. We can’t stop the world of business from growing more multifaceted. But we can and should filter it all down to the simplest, most meaningful, most actionable experience, targeted specifically to each user, and contextualized precisely to each and every occasion.

Down to a pure and beautiful crystallization.