Big Data Ain't Worth Diddly Without Big Process

Nowadays, there are two topics that I’m very passionate about. The first is the fact that spring is finally here and it’s time to dust off my clubs to take in my few first few rounds of golf. The second topic that I’m currently passionate about is the research I’ve been doing around the connection between big data and big process.

While most enterprise architects are familiar with the promise — and, unfortunately, the hype — of big data, very few are familiar with the newer concept of “big process.” Forrester first coined this term back in August of 2011 to describe the shift we see in organizations moving from siloed approaches to BPM and process improvement to more holistic approaches that stitch all the pieces together to drive business transformation.

Our working definition for big process is:

“Methods and techniques that provide a more holistic approach to process improvement and process transformation initiatives.”

As we pushed deeper into our big process research, we found that the relationship between big data and big process is crucial to driving real business value and improved business outcomes. Specifically, we found that the connection between big data and big process revolved around the “Four Cs” of:

  • Customers. You’d have to be hiding under a rock not to see that improving the customer experience is a top priority for business and IT executives. However, in order to improve the customer experience, business and IT leaders must first mine oceans of operational data to pinpoint exactly what must be done to improve the customer experience and underlying business processes.
  • Chaos. Enterprise architects and business process professionals need to come to terms with the fact that today’s business processes are not as neat, tidy, and structured as we’d been led to believe in the 20th century. Today’s business processes are chaotic and require an  understanding of the relationships between process and data in order to be able to drive better business outcomes.
  • Context. This term is quickly becoming the most overused buzzword within IT and business circles. However overused or misused, enterprise architects and business process pros must begin to build business and technology architectures that glean and intuit deeper meaning across the relationship of business events, operational data, and operational performance.
  • Cloud. In our research, we found many companies looking to push core business processes into hybrid on-premises/off-premises configurations. This means that some parts of a particular value stream might live in the cloud, while other parts of the value stream might live on internal infrastructure. While this is not a new concept, we found that most companies exploring these hybrid configurations overlooked the need to maintain data relationships across a fragmented and splintered value stream.

As companies begin to ramp up their big data initiatives, we’re finding that leading enterprise architects also see the need to bring in the big process perspective. And even for organizations that might not be focused on big data quite yet, there is still the need to begin thinking from a big process perspective to better understand the relationships and impacts between operational data and business process performance.

What do you think? How important is it to your organization to make the connection between the data that lives across different packaged applications and systems to the broader end-to-end business process? How important is it to reconcile process improvement efforts with data improvement and business intelligence initiatives?

We will explore specific strategies and emerging best practices for connecting big data and big process initiatives during my Emerging Technologies session at Forrester’s upcoming Enterprise Architecture Forums in Las Vegas and Paris. Hope to see you there!


Agreed - Big Data needs Big Process

Clay, you make great points. Big Data provides the clues that unlock the secrets of Big Customer Service (which is a process). To provide great service to customers, organizations need to rely on data to be amoeba-like in their response to issues and opportunities. This required creativity runs counter to 'organizing' and organization. In the end, being creative and flexible is a human characteristic more so than a machine, so it comes back to the non-techy solution of culture, training and empowerment of people.

Couldn't agree more that nothing is as neat as our architects and process geeks would have us believe. That's a big part of the reason BPM has struggled and Adaptive Case Management had its moment. The tug and pull of structured versus unstructured is a distractor to the problem that process defies a simple view.

Context is overused (even by me), but it is a simple word to define understanding where an idea or object 'lives'. When I think of a better word, I'll start using it.

Cloud needs new solutions that allow us to not care where the value stream is splintered. It isn't about creating any more complexity, but about hiding that complexity from the end user who doesn't need to feel the frustration and confusion of many data sources. Vendors are working on that solution as I type.

Great post. It really summarizes the BPM and architecture challenges that are the flash point for the next couple of years.

Big Data and Big Process

Some very good points Clay. I believe you really cannot take full advantage of process without the data necessary to back it up and gain insights and relationships. I especially liked this point on Chaos.

•Chaos. Enterprise architects and business process professionals need to come to terms with the fact that today’s business processes are not as neat, tidy, and structured as we’d been led to believe in the 20th century. Today’s business processes are chaotic and require an understanding of the relationships between process and data in order to be able to drive better business outcomes.

Big Data & Business Process

I'm not sure I entirely follow what "Big Process" really means, but considering the correlation between "Big Data" (I'm going to stop quoting that term now because I'm lazy and use "BD" instead) and BPM in general, I think you have some interesting observations; particularly pertaining to "Chaos" and "Context".

When looked at within any one silo, BD isn't quite so big. However, the advent of BD, at least in an enterprise form with consistent semantic meaning, crosses silos and can serve to highlight disparities in the business processes of those silos. I am beginning to see this at my own firm. Enterprise Architects are well placed to note the correlation between the two, which can be used alternatively as justification for further BD initiatives, or to enable Business Process normalization.

Like BD, the "Cloud" is the latest evolution in a particular part of the technology stack and as such, I would be hesitant to accord it the same status as your other categories - ones which describe more immutable facets of business conduct. One hopes there will always be Customers; as you point out, Chaos is endemic and Context (what the systems or operators thought they knew at the time of the business activity in question) is inherent in any business operation. The bottom line is that correlations between data and process ought to be more apparent; better yet, explicit and accessible in BD data sets.

Big Process, like Big Data is about getting small

Originally posted by Dave Duggal on Sat, 04/14/2012 - 11:17

Hi Clay,

I'm glad to see your still working on your Big Process research. I think your bullets are good outline of the scope.

Customers - ultimately business transform resources to deliver value, not every corporate process has a traditional "customer", but Big Process let's us zoom in and deliver targeted value - Goals over Procedures.

Chaos - All businesses operate in a dynamic environment, it only varies by degree. It is well over due that folks recognize 'reality' as opposed to clinging to entrenched models, which at best are shadows of the real-world. Again, Big Process notionally let's us step away from static flowcharts and look to flexibly accommodate unique demands and change in the environment so we satisfy point one - Customers.

Context - might be overused as the IT industry jumps on every term, but this is actually something few can deliver, true real-time inflight enterprise context. Marketing aside, it's a wonder that we let automation get away from us and forgot core tenets of business - standard processes, by definition support commoditization, not differentiation and value creation. Big Process let's us get small and focus on "mass customization" of user experiences.

Cloud - Great IT value, but moving middleware stacks to virtualized servers does not net you a "Cloud App" - just a brittle, hard to govern apps - the next generation of silos. It's time for folks to think different. It's time for a new app layer that supports Big Cloud Processes, not big middleware debacles.

We've got the single-layer app fabric for the cloud that knocks down traditional barriers between SOA and REST, Analytics and Transactions, Design-Time and Run-Time - it's time for process to get real.


Big Process is Where Analytics-Oriented Companies Will Live

It is terrific to see that an architectural-based agenda for big process is emerging. There has been and still is a major disconnect between business, operational users, their data and more importantly, what to do with it. This pain point is largely because of “big process” being absent.

Analytics can be simply defined is as model-based decision making. To optimize a customer’s experience, descriptive analytics can earmark the customers who are the most and least profitable, and the most loyal and least loyal. Predictive analytics can tell us what customers are at risk to leave and how to influence their behavior. Prescriptive analytics can simulate and optimize what the best outcome would be in provisioning resources to serve customers. However, such analytic models must be dynamic and aligned within the context, speed, and decision management of the business process that executes on and acts upon customer experience outcomes.

Chaos and context are at an intersection that must to be considered together in an architecture that embraces big process. Operationalizing analytics and big data as an architecture is complex because it requires meeting the needs of real-time and fixed behavior as well as static and dynamic relationships related to big process.

For example, a highly customer-focused telecommunications company recently created and operationalized a predictive customer experience index to score all of its customers to drive its relevant business processes in churn management- all the way from the sales office, to customer care, to product provisioning and pricing, to billing, and to the C-suite. Once the entire business was able to interact with actionable analytic models that were embedded in their business process, they discovered glaring problems in their decision management, performance and goals. This wasn’t surfaced by big data, reports, dashboards, cubes, and predictive model read outs. Rather, big process and analytics intelligence were needed to make big data small data for their performance results.