Posted by Rob Karel on September 28, 2007
By Rob Karel.
Quality data in the enterprise is like breathable air — you don't truly appreciate it until it's gone. Many companies don't even bother to ask whether the customer, product, asset, or any other data it captures is actually complete, valid, and useful. Other companies leave the responsibility of standardizing, cleansing, and aggregating data from source systems to their IT developers, perhaps leveraging transformation capabilities within extract, transform, and load (ETL) tools to automate this hygiene process.
Then there are those companies that have felt enough data quality-induced pain such as wasted marketing costs or low call center productivity, and have invested in data quality software that allows for the advanced definition and maintenance of rules to standardize, cleanse, enrich, match, and merge. Once an investment in data quality software is made, companies hopefully have invested also in staffing at least a handful of data quality stewards or business analysts. These data quality professionals (DQPs) can translate requirements and perspectives of quality from the business stakeholders to technical requirements that can be implemented within the DQ software.
Unfortunately, these DQPs are usually treated like the red-headed stepchild — they are expected to do their jobs with minimal executive support, funding, strategic priority, or recognition. It is assumed that because an investment in data quality software was made, there should no longer be a data quality issue. This is of course absurd: The technology is only as good as the data governance and business ownership of the data rules and definitions. If the business does not actively participate and take responsibility for the quality of data, the technology will be nothing more than yet another IT expense.
These underappreciated DQPs often spend a large percentage of their time trying to convince management to let them do what they were hired to do — improve the quality of data — but can't move the quality needle far enough through their own grassroots efforts. That is why it's ironic and sad that often a DQPs best friend may just be a data quality crisis. When senior executives learn that they have failed a compliance audit, a customer is suing for violation of their privacy preferences, or they have made a very bad strategic business decision based on faulty data, the DQP is suddenly very popular and subsequently receives plenty of priority, funding, and support.
Thankfully there are many organizations today that do put significant investment and priority in data quality, especially those in highly regulated industries like financial services and healthcare. But for the rest of the pack, data quality is still just a nice to have. And to the data quality professionals working for these laggard organizations: Keep up the good work, your time will come!