I’ve seen many Business Intelligence (BI) projects fail because of data quality issues, but are data quality issues really a BI foe?
It’s true that BI shouldn’t be pushed to a broader user base before its data is clean and reliable. Don’t do it! First impression is critical. If data isn’t reliable, users will likely lose trust and the odds of a failed project go through the roof!
With the right approach, data quality issues (or dirty data, as some call it) are actually BI’s friend.
Odds are that you have bad data in your systems. But how do you go about cleaning it? Should you only invest in business intelligence after you clean your data (trash-in trash-out)? Would a business intelligence solution be of any value before you clean the data? Would BI be of value for discovering what data issues you have, which ones are important to fix, and in what priority? These are some of the questions that many organizations fail to analyze and act on properly.
Shelve Business Intelligence Until You Clean Your Data?
Take this example from one of my clients shipping products internationally. The ShipTo address to domestic clients had lots of issues, such as the country fields populated with US, U.S., USA or blank. For their shipping processes, all these address variations worked well. Once management saw the data via the BI solution, they quickly asked IT to fix ShipTo.
That sounded like a good idea, at first. However, they hadn’t looked at their ShipTo data issues closely or discussed it. Management said, “Fix ShipTo,” and that’s what IT did. IT went through a complex and long project to guarantee that each captured address was validated against a USPS database. Later, they realized that management only needed accuracy at the Country and State levels for their regional analysis. If they had known that, they could have fixed the issue in a couple of weeks, rather than the 10 months and lots of expense.
Focus BI on Your Good Data
When the management of the company above resumed their BI use 10 months later, they realized they had plenty of valuable and reliable data all along. There were lots of KPIs to analyze their customer, reps, and inventory trends that didn’t depend at all on their bad ShipTo data. On top of that, through the BI they could realize other data capturing issues (such as the incomplete and lack of some customer and product categorizations) that were more critical than the ShipTo.
Bottom line, had they started using BI early on, they would have reached their BI project ROI about 10 months earlier!
I had another client where the CFO demanded that his business intelligence solution be deployed as is—no testing, no custom work. He said, “In my last BI project, the business intelligence team spent about 3/4 of their setup time in the data cleaning-up process, while managers were flying blind. This time, I want to see my data right away, no matter how bad it is. I want to decide if, when, and what needs to be cleaned.” At first, I tended to push him back since he’s not an IT savvy person. But I loved his attitude and enthusiasm to be part of the data QA process and it turned out to be great! Two days after the system was installed, the CFO found a couple of gems in his operation trends by slicing and dicing the data. The BI project paid for itself even before any data cleaning took place.
This CFO’s approach was so successful that I decided to recommend it to all my new clients: install the BI, train decision makers who’re championing the project, and then let them drive what’s next.
Business Intelligence to Understand Data Quality
I’ve seen a consistent trend over time. Companies that adopted BI as a data quality tool and involved management early on tend to get the best from their data. The main reason is because management can really understand the impact of the good and bad data, and set the right priorities. Sometimes, critical issues are in unforeseen processes that haven’t been implemented yet. Overall, without management direction, IT teams tend to under or overshoot efforts to clean the data, and that results in frustrations, extra costs, and many BI project failures.
It’s not only about the bad data, but about understanding where the business processes and systems are, and what needs improvement. With the right approach, bad data is a friend of BI because it brings management to observe, understand, and drive IT decisions.
After all, BI is all about supporting the decision-making process.
