by Petr Hájek, Information Management Professional
This picture was randomly taken at a small local Asian marketplace. It literally (and I think unintentionally) offers a single item for the price of 3 items. This is funny when you are buying a T-shirt. It is not funny if you are spending millions. It reminds me of a colleague of mine who always says that when a company decides to build its own business data warehouse (or BI/DHW) solution, it usually succeeds no sooner than the third attempt. I agree with him and would like to share a few ideas on why this happens and how to avoid it.
1) There are no “small data warehouses”
BI/DWH solutions are always robust. This is true regardless of the size of your organization. Their architecture consists of several data layers; they are connected to many other systems. Moreover, once a BI/DWH solution is deployed, it becomes a factory that must be kept running, like a power plant that cannot easily be left without supervision. Do not focus on the project phase itself. Very shortly after starting the project (and during most project phases), you will need to start operations and support. As soon as you deploy the first data layer, you will need at least a couple of people to take care of operations: regular (typically daily) data loads, constant database tuning and maintenance, data quality checks, etc. Later on, BI/DWH operations will consume around 50% of your overall BI human resources. Be aware that BI/DWH projects are long term and costly.
2) Do not expect business benefits right away
Since BI/DWH solutions are robust, do not imagine that you will reap business benefits* as soon as the project starts. With even the fastest project, it takes a couple of months to stabilize routine loads into the first data layers (typically stage layers, while loads into the core DWH layer come even later). You can expect to launch the first data marts after nine months—if you are really lucky. But most probably, the first outputs will be rather technical proofs of end-to-end data processing, still far from bringing any value. The first signs of business value may appear after roughly 1.5 years. The reasons why it takes so much time will be clearer after reading my third point. Do not blame your team, accusing them of inadequate effort. If anyone promises you faster results, do not believe them.
On the other hand, there are many cases where the project sponsor states a clear condition: either the project brings “something of value” within the first months or it is stopped. Yes, the sponsor has to avoid the situation where the project budget is understood as a blank check for never-ending BI/DWH development with no clear outputs. If this is a no-go decision from the sponsor, then we strongly suggest you build a BI/DWH prototype focused on a small and well-understood business subject area. After that, re-evaluate the BI/DWH project business case based on the prototype results. You will show you can deliver a small viable solution, and your sponsor will trust you more and will better understand the complexity. Then you and your sponsor can work together to set more rational expectations.
* Let’s leave aside what business and non-business benefits are for now. Unlike most business sponsors, we believe that even the integration of heterogenous business data to a single platform is a significant achievement that brings many positive externalities to the whole organization. But, we understand that business folks are not satisfied with this.
3) Reporting is a pressing symptom, but the root causes lie much deeper
Typically, the impulse to build a new BI/DWH is that users are not receiving their “reports” in time, the quality of the reports is poor, or new reports cannot be developed within a reasonable amount of time. These are often considered to be problems in existing BI/DWH solutions, but they should ask why. And they should not only ask once but follow the rule “ask five times why“. After a few deep dives, they would discover that their current BI/DWH looks the way it does because of unresolved issues in primary information systems that provide low quality data, maybe a missing or incomplete data integration process, weaknesses in master data management (the same data in different systems), inconsistent or even contradictory business definitions, or missing or messy reference data management.
My suggestion would be to start with reference data management as such before investing in BI/DWH. You will see how much effort is required to get everyone in your company to agree on unified code lists for just this one quite non-technical piece. After you go through this process, your company will be better prepared for BI/DWH.
4) Do not misuse BI/DWH solutions for operational processes
I believe that every BI/DWH solution is partly used to fix gaps in operational processes. Since a BI/DWH holds integrated data from other systems, it is simple to plug in some applications that use DWH outputs to support a critical business process. BI/DWH systems are usually more flexible than other IT applications when it comes to managing small changes, so it often happens that a requirement deploys something which would normally go under primary applications. Suddenly, you find your BI/DWH serving various purposes which are different in terms of criticality, SLAs, frequency, etc. I suggest building a data integration platform or ODS (operational data store) and keeping BI/DWH solutions for their original purposes—reporting, analytics, and business intelligence. If, for example, you need to calculate a credit score, then use a BI/DWH to help develop the right scoring function, and then deploy it in your primary system. If you load too many non-BI processes into your BI/DWH, do not be surprised when it is more expensive, less flexible, and less efficient in supporting reporting and analytics. Nobody can serve too many masters.
5) “Business intelligence is a journey, not a destination”: Avoid fixed-time/fixed-price projects
It is extremely difficult to perfectly specify what you want in a BI/DWH system so that you can exactly plan the project or any external provider can perfectly calculate the offer in terms of time, scope, and price. First of all, you usually do not know exactly what you need, what the source data is, what its quality is, and many other things. Logically, you will end up with too many boundary conditions, and you will likely sign off on a contract based on unrealistic assumptions. Very likely, these pre-requisites will not be met, and you will start spending your budget on unexpected change requests soon after the project has started. Or, the boundary conditions will be quite flexible, but then the price will cover a huge risk margin.
I suggest splitting the project into rather small and short increments. After each increment, you will be smarter and more experienced and therefore better equipped to call the next step.
Find a partner who will guide you through the whole process of building the BI/DWH solution from the beginning. If you are dissatisfied with your cooperation after the first phase, switch to someone else. Maintain the flexibility and right to choose what happens at every step without making long-term contracts at the very start. And remember that business intelligence should be always owned by your organization, not by an external provider.
Do not try to externalize your current BI/DWH problems by handing them over to an external partner. The problems will be back like a boomerang. Good BI/DWH experts are good at what they do in the field of data management / governance analytics, but they are not insurance companies for your hidden BI/DWH risk.
From my experience, I would say that the best mode of cooperation is something known as a team lease—an external partner can provide you with a well-coordinated team, which also usually uses their own methodology. Both the team’s skills and their methodology can be naturally adopted by your own internal workforces who will gradually raise your internal BI competency.
Business intelligence is not just about data and data warehouses. Generally, any kind of intelligence is more of a skill or ability. If human intelligence is a way of understanding the world around us, then business intelligence is the ability of a company or organization to understand its business, competitors, and markets. Intelligence cannot be purchased, it must be developed and learned.