When building data management enterprise solutions such as metadata storage, data warehouses, data lakes and the like, we often encounter a requirement to combine the agile approach with a guaranteed fixed scope, delivery date and budget. The term agile is undoubtedly being misused in such cases. The customer is probably referring to its idea that we will start doing something and be ready to change all of that something continuously at any time, including the essential requirements for the product and target solution. Such an idea is tempting, but among other things, it directly conflicts with the requirements of a fixed price and delivery time.
To avoid walking on the thin ice of project management terminology, I’ll take a step to the side and try to define two terms of my own. The first term is the engineering approach. It is perfect for situations where it is possible to specify in advance exactly what is to be delivered. It works when supplying anything that has been repeated many times and that there is a certain body of general knowledge for (i.e., a certain “state of the art” for the given field). The engineering approach assumes that first, the requirements are analysed; once they are approved, a detailed design of the solution is prepared; and then it is completely approved. For work done this way, costs and labour are estimated, a waterfall project plan is made and then it is implemented. (Of course, changes occur during implementation, but they are properly managed and should be relatively few. In the event of a large number of major changes, the entire project is rescheduled.) All the design, estimates and planning are based on many years of knowledge and experience (engineering) and, therefore, should be realistic, which means, among other things, that any deviations from the plan (especially regarding time or budget and possibly the scope and the quality of the supply) are traceable and explainable.
An example from a field other than IT could be construction projects, which are often repetitive and very comparable. There are many construction companies on the market which have to follow the same rules and regulations that construction students learn at various post-secondary schools. There are also metrics that can be used to assess or verify labour and cost estimates (e.g., the average cost to build a square meter of flat in a block of flats).
But the reality is that in the field of information technology there are fewer and fewer projects that are conducive to a pure engineering approach. Data projects are the best example of this in IT. The reasons are obvious. Each new data project is essentially unique, so the condition of repeatability is not met. Often, solutions are supposed to emerge that are essentially living experiments in the field. The more dynamic the environment is in which the new project is initiated, the more difficult it is for the enterprise to precisely specify the scope, structure, or even the quality required of the delivery. Additionally, projects are becoming increasingly interconnected and interdependent, leading to more frequent and more fundamental change requirements. Project outputs are often robust solutions, so the implementation is planned over a long period of time (years), during which there are many changes in the priorities and business needs of the customer. In such a context, a pure engineering approach with forward planning and consistent quantification and projection of the impacts of all changes is practically inapplicable. One of the main features of the engineering approach is, therefore, forward detailed design.
The simple yet ingenious program of evolution that has been happening in the natural world around us for billions of years has excellent results in all fields. Without any projects at all, truly miraculous things have been created such as the mitochondria in our eukaryotic cells that function as microscopic power plants. Thus far, nature’s most amazing creation is perhaps the human brain. Apparently, this was also created without a planned project, regular status meetings, a steering committee or a priority commission. There is not even any evidence that the human brain is the joint work of some enthusiastic workers in tribes or squads supported by sacrificial scrum masters under the supervision of empathic agile coaches who all, for ages, attended stand-up meetings every morning and then worked hard to develop a neural network in the cortex.
However, natural evolution has a few drawbacks: it’s unpredictable, it doesn’t have a goal and it has an incredible amount of time and resources to try all the dead ends. We would have to be really very lucky to have shareholders or customers as project sponsors who would tell us “Do whatever you want for as long as you want and with an unlimited budget. We know that sooner or later you’ll create something really amazing.” That is, of course, beyond reality. So, the question becomes how do we tame evolution to serve a predetermined goal. Maybe that seems like a contradiction, but perhaps the solution for complex projects in a dynamically changing environment is intelligently managed evolution. Intelligence is sometimes defined as something that seeks to achieve a defined goal—whether the goal is to reduce poverty in third-world countries or to turn the universe into a paper aeroplane factory. By intelligent project management, I mean a constant holistic evaluation of the situation on a project by considering appropriate changes (whether changes in the overall scope or quality of the target supply, changes in the available capacities and their expertise, changes in supply priorities, etc.). By the driving force of evolution, I mean the maximum degree of freedom and creativity of each tooth of the gear. By an intelligently managed evolutionary project, I mean the work of honest experts with a holistic perspective, who maximally support the inner freedom and creativity of everyone working on the project while at the same time gradually and actively directing them towards a rough goal which can change.
Therefore, those who implement intelligent evolution must satisfy several key prerequisites: they must be experienced professionals who are willing to constantly evaluate the situation humbly and holistically and propose changes to the goals and the means leading to them; to learn continuously; and especially, to communicate clearly and respectfully with others. I can say that at least in the area of complex data projects, these prerequisites are met by the data competence consultants at Profinit. In another article, I will try to describe how this managed evolution process works using an example of a specific project for the construction and development of a data warehouse.
Author: Petr Hájek
Information Management Advisor