In this paper I look at a major issue for those responsible for data and IT in small and medium size enterprises: how do you stitch together all those bits of systems and their data?
Top tips on Managing the Morass
1. Tolerate Ambiguity: Don't over-analyse. Accept sub-optimal solutions when people and their behaviour are most likely limiting factors. Plan fluidly for a world in flux.
2. Be Proactive: Get out there, see what is happening, and constantly sell a better world of shared company data.
3. Interface and Reconcile: Build the two together, along with ownership and responsibility.
4. Focus on added value: Don't rush to install new systems in the hope of solving data problems.
5. Fix the broken bits first: the systems or data problems that fall in the not urgent, but important category. In other words, those that have been on the to do/known list for ever. If you do you will get happy staff, happier customers and maybe even become a bit more profitable.
One ideal target is that all our company data is held in one common database, which may be served by one or more applications. Data is held only once, it is accurate and up to date. This vision was widely held 5 years ago, but the world has changed dramatically.
Even if we could marshal the resources and consensus to reach such nirvana, it would soon be out of date as the rest of the world, and our business requirements, move on. Furthermore, such a monolithic system holds back the organisation. Updates to the database structure need to be overseen by a senior committee. Downstream problems caused by structure changes or backfilling data becomes the stuff of career nightmares. The effort and risk to update the database structure makes the organisation unable to respond quickly, if at all. IT would be paralysed, and "shadow" systems (see below) would proliferate. Of course, you could replace a monolithic system with another monolithic system, either bought in or built in-house. If you have all the functionality you require, perfect data, and an unchanging business, then that could be a reasonable choice.
In a dynamic world, we may prefer to be able to treat our systems as a set of components where we can swap new bits in and out. This enables us to write or buy new components to respond to new business, to swap out other components which can no longer be supported, or where regulations have changed, or a supplier has changed their pricing model. It enables us to treat those components in different ways, for example being very strict about controls in a central accounting function, whilst prototyping with newer technologies for newer businesses. Inevitably, as many of these components will have their own "database", this moves us away from the idea of a piece of data being held only once. This creates problems with ownership, consistency and integrity, which constitute many of the real issues for most businesses' data management. Our data is held in several silos, and we need mechanisms to link them together both for daily operations as well as for reporting, ideally with some means of allocating ownership and the source of truth for any particular datum (piece of data).
The reality for most companies is intractable, in that they have a patchwork of systems, most of which will have their own distinct set of data. There may be large bespoke internal systems, others created by a software firm, either of which may exist within an environment created by an even larger software firm on which you then have a dependency. Today there will be probably be some software as a service in use, and often a website or two (or 20!) created by a marketing, or advertising agency. Any larger company may have duplicate systems from a merger or takeover, and there are probably legacy systems that are no longer in active use, but for which nobody could find the enthusiasm to migrate and tidy up all the data. There will be interfaces which work well, and some which do not. Some interfaces will be overnight batch processes, so data changes in one system may take a day or two to cascade down to other systems.
To make the management of this hodgepodge worse, there are likely to be a set of "shadow" systems. Shadow systems are those that have been created beyond the control, or knowledge of the IT department. Some of the problems with this, mostly quality related, are clearly enumerated in a short Wikipedia entry, to which I have added some points:
There is a case to be made that some departmental systems exist in the penumbra, if not quite the full shadow. Software as a service (SaaS) has made it easier for departments to implement and control their own IT services, such as a sales department running Salesforce. They will have their own budget. They will use their own resources, possibly with the help of an external agent, to get things up and running. They may tweak the software to better suit their requirements, or even use it like a development environment and create their own shadow systems within the software as a service. This can all be invisible to the IT department, who are only perceived as control freaks who would slow things down and get in the way (along the lines of Mordac the preventer of Information Systems in the Dilbert cartoons).
From a business viewpoint, SaaS is a good thing, particularly for small and medium size companies. It provides access to rich functionality for a fraction of the cost of bespoke development. You can be up and running quickly, with little business and technical risk. You don't need your own servers and data centre and all the infrastructure and staff costs that go with it. The system will be delivered through your standards compliant web browser, which means you can access it from multiple devices and operating systems: there should be minimal installation issues and IT support requirements. The physical boundaries of your office are no longer a constraint, and it is easy to extend your virtual company by giving access to advisers, customers and suppliers. We can conceptualise the SaaS silos as components of our overall system architecture. The more they are stand-alone silos, the easier they are to swap out for another system should the business requirements or the environment change. It also helps to have chunks of functionality of a size and scope that most people can cope with, and that are limited in the number of departments they service. Clear ownership and responsibility help get things done.
However, there are plenty of traps here. In an ideal world we record data only once. We can then look after the integrity and consistency of our data. In our siloed world this is not the case. Each silo may have its own version of a customer table, with a slightly different but overlapping set of customer data, for example. Static datasets may be different between our silos - we might have different country codes, different regions, different date formats. One system might be well validated to 20 job functions, whilst another has 43,000. The underlying conceptual models of systems may be different, so it is difficult to map entities between systems, never mind fields. One system may have been conscientiously tended, whilst another has data thrown in to meet an immediate client turn-around or just has an email address and nothing else. Relating the data in one system to another may not be possible in a reliable manor, and reconciliation is then impossible.
The same departmental ownership that helps getting things done can be obstructive and damaging for the business when trying to put together an approach that supports the whole company. There are also dangers that departments become, effectively, captive customers. This can become exacerbated where staff have become "certified" professionals, who then want to continue to work in that particular SaaS environment. A narrow and incremental approach means a department, or indeed the whole business, can invest too heavily in customising SaaS to their needs when an internal application or fuller use of an existing system would be more appropriate. Our idea of being able to swap components in and out of our suite of applications will meet heavy resistance from those who currently have control, an emotional ownership of their system, those who dislike change, and those with skin in the game.
There is a cost element to IT, the same as for power, water, general HR costs and other inputs. The danger in being too enthusiastic about driving down those costs is that companies lose sight of the fact that some IT costs are an investment: capital expenditure rather than expense. That investment may be more nebulous than a piece of large capital machinery standing on the factory floor, but like the physical assets it will require some maintenance effort if it is to continue to add value. When paring down those IT costs, we must be careful not to throw out the baby with the bathwater. Agile is a good approach for projects, but there is a danger that maintenance is missed, and that small updates are never quite important enough to get scheduled. This can then lead to disgruntled end-users and yet more shadow systems.
Project plans are made with the idea that IT staff are inter-changeable, but as Frederick P. Brooks Jr. noted in "The Mythical Man Month", some programmers are 10 times more productive than others, with no correlation between experience and performance. A good analyst/programmer will understand the business as well as the end-users and can then be proactive in suggesting ways forward and guiding through the fog of miscommunication that bedevils projects. In the past they would have continued a relationship with the users, probably doing the software maintenance. I worry that in an agile environment it is more difficult to continue this working relationship between users and analyst/programmers, and other IT staff deployed for a sprint will be less effective until they too are up to speed. To outsource such people might gain a manager a short-term bonus, but it is incredibly naïve.
I fought some early skirmishes in the battle against the all-powerful and controlling corporate data centre when PCs were still establishing themselves as valid grown-up business tools. Committees decided what got done, which was executed on the in-house machines using the approved tools provided by the manufacturer. User interfaces and performance were of little importance. Things have moved on. Technology is now pervasive, and IT are much more responsive to the needs of the business. However, there can still be a sense that central IT will try to impose controls on departmental computing that will slow them down and restrict what they can do. They are Mordac made flesh, out to scupper the plans of those who are at the sharp end of getting the business done. This is grossly unfair. IT must operate with a high degree of quality control, change control and general rigour. They may be responsible for information security, data protection, data quality etc. for the whole organisation, including all those shadow and departmental systems over which they have little control, some of which they will not even know exist. Responsibility without control is not a recipe for success.
This may seem a facile suggestion. It is anathema to most technical and data people, who work to bring order. We analyse, we classify, we fit things into known patterns so that we can re-use concepts and program code. We work towards a goal where everything is in order, effectiveness and efficiency are maximised, change is controlled, specification and testing are rigorous. But one of my starting points was that, even if we could fleetingly build an all-encompassing optimal solution, the world would move on and our super solution would render us unable to respond fast enough. We need to accept a set of component systems that will inevitably put our data into silos. We need to accept that some departments will want to push on with their own systems, and that they will probably get there faster without us doing it for them. We need to embrace shadow systems as a symptom of some sort of gap and thus an opportunity to do it better in the next iteration.
Avoid analysis paralysis. We will obviously want that systems map for the office wall and the PowerPoint slides, but we probably don't need the data schema for all those third party and SaaS systems, especially as they have functionality we don't use. We can treat them more like a black box, or even a cloud, and focus on the outputs, and hence the inputs. It is more important we talk to the users and understand where the systems are adding value, or not. We need a holistic approach.
It is also important that we accept reality. It is unlikely our data is complete, accurate, and up to date. Whilst humans are involved in adding and maintaining it, this will continue to be the case. I am not suggesting we give up, but that we accept some imperfection and don't insist on data that is not required, or the end-users will find ways around our controls. Be prepared to implement changes that fix things in one area but that are "no worse" in another. If we really must go through a data clean-up, then the scope must be clear and minimal and the processes and procedures to keep that data clean in place and running, otherwise momentum will just fizzle out.
Another cliché. My point is that rather than burying yourself in busy analysis and planning, you should put yourself about. There is a constant need to sell the use of data as a shared company resource, and a vision of how you can do it better. Most users grasp this anyway, but there will be others for whom the concepts and operating models that are obvious to you will not be so for them. A particular problem will be with those managers who want to control "their" data, and who can be very resistant to change. Some nagging may be required.
Moreover, you need to sit down with the users and see and understand what they actually do and what data they operate on. What is the role of that Job sheet? Why is that lever arch file on the desk? Why does the manager spend 30% of their time keying reporting data into Excel when the company has spent so much money on data lakes/warehouses or other "solutions"? An audit may be too formal a way to put this and might scare back into the shadows the owners of what could turn out to be an important part of your company data. But you really can't do your job if you don't understand what is out there in the shadows.
You may need to sell the idea of IT as an investment, rather than a cost, and that IT are there to help rather than just prevent others from doing their job. As part of this IT should be looking to inveigle their way into the departmental and shadow systems in the hope of modifying the outcomes without being heavy-handed. Some oversight and conformity to standards can be gained by helping with the more technical aspects of departmental SaaS: customisation, data load, and the building of interfaces. This also gives you the chance to sell other options should the department want to customise their instance of the SaaS system into some byzantine castle in the clouds.
I think you should also embrace the shadow systems and provide professional help to support and enhance them. This is not just because this is the space in which I mostly work. There is risk and inefficiency here, most of which is probably hidden in plain sight. A fair amount of these systems are probably superfluous and cover functionality that may already exist, or that could be yielded very quickly from the "official" systems by an IT professional. Others could be combined, or share their data, or be written to get their data via an interface or download. Tact may be required to change habits, as well as the acceptance of some sub-optimal interim solutions, and care taken to avoid scope creep. Critically, these systems are brought into the light; efficiency and data sharing can be improved, risk is better controlled, and IT are seen as facilitators. One hopes that in due course the functionality of these shadow systems will be incorporated in other applications, but by then there will doubtless be other needs being met in the shadows.
We will have to interface our various systems to get the data from one silo to another. Most modern systems will have facilities to enable this, and if not we can probably find alternative ways to do it, such as going direct to the underlying database, or using a software robot. We can, therefore, do something about the inherent inefficiencies of our siloed data architecture. However, we will still have problems with data being held more than once, and in particular how we keep this all synchronised less we end up with multiple versions of the truth that are different. If we have written all the components, then we can ensure that data is not held more than once. More likely, we will have multiple systems with some sort of customer table. For example, we can download customer data from system S (sales) to system A (accounting), where S is a SaaS system running in the cloud, A is packaged software running locally. We then have issues if the customer data in system A is updated inside system A, as it is then different from the system S data. We may wish to insist that updates are done in system S, but system A will probably have fields pertinent to its function that system S does not, for example a customer credit limit. We could add a mirrored field to S, but can we trust the users of S to enter such data and get it right? Can we wait for the interface to copy the amended data from S to A? But if we let users change data in A, they will change some data that has come from S.
One way or another, our component systems will soon be out of kilter. This is the fundamental flaw in our data architecture consisting of many component systems. It is the chronic complaint we must accept in exchange for the flexibility and dynamism (and cost) advantages. However, we can try to ameliorate the effects, and this boils down to responsibility and reconciliation. Ownership and responsibility for data must be clear and accepted. This may have to extend down to field level in some common tables replicated across a number of systems. Like many procedural solutions, this will not stop the problem, but it will help highlight who must fix things, and probably who has transgressed, when the reconciliation process tells you the systems are not in agreement. Overall data quality will improve as users learn what data they can operate on, in which system.
Putting the reconciliation process in place is an IT task, on the assumption that this will be mostly automated. Ideally the reconciliation reporting will only report anomalies, and these are passed to the relevant owners to rectify quickly. Even before an interface is in place, there should be a reconciliation process to oversee it, and identifiers in source and target systems so we can trace data. That means the reconciliation can be used as part of the interface testing, and then kept in place and monitored. This is especially important as IT, and the interface processes they have written, have the ability to create havoc with the data far beyond the capabilities of ordinary users, sometimes for months before this is noticed.
The critical point is that we cannot just install an interface and walk away. We need to design and put in place the responsibilities and processes to find and correct discrepancies, and resources are required to keep this a continual effort.
Well obvs. Unfortunately, there are a whole set of incentives and behaviours which do not encourage adding value for the business and that pervade companies from bottom to top. There is scope here for another blog, or perhaps a book. For now, I just want to focus on the need to be seen to do stuff. Some careers are made by not doing things, and we have all probably seen examples of survivors rising through the corporate ranks whilst avoiding anything that involves risk and responsibility until they reach their "level of incompetence" (as postulated in The Peter Principle). Most go-getters, however, are keen to be seen to have added value by implementing new things. Read their CVs and it will be full of how they put in the new system that contributed some magnificent gain to the company's bottom line, or throughput, or some other unambiguous measure. You will struggle to find a section that says "saved the company loads of money, risk, and management time by not implementing new system".
There is a pervasive belief that problems with data can be solved by new systems. Systems do contribute to poor data quality, but in this day and age most issues are related to human behaviour, to understanding, to perverse incentives, to ambiguous ownership and responsibility, to belief that the little bit of extra effort required is not worthwhile or appreciated. Taking the questionable data from your old system and pouring it into a new one will quickly lead to a loss in confidence in the new system. There are incentives to throwing old systems away, not least the wish of IT to keep abreast of the latest technologies for their career. Systems will have a lifetime, but there are ways to keep otherwise functional systems running.
The temptation is to come up with a grand plan, particularly if you use some consultants. This is, after all, why they are employed. This will be done after some in-depth analysis and will recommend the replacement of various systems either directly, or by incorporation into a smaller set of components to reduce the overall complexity. This may well be a valid approach. My suggestion is not to rush into this. If you spend on a new "solution" then you may find that the resources I have suggested for supporting end-user shadow systems and departmental SaaS, and for data reconciliation are no longer budgeted, especially if the new system has inherited, or generated, new problems with the data.
Consider also that it is not just important that your data is good: your users must believe that the data is good. If the users do not have confidence in it then inefficiencies and shadow systems will creep in as they double-check or re-generate the data. Furthermore, they are unlikely to take the appropriate care when updating the systems if they think someone upstream has not done their bit. The lack of confidence in the data becomes self-fulfilling. The promise of a fix via a new system six months or a year down the road sends the wrong messages about responsibility and the importance of data to the organisation. If you can fix the issues that lead to poor data as a high priority then the dynamic nature of data means it will be in much better shape by the time you do replace systems, and the users are more likely to see how important this is.
My appeal then, is to avoid the mega plan: to be more heuristic. Only to get down to the bits and bytes where necessary. To tolerate ambiguity and sub-optimal solutions. To see the situation as in constant flux, to see that value is added by functionality for the users and the pertinent and accurate data required for that, not by new replacement systems per se, or data just for the sake of completeness. Focus on fixing the known niggles, plugging the gaps, interfacing and reconciling data, supporting the business, being an evangelist for sharing quality data, and looking at where the company is going, rather than implementing a static grand design.
I hope you have found this helpful and thought provoking. I welcome comments to affirm or disagree with the points I have made. Please contact me if you would like help with managing your own morass, fitting component systems together, or making sense of a plethora of shadow systems.
John.Davis@cranfieldsoftware.co.uk
15 Nov 18.
"Dilbert Gives You the Business" by Scott Adams