Common issues when synchronising data between applications
In this first of a series of posts, I will layout some of the common issues encountered specifically when synchronising data between applications.
Synchronisation is the replication of data between two or more applications. It is usually more than a one-way push. Synchronisation is the continued updating of data to keep all applications reflecting the same view.
Firstly, there are two broad categories, each with its own set of properties, that data falls into:
It is increasingly expected that Master Data be updatable from multiple applications where each application holds its own copy of a common ‘truth’.
There is a third category: Master Data with transactional properties. An example of this is a project structure, where the project goes through several stages throughout its life and where each stage has different business rules affecting its behaviour.
Data Impedance & Mismatch – The higher the resistance or more transformation required the less able you are to synchronise it.
Data Interchange – As the sophistication, complexity and/or frequency of the synchronisation increases so does the requirement for more sophisticated means for transferring data.
Data Take On – Finally, issues around the amount of historical data, whether the load time for data is significant and whether data cleansing for old or historical data needs to take place.
In subsequent posts I will describe the various strategies available to resolve these issues. Part II will describe strategies available for synchronising data between applications, their advantages and pitfalls.
Master Data
Master Data is conceptualised as key entities used within an organisation. Customers, Employees, Items & Suppliers are all considered master data. At the simple end of synchronising, Master Data records involve a single record of data per entity. However, Master Data often has a number of child attributes such as addresses for customers, bank details for employees, and a range of attributes such as locations, suppliers and units of measure for items.
It is increasingly expected that Master Data be updatable from multiple applications where each application holds its own copy of a common ‘truth’.
Transactional Data
Whilst sometimes more complex due to their header/detail structure transactions, they are usually easier to synchronise due to the following characteristics:- Simple direction/flow – One application is the master and the other(s) are slaves thus eliminating issues such as conflicts.
- Transactions are less likely to be updated – Once a transaction is posted it has reached its final state and requires only a single push to the target application. There are however a couple of notable exceptions: Sales & Purchase Orders often need to be synchronised to reflect their current ‘state’ and because of their more complex structure and heavy business logic require careful attention. Further, Customer & Supplier Payments can have a complex many-to-many interaction with their corresponding invoice documents.
Issues Surrounding Synchronisation
Data Structure Complexity – More complex data involving header, details, sub-details, etc need strategies which address changes for each ‘level’ of data. Synchronisation & Conflict Resolution- Synchronisation needs to be considered at each level of data.
- Deciding which application, if any, is the master or whether bi-directional integration will be supported.
- Conflict Resolution – What rules should be followed where there is a conflict and is there sufficient data available to resolve a conflict and/or data storage to mediate conflicts?
- Field Level Transformation – Destructive transformation, where the result of a derived or calculated field cannot be applied in reverse or accurately reversed can prevent bi-directional synchronisation.
- Field Lengths and Type Differences – Two common issues arise: truncation of textual fields such as name, addresses & comments, and the mapping of enumerated fields i.e. drop down fields, where the choice is identified as an integer value within the source or target application.
- Data Structure Mapping – Cases where the structure of the source data does not match the target and where transformation is required to massage the data into the required shape.