businessinfomag.uk magazine 24 DATA The importance of flexibility in data migrations Done right, data migrations can help businesses enhance data analytics, safeguard against cyberattacks and improve disaster recovery, whereas half-cocked, Waterfall-inspired attempts will only lead to headaches for Ops teams, warns Alex Fishlock, CEO of Agile consultancy Catapult CX engineering, testing, regular deployments and clearly defined small batches of changes will be needed to do ‘data migration’ well. Working with an Agile vendor can help ensure the project is successful but may introduce a requirement for cultural changes and the adoption of Agile methodology. Continual improvement If an organisation attempts a data migration based on anonymised data or a limited subset, it will no doubt fail. The complete and unredacted data being migrated must be available to the software engineering team from the very start. Migrations need to happen at least daily, and tests be created for all ‘edge cases’, validating that the ‘before’ and ‘after’ are as expected. Because of Waterfall’s influence, many Ops teams see a new system as something to be developed, tested and deployed in one go, as a project that has an end date. This approach is just plain wrong. In reality, the new system is always going to be something to improve continually. Without continual improvement, the new system will die an ugly death just like the old one. By treating the migration as a continuous software engineering problem, Catapult CX can embed testing into the migration process itself and with every iteration improve the quality and prove the completeness of the new product build. In many respects, this moves the team from concentrating on a ‘project’ to now owning and managing a product. Automating everything allows us to get a first As more public and private organisations adopt artificial intelligence (AI) and data mining, combined with ever more demanding customer expectations, outdated legacy systems that struggle to keep up must be replaced. This will involve the obvious product creation but also the less glamorous extraction, transformation and loading of data from one system to another – a process known as data migration. According to Gartner, 83% of data migrations either fail outright or exceed their allotted budgets and implementation schedules due to various factors, including the inability of Ops teams to respond with flexibility to the unexpected, such as transformation flaws or missing data. Data migration is more complex than it sounds. If an organisation focuses purely on the new system's capabilities, it may run into problems. Data from the old system is quite often a mixture of unused data, captured because the ‘system’ requires it, or just plain wrong. The data structure often represents what was good for the original system, but which has constrained it over time, so that when the time comes to move to a new system a hitherto blocked backlog of business process improvements can finally be unlocked. Making structural data and business process changes are core software engineering activities and not simply a data migration problem. Because of this, proper software engineering is required and, as with all proper software cut of the new product to private beta users more rapidly. Not taking risks There are typically two types of Ops people: the maintainers and the modernisers. The first group mainly cares about keeping their current system running and will only introduce modifications to keep it alive. For this group, success is judged on the basis of whether an incident has occurred over a given period. In contrast, modernisers regularly make changes to the applications they look after. Making changes means they are frequently doing deployments, which runs the risk that something might go wrong, such as incorrect data mapping or configuration errors. However, by bringing the right expertise on board, these risks can be avoided. Remember, evolution pays dividends – a regularly updated frontend is more likely to meet changing customer requirements and stay secure. Don’t give yourself a headache with a half-cocked migration project. Instead, regularly build, test and deploy a new product, and combine that with a fully automated data migration pipeline so that your trusted users can play with the new product as it’s being built. About Catapult.cx: Catapult CX helps public and private sector clients deliver better software and services. It provides customercentric and agile product development, legacy technology modernisation, development of software-based services and is amongst the most longstanding Atlassian partners in the United Kingdom. The company holds ISO/IEC 27001:2013 and is Cyber Essentials Plus certified. 0203 773 0553 https://catapult.cx/
RkJQdWJsaXNoZXIy NDUxNDM=