An online retailer of personalised gifts, cards, and merchandise needed faster and more reliable management information.
The retailer wanted to ensure that up-to-date marketing data was constantly available in its business intelligence (BI) platform to maximise sales conversion, particularly around seasonal peaks. The BI platform is used to drive customer behaviour and conduct ad-hoc discovery, provide marketing analytics, and aid financial planning.
However, the data was not available instantly: the previous day’s data should have been available at the start of the following working day, but data pipelines either failed or simply did not complete in time, and so decisions were delayed. Moreover, data pipeline failure required manual intervention and caused further delays.
We conducted an analysis of the data platform, identifying and removing redundant or obsolete pipelines to reduce the size of the problem space. Monitoring was then introduced to examine the performance of each pipeline and to target optimisation work for maximum benefit.
The data pipelines were running on bespoke, customised infrastructure, which made maintenance and disaster recovery difficult and time consuming. To mitigate this we migrated to newer versions of key elements of the cloud infrastructure, which also reduced the attack surface area and improved security. We then built a robust scheduling solution with AWS Serverless, improving reliability and enabling faster delivery of key reports. Finally, we introduced best-practice DevOps processes to the business and introduced lifecycle management to AWS components, reducing infrastructure costs.
