Well Pharmacy: Automated prescription packing and distribution with AWS Serverless

Well Pharmacy (“Well”) is one of the UK’s largest pharmacy chains, operating over 700 high-street pharmacies and one of the UK’s largest online pharmacy services.

Well saw an opportunity to reduce the amount of time that in-store staff spend packing and preparing medicines for repeat prescription customers by using warehouse-based machinery to pre-package and label prescriptions for delivery to pharmacies.

The challenge was to integrate a new pharmacy system with existing and new warehouse machinery, enabling all of Well’s pharmacies to order from the new service. The service must ensure that the correct medicines are delivered, and that the correct instruction labels are applied to the medicine.

Infinity Works developed an API and message processing system that worked around the constraints of existing systems and provided the high availability and scalability required for healthcare systems. The effective use of AWS Serverless technology resulted in a solution that has proven itself to be extremely reliable, scalable, and that requires minimal maintenance.

Infinity Works played a critical role in the successful build and deployment of Well’s Central Fulfilment capability, which is currently the largest in the UK. Their knowledge and experience of Cloud-native and particularly AWS Serverless technology was key to delivering a flexible, stable and secure solution at pace. Central Fulfilment is now a critical business process for Well, with millions of customers prescriptions being assembled. It is reassuring to know the AWS solution will scale to the volumes required when we need it.

Claire O’ConnellDirector of IT and Digital at Well

Well wanted to reduce waiting time for customers in-store and realised that one way would be to reduce the amount of time that store colleagues spent assembling prescriptions.

Well’s recently launched online channel – a home delivery prescription service – was a success that resulted in large volumes of prescriptions.

By introducing a central fulfilment (hub and spoke) model, Well was able to introduce technology to reduce effort in pharmacies by automating some of the sorting at the warehouse. By centralising this capability, Well were able to invest in equipment that would not be practical to install in every pharmacy, due to space constraints or lack of sufficient prescription volume to balance the cost of the equipment.

Had Well not taken on this challenge, the home delivery prescription service would not have been able to continue scaling user volumes, waiting times for customers inside the pharmacy would not have decreased and Well would have risked being overtaken by competitors who had invested in medicine distribution automation technology.

When selecting technologies to achieve the desired outcomes, the effectiveness and completeness of AWS Serverless technology were a major factor. The low up-front costs of AWS generally, and especially Serverless technology meant that proof-of-concept implementations were inexpensive and fast to create.

Prior experience with AWS’s Serverless offerings had demonstrated that they met our regulatory, compliance and security requirements while being reliable, flexible, and capable of scaling up (and down) to meet user demand. The availability of transparent encryption at rest and in transit for each service we used was especially appealing.

The alternative of building systems on-premises was considered, but ultimately rejected due to the complexity of building fault tolerant message queues across multiple data centres, an application hosting platform, the complexity of secrets management on-premises where AWS IAM capabilities are not available, and the extra software engineering effort required to build server-based applications. Selecting AWS allowed us to maximise the time that we spent building application features, rather than delivering infrastructure that offers no differentiation in the marketplace.

Well chose Infinity Works as a partner because of our experience delivering healthcare solutions, often on critical National Health Service (NHS) systems, and capability to engineer high scale cloud-based applications.

Within Well, Infinity Works had built up a reputation for being low-fuss, technically excellent and suited to delivery of complex agile projects.

Infinity Works had provided much of the engineering capability and cloud infrastructure for Well’s online pharmacy offering, and had developed an understanding of Well’s business that meant they were uniquely qualified to deliver this project.

A secure API integrating with Serverless components

Infinity Works proposed that a suitable solution would be to provide a secure API enabling pharmacy systems to check stock, order prescriptions from the central fulfilment facility, and track the status of orders.

This system would need to include a mechanism for integrating into warehouse technology, and a way to collect up-to-date stock data from the warehouse inventory management systems.

Infinity Works proposed the use of AWS API Gateway due to its security features and integration with other serverless components. API Gateway provides strong protection against incoming HTTP-based attacks, rate limiting, and only supports TLS connectivity, preventing accidental misconfiguration that could result in transmission of customer data over plaintext protocols.

API Gateway is well integrated into AWS Lambda, which was selected as the compute platform for the project. AWS Lambda functions were used to build the APIs and connect to:

AWS S3 to collect stock data.
DynamoDB to store and retrieve prescription data.
SQS to broker communication with on-premises warehouse systems.

AWS Lambda was selected over other options such as EC2 because of its high scalability, low cost, low maintenance requirements, and security. Under the AWS Shared Responsibility Model, AWS manages the security of the compute platform, while we are responsible for the code that runs on the platform. This reduces the total cost of ownership by removing the need to manage the security lifecycle of the compute platform.

The database platform that drives the warehouse inventory management system was configured to push regular stock data updates to an S3 bucket. This low-cost approach simplified integration between systems, while providing fault tolerance and preventing the need for on-premises systems to accept inbound network connections.

DynamoDB was used to store prescription data due to its simplicity and reliability, low cost, and strong encryption features. The unique TTL (Time-To-Live) DynamoDB feature ensures that data is automatically deleted according to compliance policies.

Amazon Simple Queue Service (SQS) was selected as a queue mechanism to control the throughput of messages to the warehouse system, smoothing out usage spikes. A small agent written in the Go programming language runs on-premises and connects via the AWS SDK to the SQS queue system within AWS to collect messages and forward them on to the warehouse equipment, returning information back to the cloud via another queue. SQS’s built-in encryption features and built-in AWS Lambda integration allowed us to spend more time working on features than working on developing a platform or connecting systems.

SQS also allowed for scheduled maintenance of the warehouse systems – the queue processing can be paused and resumed after maintenance is complete – while the dead letter feature allowed us to be sure that all messages had been processed and to handle issues.

We chose to use AWS Serverless technology in order to minimise the amount of engineering time we would need to spend on platform engineering, prefering to focus on product features, compliance requirements, and automatic scaling to meet demand.

Deploying the power of Serverless

Number of Lambda functions	10 – used to process API requests, handle queue messages and to process streams of updates from DynamoDB.
Events per month	24M
AWS services used	Amazon API Gateway, AWS Lambda, Amazon DynamoDB, Amazon SQS, Amazon S3, Amazon Kinesis, or AWS Step Functions.
Framework	Serverless framework.
Languages	Go

Deployment

Well Pharmacy had some initial concerns about the use of Serverless due to unfamiliarity with the overall toolset and its ability to meet the stringent compliance requirements required by healthcare. By working with compliance and security teams to help them understand the security capabilities of AWS, the specific AWS services we were using, and the AWS Shared Responsibility Model, we were able to demonstrate that the solution met or exceeded all requirements.

To deploy the solution we built a continuous integration pipeline using a continuous integration (CI) platform and Serverless Framework – a command line tool that can be used to build and deploy Serverless applications by generating CloudFormation templates.

This allowed the new versions of the application to be deployed to non-production environments automatically in order for functional and user-acceptance testing to be carried out. The CI pipeline standardised the release process and the use of CloudFormation allowed us to be certain that our non-production environments matched production environments. The “pay for value” aspect of Serverless technology meant that our non-production environments cost nothing to run when not in use as per the Serverless Architectures with Lambda whitepaper.

Serverless architecture - Well Pharmacy's hub manager

Solution characteristics

The performance of the platform was tested in non-production environments by invoking the API at higher than expected volumes. There was a high degree of confidence that our production environments exactly matched non-production environments due to the use of Serverless framework to create CloudFormation templates that exactly replicate environments. During testing, the RAM size of the Lambda functions was tuned down until it had no impact on the performance of the solution.

Appropriate DynamoDB auto scaling parameters were discovered by load testing, however the solution was migrated to the on-demand DynamoDB pricing model as soon as it became available to allow us to benefit from cost-effective zero configuration auto scaling.

DynamoDB provided consistent response times at all tested scale levels. This operational consistency is one of the primary reasons that DynamoDB was selected over a traditional database solution such as MySQL.

By default Lambda functions have no permission to access other services. To enable Lambda functions to access other AWS services, permissions were applied to each Lambda function using an IAM role that only allowed access to the specific resources and operations required for that function. This protected the solution against accidental misconfiguration and privilege escalation attacks.

The solution includes monitoring for failed Lambda executions using CloudWatch, and has alerts in place to notify teams to investigate. The SQS retry policy and dead letter queue features allow the solution to handle transient network failures and unplanned maintenance of downstream systems without requiring human intervention, reducing the ongoing maintenance costs of the solution.

The need to limit parallel execution volumes to limit connectivity to backend resources such as a traditional MySQL database is supported by AWS Lambda. However, DynamoDB was able to meet performance requirements without throttling, while SQS’s batch size features allowed fine tuning of message throughput to warehouse equipment in order to prevent overloading.

By using Serverless components, the need to manage and configure a network (VPC) was eliminated resulting in lower costs (no NAT gateway or load balancer costs) and reducing the need to manage and audit firewall configuration.

Central fulfilment, lower wait times, reduced cost and reduced labour

All Well pharmacies can now benefit from central fulfilment, resulting in lower wait times for customers, reduced cost of operations, and reduced labour in the pharmacies, while the online pharmacy has been able to continue to grow its user base without excessive increases of operational costs.

The stability of the platform has been excellent over its last year in production, with no outages or system performance issues recorded.

During the development of the platform, we learned a lot about effective DynamoDB design, and would encourage anyone interested in using DynamoDB to watch Rick Houlian’s Amazon DynamoDB deep dive: Advanced design patterns talk and to gain the benefits of a “one table” DynamoDB design.

The system now processes over 1 million prescriptions each month to over 700 pharmacies. The complete system cost, including all non-production environments is around $500 per month – substantially lower than designs that do not take advantage of Serverless technology.

Serverless prescription medicines