This is a Rosetta Code post.
You need to come up with a Solution Design for an Ordering System that is fault tolerant. Orders are placed by the Admin user on behalf of Customers. The customer doesnt login to anything, they place the order via E-mail/Telephonic conversation but their details are known to the company. Orders are shipped via an external company called
Bobs Post but their API is unstable and can often go down for days, there is no other shipper we can use.
Design a system that allows an
Admin User to place orders for
Customers, the actual items (what they order) dont matter. The focus is on Solution Design for a fault tolerant system where the Customer is notified when the item(s) are shipped.
- Following KISS (Keep It Super Simple) solve the problem in the simplest manner, itterate later to allow for fault tolerance.
- Draft a few solutions and itterate on them
- The solution is Cloud Based (Focus on AWS), consider technologies for the pending decision register item
- Security (Authentication and Authorisation), consider using OAuth2 flows
- Serverless Lamda Functions to expose data / house logic
- State Machine if we want to group logic and Lamdas
- Kubernetes (EKS) for Workers
- RESTful APIs to expose data / house logic
- Persistant storage options like Relational (RDS) or Key/Value Document DB (Dynamo)
- Frontend Application. React, Angular or Vue Single Page Application (SPA) hosted in S3
- Messaging System (SQS) and Cloud Events
- Potential Cloud Design Patterns
- Call out and try mitigate all bias from the decision making
- Ubiquitous Language
- Figure out the language of the domain, this this in all conversations with stakeholders and the customer
- Status =
First lets understand the problem and use an unconventional
Sticky Note representation.
Bobs Post has an unstable API, we can unload responsability from the BFF and introduce some retry logic using the Asynchronous Request-Reply pattern and Queues. Additionally Polly can be used to retry HTTP requests.
Now add a Worker process that will process the order request. If a message is not deleted and it has been received the maximum receive count times its pushed to the configured Dead Letter Queue (DLQ)
There is still a problem with the design as
Bobs Post is unstable, so if any of the requests fail and the retry count for Polly is exhausted the process will fall over and cause possible duplication.
We can try address this by adding a key/value database like Dynamo to keep track of the process. Additionally the responsability of the process can be delegated to several workers and queues. The caveat being this brings complexity, so only introduce this when needed.
Conditional logic would be added so that should the data exist in Dynamo then dont do that step. IE: We have the user data, dont call the
User API ect.
- Third Draft: The main focus here is to place the order with
- Third Draft: The main focus here is to check for updates at
- If no update is avalible, just Queue another message to check for shipped (WARNING: Can cause infinite loop.) So potentially just dont delete and rely on the DQL.
- Third Draft: The main focus here is to notify the User that their item(s) have shipped
- Emails are a common thing to send, potentially a flow exists (Event Queue) where the request can be sent
The DLQs can be redriven.
- Another worker can poll each DLQ and re-drive
- The process could be manual
Finally once you agree with your team(s) draw as a technical sequence diagram. Also see the Miro Template.
The process above is not without fault. The SQS messages default retention period is 4 days so if
Bobs Post is down for longer than this and the maxium retry period is exhausted the system will fall over. Potentially the flow could be
SPA -> BFF -> SQS -> DYNAMO with workers that progress the status and call the APIs based on the state of the
DYNAMO database record.
My suggestion is to be pragmatic and solve problems when they are problems, if you add complexity early you could solve a problem that doesnt exist and the potential gains are lost in the complexity.