Logo

dev-resources.site

for different kinds of informations.

Delivering 100% of Webhooks

Published at
9/22/2022
Categories
reliability
architecture
statemachine
Author
sibelius
Author
8 person written this
sibelius
open
Delivering 100% of Webhooks

If something is important for your customers, you should fix it. Webhooks are one of the core features of Woovi, and even a single webhook failed causes frustration for our customers, and also reduce our trust. For this reason, we build a reliable system to ensure we always deliver at least once every webhook.

Woovi

Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.
To provide real-time capability for our e-commerce plugins, we send webhooks when a new charge is paid.

What is a Webhook/Postback URL

A Webhook/Postback URL is an HTTP endpoint that receives a new request when a new event happens.

Woovi Charge

A Woovi Charge represents a payment request. It requests a payment of a given amount from a merchant customer. Usually an e-commerce order has 1 charge to receive an instant payment.

Building a reliable system

To build a more reliable system, we used a state machine in conjunction with a fallback cron job to ensure we always deliver any webhook at least once.

State Machine to track Webhook Delivery

Webhook State Machine

For every charge paid, we emit a charge paid event.
Then the chargePaid event processor will send a webhook for the e-commerce plugin.
We noticed that some webhooks failed, and we decide to use a state machine to keep track of webhook delivering.
The state machine has 4 status:

  • PENDING - the webhook was not sent yet
  • SUCCESS - the webhook was delivered with success
  • FAILED - the webhook tried to deliver 7 times and the e-commerce plugin didn't return a success response
  • NOT_APPLY - when the merchant do not have any Webhook setup at Woovi.

Fallback cron job

We also added a fallback cron job, that runs every 5 minutes to reprocess charges that did not deliver the webhook yet (PENDING state). Charges in PENDING state.
This fallback also solves the problem when Woovi system is having some problems.

Conclusion

State machines are a great tool to make sure your system reaches the state that it was supposed to be. And fallback job can fix a wrong system state to a desired one.


If you wanna work with us, we are hiring!

reliability Article's
30 articles in total
Favicon
SRE Culture Embedding Reliability into Engineering Teams
Favicon
Understanding Idempotency in API
Favicon
Navigating Software Resiliency: A Comprehensive Classification
Favicon
60 Years of the IBM System/360: A Legacy of Reliability and Security
Favicon
Reliability in Legacy Software
Favicon
Azure Site Recovery
Favicon
A simple guide to addressing single point of failure (SPOF) while evaluating external tools
Favicon
How to design Reliable Microservice Chains using the principles of Systems Thinking.
Favicon
Reliability concepts: Availability, Resiliency, Robustness, Fault-Tolerance, and Reliability
Favicon
Lessons in Reliability: Margaret Hamilton's Software Engineering Approach
Favicon
Understanding Observability in Software Distributed Systems
Favicon
Ensuring reliability: SLOs, on-call process, and postmortems
Favicon
Building Resilient Software Architecture: Lessons Learned from the Domino Game
Favicon
10 most important Metrics you must know as a DevOps Engineer
Favicon
10 Most Effective Strategies to ensure reliability of the system
Favicon
Saving 30% on costs and improve infrastructure reliability with profiling
Favicon
"Building Secure and Reliable Systems": How Google's Approach to Security and Reliability Can Benefit Your Organization
Favicon
SLO Anti-Patterns: Real-World Lessons
Favicon
Building Resilient Systems on AWS: Avoiding Common Errors with the Well-Architected Framework
Favicon
SRE book notes: Introduction to Site Reliability Engineering
Favicon
PagerDuty Community Update: November 18, 2022
Favicon
5 key points about Immutable Infrastructure
Favicon
What about off-grid programming?
Favicon
Delivering 100% of Webhooks
Favicon
Observability is becoming mission critical, but who watches the watchmen?
Favicon
Availability Service Level Calculation
Favicon
Reliability Restaurant – How to approach software reliability as a mindset
Favicon
Delinearized Rollouts
Favicon
Submitting Changes
Favicon
Multi-Version Rollouts

Featured ones: