Concepts
Webhook
We can register a webhook with the external service. It means: call me back at a certain URL when you have updates on the request. When the PSP has completed the processing, it will invoke the HTTP request to update the payment status. (We can set up a housekeeping job to check payment status every hour.)
improve API performance
- pagination
- async logging
- caching
- payload compression
- connection pool
api gateway functionality
- parameter validation
- allow/deny-list
- authentication/authorization
- rate limit
- dynamic routing
- service discovery
- protocol conversion
- error handling
- circuit break
- logging monitoring
- cache
TinyURL
Consistent Hash
consistent hashing is special kind of hashing such that when a hash table is resized and consistent hashing is used, only k/n keys need to be remapped on average, where k is the number of keys, and n is the number of slots.
terminology
- node
- hash function
- data partition
- data replication
- hotspot
- gossip protocol
consistent hashing realworld practices
- amazon dynamo partitioning
- apache cassandra data partitioning
- discord chat application
- akamai CDN
- maglev NLB
rate limiter
- token bucket
- leak bucket
- fix window counter
- sliding window log
- sliding window counter
hard & soft rate limiting
- hard: the number of requests cannot exceed the threshold
- soft: requests can exceed the threshold of a short period
rate-limit best practices
- use client cache to avoid making frequent API calls
- understand the limit and do not send too many requests in a short time frame
- include code to catch exceptions or errors so your client can gracefully recover from exceptions
- add sufficient back off time to retry logic
manage traffic / capacity planning
- forecast the application load
- estimate capacity requirements
- plan adequate redundancy
- benchmark the application
- reserve compute capacity
- monitor utilization, and reassess requirements periodically
KV store
- Consistency: consistency means all clients see the same data at the same time no matter which node they connect to.
- Availability: availability means any client which requests data gets a response even if some of the nodes are down.
- Partition Tolerance: a partition indicates a communication break between two nodes. Partition tolerance means the system continues to operate despite network partitions.
kv core concepts
- data partition
- data replication
- consistency
- inconsistency resolution
- handling failures
- system architecture diagram
- write path
- read path
unique id generator
notification system
- user preferences and customization
- granular controls
- frequency settings
- notification group
- quiet hours
concerns
- type of notifications
- channels
- in-application
- push
- sms
- user preferences
- personalization
- time sensitivity
key considerations
- notification generation and filtering
- trigger events
- region targeting
- customization rules
- notification delivery
- channels
- message formatting
- time and rate limiting
- prioritization
- system architecture
- components
- scalability
- user interface
- settings
- notification history
- additional considerations
- accessibility
- feedback loop
web crawler
basic functions
- continously remove a url from the horizon
- issue an http get on the url
- download the contents, add the contents to the results
- parse the contents for new Urls and append any unvisited URLs to the horizon
possible features
- robots.txt handling
- crawl depth limiting
- request rate limiting
- incremental crawling
- non html media type support
- page deduplication (url canonicalization)
google doc
differential synchronization algorithm
References
- tiny url
- consistent hash
- https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf
- Maglev: A Fast and Reliable Software Network Load Balancer
- differential synchronization algorithm