Designing a Scalable News Feed System Similar to Instagram, Facebook, and Twitter

📚 Designing a news feed system similar to Facebook, Instagram, and Twitter.

📸 The system will focus on photo posts but can easily be extended to include video posts.

💻 The system will be built using relational database tables, starting with MySQL.

💡 A news feed system can consist of complex tables, but for simplicity, we start with basic relational tables.

📱 The end-to-end flow of a user creating a post involves the app, load balancer, and the post service.

🔍 Additional work behind the scenes includes post moderation, enrichment, and storing metadata.

📌 The system design for a scalable news feed involves using a change data capture stream (CDC) to make the process asynchronous.

🔍 Different consumers, such as a moderation stream processor and an enrichment stream processor, can independently process the data from the CDC stream to perform various actions on the posts.

📲 When consuming posts in a feed, the client makes a get request to the feed service, which computes and returns the user's feed as a JSON payload.

📌 The feed system works by obtaining the user's friends' user ids and getting all their posts.

⚙️ The ranking service ranks the posts based on different conditions and sends them back to the feed service.

🔁 To optimize performance, a cache is added to the feed service to pre-compute and store the post ids for each user.

🔍 The cache is used to map user IDs to a list of post IDs, saving on bandwidth and storage.

⚡ The cache allows the feed service to quickly retrieve the list of post IDs and make HTTP calls to retrieve post metadata.

🔄 The cache significantly reduces the workload of the feed service and other services when serving a user's feed.

📌 The video discusses the design of a scalable news feed system similar to Instagram, Facebook, and Twitter.

🔁 The system includes a real-time feed stream processor that updates the feed cache for each user whenever a new post is created.

🔃 Additionally, there is a scheduled batch job that iterates through all users to update their feed cache periodically.

🔍 The system design for a scalable news feed involves multiple processors that moderate posts, extract metadata, and promote posts to relevant users.

⚡ Real-time processing is used for moderating and enriching posts, while offline processing is used for reformatting and recomputing user feeds.

🌐 The feed service utilizes a cache to quickly retrieve and deliver post details to users.