Hysen Labs
Open-source project
twitter/the-algorithm avatar
twitter

the-algorithm

GitHub describes it as Source code for the X Recommendation Algorithm. The repository metadata lists Scala as its primary language. The metadata lists the AGPL-3.0 license. This article stays within the project description and details documented in the GitHub repository README.

73,676 stars13,253 forksScalaAGPL-3.0
01
DEEP OPEN-SOURCE ANALYSIS

twitter/the-algorithm: X's Recommendation Algorithm

GitHub describes it as Source code for the X Recommendation Algorithm. The repository metadata lists Scala as its primary language. The metadata lists the AGPL-3.0 license. This article stays within the project description and details documented in the GitHub repository README.

02
DEEP OPEN-SOURCE ANALYSIS

Repository scope

GitHub describes it as Source code for the X Recommendation Algorithm. The repository metadata lists Scala as its primary language. The metadata lists the AGPL-3.0 license. The README describes the project this way: X's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of posts and other content across all X product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our engineering blog.

03
DEEP OPEN-SOURCE ANALYSIS

Architecture

The README section "Architecture" states: Product surfaces at X are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:

04
DEEP OPEN-SOURCE ANALYSIS

Architecture

The README section "Architecture" states: | Type | Component | Description | |------------|------------|------------| | Data | tweetypie | Core service that handles the reading and writing of post data. | | | unified-user-actions | Real-time stream of user actions on X. | | | user-signal-service | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. | | Model | SimClusters | Community detection and sparse embeddings into those communities. | | | TwHIN | Dense knowledge graph embeddings for Users and Posts. | | | trust-and-safety-models | Models for detecting NSFW or abusive content. | | | real-graph | Model to predict the likelihood of an X User interacting with another User. | | | tweepcred | Page-Rank algorithm for calculating X User reputation. | | | recos-injector | Streaming event processor for building input streams for GraphJet based services. | | | graph-feature-service | Serves graph features for a directed pair of users (e.g. how many of User A's following liked posts from User B). | | | topic-social-proof | Identifies topics related to individual posts. | | | representation-scorer | Compute scores between pairs of entities (Users, Posts, etc.) using embedding similarity. | | Software framework | navi | High performance, machine learning model serving written in Rust. | | | product-mixer | Software framework for building feeds of content. | | | timelines-aggregation-framework | Framework for generating aggregate features in batch or real time. | | | representation-manager | Service to retrieve embeddings (i.e. SimClusers and TwHIN). | | | twml | Legacy machine learning framework built on TensorFlow v1. |

05
DEEP OPEN-SOURCE ANALYSIS

Architecture

The README section "Architecture" states: The product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.

06
DEEP OPEN-SOURCE ANALYSIS

Editorial conclusion

The repository README is the source for this review. It does not replace a local installation or an independent test.

07
DEEP OPEN-SOURCE ANALYSIS

Official sources

08
Community notes

Community notes