Tables nan kontni Overview System Architecture Components Home Mixer Thunder Phoenix Candidate Pipeline How It Works Pipeline Stages Scoring and Ranking Filtering Key Design Decisions Pwoteksyon Algorithm la pou ou feed retire, ranpli, ak filtre pòs soti nan de sous: In-Network (Thunder): Post soti nan kont ou swiv Out-of-Network (Phoenix Retrieval): Posts detekte soti nan yon korpus mondyal Tou de sous yo konbine ak rankontre ansanm lè l sèvi avèk , yon modèl transformatè ki baze sou Grok ki prezante probabilite angaje pou chak pòsyon. Pousantaj final la se yon konbinezon pwa nan angaje sa yo prezante. Phoenix Nou te elimine chak karakteristik hand-engineered ak pi heuristik soti nan sistèm la. transformatè a ki baze sou Grok fè tout soulaj la difisil pa konprann istwa angaje ou (ki sa ou te renmen, reponn, pataje, elatriye) ak lè l sèvi avèk sa a yo detèmine ki kontni se enpòtan pou ou. Architecture nan sistèm ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │ FOR YOU FEED REQUEST │ └─────────────────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │ HOME MIXER │ │ (Orchestration Layer) │ ├─────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ QUERY HYDRATION │ │ │ │ ┌──────────────────────────┐ ┌──────────────────────────────────────────────┐ │ │ │ │ │ User Action Sequence │ │ User Features │ │ │ │ │ │ (engagement history) │ │ (following list, preferences, etc.) │ │ │ │ │ └──────────────────────────┘ └──────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ CANDIDATE SOURCES │ │ │ │ ┌─────────────────────────────┐ ┌────────────────────────────────┐ │ │ │ │ │ THUNDER │ │ PHOENIX RETRIEVAL │ │ │ │ │ │ (In-Network Posts) │ │ (Out-of-Network Posts) │ │ │ │ │ │ │ │ │ │ │ │ │ │ Posts from accounts │ │ ML-based similarity search │ │ │ │ │ │ you follow │ │ across global corpus │ │ │ │ │ └─────────────────────────────┘ └────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ HYDRATION │ │ │ │ Fetch additional data: core post metadata, author info, media entities, etc. │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ FILTERING │ │ │ │ Remove: duplicates, old posts, self-posts, blocked authors, muted keywords, etc. │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ SCORING │ │ │ │ ┌──────────────────────────┐ │ │ │ │ │ Phoenix Scorer │ Grok-based Transformer predicts: │ │ │ │ │ (ML Predictions) │ P(like), P(reply), P(repost), P(click)... │ │ │ │ └──────────────────────────┘ │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌──────────────────────────┐ │ │ │ │ │ Weighted Scorer │ Weighted Score = Σ (weight × P(action)) │ │ │ │ │ (Combine predictions) │ │ │ │ │ └──────────────────────────┘ │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌──────────────────────────┐ │ │ │ │ │ Author Diversity │ Attenuate repeated author scores │ │ │ │ │ Scorer │ to ensure feed diversity │ │ │ │ └──────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ SELECTION │ │ │ │ Sort by final score, select top K candidates │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ FILTERING (Post-Selection) │ │ │ │ Visibility filtering (deleted/spam/violence/gore etc) │ │ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │ RANKED FEED RESPONSE │ └─────────────────────────────────────────────────────────────────────────────────────────────┘ konpozan Home Mizik Location: home-mixer/ Lay la òkestrasyon ki monte Feed pou ou. Li sèvi ak ranje ak etap sa yo: CandidatePipeline Stage Description Query Hydrators Fetch user context (engagement history, following list) Sources Retrieve candidates from Thunder and Phoenix Hydrators Enrich candidates with additional data Filters Remove ineligible candidates Scorers Predict engagement and compute final scores Selector Sort by score and select top K Post-Selection Filters Final visibility and dedup checks Side Effects Cache request info for future use Rechèch pou Hydratè Fetch kontèks itilizatè a (historik angaje, lis ki anba a) sous Retrieve kandida soti nan Thunder ak Phoenix Hydratè Enrich kandida ak done adisyonèl Filtre Retire kandida ki pa kwalifye Score nan Prezante angaje ak kalkil pousantaj final Seleksyon Soti nan pousantaj ak chwazi tèt K Post-seleksyon Filtre Final vizibilite ak dedup tcheke Efè Side Cache mande enfòmasyon pou itilize nan pwochen sèvè a ekspoze yon fenèt gRPC ( ) ki retire posts ranked pou yon itilizatè bay. ScoredPostsService Soumèt Location: thunder/ Yon magazen pòs nan memwa ak reyèl-tan piki pipeline ki swiv post ki sot pase soti nan tout itilizatè yo. Li: Konsome pòs kreye / retire evènman soti nan Kafka kenbe magazen pou chak itilizatè pou pòs orijinal, repons / reposts, ak videyo pòs Serves "in-network" post kandida soti nan kont ki itilizatè a mande swiv Otomatikman retire pòs ki pi ansyen pase peryòd retansyon Thunder pèmèt sub-milisekond rechèch pou kontni nan rezo san yo pa frape yon baz done ekstèn. Phoenix nan Location: phoenix/ konpozan ML ak de fonksyon prensipal: 1. Retrieval (Two-Tower modèl) Jwenn posesyon ki enpòtan soti nan rezo: User Tower: Kode karakteristik itilizatè ak istwa angaje nan yon embedding Candidate Tower: Kode tout pòs yo nan embedings Rechèch Similarity: Retrieves Top-K posts via dot pwodwi similite 2. Ranking (Transformateur ak izolasyon kandida) Prezante chans yo angaje pou chak kandida: Li pran konteks itilizatè (historik angaje) ak posesyon kandida kòm entwodiksyon Sèvi ak atansyon espesyal maske pou kandida yo pa kapab atann nan youn ak lòt pwodiksyon probabilite pou chak kalite aksyon (tankou, repons, repost, klike, elatriye) nan Pou dokimantasyon detaye nan enstriktè. phoenix/README.md Kandida nan Pipeline Location: candidate-pipeline/ Yon anviwònman re-usab pou bati pipelines rekòmandasyon. Defini karakteristik pou: Trait Purpose Source Fetch candidates from a data source Hydrator Enrich candidates with additional features Filter Remove candidates that shouldn't be shown Scorer Compute scores for ranking Selector Sort and select top candidates SideEffect Run async side effects (caching, logging) Source Fetch kandida soti nan yon sous done Hydrator Enrich kandida ak karakteristik adisyonèl Filter Retire kandida ki pa ta dwe montre Scorer Kalkile pousantaj pou rankings Selector Rated ak chwazi kandida tèt SideEffect Run asynchronous efè segondè (Caching, logging) Framework la kouri sous ak idratè paralèlman kote posib, ak konfigirab manyen erè ak logging. Ki jan li travay Etap nan Pipeline Query Hydration: Retounen istwa a ak metadata nan angajman ki sot pase a nan itilizatè a (pou egzanp lis sa a) : Retrieve candidates from: Candidate Sourcing : Recent posts from followed accounts (in-network) Thunder : ML-discovered posts from the global corpus (out-of-network) Phoenix Retrieval : Enrich candidates with: Candidate Hydration Core post data (text, media, etc.) Author information (username, verification status) Video duration (for video posts) Subscription status : Remove posts that are: Pre-Scoring Filters Duplicates Too old From the viewer themselves From blocked/muted accounts Containing muted keywords Previously seen or recently served Ineligible subscription content : Apply multiple scorers sequentially: Scoring : Get ML predictions from the Phoenix transformer model Phoenix Scorer : Combine predictions into a final relevance score Weighted Scorer : Attenuate repeated author scores for diversity Author Diversity Scorer : Adjust scores for out-of-network content OON Scorer Seleksyon: Sort pa pousantaj ak chwazi kandida yo K tèt Post-selection Processing: Validasyon final nan post kandida yo dwe serve Ranje ak Ranking Yon modèl transformatè ki baze sou Phoenix Grok prezante probabilite pou plizyè kalite angaje: Predictions: ├── P(favorite) ├── P(reply) ├── P(repost) ├── P(quote) ├── P(click) ├── P(profile_click) ├── P(video_view) ├── P(photo_expand) ├── P(share) ├── P(dwell) ├── P(follow_author) ├── P(not_interested) ├── P(block_author) ├── P(mute_author) └── P(report) nan Konbine sa yo nan yon pousantaj final: Weighted Scorer Final Score = Σ (weight_i × P(action_i)) Aksyon pozitif (tankou, repòte, pataje) gen pwa pozitif. Aksyon negatif (blòk, mute, rapòte) gen pwa negatif, pouse kontni ki itilizatè a ta ka pa renmen. Filtrasyon Filtre kouri nan de etap: Pre-Scoring Filters: Filter Purpose DropDuplicatesFilter Remove duplicate post IDs CoreDataHydrationFilter Remove posts that failed to hydrate core metadata AgeFilter Remove posts older than threshold SelfpostFilter Remove user's own posts RepostDeduplicationFilter Dedupe reposts of same content IneligibleSubscriptionFilter Remove paywalled content user can't access PreviouslySeenPostsFilter Remove posts user has already seen PreviouslyServedPostsFilter Remove posts already served in session MutedKeywordFilter Remove posts with user's muted keywords AuthorSocialgraphFilter Remove posts from blocked/muted authors DropDuplicatesFilter Remove duplicate post id CoreDataHydrationFilter Retire pòs ki pa te kapab hidrate metadata debaz AgeFilter Retire pòs ki pi ansyen pase prag SelfpostFilter Remove itilizatè pwòp posts RepostDeduplicationFilter Dedupe reposts nan menm kontni IneligibleSubscriptionFilter Remove paywalled kontni itilizatè pa ka jwenn aksè PreviouslySeenPostsFilter Remove posts itilizatè a te deja wè PreviouslyServedPostsFilter Retire pòs ki deja te serve nan sesyon MutedKeywordFilter Remove posts ak keywords nan itilizatè a muted AuthorSocialgraphFilter Remove posts from bloke / muted otè Post-Selection Filters: Filter Purpose VFFilter Remove posts that are deleted/spam/violence/gore etc. DedupConversationFilter Deduplicate multiple branches of the same conversation thread VFFilter Retire pòs ki te retire/spam/violence/gore elatriye DedupConversationFilter Deduplikasyon plizyè branch nan menm konvèsasyon thread Klavye konsepsyon Decisions 1. Pa gen karakteristik Hand-Engineered Sistèm la depann konplètman sou transformatè a ki baze sou Grok yo aprann relektivite soti nan sekans entèaktif itilizatè. Pa gen enjenyè karakteristik manyen pou relektivite kontni. Sa a diminye konplisite nan pipelines done nou yo ak enfrastrikti sèvis. Candidate Isolation nan Ranking Pandan konvèsyon transformer, kandida yo pa ka atake nan youn ak lòt - sèlman nan konte a itilizatè. Sa asire pousantaj la pou yon pòs pa depann sou ki lòt pòs yo nan batch la, fè pousantaj yo konsistan ak cacheable. 3. Embeddings ki baze sou hash Tou de retrieval ak ranking sèvi ak fonksyon miltip hash pou entegre rechèch 4. Multi-aksyon pratik Plis pase predi yon sèl "relevance" pousantaj, modèl la prezante probabilite pou plizyè aksyon. 5. Composable Pipeline Arkitektur nan crate bay yon anviwònman fleksib pou bati pipelines rekòmandasyon ak: candidate-pipeline Separe egzekisyon pipeline ak kontwole soti nan lojik biznis Ekzekisyon paralèl nan etap endividyèl ak manyen erè graciously Fasil ajoute nouvo sous, idrasyon, filtè ak pousantaj Remak: Implemantasyon an nan transformatè a se pòte soti nan vèsyon an open source Grok-1 pa xAI, adapte pou ka itilize sistèm rekòmande. Remak: Implemantasyon an nan transformatè a se pòte soti nan vèsyon an open source Grok-1 pa xAI, adapte pou ka itilize sistèm rekòmande. Grok-1 Release nan sous louvri Pwojè sa a se lisans anba lisans la Apache 2.0. Pwojè sa a se yon lisans anba . Apache lisans 2.0
