Job Summary:\n\nBrainstorm new and creative marketing ideas \u0026 Developing social media strategies\nResearch industry-related topics (combining online sources, interviews, and studies)\nManage social media campaigns on Facebook, Google Ad Words, email marketing and other media efficiently and economical\nPreparing weekly updates and reports to track growth and success rates of campaigns\nAbout the Internship\nSelected intern's day-to-day responsibilities include:\n\nHandle WordPress website development and design\nManage SEO, SMO, on-page \u0026 off-page SEO\nDevelop micro website\nDevelop a Vblog website and update\nPromote and advertise on various digital platforms\nKey Skills\nSocial Media Marketing, Facebook Ads, Digital Marketing\nBenefits\nMentorship, Incentives on a good performance, Flexible work hours, Education assistance\nContact: 7021198277

RankerZ Media

Ugh! To think that this article started from a rant about my K-drama search results. SMH. I’m thrilled that it did because it led me down the rabbit hole of Algorithms, Deep Personalization, Search optimization, and…wait for it… ! That site is to my brain what a pâtisserie is to my eyeballs and belly. I digress, but seriously, if you haven’t checked it out, you should. Netflix’s Research Library It is due to my curiosity that I decided to write this article regarding optimizing Netflix's ML ranker outcomes for a more efficient and accurate deep personalization experience of subscribers’ long and short-term viewing preferences — to address both Fetch and Explore intents. I found writing this article very interesting because it takes the business and customer angles, as well as the infrastructure, into consideration. Context Netflix seems to focus more on home-page recommendations that enhance users’ pre-query experience and less on post-search experience. To some extent, this is great because if it were to be the other way around from the beginning, we’d all have dropped Netflix years ago. After all, the user experience (UX) would have been a hot mess! According to Das et al. (2022) in their paper on , even though most of the content discovery happens on the home page, a large amount still comes from Search. So, I can’t help but wonder, if this is the case and Netflix has solidified its , how come members still don’t have some control over their experience, especially during Search? As a Product Manager, I understand focussing on features that affect more users, but sometimes, paying attention to the seemingly insignificant 20% that don’t seem like urgent user needs could lead to 80% of our desired product outcomes. query-facet mapping and its applications UX dominance amongst its competitors There’s been some recent about Netflix’s customer satisfaction ratings in the US/Canada region, and the data shows that there seems to be quite some churn in this region despite Netflix’s growing presence in . Nonetheless, it can be argued that that region was the primary market for the product. Thus, it’s a more mature market. Therefore, it is highly likely that most other markets will follow the same trajectory as the product/markets mature and other market players arise in the mid to long term. buzz other regions The way I see it, as stated in a tweet, “…Real value from AI always hinges on solving the hard 20%- and this almost always requires an entirely different set of approaches.” thus, Netflix must start to look into the features that they don’t focus much on — based on the evidence, that’d be query-related recommendations and Search. What if the “different set of approaches” is simple? Alex Ratner Let’s be real: Netflix’s recommender systems are SUPREME (All Hail!), and the recommendations on the homepage are great a lot of the time, but it becomes more stressful when I need to find films within a genre or category that falls within Netflix’s “movie genome” (“This show is …” e.g., Swoonworthy) because . From my research, which included an admittedly ‘leading’ and informal Twitter with a small sample size, interview sessions, and scouring tons of forums on the internet for reasons why people don’t like Netflix, recommendations and search relevancy seem to be recurring customer issues. These hangups don’t stop the users from clocking their “Hours spent watching…” metric because the recommendations are decent enough, but from the responses I got, it’s almost like a nagging but dull pain point that subscribers have. They know it can be better, but it works fine to accomplish their primary goal, and they’ve used Netflix for so long that it’s a pain that they have learned to deal with. the recommendations on the homepage are finite poll I want to be able to find more movies that meet my specific taste so that my recommendations are more accurately tailored to my preferences. As a subscriber, what do I want? According to their Research website, they want me, the subscriber, to spend less time searching and more time watching what I like. What does Netflix want? So, what if I’m able to input search components once in a blue moon, and my filters are so detailed that the queries finetune the recommender systems to provide better in-session pre-query recommendations during Fetch and Explore user intent actions such as idly scrolling/hovering through recommendations lists and typing to search respectively? For instance, this has happened to me too many times than I can count and apparently so many others experience the same; Scenario: Uchenna prefers to watch Korean TV shows in the romance genre that are swoonworthy, comedy and heartfelt. It doesn’t mean that if Netflix recommends a Chinese TV show that is in the romance genre with genome tags for ‘Bittersweet, Heartfelt and Emotional’, she wouldn’t watch it. She most likely would or she would have to scroll for several minutes looking for something that meets her current mood and preference. Netflix’s Problems In my humble opinion, these are Netflix’s problems; Firstly, some of its core metrics are extrinsically motivated, quite presumptive, and based on the premise that success is in the completion of the main goal, which is, watching something on the platform in video format. In a perfect product world with business KPIs at the forefront of decision-making, yes, this is right, but in a world with flawed humans with varying moods, preferences, and intents, this is a flawed and unempathetic approach — it does not take into account that I might enjoy watching something that was recommended to me, but it doesn’t mean that it is what I would usually go for or it’s what my mood calls for at the point of selection. Because Netflix is in the growth stage of the product lifecycle, it seems the current metrics are based on customer acquisition and retention without much focus on customer satisfaction. However, the bottom line is that I’m generally happy, but my need at that moment hasn’t been met. In addition, in a scenario where I have to find films with labels that are close to what I have in mind, it’d take longer for me to find something, which then explains why most people are likely to watch something that’s been recommended to them by a friend instead of searching or always following the recommendations. Secondly, another recurring complaint on the internet pertains to the tags. The genre tags are inconsistent and do not follow regular genres. In addition, the genome tags and categories seem to be inconsistent and disorganized as well. With inconsistent and scattered labeling of video content, the likelihood of the recommender systems producing pre-query and post-query recommendations that are exactly like the ‘previously watched’ content and/or the probability of accurately predicting the subscriber’s taste becomes diminished. Thus, Netflix has a great recommender systems infrastructure, but the labels aren’t organized properly enough to help the customer decide as well as help the systems make fully optimized anticipatory decisions. For instance, last night, I started watching a TV show with genome tags ‘Swoonworthy, comedy, quirky,’ and there’s absolutely nothing comedic about this show. It’s quirky and romantic but definitely not comedy—wrong labeling. Lastly, e-commerce, content-sharing, and video tech companies are beginning to optimize their Search function despite having great algorithms that predict content based on user preferences but Netflix’s search function barely even has any filters. I found lots of comments online during my research, but one of them seems to encapsulate everything I know is wrong with Netflix. Recommendations Option 1: Optimization of Search Filters and Creation of ‘Saved Search Labels Applying optimized search filters and member pre-selected ‘saved search’ results could help improve Netflix’s recommender systems’ outcomes for pre-query in-session and long-term recommendations. Using a saved search to establish a baseline for behavior and user interest. User interactions with the baseline recommendation results can then help predict user preferences within the confines of that saved search, e.g., a Saved search named Quirky K-Drama can show more related results as the user interacts with the recommendation results — ‘Because you saved Quirky K-Drama.’ Below is a design of the Advanced Search/Saved Search feature recommendation. Option 2: Creating Expandable Pre-Query Recommendations Carousel Lists ‘Because you watched…’ and ‘Because you liked…’ rows/ recommendations are finite, which puts a limit on the subscriber’s explore intent. Currently, the recommender system provides a carousel row for each and then more carousel rows for movies with similar genome tags to what you’ve watched in the past. This creates quite a lot of clutter and diminishes the user’s experience because they have to scroll down to view films within each genome tag. Instead of creating too many lists/carousels that make the user continuously scroll downwards and sideways to find videos within mislabeled genome tags, Netflix can optimize the ‘Because you liked…’ and ‘Because you watched…’ carousels by opening them up to become expendable. This way, there’s not much of a break in the user’s exploratory actions, thus creating a better experience for them. Option 3: Creating Genome Tags Ratings to Improve Labelling and Algorithmic Outcomes I’m thinking out loud (on paper), but just keep up with me, and hopefully, this written soliloquy will be coherent enough for you to understand the point that I’m trying to make. There’s no way that humans can possibly watch every video in the Netflix library to judge the genome tags and relabel them correctly. Besides, if this were possible, the labels might even be more disorganized because the perception of a video is, to some extent, subjective. Bringing that bias into the mix might make the data more inaccurate. But could there be a way for subscribers to rate a video’s genome tag accuracy? If this happens, the recommender systems can use the data to properly label the video content. For instance, Grammarly does something similar with its text tone detector ratings. I believe these ratings feed into their algorithm that helps predict and detect the tone of text. The drawback to this is that Netflix might have to introduce a more detailed rating system for videos, which is a feature that was discontinued many years ago. However, if there’s a way to introduce the genome tag rating system even for three months, I believe that predictions will be better. Open Beta testing provides the perfect environment and timeframe for Netflix to allow subscribers to help train the model for more accurate predictions. Potential Limitations in Implementing My Recommendations It is vital to figure out whether the cost of implementing these ‘delightful’ feature recommendations can justify the required investment because, at the end of the day, Netflix is a business, and cost-benefit analysis is a given. There was not much Netflix-specific data to analyze; thus, the cost-benefit analysis outcome is unknown, and the feasibility of implementing these suggestions cannot be determined with any certainty. Cost-Benefit Dilemma: Qualitative research for this article was done using a random sampling approach on a very small cumulative sample size of about 30 people. Other feedback gathered from different online forums might also be indicative of a recurring customer pain point. However, Netflix has over 200 million subscribers; thus, even though it may seem like these are widespread customer complaints, they might not necessarily be representative of the broader subscriber population size as the small sample size comes with the probability of a sampling error (Phew! What a long sentence). Anyway, only Netflix’s data can truly confirm the validity and importance of these issues to their users. Regardless, Hypothesis testing and further research would be required to know if these recommendations are worth implementing. Without testing, implementing these would be a shot in the dark. Risk of Sampling Error: These suggestions are presumptive that the features haven’t already come up on the Netflix Product team’s backlog or brainstorming board, and they either haven’t been prioritized or have been canned because they aren’t feasible. Potentially Erroneous Assumption of Priority: Recommendation Option 3 assumes that genome tags are determined by Netflix’s AI, however, there's a possibility that in reality, genome tags are added manually during the uploading process e.g. Think: How labels and categories are added to products in a POS system. If that's the case, then we are back to square one — human error due to subjectivity — this might account for discrepancies. However, on second thought, despite this change in scenario, I believe that the validity and viability of the genome tags ratings recommendation (Option 3) could very well still stand. Just think about it for a second. Unfortunately, unless you work at Netflix, this information is unknown, so let’s just stick with our imagination and inferences, okay? :) Reality-Hypothesis Scenario Divergence: Conclusion Netflix arguably has the best Recommender systems in the game; its subscriber growth rate is increasing at a healthy pace due to expansion strategies, and its UX is probably the best in the competitive landscape as well. I envisage that they will continue to hold on to the majority of the market share for at least the next five years. However, what happens if they eventually manage to capture the streaming markets in most target countries? What happens when they can't compete on price with local and international competitors? What happens when, like the US, most of these markets mature, and customers in these countries begin to look for better customer experiences with their video streaming? What happens when innovation in other streaming service companies, like Hulu, AppleTV, and HBO, outpace Netflix's holy grail recommender systems? Ultimately, in the words of Steven Van Belleghem , “What if customers want more than excellent service?” I'm sure the amazingly talented people of Netflix are already thinking about these potential issues. These are long-term questions, but I believe that in the meantime, it's time to tweak Netflix's recommender systems to harness its optimum potential for this much-loved service to provide much better customer satisfaction. Customer experience (CX) is currently Netflix’s strength; however, customer satisfaction is slipping. A great product can have amazing CX, but if the customers want more than just excellent service to stay satisfied, more will need to be done. This is where these recommendations come in. According to Former VP of Product at Netflix, Gibson Biddle, the Netflix product team is not just focused on understanding the customer’s current wants and needs but instead on ‘inventing and delivering on unanticipated future needs.’ I also came across an article in titled ‘Stop Trying to Delight Your Customers’, and one statement caught my attention, “According to conventional wisdom, customers are more loyal to firms that go above and beyond. But our research shows that .” Harvard Business Review exceeding their expectations during service interactions makes customers than simply meeting their needs only marginally more loyal Source: (Former VP of Product at Netflix ) Gibson Biddle Therefore, typical metrics for customer satisfaction, such as NPS, might not capture the sentiments observed during my research. Focus groups and interviews are the best research methods for figuring out such “nagging-but-dull” pain points that customers have, as they don't cause churn per se. However, like a toothache, over time, the dull pain points eventually might be the reason for customers to migrate to a different service provider. Thus, this is the reason why I strongly agree with Dixon et al. (2010) in their HBR article that states that Customer Satisfaction and Customer Retention/Loyalty are more likely to be captured using Customer Effort Score as opposed to Net Promoter Score or CSAT and that’s because you learn more when asking the main question in CES — “How much effort did it take the customer to achieve their desired goal.” I believe that if Netflix asks this question often, they will find the reason why customer experience (CX) is great but customer satisfaction (CS) and loyalty (especially in the US) are disproportionately lower, and the results would help deepen personalization efforts on the platform. In the context of Netflix, Deep personalization stands at the intersection between Customer Satisfaction, Customer Experience, Customer Loyalty, and an optimized AI system. Supercharge Deep Personalization using features based on insights from research methods such as CES, the ML ranker becomes more efficient in determining user preferences. The hypotheses in this article address business problems and user problems and encourage revisiting the current product metrics at Netflix. It’s a win-win for all, right? Well, it’s up to you to decide. In a nutshell, there’s still a lot that the Netflix ML/AI systems are capable of. There’s still a lot that Netflix can do better to acquire and retain subscribers if they ask the right questions and implement features that reduce the users’ effort, create a better experience, and enhance the efficiency of the recommender systems. Thus far, they have done a great job, but it can certainly be better. The question is whether the cost of enhancement is commensurate with the benefit of a more efficient platform. What are your thoughts on this subject? Do the pain points resonate with you? Also published . here

Enhancing Netflix's Deep Personalization: The Full Potential Of Its Current AI Recommender Systems

How I Got My Startup Funded By Jason Calacanis

’s feed has come a long way since the early days of assembling the machine-learning infrastructure that powers it. Recently, a major update to this infrastructure was released. We caught up with the people behind it to discuss how the principle of being people-centric translates to technical terms and implementation. LinkedIn How do data and machine learning-powered algorithms work to control newsfeeds and spread stories? How much of that is automated, how much should you be able to understand and control, and where is it all headed? That was the introduction to my , exploring what was then a newly revamped LinkedIn newsfeed in terms of how the underlying machine learning algorithm worked and what the principles governing it were. Six years is a long time in technology. Back then, it was the early days of machine learning (aka “AI”) entering the mainstream. Today we live in the ChatGPT era. 2017 article for ZDNet In the last year interest in AI has peaked, and it seems like there are new developments almost on a weekly basis. LinkedIn could not possibly have stayed behind, and its models and infrastructure have evolved significantly. Recently, LinkedIn Engineering published a blog post titled “ “. machine-learning Enhancing homepage feed relevance by harnessing the power of large corpus sparse ID embeddings This blog post is a deep dive into LinkedIn’s newly revamped machine learning models and infrastructure that power its newsfeed. Unlike 2017, these changes are not directly perceptible. There are no new features per se, but rather, improvements in the underlying technology with the goal of enhancing the feed’s relevance. We caught up with Jason Zhu, Staff Software Engineer, and Tim Jurka, Senior Director of Engineering at LinkedIn, to discuss how the principle of being people-centric translates to technical terms. Click to listen to the conversation on the here Orchestrate all the Things podcast Human in the loop The gist of the update is that LinkedIn’s model, powering the feed, can now handle a larger number of parameters, resulting in higher-quality content delivery. In the process, Zhu’s team also ended up upgrading the hardware infrastructure that powers the model. Zhu is a member of the LinkedIn Foundational AI Technologies team. He defines the team’s mission as “prototyping and building the foundations behind those advanced algorithms that can benefit multiple vertical use cases at LinkedIn”. Zhu provided a good overview of LinkedIn’s previous major update to the feed in 2021, how that led to the latest update, as well as insights into the inner workings of LinkedIn’s approach. However, we were also interested in finding out what else has changed compared to LinkedIn’s 2017 incarnation of the feed. Most notably, LinkedIn’s spokespeople in 2017 emphasized the , and we were wondering if that is still a thing. As Jurka put it, it’s definitely still a thing, probably even more of a thing now than it was then: human-in-the-loop approach “It boils down to two applications for the feed. The first is we’re really trying to prioritize insightful, authentic, knowledge-oriented content in the LinkedIn feed, and it’s a very hard problem to get right just with AI alone. But we have an entire team of award-winning journalists led by our editor-in-chief, Dan Roth. We’ve actually created AI algorithms that train on the content that they curate and believe is really the cream of the crop in terms of LinkedIn insights. That AI algorithm then tries to say – Would an editor promote this on LinkedIn’s platform? And if so, let’s try to distribute on the platform and see which audiences resonate with that particular piece of content. The second is just with large language models, really taking the world by storm. The way that we make those high quality is by fine-tuning them using techniques like reinforcement learning with human feedback. And so large language models, again, are presenting yet another way where human loop is much more deeply integrated into the AI workflow versus where it was maybe five or six years ago”, Jurka said. The human-in-the-loop approach also applies to spam detection. While LinkedIn employs AI-powered spam detection, there is also the option for users to report spam as well as content moderators to look out for anything the AI may have missed. Jurka went on to add that as far as spam detection for the feed is concerned, some things have stayed the same, while others have changed. The three-bucket classification system described in 2017 has not changed much. What has changed is that LinkedIn has invested a lot more in deeper content understanding, aiming to understand the intent behind each post. For example, is it somebody trying to share a job opportunity? Is it somebody trying to share their opinion about the news? Getting more granular about the content helps LinkedIn know how to distribute it across the platform. From linear to neural, from single to multiple points of view As Jurka shared, the evolution of the feed ranking algorithm really follows the evolution of the LinkedIn feed. Compared to 2017, there are . A lot of the complexity is about making sure that the AI models can capture all those diverse use cases. And this is where understanding intent is also relevant. more members using the feed and more content creators publishing on LinkedIn “For example, you might have a post from a member mentioning that they raised their Series A for their startup, and now they’re hiring. And that post can be perceived through three very different lenses. If you’re a job seeker, that post might actually be an entry point for you to reach out and get a job, and the AI model has to understand that value to you. If you’re a first-degree connection of that individual, it might be just to congratulate them and say, Hey, I’m just checking to see you raised Series A. Congrats. If it’s somebody who’s in the venture capital space, this is actually an insight where they are like, I didn’t know that this particular company raised Series A”, Jurka said. Picking up on that, Zhu explained that LinkedIn’s feed contains a heterogeneous list of updates from members, connections and a variety of recommendations such as jobs, people, and articles. The objective is to provide a personalized ranked list for each member, to help professionals on the platform be more productive and successful. To achieve this, LinkedIn has adopted a two stage ranking process. There is something called the first pass ranker which is applied for each type of update. So the top-k candidates of each type of update are selected by individual algorithms before being sent to the second pass ranker for a final green ranking. LinkedIn’s models are presented with a set of features from both member and content sets. The model is trying to predict a set of likelihood of responses such as the likelihood of starting a conversation or the likelihood of performing certain actions on a particular post. Previously, LinkedIn transitioned its ranking from a linear-based model to a deep learning / neural network-based model. Zhu noted that the deep learning model is more effective because it introduces non-linearities through activation functions. That enables the models to capture more complex and non-linear relationships in the data and learn more powerful representations. Instead of an individual linear model to predict each different response separately, a multitask deep learning model shares parameters between different tasks. So this learning of each task can benefit the other tasks through transfer learning, Zhu said. However, he added, LinkedIn’s data is highly skewed between different types of responses. There are more click responses than engagement, for example. Therefore, cautious sampling and re-weighting is needed to avoid negative task interference during the model training process, because all those tasks are sharing parameters. Harnessing the power of large corpus sparse ID embeddings Zhu introduced LinkedIn’s previous major update of the feed, . As he shared, LinkedIn’s model size has grown by a factor of 500, while the data sets have also grown by a factor of 10. In addition, the training period was expanded and more training data is now sampled. This also presents some challenges in terms of overcoming bias. namely using multi-task learning using TensorFlow However, this was just the starting point for the latest update Zhu’s team implemented. Their objective was to optimize for engagement, which means improving the prediction of likelihood in terms of how users will interact with different posts. Then by enhancing the prediction of this likelihood, LinkedIn can better rank feed items to provide more engaging experience to users. The key to this was sparse . Let’s see what these are and how they are relevant. embedding vectors As a LinkedIn user interested in AI, you may have interacted with the #AI and #ML hashtags. Another user has interacted with the #generativeAI hashtag. These are different hashtags. However, via embeddings, their semantic meanings in the embedding space are close. As a result, the model can understand you may have similar tastes as mine, and the model can recommend similar content in the AI domain to both users. For each content item on LinkedIn, and for each user, the recommendation system uses a vector of embeddings. This means that those vectors will be very large, but also very sparse. They will be large, i.e. have many dimensions, because there are many different topics and corresponding hashtags. But they will also be sparse because not all users have interacted with all hashtags. Zhu’s focus is on transforming large corpus sparse ID features (such as hashtag ID or item/post ID) into embedding space using embedding lookup tables with hundreds of millions of parameters trained on multi-billions of records. For example, Zhu noted, that there are millions of dimensions for member IDs because there are millions of members on LinkedIn. “For each member the goal is to learn a personalized tensor representation. This tensor representation will encode user preferences, such as interaction with other users or hashtag preferences in a fixed period. All this information will be encoded in a dense vector during training. By presenting the model with such a personalized vector, the model will be able to capture the dynamic, shifting world as well as member preferences better”, Zhu said. Sparse features can be interpreted as categorical one-hot representations where the cardinality is several millions, Zhi noted. In these representations, all entries are zero except one, which corresponds to an ID index. That could be a hashtag ID or a member ID, for example. These are transformed into a low dimensional continuous dense space, called the embedding space, using embedding lookup tables with hundreds of millions of parameters trained on multibillion records. As Zhu noted, the embedding table itself is highly coupled with the data set and the problem at hand. However, the technique is definitely generalizable. The same approach is already leveraged in different use cases within LinkedIn such as job recommendation and Ads recommendation. “We are adopting a very similar strategy of converting those sparse IDs to the dense representation. We believe that’s a scalable way of developing AI at LinkedIn”, Zhu said. Scaling up, going forward Zhu’s work also touched upon hardware upgrades. As he explained, the team was facing challenges in serving dozens of large models in parallel on their serving host after enlarging the model size by 100x. The host was just not designed to be able to handle such a scale, so the team chose to serve the model with a two-stage strategy. In the first stage, a global model is trained in the usual way. ID features are converted to dense vectors from large embedding tables first and then these dense vectors are sent to a deep along with existing features for implicit feature interactions. neural network In the second stage, the global model is split at the boundary of the embedding table and deep neural network. From the deep neural network’s perspective, it only needs a set of dense vectors, which is the embedding representation of those sparse features for predicting the result. The ID conversion step is not necessary to present it on the critical path of model serving. Embeddings are converted offline and pushed to high-performance key-value stores. Therefore, these embedding tables don’t have to be hosted in the serving host which has limited memory. This strategy is used while in parallel LinkedIn is working on upgrading the host. Once the upgrade is complete, Zhu said, everything will be moved in memory and memory optimizations will be applied. The team adopted and extended the open-source framework . Horovod Zhu identified a few directions his team would like to explore going forward. In addition to serving both memory-intensive and compute-intensive models, scaling model size by adding more sparse features is another objective. The team is also exploring something called continuous training or incremental training to capture the dynamics of a shifting world. As Zhu explained, embeddings currently are trained on a fixed period basis. This could potentially introduce the problem of inconsistent offline vs. online results or diminished gains over a long period of time. Zhu thinks that by having more frequent retrieval, embeddings can capture the dynamics of the system and better predict a personalized feed for LinkedIn members. Also published . here Click to listen to the conversation on the here Orchestrate all the Things podcast

LinkedIn Feed Evolution: More Granular and Powerful Machine Learning, Humans Still in the Loop

Imagine writing a piece of software that could understand, assist, and even generate code, similar to how a seasoned developer would. Well, that’s possible with . Leveraging advanced models such as VectorStores, Conversational RetrieverChain, and LLMs, takes us to a new level of code understanding and generation. LangChain LangChain In this guide, we will reverse engineer Twitter’s recommendation algorithm to better understand the code base and provide insights to craft better content. We’ll use OpenAI’s embedding technology and a tool called to make the code understandable, and an LLM hosted on called to converse with the code. Activeloop DeepInfra Dolly When we’re done, we’re going to be able to shortcut the difficult work it will take to understand the algorithm by asking an AI to give us answers to our most pressing questions, rather than spending weeks sifting through it ourselves. Let’s begin. A Conceptual Outline for Code Understanding with LangChain is a very helpful tool that can analyze code repositories on GitHub. It brings together three important parts: VectorStores, Conversational RetrieverChain, and an LLM (Language Model) to assist you with understanding code, answering questions about it in context, and even generating new code within GitHub repositories. LangChain The Conversational RetrieverChain is a system that helps find and retrieve useful information from a VectorStore. It uses smart techniques like context-aware filtering and ranking to figure out which code snippets and information are most relevant to the specific question or query you have. What sets it apart is that it takes into account the history of the conversation and the context in which the question is asked. This means it can provide you with high-quality and relevant results that specifically address your needs. In simpler terms, it’s like having a smart assistant that understands the context of your questions and gives you the best possible answers based on that context. Now, let’s look into the LangChain workflow and see how it works at a high level: Index the code base The first step is to clone the target repository you want to analyze. Load all the files within the repository, break them into smaller chunks, and initiate the indexing process. If you already have an indexed dataset, you can even skip this step. Embedding and Code Store To make the code snippets more easily understandable, LangChain employs a code-aware . This model helps in capturing the essence of the code and stores the embedded snippets in a VectorStore, making them readily accessible for future queries. embedding model In simpler terms, LangChain uses a special technique called code-aware embedding to make code snippets easier to understand. It has a model that can analyze the code and capture its important features. Then, it stores these analyzed code snippets in a VectorStore, which is like a storage place for easy access. This way, the code snippets are organized and ready to be quickly retrieved when you have queries or questions in the future. Query Understanding This is where your LLM comes into play. You can use a model like to process your queries. The model is used to analyze your queries and understand the meaning behind them by considering the context and extracting important information. By doing this, the model helps LangChain accurately interpret your queries and provide you with precise and relevant results. databricks/dolly-v2–12b Construct the Retriever: Once your question or query is clear, the Conversational RetrieverChain comes into play. It goes through the VectorStore, which is where the code snippets are stored, and finds the code snippets that are most relevant to your query. This search process is very flexible and can be customized to fit your requirements. You have the ability to adjust the settings and apply filters that are specific to your needs, ensuring that you get the most accurate and useful results for your query. Build the Conversational Chain Once you have set up the retriever, it’s time to build the Conversational Chain. This step involves adjusting the settings of the retriever to better suit your needs and applying any additional filters that might be required. By doing this, you can narrow down the search and ensure that you receive the most precise, accurate, and relevant results for your queries. Essentially, it allows you to fine-tune the retrieval process to obtain the information that is most useful to you. Ask questions: Now comes the exciting part! You can ask questions about the codebase using the ConversationalRetrievalChain. It will generate comprehensive and context-aware answers for you. Your LLM, being part of the Conversational Chain, takes into account the retrieved code snippets and the conversation history to provide you with detailed and accurate answers. By following this workflow, you’ll be able to effectively use LangChain to gain a deeper understanding of code, get context-aware answers to your questions, and even generate code snippets within GitHub repositories. Now, let’s see it in action, step by step. Step-by-Step Guide Let’s dive into the actual implementation. Acquiring the Keys To get started, you’ll need to register at the respective websites and obtain the API keys for Activeloop, DeepInfra, and OpenAI. Setting up the Indexer.py file Create a Python file, e.g., indexer.py, where you’ll index the data. Import the necessary modules and set the API keys as environment variables: import os
from langchain.document_loaders import TextLoader
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import DeepLake
os.environ['OPENAI_API_KEY'] = 'YOUR KEY HERE'
os.environ['ACTIVELOOP_TOKEN'] = 'YOUR KEY HERE'
embeddings = OpenAIEmbeddings(disallowed_special=()) Embeddings, in plain English, are representations of text that capture the meaning and relatedness of different text strings. They are numerical vectors, or lists of numbers, that are used to measure the similarity or distance between different text inputs. Embeddings are commonly used for various tasks such as search, clustering, recommendations, anomaly detection, diversity measurement, and classification. In search, embeddings help rank the relevance of search results to a query. In clustering, embeddings group similar text strings together. Recommendations leverage embeddings to suggest items with related text strings. Anomaly detection uses embeddings to identify outliers with little relatedness. Diversity measurement involves analyzing the distribution of similarities among text strings. Classification utilizes embeddings to assign text strings to their most similar label. The distance between two embedding vectors indicates how related or similar the corresponding text strings are. Smaller distances suggest high relatedness, while larger distances indicate low relatedness. Cloning and Indexing the Target Repository Next, we’ll clone the Twitter algorithm repository, load, split, and index the documents. You can clone the algorithm from . this link root_dir = './the-algorithm'
docs = []
for dirpath, dirnames, filenames in os.walk(root_dir):
for file in filenames:
try:
loader = TextLoader(os.path.join(dirpath, file), encoding='utf-8')
docs.extend(loader.load_and_split())
except Exception as e:
pass This code is traversing through a directory and its subdirectories (os.walk(root_dir)). For each file it encounters (filenames), it attempts to perform the following steps: It creates a TextLoader object, specifying the path of the file it is currently processing and setting the encoding to UTF-8. (os.path.join(dirpath, file)) It then calls the method of the TextLoader object, which likely reads the contents of the file, performs some processing or splitting operation, and returns the resulting text data. load_and_split() The obtained text data is then added to an existing list called docs using the method. extend() If any exception occurs during this process, it is caught by the try-except block and simply ignored (`pass`). Basically, this code snippet is recursively walking through a directory, loading and splitting text data from files, and adding the resulting data to a list called docs. Embedding Code Snippets Next, we use OpenAI embeddings to embed the code snippets. These embeddings are then stored in a VectorStore, which will allow us to perform an efficient similarity search: from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(docs)
username = "mikelabs" # replace with your username from app.activeloop.ai
db = DeepLake(dataset_path=f"hub://{username}/twitter-algorithm", embedding_function=embeddings, public=True) #dataset would be publicly available
db.add_documents(texts)print(“done”) This code imports the class and initializes an instance of it with a chunk size of 1000 characters and no overlap. It then splits the provided docs into smaller text chunks using the split_documents method and stores them in the texts variable. CharacterTextSplitter Next, it sets the username (the one you used to sign up for Activeloop!) and creates a DeepLake instance called db with a dataset path pointing to a publicly available dataset hosted on “app.activeloop.ai” under the specified username. The embedding_function handles the embeddings needed. Finally, it adds the texts to the db using the add_documents method, presumably for storage or further processing purposes. Run the file, then wait a few minutes (it may appear to hang for a bit… usually no more than 5 minutes). Then, on to the next step. Utilizing dolly-v2–12b to Process and Understand User Queries Now we set up another Python file, question.py, to use , a language model available in the platform, to process and understand user queries. dolly-v2–12b DeepInfra Constructing the Retriever We construct a retriever using the VectorStore we created earlier. db = DeepLake(dataset_path="hub://mikelabs/twitter-algorithm", read_only=True, embedding_function=embeddings) #use your username
retriever = db.as_retriever()
retriever.search_kwargs['distance_metric'] = 'cos'
retriever.search_kwargs['fetch_k'] = 100
retriever.search_kwargs['maximal_marginal_relevance'] = True
retriever.search_kwargs['k'] = 10 Here’s a breakdown of what the code is doing: The code initializes a DeepLake object called db. It reads the dataset from the path specified as “hub://mikelabs/twitter-algorithm”. It’s worth noting that you need to replace “mikelabs” with your own username! The db object is then transformed into a retriever using the as_retriever() method. This step allows us to perform search operations on the data stored in the VectorStore. Several search options are customized by modifying the dictionary: retriever.search_kwargs The is set to 'cos', indicating that cosine similarity will be used to measure the similarity between text inputs. Imagine you have two vectors representing different pieces of text, such as sentences or documents. Cosine similarity is a way to measure how similar or related these two pieces of text are. distance_metric To calculate cosine similarity, we look at the angle between the two vectors. If the vectors are pointing in the same direction or are very close to each other, the cosine similarity will be close to 1. This means that the text pieces are very similar to each other. On the other hand, if the vectors are pointing in opposite directions or are far apart, the cosine similarity will be close to -1. This indicates that the text pieces are very different or dissimilar. A cosine similarity of 0 means that the vectors are perpendicular or at a 90-degree angle to each other. In this case, there is no similarity between the text pieces. In the code above, cosine similarity is used as a measure to compare the similarity between text inputs. It helps determine how closely related two text pieces are. By using cosine similarity, the code can rank and retrieve the top matches that are most similar to a given query. The parameter is set to 100, meaning that the retriever will retrieve the top 100 closest matches based on cosine similarity. fetch_k The is set to , suggesting that the retriever will prioritize diverse results, rather than returning highly similar matches. maximal_marginal_relevance True The parameter is set to 10, indicating that the retriever will return 10 results for each query. k Building the Conversational Chain We use the ConversationalRetrievalChain to link the retriever and the language model. This enables our system to process user queries and generate context-aware responses: model = DeepInfra(model_id="databricks/dolly-v2-12b")
qa = ConversationalRetrievalChain.from_llm(model,retriever=retriever) The ConversationalRetrievalChain acts as a connection between the retriever and the language model. This connection allows the system to handle user queries and generate responses that are aware of the context. Asking Questions We can now ask questions about the Twitter algorithm codebase. The answers provided by the ConversationalRetrievalChain are context-aware and directly based on the codebase. questions = ["What does favCountParams do?", ...]
chat_history = []

for question in questions:  
    result = qa({"question": question, "chat_history": chat_history})
    chat_history.append((question, result['answer']))
    print(f"-> **Question**: {question} \n")
    print(f"**Answer**: {result['answer']} \n") Here’s some example questions, taken from the LangChain docs: questions = [
"What does favCountParams do?",
"is it Likes + Bookmarks, or not clear from the code?",
"What are the major negative modifiers that lower your linear ranking parameters?",
"How do you get assigned to SimClusters?",
"What is needed to migrate from one SimClusters to another SimClusters?",
"How much do I get boosted within my cluster?",
"How does Heavy ranker work. what are it’s main inputs?",
"How can one influence Heavy ranker?",
"why threads and long tweets do so well on the platform?",
"Are thread and long tweet creators building a following that reacts to only threads?",
"Do you need to follow different strategies to get most followers vs to get most likes and bookmarks per tweet?",
"Content meta data and how it impacts virality (e.g. ALT in images).",
"What are some unexpected fingerprints for spam factors?",
"Is there any difference between company verified checkmarks and blue verified individual checkmarks?",
] And here’s a sample answer that I got back: **Question**: What does favCountParams do? 
**Answer**: FavCountParams helps count your favorite videos in a way that is friendlier to the video hosting service (i.e., TikTok). For example, it skips counting duplicates and doesn't show recommendations that may not be relevant to you. Resources Here are some additional resources you might find helpful: Activeloop documentation AIModels.fyi LangChain guides OpenAI embeddings documentation Conclusion Throughout this guide, we explored reverse engineering Twitter’s recommendation algorithm using LangChain. By leveraging AI capabilities, we save valuable time and effort, replacing manual code examination with automated query responses. LangChain is a powerful tool that revolutionizes code understanding and generation. By using advanced models like VectorStores, Conversational RetrieverChain, and an LLM hosted on a service like DeepInfra, LangChain empowers developers to efficiently analyze code repositories, provide context-aware answers, and generate new code. LangChain’s workflow involves indexing the code base, embedding code snippets, processing user queries with language models, and utilizing the Conversational RetrieverChain to retrieve relevant code snippets. By customizing the retriever and building the Conversational Chain, developers can fine-tune the retrieval process for precise results. By following the step-by-step guide, you can leverage LangChain to enhance your code comprehension, obtain context-aware answers, and even generate code snippets within GitHub repositories. LangChain opens up new possibilities for productivity and understanding. What will you build with it? Thanks for reading! Also published . here

What The Tweet? Reverse-Engineering the Twitter Algorithm with LangChain

FORTUNE TELLING

In the ever-evolving digital landscape, search engines play an increasingly crucial role in powering search functionalities across various platforms. Among the popular search engines, and stand out with their unique offerings. Meilisearch Manticore Search However, choosing the right search engine for your project requires a thorough understanding of their performance, use cases, and limitations. This article aims to provide a comparison of Meilisearch and Manticore Search, focusing on their feature set and data ingestion and search performance in three real-world benchmarks: 10 million NGINX logs, Hacker News 1.1 million docs dataset, and Hacker News 116 million docs dataset all available at . All the performance test scripts, configurations and the data collections are and reproducible. DB Benchmarks publicly available Full-text Search Relevance Both Manticore and Meilisearch position themselves as full-text search engines. The key element in full-text search engines is how they rank documents during a search. Choosing the right search ranking algorithm is crucial to ensure users can find the information they need with precision and recall. In the context of full-text search relevance, it is essential to understand how these algorithms work and how they contribute to providing accurate and meaningful search results. Manticore Search is very flexible in controlling search ranking and exposes dozens of ranking factors; however, by default, it employs the classical BM25 algorithm and its derivatives. is a well-established information retrieval algorithm that calculates the relevance of documents based on term frequency and inverse document frequency. BM25 An for the BEIR (Benchmarking and Evaluation of Information Retrieval) benchmark demonstrates Manticore Search’s commitment to search relevance. BEIR is an evaluation framework that measures the performance of information retrieval systems on various tasks, such as document retrieval and question-answering. The results of the BEIR benchmark can be found here: ongoing pull request . https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZqX14nmCCPVlE_y3a_y5KkYI/edit#gid=0 In contrast, Meilisearch claims to offer good search relevance, but there are no public benchmarks available to substantiate this assertion. According to a discussion on , Meilisearch users have mentioned its search relevance, but without any empirical evidence, it is difficult to compare its performance to Manticore Search objectively. Hacker News Overall, Manticore Search’s use of proven ranking algorithms and participation in the BEIR benchmark highlights its commitment to providing highly relevant search results, making it a reliable choice for various applications. While Meilisearch may excel at full-text search relevance too, it is difficult to make a definitive statement since there are no established benchmarks and the algorithm used is not widely known. Index Size and Data Ingestion Manticore Search demonstrates its ability to effectively handle large datasets (e.g. or simply ) through the use of row-wise and columnar storages. The columnar approach is specifically designed to accelerate search performance and lower RAM consumption on large datasets. In contrast, Manticore Search’s default row-wise storage offers unbeatable performance on small and medium datasets. This flexibility makes Manticore Search an ideal choice for a wide range of applications. 1.7 billion docs taxi rides test Craigslist.org Meilisearch, on the other hand, struggles with larger datasets, as we could not load the into the search engine even after 2 days of loading. Furthermore, Meilisearch experiences a degradation in performance when loading documents. As the dataset grows, the time it takes to load each subsequent batch of documents increases. This performance issue indicates that Meilisearch has a problem with data scalability and could be problematic for applications that require real-time data ingestion or indexing of large datasets. Meilisearch processes document updates in a single queue, which can lead to bottlenecks and reduced performance over time. Hacker News larger dataset It is crucial to note that document updates in Meilisearch are not instantly reflected in search queries. This is because Meilisearch employs an asynchronous task queue for handling updates, ensuring search performance remains stable even during intensive indexing operations. When updating a document, the change is added to the task queue and processed by the engine in the background. Once the task is completed, the updated data becomes available in the search results. The processing time can vary depending on the update size and server resources. To monitor task status, you can utilize the , which offers information on task progress and completion. Tasks API Manticore offers real-time inse rt, replace, and delete capabilities, allowing changes to be immediately visible as soon as the query is complete. In summary, while Meilisearch provides fast and efficient search capabilities, keep in mind that updates to documents might not be immediately visible in search results due to the asynchronous task processing. Search Performance Meilisearch is known for its impressive speed, . However, its performance is most noticeable when working with small datasets. As the dataset size increases, Meilisearch’s performance may decline. outperforming Elasticsearch in many cases Manticore Search consistently delivers fast query performance for various query types and dataset types, outperforming both Meilisearch and . With optimized row-wise and columnar indexing methods, Manticore ensures a responsive search experience, crucial for maintaining user engagement in high-performance applications. Elasticsearch In contrast, Meilisearch struggles with efficiently handling large datasets and suffers from performance degradation during document loading. Therefore, Manticore is the superior choice for those who don’t want to worry about their dataset size. Benchmark Tests Hacker News Small Dataset (Hacker News Comments) The Hacker News small dataset benchmark, which features a collection of 1.1 million curated Hacker News comments with numeric fields (source: ), highlights the higher search performance of Manticore Search over Meilisearch. The dataset contains textual data from comments and numeric fields such as upvotes, timestamps, and user IDs. The benchmark test involves running full-text and analytical queries to assess the search engines’ capabilities. https://zenodo.org/record/45901/ The benchmark results can also be verified through . this link Unfortunately, Meilisearch is not capable of executing many types of queries, such as aggregation queries and those with negative full-text search terms. An interesting aspect of this benchmark is the significant difference in disk space usage between the two search engines: [email protected] /perf/test_engines/tests/hn_small/manticore # du -sh idx
1.1G    idx


[email protected] /perf/test_engines/tests/hn_small/meilisearch # du -sh .
38G     . Meilisearch requires more disk space to store the same dataset compared to Manticore Search. 34x In terms of data loading performance it took: Meilisearch 31 minutes Manticore 65 seconds to fully complete data loading. Hacker News Large Dataset (116 million comments) This test involves the same 1.1 million curated Hacker News comments dataset (source: ), but multiplied 100 times, resulting in about . The benchmark covers both full-text and analytical queries, making it an excellent test case for evaluating search engine capabilities on a larger scale. https://zenodo.org/record/45901/ 116 million documents Meilisearch couldn’t load the data in 2 days. Its performance of inserts degraded as the database grew. We attempted to optimize it but were unsuccessful since all batches, even when we tried to make them parallel, went into a single queue. As a result, we couldn’t achieve any improvement in data loads for Meilisearch. It took Meilisearch about 2 days to load only 38% of the data, which already consumed over 850 GB of disk space. This is a stark contrast to Manticore Search, which stored the entire dataset using approximately 100 GB of disk space and took 2 hours 9 minutes to load using a single CPU core (which is virtually linearly scalable). The inability of Meilisearch to process the entire Hacker News large dataset highlights its challenges in managing and scaling with more extensive data collections. Manticore Search’s superior performance in this benchmark underscores its capacity to handle large-scale search requirements, making it a more suitable choice for applications with larger data collections. Since we couldn’t load the data into Meilisearch, you can check the Manticore-only results . here 10 million NGINX logs This test is based on a dataset containing 10 million NGINX logs. The source of this dataset is . Web server logs register various events, providing valuable insights into website visitors, user behavior, crawlers accessing the site, business intelligence, security issues, and more. The benchmark uses a curated list of typical queries that a random DevOps engineer might run. Kaggle Manticore Search and Meilisearch exhibited a significant difference in disk space usage for the dataset. Manticore Search used 4.4 GB of disk space, while Meilisearch consumed 69 GB, which is approximately than Manticore. Although the difference is less dramatic than the Hacker News small dataset test, it is still noteworthy, especially considering the Logs10m dataset contains less text data. 15 times more It took Meilisearch around to fill up the data, whereas Manticore finished in . 20 minutes 6 minutes You can find the detailed comparison of the performance results using the . Please take note that many empty results are simply due to Meiliesarch being unable to handle certain types of queries. As a result, these queries were skipped during the benchmarking process. provided link Manticore Search Features Comparison of and Meilisearch Full-text matching ✅ Manticore: over 20 full-text operators. Percolate search (search in reverse). ❌ Meilisearch: very simple: AND and phrase search. No percolate search. Search Relevance ✅ Manticore employs tried-and-proven classical ranking algorithms (BM25, BM15). The relevance is benchmark-proven. 7 built-in rankers and a custom ranker with . 20+ ranking factors ❌ Meilisearch claims good search relevance but lacks public benchmarks for validation. . 6 ranking rules Storage ✅ Manticore: own row-wise storage for small/medium datasets, own columnar storage with lower RAM requirements suitable for larger datasets ❌ Meilisearch: with all its advantages, disadvantages, and consequences: e.g., 205GB virtual memory requirement for a 9.1 MB dataset seems odd. LMDB Index Size and Data Loading ✅ Manticore accommodates large datasets with columnar and row-wise indexing methods. Easily sync data from MySQL, PostgreSQL, MS SQL, and any other database that supports ODBC, XML, and CSV. True real-time transactional inserts, replaces, and deletes. Binary log. In-place attribute value updates. ❌ Meilisearch has difficulty with larger datasets and experiences performance degradation during document loading. You can upload CSV and JSON. Only asynchronous addition of documents. No in-place updates. Schema ✅ Manticore: Auto-schema. Auto-ID. All attributes are filterable, sortable, and groupable by default. ❌ Meilisearch: Auto-schema. ID can be picked automatically from the document. All fields are full-text searchable by default, but attributes are not filterable or sortable. You must decide on the schema before loading data into the index to avoid full reindexing. Search Performance ✅ Manticore outperforms Meilisearch in search performance. ❌ Meilisearch is less suitable for applications requiring fast and scalable search functionality. High availability ✅ Manticore: replication, distributed tables supporting remote agents with mirroring, and several HA strategies. ❌ Meilisearch: no replication, no distributed searching, no mirroring. Typo Tolerance ✅ Meilisearch offers easier typo tolerance. ❌ Manticore can handle typo tolerance but demands a higher effort in the app. Search preview ✅ Meilisearch features a helpful search preview - a built-in UI for searching through data in the instance. ❌ Manticore does not have this feature. Tokenization ✅ Manticore: highly flexible tokenization: token chars, blended chars, ignored chars, regular expression tokenization rules, etc., wordforms, stopwords, synonyms, option to create tokenization plugins, morphology for various languages based on stemmers and lemmatizers. ❌ Meilisearch: tokenizer depends on the language: Unicode segmenter for most languages, specific tokenizers for Chinese, Japanese, Hebrew, and Thai. Synonyms. Stopwords. Authentication ✅ Meilisearch: built-in authentication. ❌ Manticore: no built-in authentication. Interfaces ✅ Manticore: SQL-first, you can connect using a MySQL client. HTTP JSON interface. Binary interface for extremely low response times. Clients for: PHP, Python, JavaScript, Java, C#, Elixir, Golang. ❌ Meilisearch: HTTP JSON interface. Clients for: JavaScript, Python, PHP, Java, Ruby, Golang, C#, Rust, Swift, Dart. Use Cases ✅ Manticore: log search, e-commerce platforms, content-rich websites, enterprise applications. ❌ Meilisearch: small-scale projects with limited data and search requirements. Use Cases Use cases for Manticore Search Manticore Search can efficiently manage large product catalogs, providing relevant search results for customers with its . This improves conversion rates and enhances the overall shopping experience, making it a highly sought-after feature for e-commerce platforms. E-commerce platforms: advanced faceted functionality Manticore Search can index and search through extensive content libraries, such as news sites, blogs, or knowledge bases. With proper full-text ranking, it ensures users find the information they need quickly and effectively, contributing to higher user engagement. Content-rich websites: Manticore Search’s scalability and advanced search capabilities make it ideal for large-scale enterprise applications, including customer relationship management (CRM) systems, document management systems, and intranet portals, where accurate and efficient search functionality is critical. Enterprise applications: Manticore Search is great for searching in logs, as it can efficiently handle and search through huge logs. Its speed and performance make it an excellent choice for log analysis and monitoring. Log search: Use cases for Meilisearch Small-scale projects: Meilisearch’s lightweight nature and ease of deployment make it suitable for small projects with limited data and search requirements, such as small-scale e-commerce, personal websites, local directories, or simple web applications, where fast data loading, advanced search features and scalability are not critical factors. Conclusion When choosing a search engine for your project, it is crucial to consider factors such as search relevance, scalability, and performance. Manticore Search stands out as the superior choice for diverse applications and use cases, ensuring optimal search performance and relevance regardless of dataset size. Its advanced search and analytics capabilities make it a reliable choice for projects that demand high-performance search functionality. Meilisearch is suitable for small projects where advanced search features and scalability are not critical factors. Ultimately, the choice between Manticore Search and Meilisearch will depend on your specific needs and project requirements. Also published here.

RANKERZ MEDIA

rankerzmedia.com

-- employees

Since n.d.

COMPANY RANKING

RANKERZ MEDIA

EVERGREEN INDEX #8966

Representing this company? Click here to claim and customize this page!

RankerZ Media WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

RANKERZ MEDIA

rankerzmedia.com

-- employees

Since n.d.

COMPANY RANKING

RANKERZ MEDIA

EVERGREEN INDEX #8966

RELATED COMPANIESRANK

STORIES ABOUT RANKERZ MEDIA

Representing this company? Click here to claim and customize this page!

RankerZ Media WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!