Nestimate

How I approached solving an interview task for autonomous driving from 3 different perspectives: RANSAC, PCA, and Ordinary Least Squares (OLS).

RANSAC, OLS, PCA: 3 Ways to Draw a Straight Line Across a Set of Points

Estimating IT projects is a pain. When I started my path and tried to estimate my time spent while being a developer, I always underestimated things. Every time there would appear a job I didn’t account for. Colleagues told me to multiply my estimates by 2, 3, the number Pi. Only it didn’t help to increase the estimation accuracy, just added other problems. For example, when I had to explain where the high numbers came from.

A Guide to Making Estimates for Software Development Projects

Ask any developer to estimate how long it will take for them to finish a project. You will see the loathing in their eyes. And for good reason. Estimates have been wrongly used for decades by a lot of managers who then hold the team accountable to their estimate as if it were the actual deadline. Even more frustrating, in many cases those managers only act on the lowest number they hear from you! It’s as if they had a min() function, and they just keep trying to get the number lower. But engineering is not magic!! No wonder the hashtag #NoEstimates has become famous.

How to Make Estimates Finally Useful to Developers

Hopefully I caught your attention with the controversial (and admittedly hyperbolic) title. Good. Let me restate it in a slightly more elegant and less clickbaity form:

You have to choose between Software Delivered on Time and Good Software

If you could travel back in time and change one thing in a delivered project, what would it be?

Decisions

‘When will it be done?’ is one of the most common and difficult questions to answer in software development.

Estimation — how can we estimate with confidence in software delivery?

This tutorial shows you how to build a WhatsApp bot that accepts images, analyzes them using AI vision models, and responds with intelligent insights. 

How to Build an AI-Powered WhatsApp Bot That Analyzes Images Using Python and Vision Models

AI’s biggest bottleneck isn’t code, it’s infrastructure. Power, compute and governance now define who can actually scale AI.

AI Power Isn’t Just About “Better Models.” It’s About Who Controls the Systems They Run On

Your AI model isn’t failing; your data is. Learn how clean, verified data improves model accuracy and how easy it is to fix with APIs.

Your AI Model Isn’t Broken. Your Data Is

Do valuations cause crashes? Use Causal AI & EODHD data to prove how profitability and beta drive downside risk during market shocks. Move beyond correlation.

I Built a Causal AI Model to Find What Actually Causes Stock Drawdowns

How can we know ours ads are making impact that we aim for? What if targeted ads are not working the way we want them to?

How to Use Propensity Score Matching to Measure Down Stream Causal Impact of an Event

A deep dive into the Internet Archive's custom tech stack.

The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting

Pepeto’s exchange-first model converts every trade into native token demand, creating compounding growth loops that speculation-only memecoins lack.

Next 100x Meme Coin? How Pepeto's Exchange-First Model Creates Compounding Demand

Learn to create reusable components in React for better code organization and maintenance, using ComponentProps and HTMLAttributes interfaces for adaptability..

Creating Effective And Optimized Next.JS Reusable Components.

Explore the transformative impact of machine learning on modern business growth strategies. 

Leveraging ML Algorithms for Business Growth Optimization

Building a web component that renders one picture at a time. 

A Simple Web Component for Building a Slideshow

Create an onboard screen with smooth animations in Flutter for both Android and iOS. Use shared preference to onboard only once, saving you and your users time.

How To Create An Onboard Screen In Android and IOS with Flutter?

RAIN can create a mental space to by practising awareness of our feelings, thoughts, and behaviours without being caught in their narrative. 

RAIN: How to Recognize, Allow, Investigate, and Nurture Emotions

By learning and applying new words and mental concepts, we will not simply react to what happens to us but stop, ask, revise.

Learning New Words Can Help You Recognize Your Emotions

Estimating the price of a home is both science and art. Home characteristics, such as square footage, location or the number of bathrooms, are given different weights according to their influence on home sale prices in each specific geography over a specific period of time, resulting in a set of valuation rules, or models that are applied to generate each home’s Zestimate. Specifically, some of the structured data that is used in the estimation include Physical attributes: Location, lot size, square footage, number of bedrooms and bathrooms and many other details. Tax assessments: Property tax information, actual property taxes paid, exceptions to tax assessments and other information provided in the tax assessors’ records. Prior and current transactions: Actual sale prices over time of the home itself and comparable recent sales of nearby homes.

Using object detection in home pictures for improving price estimation

There has been a high tide wave in the increment of fantasy sports app lovers all over the world. Fantasy sports app can be seen as a perfect partner for a sports lover who wants to get involved in the sports in any which way.

Top 10 Fantasy Sports App Development Companies

STEP 1: Create divs inside of your page that will contain the carousel elements

Creating a Carousel Using Only Jade(Pug), Javascript, jQuery and CSS

This is a Plain English Papers summary of a research paper called <a href="https://www.aimodels.fyi/papers/arxiv/scaling-embeddings-outperforms-scaling-experts-language-models?utm_source=hackernoon&utm_medium=referral">Scaling Embeddings Outperforms Scaling Experts in Language Models</a>. If you like these kinds of analysis, join <a href="https://www.aimodels.fyi/?utm_source=hackernoon&utm_medium=referral">AIModels.fyi</a> or follow us on <a href="https://x.com/aimodelsfyi">Twitter</a>.

<h2>Overview</h2>
<ul>
<li>Researchers compared two approaches to making language models more efficient: scaling up the embedding layer versus scaling up expert networks</li>
<li>Embedding scaling consistently outperforms expert scaling across different model sizes</li>
<li>The study introduces an n-gram embedding layer as an alternative architecture for handling vocabulary</li>
<li>Results suggest that how models process input words matters more than having specialized computational pathways</li>
<li>The findings challenge the current trend toward mixture-of-experts designs in large language models</li>
</ul>
<h2>Plain English Explanation</h2>
Language models need to handle millions of words, and this creates a real computational problem. The traditional solution involves storing a massive lookup table where each word gets mapped to a set of numbers that represent its meaning. As models grow larger, this lookup table becomes enormous.
Recently, many researchers have focused on a different approach: instead of making the lookup table bigger, they've been adding more specialized experts—think of these as different mini-networks that each handle specific types of inputs. The idea sounds good in theory. Different experts could specialize in different linguistic patterns, similar to how different people have different expertise.
This paper challenges that assumption. The researchers found something counterintuitive: making the lookup table smarter and larger actually works better than adding more experts. It's like discovering that investing in a better dictionary outperforms hiring specialists who only know particular subjects.
The researchers also developed a new way to organize the embedding layer using n-grams, which are small sequences of words. Instead of treating each word in isolation, this approach captures relationships between consecutive words. This turns out to be a more efficient use of computational resources than the expert-based approach that has become popular in recent years.
<h2>Key Findings</h2>
<ul>
<li>Embedding scaling consistently wins: Across all tested model sizes, scaling the embedding layer produces better performance per unit of computation than scaling expert networks</li>
<li>N-gram embeddings improve efficiency: The proposed n-gram embedding layer outperforms both standard embeddings and mixture-of-experts approaches</li>
<li>Timing matters for integration: The optimal point to integrate the n-gram embedding layer occurs at specific depths in the model architecture</li>
<li>Computational efficiency gains are substantial: The embedding approach achieves superior results while using fewer computational resources than comparable expert-based systems</li>
<li>The trend toward experts may be misguided: Current industry momentum toward <a href="https://aimodels.fyi/papers/arxiv/training-sparse-mixture-experts-text-embedding-models?utm_source=hackernoon&utm_medium=referral">mixture-of-experts designs</a> appears to overlook a simpler, more effective scaling direction</li>
</ul>
<h2>Technical Explanation</h2>
The paper presents a direct comparison between two architectural choices that have competed for attention in recent model development. The embedding layer sits at the front of the model and converts discrete word tokens into continuous numerical representations. The researchers scaled this component by increasing its dimensionality and capacity, allowing it to capture richer information about each word and its context.
The alternative approach uses mixture-of-experts (MoE) layers scattered throughout the model. These layers dynamically route different inputs to different computational pathways based on what needs to be processed. While this sounds efficient, the paper's experiments reveal it consumes more resources than simply improving the front-end representation.
The n-gram embedding innovation builds on this finding. Rather than treating words as isolated tokens, the system considers small sequences of words together. This allows the embedding layer to learn patterns that span multiple tokens, capturing linguistic structure more efficiently. The researchers tested where to place this layer within the model's architecture and found specific depths where it delivers maximum benefit.
The experimental design compared models across multiple scales, ensuring results held across different sizes rather than appearing only in specific configurations. This approach strengthens the findings since it demonstrates the scaling behavior across the range that matters for practical deployment.
These results have implications for <a href="https://aimodels.fyi/papers/arxiv/parameters-vs-flops-scaling-laws-optimal-sparsity?utm_source=hackernoon&utm_medium=referral">optimal sparsity in scaling laws</a>. The research suggests that current estimates about how to efficiently allocate compute may undervalue the contribution of input representation. When practitioners make <a href="https://aimodels.fyi/papers/arxiv/scaling-laws-across-model-architectures-comparative-analysis?utm_source=hackernoon&utm_medium=referral">architectural decisions informed by comparative scaling analysis</a>, they should weigh embedding improvements alongside expert multiplication.
<h2>Critical Analysis</h2>
The research presents compelling empirical results, but several limitations deserve consideration. The experiments measure performance on standard benchmarks, which may not capture all types of linguistic tasks. Some domains or languages might benefit from the specialized routing that experts provide, even if experts underperform on average.
The paper doesn't deeply explore why embedding scaling wins. Understanding the mechanism would strengthen the findings considerably. Does the n-gram approach work because it captures grammatical patterns? Or does success simply result from having more parameters in a position where they affect all tokens equally? The answer matters for predicting how this approach would perform on different types of data.
The computational efficiency claims merit scrutiny. The paper measures certain efficiency metrics, but implementation details matter enormously in practice. Experts might be more efficient on specialized hardware or with particular optimization techniques not covered in this analysis. Real-world deployment involves considerations beyond the academic comparison presented here.
There's also a question about whether embedding scaling remains superior when combined with other modern techniques. Researchers have been exploring <a href="https://aimodels.fyi/papers/arxiv/towards-greater-leverage-scaling-laws-efficient-mixture?utm_source=hackernoon&utm_medium=referral">leverage in scaling laws</a> through various architectural innovations. The paper's comparison focuses on these specific approaches in isolation, which may not reflect how they interact with other advances.
The findings challenge industry momentum, which makes external validation particularly important. Independent research groups should replicate these results across different training regimes and model families before the field completely shifts away from expert-based approaches. Premature consensus could waste resources if the findings don't generalize as expected.
<h2>Conclusion</h2>
This research provides evidence that the recent industry focus on mixture-of-experts architectures may have overlooked a simpler path to scaling language models efficiently. By investing in better input representations through embedding scaling and n-gram techniques, models achieve superior performance without the added complexity of routing mechanisms.
The practical implication is straightforward: teams building large language models should reconsider their architectural assumptions. Resources devoted to expert networks might deliver greater returns if redirected toward embedding innovation. This doesn't mean experts have no role, but their prominence in recent designs appears disproportionate to their actual contribution.
The broader lesson concerns how research directions become self-reinforcing. Once enough prominent projects adopt a particular approach, that approach gains legitimacy regardless of alternative solutions. This work demonstrates the value of stepping back and comparing foundational choices rather than following established momentum. For the field, it suggests that <a href="https://aimodels.fyi/papers/arxiv/scaling-laws-across-model-architectures-comparative-analysis?utm_source=hackernoon&utm_medium=referral">scaling laws across different architectures</a> deserve continuous re-examination as new techniques emerge.
The findings open space for further investigation into why input representation matters more than computational specialization, and whether hybrid approaches might combine the benefits of both strategies. As language models continue their trajectory toward larger scales, this distinction between scaling embeddings and scaling experts will likely become increasingly important for resource-conscious research teams and companies.

<hr/>Original post: <a href="https://www.aimodels.fyi/papers/arxiv/scaling-embeddings-outperforms-scaling-experts-language-models?utm_source=hackernoon&utm_medium=referral">Read on AIModels.fyi</a>

MoE Isn’t the Only Path: Bigger Embeddings Win on Efficiency

Nestimateis in the Investment Tech Industry

NESTIMATE

mynestimate.com

9 emps

Since 2022

Nestimate's stories on HackerNoon

RANSAC, OLS, PCA: 3 Ways to Draw a Straight Line Across a Set of Points

A Guide to Making Estimates for Software Development Projects

How to Make Estimates Finally Useful to Developers

You have to choose between Software Delivered on Time and Good Software

Decisions

Estimation — how can we estimate with confidence in software delivery?

How to Build an AI-Powered WhatsApp Bot That Analyzes Images Using Python and Vision Models

AI Power Isn’t Just About “Better Models.” It’s About Who Controls the Systems They Run On

Your AI Model Isn’t Broken. Your Data Is

I Built a Causal AI Model to Find What Actually Causes Stock Drawdowns

How to Use Propensity Score Matching to Measure Down Stream Causal Impact of an Event

The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting

Next 100x Meme Coin? How Pepeto's Exchange-First Model Creates Compounding Demand

Creating Effective And Optimized Next.JS Reusable Components.

Leveraging ML Algorithms for Business Growth Optimization

A Simple Web Component for Building a Slideshow

How To Create An Onboard Screen In Android and IOS with Flutter?

RAIN: How to Recognize, Allow, Investigate, and Nurture Emotions

Learning New Words Can Help You Recognize Your Emotions

Using object detection in home pictures for improving price estimation

Top 10 Fantasy Sports App Development Companies

Creating a Carousel Using Only Jade(Pug), Javascript, jQuery and CSS

MoE Isn’t the Only Path: Bigger Embeddings Win on Efficiency

Representing this company? Click here to claim and customize this page!

Reviews About Nestimate

Nestimate WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps