Dr. V K Vijayakumar, Chief Investment Strategist at Geojit Financial Services, indicated that the market is expected to remain in a consolidation phase in the short term. A positive sign is the halt in continuous selling by foreign institutional investors (FIIs), which may boost confidence ...

Stocks to buy or sell: Osho Krishan of Angel One suggests buying HAL, Jio Financial Services today | Stock Market News

Hindustan Aeronautics Limited (HAL) has appointed Barenya Senapati as its new Director (Finance). Senapati has 29 years of experience and previously held the position of Executive Director (Finance).

HAL Appoints Barenya Senapati as New Director (Finance): Rediff Moneynews

HAL appoints Barenya Senapati as Finance Director, bringing extensive experience in accounting, financial planning, and treasury management.

Hindustan Aeronautics Ltd (HAL) has announced the appointment of Barenya Senapati as the Finance Director. - The Hindu BusinessLine

The Defence Ministry has approved a Rs 60,000 crore upgrade of India&#x27;s Su-30MKI fighter jet fleet, led by <strong>HAL</strong> and supported by DRDO. This project aims to enhance the aircraft&#x27;s capabilities with new radars, mission control systems, and weapon systems, reflecting India&#x27;s push for self-reliance ...

Sukhoi fighter jet: IAF's Sukhoi fighter jets to get a Rs 60,000 crore booster: Here are all the upgrades for the new Su-30MKI - The Economic Times

<strong>HAL</strong> Dividend 2024, Hindustan Aeronautics Dividend Record Date: The counter has a 52-week high of Rs 3,130 and a 52-week low of Rs 1,237.

Rs 22 Dividend: HAL shares fall over 1% as stock trades ex-date today | Zee Business

<strong>HAL</strong> dividend record date 2024: <strong>HAL</strong> had announced an interim dividend of 440 per cent, or Rs 22 per equity share earlier this month. ​Regarding the record date for <strong>HAL</strong>&#x27;s interim dividend, the corporation has said that, &quot;as informed earlier, record date for the payment of first interim dividend ...

HAL Shares Nearing Dividend Record Date; Know Last Date To Avail 440 pc Monetary Reward | Markets News, Times Now

HAL is willing to recruit qualified applicants for the position of GDMO for a period of 2 years. There are 02 vacant seats for the mentioned post.

HAL Recruitment 2024: Notification Out, Check Post, Qualification, Salary and Process to Apply

The Defence Acquisition Council (DAC) led by Defence Minister Rajnath Singh has given the green light to capital acquisition proposals worth Rs 84,560 crore to enhance the strength of the Armed Forces and Indian Coast Guard. The approval emphasizes the nations commitment to self-reliance by ...

HAL, BEL, BDL: Defence Stocks To Be In Focus Next Week; A Rs 80,000 Crore Order May Drive Bulls! - Goodreturns

Since I started working for , I have tried to deepen my understanding of REST via various means. Did you read my review of ? Apache APISIX API Design Patterns book In the current literature, REST is generally promoted as the best thing since sliced bread. Yet, it comes with lots of challenges. In 2010(!), Martin Fowler wrote a post on the . He lists three steps for an API to become REST: glory of REST truly In each of these steps, issues lurk. This blog post focuses on listing some of them and providing hints at ways to solve them. Resources REST emerged from the cons of SOAP. SOAP provides a single endpoint and executes code depending on the payload. The idea of REST is to provide multiple endpoints, which each executes different code. I'll be honest; there are a few issues at this stage. The biggest one relates to guessing one identity from an existing one. If resource ids are sequential or even only numeric, it's easy to guess other resources' endpoints, , from to . e.g. /customers/1 /customers/2 The solution is to use non-sequential non-numeric ids, , . i.e. Universally unique identifiers Let's walk up the REST maturity model. HTTP Verbs HTTP verbs are the next step toward the glory of REST. They come from interactions with HTML "back in the day". Interactions came from Create Read Update Delete(CRUD) operations. It's pretty straightforward: Operation Verb Create POST Read GET Update PUT PATCH Delete DELETE The main problem with APIs is that you need to go beyond CRUD. Let's imagine a concrete example with a bank transfer: it takes money from one account and moves it to another one. How shall we model it? We could use the origin account as the resource, , . The target account, the amount, etc., can be passed as query parameters or in the body. But what HTTP verb shall we use? e.g. /accounts/a1b2c3d4e5f6 It changes the identified resource indeed, but it has "side-effects": it also changes another resource, the target account. Here are a couple of options on how to manage the HTTP verb: Use because it changes the source resource. It's misleading because it doesn't tell about side effects. POST Use a dedicated HTTP verb, , . It's not self-explanatory and is opposite to REST principles. e.g. TRANSFER Use with a so-called . Custom methods are a Google : POST custom method API Improvement Proposal Custom methods should only be used for functionality that can not be easily expressed via standard methods; prefer standard methods if possible, due to their consistent semantics. The HTTP URI use a character followed by the custom verb. must : Here's our bank account transfer URI: . /accounts/a1b2c3d4e5f6:transfer What's the best alternative? "It depends". Hypermedia Fowler describes Hypermedia Controls as the ultimate step to reaching the glory of REST. It's nowadays known as Hypermedia As The Engine of Application State(HATEOAS): With HATEOAS, a client interacts with a network application whose application servers provide information dynamically through hypermedia. A REST client needs little to no prior knowledge about how to interact with an application or server beyond a generic understanding of hypermedia. -- HATEOAS on Wikipedia HATEOAS is a concept; here's a possible implementation taken from Wikipedia. When one requests a bank account, say , the response contains links to actions possible with this specific bank account: /accounts/a1b2c3d4e5f6 {
  "account": {
    "account_number": "a1b2c3d4e5f6",
    "balance": {
      "currency": "USD",
      "value": 100.00
    },
    "links": {
      "_self": "/accounts/a1b2c3d4e5f6",
      "deposit": "/accounts/a1b2c3d4e5f6:deposit",
      "withdrawal": "/accounts/a1b2c3d4e5f6:withdrawal",
      "transfer": "/accounts/a1b2c3d4e5f6:transfer",
      "close-request": "/accounts/a1b2c3d4e5f6:close-request"
    }
  }
} If the balance is negative, only the deposit link will be available: {
  "account": {
    "account_number": "a1b2c3d4e5f6",
    "balance": {
      "currency": "USD",
      "value": 100.00
    },
    "links": {
      "_self": "/accounts/a1b2c3d4e5f6",
      "deposit": "/accounts/a1b2c3d4e5f6:deposit",
    }
  }
} A common issue with REST is the lack of standards; HATEOAS is no different. The first attempt to bring some degree of standardization was the , HAL. Note that it was incepted in 2012; the latest version dates from 2016, and it's still in draft. JSON Hypertext Application Language aka Here's a quick diagram that summarizes the proposal: We can rework the above with HAL as the following: GET /accounts/a1b2c3d4e5f6 HTTP/1.1
Accept: application/hal+json

HTTP/1.1 200 OK
Content-Type: application/hal+json

{
  "account": {
    "account_number": "a1b2c3d4e5f6",
    "balance": {
      "currency": "USD",
      "value": 100.00
    },
    "_links": {                                    <1>
      "self": {                                    <2>
        "href" : "/accounts/a1b2c3d4e5f6",
        "methods": ["GET"]                         <3>
      },
      "deposit": {
        "href" : "/accounts/a1b2c3d4e5f6:deposit", <4>
        "methods": ["POST"]                        <3>
      }
    }
  }
} Available links Link to self Tell which HTTP verb can be used Link to deposit Another attempt at standardization is , Web Linking. It describes the format and contains a , , and . The most significant difference with HAL is that RFC 8288 communicates links via HTTP response headers. RFC 8288 aka link relationship registry e.g. alternate copyright HTTP/2 200 OK

Link: </accounts/a1b2c3d4e5f6> rel="self";
                               method="GET",                           <1>
      </accounts/a1b2c3d4e5f6:deposit> rel="https://my.bank/deposit";
                                       title="Deposit";
                                       method="POST"                   <2>
{
  "account": {
    "account_number": "a1b2c3d4e5f6",
    "balance": {
      "currency": "USD",
      "value": 100.00
    }
  }
} Link to the current resource with the non-standard relation type self Link to deposit with the extension relation type and an arbitrary target attribute https://my.bank/deposit title Other alternative media types specifications are available. Name Description Provided by Uniform Basis for Exchanging Representations The UBER document format is a minimal read/write hypermedia type designed to support simple state transfers and ad-hoc hypermedia-based transitions. This specification describes both the XML and JSON variants of the format and provides guidelines for supporting UBER-encoded messages over the HTTP protocol. Individuals Collection+JSON Collection+JSON is a JSON-based read/write hypermedia-type designed to support the management and querying of simple collections Individual JSON: API JSON: API is a specification for how a client should request that resources be fetched or modified, and how a server should respond to those requests. JSON: API can be easily extended with extensions and profiles. Individuals Siren Siren is a hypermedia specification for representing entities. As HTML is used for visually representing documents on a Web site, Siren is a specification for presenting entities via a Web API. Siren offers structures to communicate information about entities, actions for executing state transitions, and links for client navigation. Individual Application-Level Profile Semantics An ALPS document can be used as a profile to explain the application semantics of a document with an application-agnostic media type (such as HTML, HAL, Collection+JSON, Siren, etc.). This increases the reusability of profile documents across media types. IETF Bonus: HTTP Response Status What Fowler's post doesn't mention is the HTTP response status. Most readers are familiar with the status ranges: Informational responses: 100 – 199 Successful responses: 200 – 299 Redirection messages: 300 – 399 Client error responses: 400 – 499 Server error responses: 500 – 599 Likewise, most are also with regularly-found HTTP status: , 200 OK , 301 Moved Permanently , 302 Found , 401 Unauthorized , 403 Forbidden and 404 Not Found . 500 Internal Server Error The problem is that beyond these simple cases, it's a mess. For example, look at this : "Which HTTP status code means Not Ready Yet, Try Again Later?" Here is a summary of the proposed answers, from the most upvoted to the lowest: StackOverflow question 503 Service Unavailable 202 Accepted (accepted answer) 423 Locked 404 Not Found 302 Found 409 Conflict 501 Not Implemented (downvoted) It's not a straightforward answer; there was a lot of debate around the alternatives. For the record, I think the accepted answer is the right one. That's already a lot on the designer side, but the client side contains a lot of uncertainty, too, as some big APIs providers use their . own HTTP status codes Conclusion The "glory of REST" doesn't mean much. There's no univocal semantics to rely on, despite any opposite claim. As it is, it depends mainly on the implementation and interpretation: both require documenting the custom behavior instead of relying on a shared specification. SOAP's biggest flaw was its complexity and its focus on big companies, but it at least provided a shared set of standard specifications. The industry replaced it with REST, not a specification but an architectural site. REST is simpler and, thus, more approachable, but it requires a lot of custom effort, which changes from project to project. There are initiatives to provide some standardization, but they are few, and some are at odds with others. Moreover, they have low traction, so people don't know them, which creates a vicious circle. I hardly advocate getting back to SOAP, though I sure miss it sometimes. Originally published at on January 22nd, 2023 A Java Geek

The Glory of REST: Is It Really That Glorious? 

The story goes that Caesar, to exchange secret messages with his generals in the field, used to shave his messengers bald and write the message on their scalps. He would then wait a few months for their hair to grow back and send them off to their destinations. Only his trusted generals would know of this method to exchange messages, thus delivering his orders securely (albeit a few months late). This impractical, and rather silly, example describes what cryptology is all about. Cryptology is the science of securing communication. The term is taken from the Greek ("hidden, secret") and ("word"). In essence, it is about "hiding words". It is only one pillar in the science of Cybersecurity, but it is a significant one. At the base of this pillar are the subdivisions of cryptology: Cryptography and Cryptanalysis. The two can almost be seen as complementary. Where cryptography is the practice of transforming communications into a form that is unreadable to eavesdroppers, cryptanalysis is the practice of trying to unravel those transformed messages by an unintended recipient. kryptós lógos And it is here, at the base of this pillar, where one proof of a problem could cause the collapse of Cybersecurity. So, enter the Pantheon, this temple of Cybersecurity if you will, and let me guide you to where the cracks in the Corinthian columns begin to show. There are many different algorithms designed to keep messages secure. I'm going to tell you a bit about some of those systems so that the foundation of these algorithms becomes clear. If you've read anything about cryptography, chances are you've come across a concept called the "Caesar cipher". This is another example of a method used to secure messages. It is also attributed to Caesar, although this one doesn't involve a messenger's locks (pun intended) to keep the message safe. Instead, this cipher introduces an important concept that a significant portion of cryptographic algorithms are based on: Modular Arithmetic. I know, I know. It sounds scary, right? What if I told you, it's so simple that you use it every day? See, your clock is based on modular arithmetic as well. When you're trying to figure out how much sleep you're going to get if you have to wake up at 06:00, as you're finishing up that deadline at 23:00, you know not to count past 24:00. The result is always going to be less than 24:00. You're essentially calculating modulo 24. This is modular arithmetic, and it is used extensively in cryptography. Even in its simplicity, though, modular arithmetic holds many secrets that aren't immediately evident. Many cryptographic algorithms rely on these secrets. Caesar's cipher is a symmetric cipher. This means that both parties use the same key. Let's say the key is 3. The sender would shift every letter of his message by 3. This is called encryption. The receiver needs to know the key, and would then shift back every letter by 3, obtaining the original message. This is called decryption. Compare this to the bald messenger method. Encryption is letting the messenger's hair grow over the message, decryption is shaving the messenger bald. Now, is the Caesar cipher very secure? Well, not really. Anyone can easily try all the different keys, of which there are 26 in total, and find the correct one to unravel the message. This is called a Brute Force Attack. While the cipher is not very secure, it does adhere to an important principle that Caesar's bald messengers do not adhere to. Kerckhoffs principle states: "A cipher should not require secrecy, and it should not be a problem if it falls into enemy hands." In other words, the principle states that only the key should remain secret. The method may be made public, but without the key, the message should remain secure. If Caesar's formerly bald messenger gets intercepted by the enemy and the interceptor is aware of the method used by Caesar, the poor messenger will be shaved bald again in no time, revealing the message on his scalp. With Caesar's cipher, however, even if the interceptor knows the method of encryption, he would still have to try 26 different keys to uncover the message. Caesar's cipher is one of the simplest ciphers in existence, but its simplicity nicely demonstrates aspects of modular arithmetic. Other common symmetric cryptosystems are DES, 3DES, and AES. They essentially do the same thing as Caesar's cipher, albeit way more complex and harder to brute force. If you want to read more about these systems, I highly recommend reading the . AES Comic Let's introduce another important concept. Asymmetric cryptography (or public-key cryptography). The difference is that instead of the two parties having the same key, they have their own public and secret keys. Before we dive into RSA, here's a cool tidbit about it for you. When RSA was invented by Ron Rivest, Adi Shamir, and Leonard Adleman, it was illegal for them to export the algorithm or even its description, as part of munitions control in the US. RSA was considered so powerful that it was to be kept a military secret. As a freedom of speech protest, they wore T-shirts with the algorithm printed on them to conventions and talks. RSA is the most popular public-key cryptosystem. It's a super cool system that perfectly illustrates what a lot of modern cryptography is based on. We're going to get a bit technical here but bear with me. The fundamental theorem of arithmetic states that any positive number can be represented in exactly one way as a product of one or more primes. For example, the number 77 can be represented by multiplication of 7 (a prime) and 11 (a prime). There is no other way to represent 77 in primes apart from changing the order of the primes. Keep this in mind while I explain RSA. Before a message can be sent, we need to do some setup. Say, Bob wants to send a message to Alice and there is some eavesdropper called Eve. Alice, as the receiver, chooses two large prime numbers. She multiplies these two numbers to get the result m, the modulus. We're going to keep it simple and small in our example. Alice chooses 7 and 11 as prime numbers, resulting in the modulus 77. She publishes the modulus 77 and some chosen encryption exponent e. The combination of the modulus 77 and the encryption exponent e is called her public key. With Alice's public key Bob can encrypt the message he wants to send. In this case, the message is a number x. Encryption in RSA is done by taking x^e mod m. This results in ciphertext y and sends this to Alice. Now, Alice can do some magic with the prime factors 7 and 11, which results in a piece of information called d (for decryption). This is Alice's secret key. All Alice has to do now is take y^d, which will result in Bob's original message x. Phew, that was a lot, but the hard part is over! Now, notice that only Alice knows prime factors 7 and 11. This means that only Alice can do that magic trick to get information d.  But if you're sharp you would disagree with me here. You'd say: "Since 77 is published and everyone has access to it, Eve can easily figure out that 77 = 7*11 and do the magic trick as well to obtain d, with which she can get the original message x". I would agree with you here. Then I would tell you: "What are the primes of the number 30142741". Chances are you won't be able to tell me the answer. I can very easily tell you the answer though since all I did was take two prime numbers and multiply them to get 30142741.  And if I told you those two numbers, you could just as easily verify that the product is 30142741. So indeed, RSA relies on Eve (or you) not being able to find those two primes. In this case, the primes are 6037 and 4993. See how difficult prime factorization is, but how easy it is to verify the correct primes? That's it!  We are finally at the base of the pillar. Do you see the cracks? Probably not. Let me clarify. So, the transformations we apply to the message (encryption) need to be invertible for the receiver to be able to decrypt it. The receiver can invert the message because she is in possession of some extra information. Meanwhile, any interceptor, not in possession of that information, will not be able to decrypt the message. In RSA, for example, the interceptor would need to find the two prime factors of the modulus. Finding the two prime factors of a small number like 77 is, of course, easy, but prime factorization becomes much harder when a large number is chosen. Verifying that the two primes are factors of the modulus is always relatively easy since you just do multiplications. If prime factorization was easy or if there was an algorithm that can solve the problem relatively fast, then RSA would become obsolete. It relies on prime factorization being hard to solve, but easy to verify. We can make a distinction between two sets of problems. There are problems for which we know a way to find the answer quickly, for example, multiplication or squaring. This set is called "class P". The other class, "NP" is a set of problems for which there are no known ways to find an answer quickly, but when you have some information, the answer is easily verifiable. Prime factorization belongs to this class. But what if there is a way to solve a prime factorization problem easily? This would imply that prime factorization belongs to the P class. In fact, maybe we could prove that all NP problems are actually P problems. The conclusion would be that P = NP. This conclusion would have disastrous consequences for cryptography. As we've seen, RSA relies on prime factorization being difficult to solve. But RSA is only one of many algorithms that rely on NP problems and prime factorization is only one example of an NP problem. Other symmetric and asymmetric ciphers used for your financial transactions, your e-mail communication, etc. would be rendered obsolete. Bitcoin and other cryptocurrencies would suffer severe consequences as well since Bitcoin relies on cryptographic hashing, specifically, the SHA-256 hashing algorithm. For a new block to be accepted in the blockchain, this new block needs to contain a proof-of-work. In short, the proof is easy for any node in the network to verify, but generating the correct number is extremely hard. Proof of P = NP could on the other hand have positive consequences for other fields. Many mathematical problems remain hard to solve. Just imagine the effects it could have on mathematics! Proof of P ≠ NP would mean that our cryptographic algorithms are secure and that the title of this article is just clickbait. But at least I offered you the chance to win a million dollars! That's the reward the Clay Mathematics Institute will award to the first person to solve the "P vs NP" problem. So, give it a shot! Maybe you could be the one to break modern cryptography and send us back to using bald messengers for our daily communication. Also Published Here

An Insecure Pillar in Cybersecurity

Un pilar resquebrajado en ciberseguridad

Un pilier non sécurisé de la cybersécurité

साइबर सुरक्षा में एक असुरक्षित स्तंभ

サイバーセキュリティにおける安全性の低い柱

Um Pilar Inseguro na Segurança Cibernética

Một trụ cột không an toàn trong an ninh mạng

网络安全中的不安全支柱

Hallucinations by Design: Part 4 - Fine-tuning Your Way Out of Vector Nightmares

Optimizing Next.js applications involves combining improvements in both your code and underlying infrastructure. By minimizing unnecessary re-renders, fine-tuning image handling, employing caching techniques, using dynamic imports, and integrating server components, you can significantly boost performance, reduce load times, and increase overall efficiency. Before exploring specific optimizations, it’s important to grasp how Next.js renders and delivers content: Server-Side Rendering (SSR): HTML is generated on every request.
Static Site Generation (SSG): Pages are pre-built during the build process.
Incremental Static Regeneration (ISR): Static pages are updated after deployment.
Client-Side Rendering (CSR): Dynamic content is managed after the initial load. Server-Side Rendering (SSR): HTML is generated on every request. Server-Side Rendering (SSR): Static Site Generation (SSG): Pages are pre-built during the build process. Static Site Generation (SSG): Incremental Static Regeneration (ISR): Static pages are updated after deployment. Incremental Static Regeneration (ISR): Client-Side Rendering (CSR): Dynamic content is managed after the initial load. Client-Side Rendering (CSR): Each rendering method has unique performance implications, which will be discussed throughout this article. 1. Component-Level Optimizations 1. Component-Level Optimizations Preventing Unnecessary Re-rendersIn React, re-renders can occur unnecessarily when functions defined within the render method are recreated, leading to updates in child components. To avoid this, you can memoize functions with useCallback and wrap components in memo so they only update when their props change. This approach helps cut down on extra computations and boosts performance. // Bad practice: Creating functions inside render function ProductCard({ product }) {
  const handleClick = () => {
    console.log('Product clicked:', product.id);
  };
  
  return <div onClick={handleClick}>{product.name}</div>;
} function ProductCard({ product }) {
  const handleClick = () => {
    console.log('Product clicked:', product.id);
  };
  
  return <div onClick={handleClick}>{product.name}</div>;
} //  Better approach: Memoize functions and components import { useCallback, memo } from 'react';

const ProductCard = memo(function ProductCard({ product, onProductClick }) {
  return <div onClick={() => onProductClick(product.id)}>{product.name}</div>;
});

function ProductList({ products }) {
  const handleProductClick = useCallback((id) => {
    console.log('Product clicked:', id);
  }, []);
  
  return (
    <div>
      {products.map(product => (
        <ProductCard 
          key={product.id} 
          product={product} 
          onProductClick={handleProductClick}
        />
      ))}
    </div>
  );
} import { useCallback, memo } from 'react';

const ProductCard = memo(function ProductCard({ product, onProductClick }) {
  return <div onClick={() => onProductClick(product.id)}>{product.name}</div>;
});

function ProductList({ products }) {
  const handleProductClick = useCallback((id) => {
    console.log('Product clicked:', id);
  }, []);
  
  return (
    <div>
      {products.map(product => (
        <ProductCard 
          key={product.id} 
          product={product} 
          onProductClick={handleProductClick}
        />
      ))}
    </div>
  );
} Optimizing Context API Usage Optimizing Context API Usage Using one large React Context for all states can lead to unneeded re-renders, which hampers performance. Instead, break the context into smaller segments based on domain and how often they change. For example, having separate contexts for user data, theme, and cart state ensures that only the components that rely on a specific piece of data will re-render. Additionally, employing custom hooks for state management can keep your code modular and easier to maintain. // Problematic: Single large context causing frequent re-renders const AppContext = createContext();

function AppProvider({ children }) {
  const [user, setUser] = useState(null);
  const [theme, setTheme] = useState('light');
  const [cart, setCart] = useState([]);
  
  const value = { user, setUser, theme, setTheme, cart, setCart };
  
  return <AppContext.Provider value={value}>{children}</AppContext.Provider>;
} const AppContext = createContext();

function AppProvider({ children }) {
  const [user, setUser] = useState(null);
  const [theme, setTheme] = useState('light');
  const [cart, setCart] = useState([]);
  
  const value = { user, setUser, theme, setTheme, cart, setCart };
  
  return <AppContext.Provider value={value}>{children}</AppContext.Provider>;
} // Better: Split contexts by domain and update frequency const UserContext = createContext();
const ThemeContext = createContext();
const CartContext = createContext();

function AppProviders({ children }) {
  return (
    <UserContext.Provider value={useUserState()}>
      <ThemeContext.Provider value={useThemeState()}>
        <CartContext.Provider value={useCartState()}>
          {children}
        </CartContext.Provider>
      </ThemeContext.Provider>
    </UserContext.Provider>
  );
} const UserContext = createContext();
const ThemeContext = createContext();
const CartContext = createContext();

function AppProviders({ children }) {
  return (
    <UserContext.Provider value={useUserState()}>
      <ThemeContext.Provider value={useThemeState()}>
        <CartContext.Provider value={useCartState()}>
          {children}
        </CartContext.Provider>
      </ThemeContext.Provider>
    </UserContext.Provider>
  );
} 2. Data Fetching Optimizations 2. Data Fetching Optimizations Implementing SWR or React Query Implementing SWR or React Query In Next.js, using useEffect may lead to repeated requests. To streamline data fetching, SWR and React Query offer robust solutions through caching, revalidation, and error management. SWR employs a "stale-while-revalidate" strategy to provide immediate cached data while updating it in the background, whereas React Query offers extra capabilities like query invalidation and pagination, enhancing both performance and the overall user experience. // Naïve approach: Fetching on each component mount function ProductDetail({ productId }) {
  const [product, setProduct] = useState(null);
  const [loading, setLoading] = useState(true);
  
  useEffect(() => {
    async function fetchProduct() {
      setLoading(true);
      const res = await fetch(`/api/products/${productId}`);
      const data = await res.json();
      setProduct(data);
      setLoading(false);
    }
    
    fetchProduct();
  }, [productId]);
  
  if (loading) return <p>Loading...</p>;
  
  return <div>{product.name}</div>;
} function ProductDetail({ productId }) {
  const [product, setProduct] = useState(null);
  const [loading, setLoading] = useState(true);
  
  useEffect(() => {
    async function fetchProduct() {
      setLoading(true);
      const res = await fetch(`/api/products/${productId}`);
      const data = await res.json();
      setProduct(data);
      setLoading(false);
    }
    
    fetchProduct();
  }, [productId]);
  
  if (loading) return <p>Loading...</p>;
  
  return <div>{product.name}</div>;
} //Better: Using SWR for caching, revalidation, and error handling import useSWR from 'swr';

function ProductDetail({ productId }) {
  const { data: product, error, isLoading } = useSWR(
    `/api/products/${productId}`,
    fetcher
  );
  
  if (isLoading) return <p>Loading...</p>;
  if (error) return <p>Error loading product</p>;
  
  return <div>{product.name}</div>;
} import useSWR from 'swr';

function ProductDetail({ productId }) {
  const { data: product, error, isLoading } = useSWR(
    `/api/products/${productId}`,
    fetcher
  );
  
  if (isLoading) return <p>Loading...</p>;
  if (error) return <p>Error loading product</p>;
  
  return <div>{product.name}</div>;
} Leveraging getStaticProps and getServerSideProps Leveraging getStaticProps and getServerSideProps In Next.js, getServerSideProps retrieves data on every request, making it perfect for dynamic content like news. On the other hand, getStaticProps runs during the build process, boosting performance for static pages. Incremental Static Regeneration (ISR) enables periodic updates without rebuilding the entire site, striking a balance between freshness and efficiency. // For frequently changing data that needs SEO export async function getServerSideProps() {
  const res = await fetch('https://api.example.com/news');
  const news = await res.json();
  
  return { props: { news } };
} export async function getServerSideProps() {
  const res = await fetch('https://api.example.com/news');
  const news = await res.json();
  
  return { props: { news } };
} // For relatively static data with occasional updates export async function getStaticProps() {
  const res = await fetch('https://api.example.com/products');
  const products = await res.json();
  
  return { 
    props: { products },
    // Re-generate at most once every hour
    revalidate: 3600
  };
} export async function getStaticProps() {
  const res = await fetch('https://api.example.com/products');
  const products = await res.json();
  
  return { 
    props: { products },
    // Re-generate at most once every hour
    revalidate: 3600
  };
} 3. Image and Asset Optimization 3. Image and Asset Optimization The Next.js <Image> component boosts performance by lazy loading and adjusting to different screen sizes. It's best to avoid using layout="fill" without a proper parent container to prevent unexpected layout shifts. Instead, set explicit width and height for consistency and use placeholder="blur" to enhance the user experience. Also, update your next.config.js to support modern image formats like AVIF and WebP for optimal performance. // Anti-pattern: Not specifying proper dimensions / <Image src="/product.jpg" alt="Product" layout="fill" />

// Better: Specify explicit dimensions and use proper layout
<Image 
  src="/product.jpg" 
  alt="Product" 
  width={300} 
  height={200} 
  placeholder="blur"
  blurDataURL="data:image/jpeg;base64,..."
/> <Image src="/product.jpg" alt="Product" layout="fill" />

// Better: Specify explicit dimensions and use proper layout
<Image 
  src="/product.jpg" 
  alt="Product" 
  width={300} 
  height={200} 
  placeholder="blur"
  blurDataURL="data:image/jpeg;base64,..."
/> For maximum performance, configure your next.config.js to optimize images: module.exports = {
  images: {
    deviceSizes: [640, 750, 828, 1080, 1200, 1920, 2048, 3840],
    imageSizes: [16, 32, 48, 64, 96, 128, 256, 384],
    domains: ['cdn.yoursite.com'],
    formats: ['image/avif', 'image/webp'],
  },
}; module.exports = {
  images: {
    deviceSizes: [640, 750, 828, 1080, 1200, 1920, 2048, 3840],
    imageSizes: [16, 32, 48, 64, 96, 128, 256, 384],
    domains: ['cdn.yoursite.com'],
    formats: ['image/avif', 'image/webp'],
  },
}; 4. Route and Bundle Optimization 4. Route and Bundle Optimization Dynamic Imports for Code Splitting Dynamic Imports for Code Splitting Dynamic imports in Next.js help minimize the initial JavaScript load by only bringing in components when they're actually needed. This is particularly beneficial for heavy libraries such as charts or maps. With next/dynamic, you can lazy-load components with a fallback UI, which enhances both performance and user experience. For components that only run on the client side, it's advisable to disable server-side rendering (ssr: false) to ensure they load efficiently without impacting the server-rendered content //Importing everything upfront
import { Chart } from 'heavy-chart-library';

//Dynamic import when needed
import dynamic from 'next/dynamic';

const Chart = dynamic(() => import('heavy-chart-library').then(mod => mod.Chart), {
  loading: () => <p>Loading chart...</p>,
  ssr: false // Disable SSR for components that don't support it
}); //Importing everything upfront
import { Chart } from 'heavy-chart-library';

//Dynamic import when needed
import dynamic from 'next/dynamic';

const Chart = dynamic(() => import('heavy-chart-library').then(mod => mod.Chart), {
  loading: () => <p>Loading chart...</p>,
  ssr: false // Disable SSR for components that don't support it
}); Route-Based Code Splitting Route-Based Code Splitting Next.js automatically divides code by route, ensuring that only the necessary parts load for the current page. You can further enhance this by prefetching routes that users might visit next. For example, triggering router.prefetch(href) on a mouse enter event lets Next.js load the page in the background, leading to faster navigation and an improved user experience. // Pre-fetch routes that are likely to be visited import { useRouter } from 'next/router';

function NavLink({ href, children }) {
  const router = useRouter();
  
  return (
    <a 
      href={href}
      onClick={(e) => {
        e.preventDefault();
        router.push(href);
      }}
      onMouseEnter={() => router.prefetch(href)}
    >
      {children}
    </a>
  );
} import { useRouter } from 'next/router';

function NavLink({ href, children }) {
  const router = useRouter();
  
  return (
    <a 
      href={href}
      onClick={(e) => {
        e.preventDefault();
        router.push(href);
      }}
      onMouseEnter={() => router.prefetch(href)}
    >
      {children}
    </a>
  );
} Conclusion Conclusion The key to optimizing Next.js performance depends on making strategic decisions between code and infrastructure elements. This article examines how rendering methods such as SSR, SSG, ISR and CSR affect performance. The article then explores essential optimization techniques which include stopping unnecessary re-renders and enhancing data fetching through SWR or React Query as well as image optimization and dynamic imports and route-based code splitting. The implementation of these best practices leads to the development of fast and efficient Next.js applications which provide users with a seamless experience. The collective effect of minor performance enhancements leads to better scalability and performance of web applications through time reduction and improved responsiveness and navigation smoothness.

Cut Load Times in Half with These Next.js Tweaks

In my previous installment (READ HERE) , I explored how embedding models struggle with basic language variations like unit of measurements and domain specific sentences. The community response was remarkable – clearly, many of you have encountered similar challenges in your work. Today, I'm expanding our investigation to uncover even more concerning blind spots I've identified through extensive testing. These fundamental issues have profound implications for how we approach AI system development. READ HERE) READ HERE This article is the third entry in my Hallucinations by Design series, building directly on our examination of embedding hallucinations. For optimal comprehension, I strongly suggest reading the previous articles first (HERE and HERE)  to establish the essential background needed to fully appreciate the concepts we'll be discussing here. This sequential approach will provide you with a more coherent understanding of these critical issues. (HERE and HERE) HERE HERE Statistical significance gets completely reversed My statistician colleague turned pale when I showed him this. The model rated "The results showed a significant difference (p<0.05)" and "The results showed no significant difference (p>0.05)" at 0.94 similarity. He just kept shaking his head. "That's... those are opposites. That's the whole point of statistical testing." "The results showed a significant difference (p<0.05)" and "The results showed no significant difference (p>0.05)" at 0.94 similarity. The embeddings couldn't distinguish between statistically significant and insignificant findings. Researchers searching for proven effects were shown a mix of validated and invalidated studies. Do you think scientists making research decisions appreciate confusing evidence with non-evidence? I am sure I wouldn't want my research funding wasted pursuing effects that studies actually disproved. The embeddings couldn't distinguish between statistically significant and insignificant findings. Statistical significance is the cornerstone of empirical research. When your model can't distinguish between "proven effect" and "no proven effect," you've undermined the entire scientific method. We're basically working with models that ignore p-values despite analyzing text where these values determine whether a finding is considered valid. Identity versus similarity gets muddled My materials science colleague laughed out loud when I showed her this one. The model gave a 0.97 similarity score to "This material is aluminum" versus "This material resembles aluminum." She stopped laughing when I explained how this was affecting her lab's search results. Being something and looking like something are fundamentally different! The model gave a 0.97 similarity score to "This material is aluminum" versus "This material resembles aluminum." The embeddings couldn't distinguish between materials that actually were a substance and those that merely looked or behaved similarly. Engineers searching for aluminum components were getting results for aluminum-like alloys with completely different properties. Do you think aerospace engineers appreciate getting information about aluminum-like materials when structural integrity depends on actual aluminum? I am sure I wouldn't want to fly in a plane built with "sort of like aluminum" parts. Engineers searching for aluminum components were getting results for aluminum-like alloys with completely different properties. The distinction between identity and similarity is fundamental to precise communication. When your model treats "X is Y" as equivalent to "X resembles Y," you've lost the ability to make definitive identifications. We're basically working with models that blur categorical boundaries despite analyzing text where precise identification is often the entire point. When your model treats "X is Y" as equivalent to "X resembles Y," you've lost the ability to make definitive identifications. Presuppositions vanish into thin air Now a bit from philosophy subject and this one also bothered me on a deep level. The embedding model rated "What caused the system to fail?" and "Did the system fail?" at a 0.93 similarity. That completely misses the point! The first question assumes failure happened, while the second asks if it happened at all. That's Logic 101! The embedding model rated "What caused the system to fail?" and "Did the system fail?" at a 0.93 similarity. The embeddings couldn't distinguish between questions presupposing a condition and questions asking if the condition existed. Support agents investigating system failures were getting mixed with cases questioning whether failures had occurred at all. Do you think IT managers trying to resolve incidents appreciate wasting time on cases where nothing actually broke? I am sure I wouldn't want my support team chasing phantoms instead of fixing real problems. Presuppositions fundamentally change what a sentence means. When your model treats "Why is X true?" as equivalent to "Is X true?", you've lost the ability to understand what is being assumed versus questioned. We're basically working with models that miss embedded assumptions despite analyzing text where these assumptions often contain critical information. hy is X true?" as equivalent to "Is X true?", you've lost the ability to understand what is being assumed versus questioned. Percentages get completely inverted Looking at pharmaceutical data gave me a headache that no embedding model could properly classify. Get this: "Only 5% of patients experienced side effects" versus "Up to 95% of patients experienced side effects" scored a 0.90 similarity. In what universe are those remotely the same? One's a remarkably safe drug, the other would probably never get FDA approval! Get this: "Only 5% of patients experienced side effects" versus "Up to 95% of patients experienced side effects" scored a 0.90 similarity. I discovered this building a pharmaceutical research database. The algorithm couldn't distinguish between dramatically different safety profiles. Researchers looking for treatments with minimal side effects were shown options with nearly universal adverse effects. Do you think doctors prescribing medications appreciate confusing remarkably safe drugs with overwhelmingly problematic ones? I am sure I wouldn't want to take medication with a 95% side effect rate when I thought it was 5%. Percentages express fundamentally different magnitudes that often determine risk assessments and decision-making. When your model treats "5%" as similar to "95%," you've lost the ability to understand statistical significance. We're basically working with models that see percentages as decorative rather than substantive despite analyzing text where these values drive critical decisions. When your model treats "5%" as similar to "95%," you've lost the ability to understand statistical significance. There is a way to fix these kinds of issues and they will be covered later. Let's understand more issues with embeddings. Metaphors versus literal language - all the same! Remember when embedding models were supposed to understand context? Well, they don't. "The market is climbing a wall of worry" versus "Rock climbers are scaling a worrying wall" scored an 0.89 similarity. Anyone with basic reading comprehension knows one's a financial metaphor and one's about actual mountain climbing. Remember when embedding models were supposed to understand context? Well, they don't. "The market is climbing a wall of worry" versus "Rock climbers are scaling a worrying wall" scored an 0.89 similarity. I discovered this building a financial news analysis system. The algorithm couldn't distinguish between metaphorical and literal language. Investors searching for market analyses were getting mixed results including actual rock climbing stories. Do you think traders making investment decisions appreciate getting literal climbing articles mixed with financial analyses? I am sure I wouldn't want my retirement savings influenced by articles about rock scaling techniques. Metaphorical language is pervasive in specialized domains like finance, medicine, and law. When your model confuses metaphors with their literal interpretations, you've lost the ability to understand domain jargon. We're basically working with models that miss figurative meaning despite analyzing text filled with specialized metaphors that domain experts instantly recognize. Extensional versus intensional references get confused I'm not an astronomer, but even I know this is wrong. "The Morning Star is visible at dawn" versus "The Evening Star is visible at dawn" scored a 0.93 similarity. Here's the kicker - both refer to Venus, but only one statement is actually true! The Evening Star (Venus in the evening) isn't visible at dawn, by definition. "The Morning Star is visible at dawn" versus "The Evening Star is visible at dawn" scored a 0.93 similarity. The embeddings couldn't distinguish between different ways of referring to the same object when those references carried different truth values. Astronomers searching for accurate observation times were shown contradictory information. Do you think researchers planning observations appreciate getting objectively false viewing times? I am sure I wouldn't want to wake up at dawn to see something that's only visible at dusk. Different ways of referring to the same entity often carry different contextual implications. When your model treats all references to an entity as interchangeable, you've lost the ability to preserve context-dependent truth. We're basically working with models that collapse referential distinctions despite analyzing text where these distinctions determine factual accuracy. Domain-specific thresholds get completely missed My doctor friend nearly had a heart attack when I showed her this test result. "The patient's fever was 101°F" versus "The patient's fever was 104°F" scored a 0.97 similarity. "Are you KIDDING me?" she shouted. "That's the difference between 'take some Tylenol' and 'get to the ER immediately'!" "The patient's fever was 101°F" versus "The patient's fever was 104°F" scored a 0.97 similarity. The embeddings couldn't distinguish between clinically significant temperature thresholds. Doctors searching for cases of dangerous fevers were getting mixed results including mild temperature elevations. Do you think emergency physicians appreciate getting non-urgent cases mixed with life-threatening ones? I am sure I wouldn't want my dangerously ill child triaged as having a mild fever. Domain-specific thresholds often represent critical decision boundaries. When your model treats "just above normal" the same as "critically elevated," you've lost the ability to distinguish between routine and emergency situations. We're basically working with models that see numbers as interchangeable despite analyzing text where small numerical differences represent hugely different clinical situations. Date formats cause international incidents Ever missed a deadline because of date format confusion? Our models do it consistently. "Submit your application by 12/10/2023" versus "Submit your application by 10/12/2023" scored an almost perfect 0.99 similarity. Depending on whether you're in the US or Europe, those dates are two months apart! "Submit your application by 12/10/2023" versus "Submit your application by 10/12/2023" scored an almost perfect 0.99 similarity. The embeddings couldn't distinguish between MM/DD and DD/MM formats. Students applying to universities were missing deadlines because date formats were interpreted differently across countries. Do you think applicants appreciate missing life-changing opportunities because of date format confusion? I am sure I wouldn't want my future derailed because an AI couldn't tell October from December. The embeddings couldn't distinguish between MM/DD and DD/MM formats Date format ambiguity isn't just annoying - it can have legal, financial, and personal consequences. When your model treats different date formats as identical, you've introduced a cultural bias that particularly impacts international systems. We're basically working with models that ignore date format conventions despite analyzing text where these distinctions can determine whether something is on time or hopelessly late. Date format ambiguity isn't just annoying - it can have legal, financial, and personal consequences. The Truth and the Results Here is the comparison between msmarco-distilbert-base-tas-b, all-mpnet-base-v2, and open-ai-text-embedding-3-large, and you will notice that there is no significant difference between the output of these models. msmarco-distilbert-base-tas-b embedding score across different test cases msmarco-distilbert-base-tas-b embedding score across different test cases all-mpnet-base-v2 embedding score across different test cases all-mpnet-base-v2 embedding score across different test cases openai-text-embedding-3-large embedding score across different test cases openai-text-embedding-3-large embedding score across different test cases Cannot stress more.. Look, embeddings are amazingly useful despite these problems. I'm not advocating against using them, but rather, it's crucial to approach them cautiously. Here's my battle-tested advice after dozens of projects and countless failures: Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application.


Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards.


Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential.


Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Here's the most important thing I've learnt:**these models don't understand language the way humans do – they understand statistical patterns. When I stopped expecting human-like understanding and started treating them as sophisticated pattern-matching tools with specific blind spots, my systems got better. Much better. The blind spots I've described aren't going away anytime soon – they're baked into how these models work. But if you know they're there, you can design around them. And sometimes, acknowledging a limitation is the first step toward overcoming it. Note: I have many more such cases found through experiments and I will not be covering them instead,I will start covering solution to each of the problems mentioned related to embeddings in my subsequent post. Note The next continuation article will be coming out soon. Stay tuned!! READ PART-1 here
READ PART-2 here READ PART-1 here READ PART-1 here READ PART-1 here READ PART-2 here READ PART-2 here READ PART-2 here

Hallucinations by Design - (Part 3): Trusting Vectors Without Testing Them

Caption: embeddings and its complexities Caption embeddings and its complexities READ PART-1 here (https://hackernoon.com/hallucination-by-design-how-embedding-models-misunderstand-language) READ PART-1 here (https://hackernoon.com/hallucination-by-design-how-embedding-models-misunderstand-language) https://hackernoon.com/hallucination-by-design-how-embedding-models-misunderstand-language Last month, I shared how embedding models hallucinate when handling simple language variations like negation and capitalization. The response was overwhelming – seems I'm not the only one who's been burnt by these issues. Today, I'm diving deeper into even more troubling blind spots I've discovered through testing. These are the kinds that keep me up at night and make me question everything about how we're building AI systems. This is the second part in the series on Hallucinations by Design. It is a continuation of our previous discussion on how embeddings hallucinate.  To get the most out of this article, I highly recommend reading the linked article first, as it lays the foundational concepts necessary to fully grasp the ideas explored here. By doing so, you'll have a seamless learning experience and a deeper understanding of the topic. how embeddings hallucinate Hypothetical vs. actual? Just details! Here's where things get truly disturbing. When I ran "If the treatment works, symptoms should improve" against "The treatment works and symptoms have improved", the similarity score hit 0.95. I sat staring at my screen in disbelief. One's speculating about potential outcomes; the other's reporting confirmed results! When I ran "If the treatment works, symptoms should improve" against "The treatment works and symptoms have improved", the similarity score hit 0.95. I hit this problem working on a clinical research document. The search couldn't distinguish between hypothesized treatment outcomes and verified results. Doctors searching for proven treatments were getting mixed results with unproven hypotheses. Do you think physicians making treatment decisions appreciate confusing speculation with evidence? I am sure I wouldn't want my medical care based on "might work" rather than "does work". Again, think about all the cases where distinguishing hypotheticals from facts is essential - scientific research, medical trials, legal precedents, and investment analyses. When your model conflates "if X then possibly Y" with "X happened and caused Y", you've completely misunderstood the epistemic status of the information. We're basically working with models that can't tell the difference between speculation and confirmation despite analyzing text where this distinction determines whether something is reliable information or mere conjecture. Temporal order? Whatever order! Embedding models see "She completed her degree before starting the job" and "She started her job before completing her degree" as NEARLY identical – ridiculous 0.97 similarity score. One's a traditional career path; the other's working while studying. Completely different situations! She completed her degree before starting the job" and "She started her job before completing her degree" as NEARLY identical – ridiculous 0.97 similarity score. I found this while building a resume screening system. The embeddings couldn't distinguish between candidates who finished their degrees before working and those who were still completing studies. Hiring managers wasted hours interviewing candidates who didn't meet their basic qualification requirements. Do you think busy recruiters appreciate having their time wasted with mismatched candidates? I am sure I wouldn't want my hiring pipeline filled with noise. Think about all the cases where sequence is crucial – medical treatment protocols, legal procedural requirements, cooking recipes, assembly instructions, and chemical formulations. When your model can't tell "A before B" from "B before A," you've lost fundamental causal relationships. We're basically working with models that treat time as an optional concept despite analyzing text that's full of critical sequential information. Quantitative thresholds vanish into thin air This one actually made me spill my coffee. Embedding models see "The company barely exceeded earnings expectations" and "The company significantly missed earnings expectations" as SHOCKINGLY similar – 0.93 similarity score. Exceeded versus missed! These mean opposite things in finance! "The company barely exceeded earnings expectations" and "The company significantly missed earnings expectations" as SHOCKINGLY similar – 0.93 similarity score. If you are building a financial news analysis system, the embeddings wouldn’t distinguish between positive and negative earnings surprises – literally the difference between stock prices going up or down. Investors making trading decisions based on our summaries were getting completely contradictory information. Do you think people risking actual money appreciate getting fundamentally wrong market signals? I am sure I wouldn't want my retirement account guided by such confusion. Now, think about all the cases where crossing a threshold changes everything – passing vs. failing grades, healthy vs. dangerous vital signs, profitable vs. unprofitable businesses, compliant vs. non-compliant regulatory statuses. Your model loses its ability to make meaningful distinctions when it cannot distinguish between barely meeting the target and completely missing it. We're basically working with models that don't understand the concept of thresholds despite analyzing text that's constantly discussing whether targets were met or missed. Scalar inversions get completely flipped The absurdity just keeps piling up. During testing, I found that "The meeting ran significantly shorter than planned" and "The meeting ran significantly longer than planned" scored a 0.96 similarity. I was in complete shock. These sentences describe completely opposite situations – time saved versus time wasted! "The meeting ran significantly shorter than planned" and "The meeting ran significantly longer than planned" scored a 0.96 similarity. I encountered this with project management documents. The search couldn't distinguish between schedule overruns and efficiencies. Managers searching for examples of time-saving techniques were getting shown projects with serious delays. Do you think executives tracking project timelines appreciate getting the exact opposite information they asked for? I am sure I would be furious if I were preparing for a board meeting with such backward data. Think about all the cases where direction on a scale is crucial – cost savings vs. overruns, performance improvements vs. degradations, health improvements vs. declines, and risk increases vs. decreases. When your model treats "much higher than" as interchangeable with "much lower than", you've lost the ability to track directional change. We're basically working with models that don't understand opposing directions despite analyzing text filled with comparative assessments. Domain-specific opposites look like synonyms Medical documents I couldn't believe what I was seeing in the healthcare tests. "The patient presents with tachycardia" versus "The patient presents with bradycardia" returned a 0.94 similarity score. For non-medical folks, that's like confusing a racing heart with one that's dangerously slow – conditions with opposite treatments! "The patient presents with tachycardia" versus "The patient presents with bradycardia" returned a 0.94 similarity score. I discovered this while working on a symptom-matching system for electronic health records. The embedding model couldn't distinguish between fundamentally different medical conditions that require opposite treatments. Physicians searching for cases similar to a patient with a racing heart were shown cases of patients with dangerously slow heartbeats. Do you think doctors making time-sensitive decisions appreciate getting contradictory clinical information? I am sure I wouldn't want my treatment based on the opposite of my actual condition. In the field of medicine, these distinctions can have significant consequences. Tachycardia might be treated with beta-blockers, while bradycardia might require a pacemaker – giving the wrong treatment could be fatal. We're basically working with models that can't distinguish between opposite medical conditions despite analyzing text where this distinction determines appropriate care. Legal documents The legal tests were just as bad. When comparing "Plaintiff bears the burden of proof" with "Defendant bears the burden of proof", the model returned a staggering 0.97 similarity. Let that sink in. These statements literally determine which side has to prove their case in court! Mixing these up could lose you your lawsuit. When comparing "Plaintiff bears the burden of proof" with "Defendant bears the burden of proof", the model returned a staggering 0.97 similarity. The search couldn't distinguish between fundamentally different legal standards and responsibilities. Lawyers researching precedents about plaintiff burdens were shown cases discussing defendant burdens. Do you think attorneys preparing for trial appreciate getting precisely backward legal standards? I am sure I wouldn't want my lawsuit built on completely inverted legal principles. In legal contexts, who bears the burden of proof often determines the outcome of a case. When your model can't distinguish which party has which responsibilities, you've undermined the entire basis of legal reasoning. We're basically working with models that confuse legal roles despite analyzing text where these distinctions define how justice functions. Units of measurement I had to run this test multiple times because I couldn't believe the results. "The procedure takes about 5 minutes" versus "The procedure takes about 5 hours" scored a whopping 0.97 similarity. Is this for real? That's a 60x time difference! Imagine waiting for your "5-minute" appointment that actually takes 5 hours. "The procedure takes about 5 minutes" versus "The procedure takes about 5 hours" scored a whopping 0.97 similarity I found this while building the same healthcare system. The embeddings couldn't distinguish between brief and lengthy procedures. Clinic managers trying to schedule short procedures were being shown lengthy operations that would block their surgery suites for entire days. Do you think medical facilities with tight scheduling constraints appreciate having their entire day's workflow disrupted? I am sure I wouldn't want my hospital running 60x behind schedule. Units of measurement fundamentally change meaning. When your model treats "5 minutes" and "5 hours" as essentially identical, you've lost the ability to understand magnitude. We're basically working with models that ignore units despite analyzing text where units determine whether something is trivial or significant. More measurement problems And it just gets worse from there. During the use of the same healthcare documents, I found "The tumor is 2 centimeters in diameter" and "The tumor is 2 inches in diameter" scored an alarming 0.98 similarity. For context, that's the difference between a potentially minor tumor and one that's 2.54x larger – often the threshold between "watch and wait" versus immediate surgery. "The tumor is 2 centimeters in diameter" and "The tumor is 2 inches in diameter" scored an alarming 0.98 similarity. The embeddings couldn't distinguish between metric and imperial measurements. Oncologists researching treatment options for small tumors were being shown cases of much larger growths. Do you think cancer specialists appreciate getting case studies that aren't remotely comparable to their patients? Even speed limits get confused. Models treat "Maintain speeds under 30 mph" and "Maintain speeds under 30 kph" as HIGHLY similar – a problematic 0.96 similarity score. That's the difference between 30 and 18.6 miles per hour – enough to determine whether an accident is fatal! Even speed limits get confused. Models treat "Maintain speeds under 30 mph" and "Maintain speeds under 30 kph" as HIGHLY similar – a problematic 0.96 similarity score. That's the difference between 30 and 18.6 miles per hour – enough to determine whether an accident is fatal! Converting between units isn't just a mathematical exercise – it fundamentally changes recommendations, safety parameters, and outcomes. We're basically working with models that think numbers without units are sufficient despite analyzing text where the units completely transform the meaning. The Truth and the Results Here is the comparison between msmarco-distilbert-base-tas-b, all-mpnet-base-v2, and open-ai-text-embedding-3-large, and you will notice that there is no significant difference between the output of these models. msmarco-distilbert-base-tas-b, all-mpnet-base-v2, and open-ai-text-embedding-3-large, and you will notice that there is no significant difference between the output of these models. ***msmarco-distilbert-base-tas-b embedding score across different test cases*** ***msmarco-distilbert-base-tas-b embedding score across different test cases*** ***all-mpnet-base-v2 embedding score across different test cases*** ***all-mpnet-base-v2 embedding score across different test cases*** ***openai-text-embedding-3-large embedding score across different test cases*** ***openai-text-embedding-3-large embedding score across different test cases*** Just to repeat.. Look, embeddings are amazingly useful despite these problems. I'm not advocating against using them, but rather, it's crucial to approach them cautiously. Here's my battle-tested advice after dozens of projects and countless failures: Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application.


Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards.


Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential.


Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. **Here's the most important thing I've learnt:**these models don't understand language the way humans do – they understand statistical patterns. When I stopped expecting human-like understanding and started treating them as sophisticated pattern-matching tools with specific blind spots, my systems got better. Much better. The blind spots I've described aren't going away anytime soon – they're baked into how these models work. But if you know they're there, you can design around them. And sometimes, acknowledging a limitation is the first step toward overcoming it. Note: I have many more such cases found through experiments, and I will cover them in my subsequent post. Note Note The next continuation article will be coming out soon. Stay tuned!!

Hallucinations by Design (Part 2): The Silent Flaws of Embeddings & Why Your AI Is Getting It Wrong

ቅዠቶች በንድፍ (ክፍል 2)፡ የመክተት ፀጥታ ጉድለቶች እና የእርስዎ AI ለምን እየተሳሳተ ነው

Halucinacije po dizajnu (2. dio): tihe mane ugrađivanja i zašto vaš AI griješi

Hallucinaciones por diseño (Parte 2): Las carencias silenciosas de las incorporaciones y por qué tu IA se está equivocando

設計による幻覚（パート 2）：埋め込みの隠れた欠陥と AI が間違える理由

यो बर्ष महानगरको अभियान — सम्पदा संरक्षण तथा सम्बर्धन, पर्यटन प्रवर्धनका लागि पूर्वाधार Next Post

Halucynacje według projektu (część 2): ciche wady osadów i dlaczego Twoja sztuczna inteligencja robi to źle

Alucinações por Design (Parte 2): As falhas silenciosas dos embeddings e por que sua IA está errando

Галюцинации по дизайну (часть 2): молчаливые недостатки встраиваний и почему ваш ИИ ошибается

Hallucinations by Igishushanyo (Igice cya 2): Amakosa acecetse yo gushushanya & Impamvu AI yawe irimo Kubona nabi

ERROR: invalid target_lang (tk)

Задумані галюцинації (частина 2): тихі недоліки вбудовування та чому ваш штучний інтелект помиляється

Hallucination by Design: How Embedding Models Misunderstand Language 

Галюцинацыя з-за дызайну: як убудаваныя мадэлі няправільна разумеюць мову

Халюцинация по дизайн: Как моделите за вграждане не разбират езика 

Halucinacije po dizajnu: Kako ugrađeni modeli pogrešno razumiju jezik

Hallucinación por diseño: cómo los modelos de embalaje malinterpretan el lenguaje

ERROR: invalid target_lang (fa-AF)

Hallucination par conception : comment les modèles d'intégration méconnaissent le langage

डिजाइन द्वारा हल्काइयां: कैसे एम्बेडमेंट मॉडल भाषा को गलत समझते हैं

デザインによる幻覚:組み込まれたモデルが言語を誤解する方法

Udanganyifu kwa Usanifu: Jinsi Miundo ya Kupachika Haielewi Lugha 

ภาพหลอนโดยการออกแบบ: โมเดลฝังตัวทำให้เข้าใจภาษาผิดได้อย่างไร

I-Hallucination by Design: Indlela I-Embedding Models Imininingwane Isilinganiso

With the rapid development of the Internet, email—one of the most important tools for information exchange—has seen an ever-growing demand for data processing and task scheduling. As a leading email service provider in China, NetEase Mail introduced the DolphinScheduler platform to better cope with the challenges of processing massive amounts of data and scheduling tasks. It has carried out in-depth deployment and optimization practices in actual use. Project Background and Selection Development History of NetEase Mail Since its launch in 1997, NetEase Mail has undergone multiple important development stages, evolving from 126 Mail, 163 Mail to Mail Master, continuously enriching its product lines and service offerings. Today, NetEase Mail has built a diversified business system that includes free mail, enterprise mail, VIP mail, and more, providing stable, secure, and efficient email services to a massive number of users. Data Application Scenarios In the daily operations of NetEase Mail, there is a need to process massive amounts of business log data. These data require permanent cold backup as well as hot backup storage for over half a year, and business logs are processed and stored separately as offline (HDFS) and real-time (ClickHouse). Meanwhile, to ensure business availability, critical links such as core email sending/receiving and user login authentication mechanisms require efficient data processing and task scheduling support. Background of Selection and Advantages of DolphinScheduler In the face of ever-growing data processing needs, the NetEase Mail team compared multiple open-source scheduling platforms. DolphinScheduler stood out due to its support for various scheduling mechanisms, strong stability, high ease of use, and rich functionality. It is able to support a variety of data scenarios, meeting the complex and diverse task scheduling requirements of NetEase Mail. Platform Deployment and Current Usage The Upgrade and Transformation Path Based on DS After introducing the DolphinScheduler platform, the NetEase Mail team embarked on a continuous path of optimization and upgrade transformation. Starting with preliminary research and selection comparisons, then resolving issues encountered during usage, and finally performing secondary development to continuously improve the platform’s ease of use and user experience, the team has been relentlessly advancing the platform’s perfection. After the selection was confirmed in March 2023, the initial transformation based on the alarm functionality was completed in December 2023, and by March 2024, stable operation and ongoing exploration were achieved. Data Architecture and Current Usage Currently, within the NetEase Mail department, the DolphinScheduler platform has been deployed with a 3-Master, 5-Worker architecture and runs on Kubernetes (k8s). The platform supports over 1,200 offline scheduling tasks for services such as anti-spam, risk control, AI, and more, with daily executions exceeding 50,000 times. At the data architecture level, by integrating DolphinScheduler with Streampark, a combination of offline scheduling and real-time Flink task processing has been achieved. Meanwhile, self-developed components such as the metadata center and data portal form the data service layer, providing support for data management and services. The introduction of the platform has significantly improved data development efficiency, reduced operation and maintenance costs, and ensured the stable output of business data, thereby strongly supporting rapid business iteration and innovation. Task Types and Application Support The DolphinScheduler platform has become the primary task scheduling platform within the department, offering a wide variety of task types including Spark, Shell, SQL, SeaTunnel, Python, and more. These tasks provide solid data support for downstream applications such as metadata management, BI reports, and data R&D, meeting the data processing needs of different business scenarios. Integration of Data Distribution Functionality and Optimization Practices Integration of Data Distribution Functionality Regarding data scheduling and distribution, the NetEase Mail team identified several issues that needed to be addressed to improve the efficiency and capability of data processing, including: Frequent data processing requests Frequent data processing requests Frequent data processing requests from non-R&D personnel require the support of data developers. Low willingness to build intermediate tables Low willingness to build intermediate tables The product and QA teams show little willingness to build intermediate tables, as it takes a long time for them. There are thresholds in data synchronization configuration There are thresholds in data synchronization configuration The configuration of data synchronization tasks between heterogeneous data sources presents certain thresholds, requiring specialized data development support. The synchronization task development process is relatively long The synchronization task development process is relatively long The complete development process for data synchronization tasks is lengthy, involving steps such as table creation, synchronization configuration creation, and scheduling task creation. To address these issues, the NetEase Mail team integrated a data distribution functionality into the DolphinScheduler platform. The overall approach is Intermediate Table Creation + Synchronization Task Configuration Generation + Scheduling Task Creation, which enhances the efficiency of intermediate data processing and provides a one-stop process for constructing data processing tasks. Intermediate Table Creation + Synchronization Task Configuration Generation + Scheduling Task Creation This functionality offers two modes—quick configuration and custom configuration. It can automatically generate table DDL statements based on form parameters and execute table creation logic, simultaneously generating intermediate processing SQL and data synchronization task configurations, and finally calling internal Dolphin methods to create workflows and scheduling tasks. The implementation of this functionality has reduced the average processing time of intermediate data from 1 hour to within 20 minutes, increasing development efficiency by 67%, and it has already supported over 40 online data processing tasks, with the number of users continuously increasing. Optimizing Offline Scheduling Methods to Enhance Fault Recovery Capability The original scheduling mechanism, when handling dependent task failures, would cause the dependent nodes to directly fail, making it impossible to rerun the incomplete data chain with one click. This increased the operational burden on data developers and prolonged fault recovery times. To address this issue, the team redesigned the waiting and failure signal processing logic in the getModelDependResult method of the DependentExecute class in the dolphinscheduler-master module. When the dependency task status is FAILED, the status in the dependResultList is changed to WAITING, so that downstream tasks receive a WAITING status instead of FAILED. getModelDependResult DependentExecute FAILED dependResultList WAITING WAITING FAILED This optimization enables one-click recovery and rerun of the data chain, reducing the cost of manual intervention, enhancing the recovery efficiency and intelligence of the task chain, accelerating the fault recovery speed of the data chain, and ensuring the timely recovery and output of business data. SeaTunnel Component Integration and Optimization, Enhancing Data Synchronization Efficiency To meet the large-scale data synchronization needs of new business, the team introduced the SeaTunnel data integration tool and deployed it on the k8s cluster in a separated cluster mode. By optimizing the SeaTunnel plugins, NetEase Mail offers both custom and quick configuration modes, thereby lowering the usage threshold. Implementation Principle In terms of implementation, the solution supports both form-based configuration and custom configuration. The form-based configuration mode generates the interaction logic for SeaTunnel (ST) configuration through form parameters and supports the configuration of additional custom parameters as well as task-level JVM configurations. The backend receives form parameters to generate the necessary configuration for the task execution context. It designs the IConfigGenerator interface to implement the logic for generating Source and Sink for each data source, and finally, the SeatunnelConfigGenerator class is used to generate the final ST configuration. IConfigGenerator SeatunnelConfigGenerator Deployment and Tuning In terms of deployment and tuning, the SeaTunnel cluster adopts a 2-Master, 5-Worker architecture to achieve high availability and synchronization performance. By optimizing the sharding logic for HDFS, HIVE, and MultiTable Source, the reader receives more balanced shards, which enhances data synchronization performance. The related optimizations have been submitted as PRs to the community. In terms of parameter tuning, in response to the large-scale data synchronization requirement from HDFS to Doris, after researching and tuning the parameters of the DorisSink code, a data transfer rate of 2 million records per second was achieved, greatly enhancing data synchronization performance. Project Practice Case: Task Migration and Resource Isolation Practice Efficient Migration of Mammoth Platform Tasks to DolphinScheduler The Mammoth platform is a big data platform software designed for internal use within NetEase Group, and some of the dependencies between Mammoth tasks are very complex, requiring analysis and organization. If done manually, the efficiency would be low, and omissions or mistakes would likely occur. Eventually, after researching the representation of task dependencies on the DolphinScheduler platform, it was decided to migrate to DolphinScheduler. Regarding the choice of migration method, if a manual migration approach were adopted, it would not only be time-consuming and labor-intensive but also prone to migration errors due to the complexity of task dependencies, thereby affecting the overall stability of tasks. Based on various considerations, after discussion and research, NetEase Mail decided to adopt an automated synchronization solution. This solution automatically collects task metadata and task lineage from the old platform, converts it into the task configuration format of the Dolphin platform, and simultaneously adds task dependencies. Finally, workflows are quickly created through the Dolphin interface. To achieve this process, NetEase Mail used the PyDolphinScheduler synchronization tool officially provided by DolphinScheduler—a Python API that allows workflows to be defined using Python code, namely, “workflow as code.” In addition, NetEase Mail also collected the metadata and lineage of Mammoth tasks through the metadata system, batch rewriting Mammoth tasks into DolphinScheduler tasks, and automatically adding dependency nodes according to the lineage. This practice efficiently completed the migration of over 300 Mammoth tasks, ensuring a smooth business transition, achieving one-click batch migration, greatly saving labor costs, and providing valuable experience for similar task migration scenarios in the future. Worker Group Isolation Practice The next project practice is the resource isolation practice based on Worker grouping in DolphinScheduler. In the daily use of the NetEase Mail DolphinScheduler platform, a key business scenario is to support QA’s online monitoring scheduling tasks. These tasks are characterized by high scheduling frequency, mostly at the minute level; there is a large number of tasks, with the number of monitoring-related tasks currently reaching 120+. The execution of these monitoring scheduling tasks is currently mainly achieved through the SHELL task type by calling a self-developed ETL task processing tool jar package. This jar package provides additional extension functionalities during task execution, such as idempotency checks, timeout retries, etc. However, this kind of invocation will start many JVM processes on the worker nodes. At this time, if some critical scheduling tasks are also scheduled to the same worker node, and an OOM killer is triggered due to insufficient memory, these critical tasks—generally having higher memory usage—may receive a higher OOM score and are more likely to be killed, resulting in unstable execution of the critical tasks. In addition, some T+1 synchronization tasks executed in the early morning require a large amount of resources and are not suitable to be scheduled on the same node as other tasks, which may affect the execution performance and stability of other tasks on the same worker. Therefore, the NetEase Mail team considered using a Worker group isolation solution to address the above issues. Through Worker group isolation, the team separated high-frequency scheduling tasks such as real-time monitoring from other tasks, ensuring the stable execution of core OKR-related schedules and enhancing the overall stability and reliability of task execution. In practical implementation, NetEase Mail created different Worker groups on the platform containing different Worker nodes, including groups such as Default, AI, Monitoring, Hadoop, and large-resource tasks, to avoid issues of task blocking and resource contention, optimize resource allocation, and enhance overall resource utilization and performance of the platform. This practice effectively resolved issues such as task blocking caused by high scheduling frequency of QA monitoring tasks and worker node OOM problems triggered by the high resource demands of early morning T+1 tasks, thereby reducing operational risks. Summary and Future Outlook Practice Summary By introducing the DolphinScheduler platform and conducting a series of deployment and optimization practices, the NetEase Mail team has achieved significant results in improving data development efficiency, reducing operational costs, and ensuring the stable output of business data. The platform’s secondary development closely aligns with business needs, emphasizing enhanced user experience and development efficiency, while the team continuously focuses on platform optimization and improvement and actively contributes to the community, driving the platform’s ongoing development. Platform Value and Benefits The DolphinScheduler platform plays an important role in NetEase Mail’s business. It not only enhances the efficiency and stability of data development but also meets the diverse needs of the business through optimized task scheduling processes, ensuring the timely output of data and strongly promoting the sustained development of the mail business. Experience Sharing and Insights During the process of platform deployment and optimization, the NetEase Mail team has accumulated rich practical experience. These experiences provide important reference value for other enterprises when choosing and using task scheduling platforms. The team emphasizes that secondary development should be closely integrated with actual business needs, always prioritizing user experience and development efficiency. At the same time, continuously focusing on platform optimization and improvement and actively participating in open-source community building can drive the platform’s continuous refinement and development. Future Outlook Looking ahead, the NetEase Mail team plans to continue exploring and advancing in the following directions: Embrace AI: Integrate AI and LLM capabilities to achieve more intelligent and user-friendly data processing ETL processes, enhancing the automation and intelligence level of data processing.
Data Governance: Integrate DolphinScheduler scheduling data with the internal metadata center to achieve intelligent collection and analysis of data/task lineage and data maps, providing strong support for data governance.
Platform Optimization: Further optimize the performance and functionality of the DolphinScheduler platform to enhance its stability and reliability, better meeting the growing data processing demands.
Embrace DATA OPS: Achieve integration and consolidation between the DolphinScheduler platform and other data platform systems, promote the automation of data integration and transmission, and build a more efficient data ecosystem. Embrace AI: Integrate AI and LLM capabilities to achieve more intelligent and user-friendly data processing ETL processes, enhancing the automation and intelligence level of data processing. Embrace AI Data Governance: Integrate DolphinScheduler scheduling data with the internal metadata center to achieve intelligent collection and analysis of data/task lineage and data maps, providing strong support for data governance. Data Governance Platform Optimization: Further optimize the performance and functionality of the DolphinScheduler platform to enhance its stability and reliability, better meeting the growing data processing demands. Platform Optimization Embrace DATA OPS: Achieve integration and consolidation between the DolphinScheduler platform and other data platform systems, promote the automation of data integration and transmission, and build a more efficient data ecosystem. Embrace DATA OPS In Conclusion The deployment and optimization practice of NetEase Mail based on the DolphinScheduler platform not only solved the current challenges of task scheduling and data processing but also laid a solid foundation for future development. Through continuous technological innovation and practical exploration, NetEase Mail will continue to provide users with higher-quality and more efficient email services, while also contributing more to the development of the open-source community.

NetEaseMail Slashes Dev Time by More Than Half With DolphinScheduler

When we started building our AI-powered customer support tool last year, we were more focused on performance than philosophy. Like a lot of startups, we were chasing speed - how quickly could we ship an MVP that worked well enough to impress investors and land our first batch of users? But somewhere between the third late-night brainstorming session and the fifth sprint demo, I realized we were drifting into uncomfortable territory. The faster we moved, the less time we spent asking the uncomfortable questions: Are we respecting user privacy? Is the model perpetuating bias? Do users even know how much of their data is being used? This is the story of how my team learned to balance innovation with ethics, what we got right, what we messed up, and how other developers can (hopefully) avoid the same mistakes. Step 1: Admit That Ethics is a Feature - Not a Roadblock At first, I’ll be honest - any time someone brought up "ethics" during a sprint review, it felt like a speed bump. We wanted to ship a working chatbot, not write a manifesto. But that mindset shifted the first time we saw hallucinated responses that included completely made-up user data. Our model, trained on support tickets, was pulling in out-of-context names and product details from unrelated conversations. It was a privacy nightmare waiting to happen. That’s when we realized that ethics isn’t a compliance box to check after launch. It’s a core feature. Just like uptime, performance, or UX, ethical design needed a dedicated slot in our roadmap. Ignoring it would break the product, just in a less visible way. Step 2: Define What Ethical AI Means for Your Product “AI ethics” sounds big and academic, but in practice, it’s painfully specific. Every product has different risks. In our case - a customer support assistant - we identified three core principles: Transparency: Users should know when they’re talking to AI, not a human.
Data Privacy: We should minimize what data we collect and be crystal clear about how it’s used.
Bias Mitigation: The AI shouldn’t favor certain types of users or responses over others. We didn’t invent these from scratch. We borrowed heavily from the EU’s AI Act, Google’s AI Principles, and a few excellent research papers on algorithmic bias in customer service. But the key was making them concrete enough for engineers to act on, not just a slide deck for investors. Step 3: Bake Ethical Checks Into Development - Not Just a Legal Review One thing we learned quickly: ethics reviews can’t just live with legal or compliance teams. Developers need to own them too. For us, that meant adding a simple "Ethics Check" to our pull request template. Every time someone opened a PR, they had to answer three questions: Does this code access or store new user data? If yes, why is it necessary?
Could this feature create a bias (e.g., favoring certain customers)? If yes, what’s the mitigation?
Is this change visible to the user? If not, should it be? These questions forced us to think before we shipped, and they caught issues we would have completely missed in a traditional security review. Step 4: Be Transparent - Even When It’s Uncomfortable Early on, we decided to add an “AI Transparency Notice” to every conversation handled by our assistant. At first, it felt like a weird over-explaining move - does anyone care if a chatbot is powered by GPT or rule-based logic? Turns out that yes, they do. User feedback showed us that knowing an AI was involved changed expectations. Customers were more forgiving of weird responses when they knew it wasn’t a human. More importantly, they appreciated the honesty. That transparency earned us trust - and trust, it turns out, is one of the hardest things to build into an AI product. Step 5: Build for Audits - Because Someone Will Ask Questions One thing we didn’t anticipate was just how much investors, partners, and even potential customers would care about our AI’s decision-making process. “Explain ability” went from an optional academic feature to a dealbreaker in serious negotiations. To prepare, we built audit trails into every major decision the AI made - what data it accessed, what confidence scores it assigned, and why it chose one response over another. We even created a developer-facing “bias dashboard” that tracked how often certain types of users got escalated to a human rep. This wasn’t just ethical window dressing - it saved us during multiple sales calls. When a prospect asked, “How do you ensure the AI isn’t biased against non-English speakers?” we could show, in real time, how our bias detection system worked. That transparency turned ethical practices into a competitive advantage. Final Thoughts: Ethics is a Culture, Not a Checklist If there’s one thing I’d tell every team building with AI in 2025, it’s this: You can’t bolt ethics onto a product after launch**.** It has to be part of your engineering culture from day one. We didn’t get everything right. There were moments when we prioritized speed over principles, and we had to walk back bad decisions. But by treating ethical design as a core part of our development process - not a side quest for legal - we built a product we’re proud of. If you’re a developer, product manager, or founder wrestling with similar questions, my advice is simple: Talk about ethics early, often, and out loud. Your users (and your future self) will thank you. When we started building our AI-powered customer support tool last year, we were more focused on performance than philosophy. Like a lot of startups, we were chasing speed - how quickly could we ship an MVP that worked well enough to impress investors and land our first batch of users? But somewhere between the third late-night brainstorming session and the fifth sprint demo, I realized we were drifting into uncomfortable territory. The faster we moved, the less time we spent asking the uncomfortable questions: Are we respecting user privacy? Is the model perpetuating bias? Do users even know how much of their data is being used? This is the story of how my team learned to balance innovation with ethics, what we got right, what we messed up, and how other developers can (hopefully) avoid the same mistakes. Step 1: Admit That Ethics is a Feature - Not a Roadblock At first, I’ll be honest - any time someone brought up "ethics" during a sprint review, it felt like a speed bump. We wanted to ship a working chatbot, not write a manifesto. But that mindset shifted the first time we saw hallucinated responses that included completely made-up user data. Our model, trained on support tickets, was pulling in out-of-context names and product details from unrelated conversations. It was a privacy nightmare waiting to happen. That’s when we realized that ethics isn’t a compliance box to check after launch. It’s a core feature. Just like uptime, performance, or UX, ethical design needed a dedicated slot in our roadmap. Ignoring it would break the product, just in a less visible way. Step 2: Define What Ethical AI Means for Your Product Your “AI ethics” sounds big and academic, but in practice, it’s painfully specific. Every product has different risks. In our case - a customer support assistant - we identified three core principles: Transparency: Users should know when they’re talking to AI, not a human. Data Privacy: We should minimize what data we collect and be crystal clear about how it’s used. Bias Mitigation: The AI shouldn’t favor certain types of users or responses over others. Transparency: Users should know when they’re talking to AI, not a human. Transparency: Data Privacy: We should minimize what data we collect and be crystal clear about how it’s used. Data Privacy: Bias Mitigation: The AI shouldn’t favor certain types of users or responses over others. Bias Mitigation: We didn’t invent these from scratch. We borrowed heavily from the EU’s AI Act, Google’s AI Principles, and a few excellent research papers on algorithmic bias in customer service. But the key was making them concrete enough for engineers to act on, not just a slide deck for investors. Step 3: Bake Ethical Checks Into Development - Not Just a Legal Review One thing we learned quickly: ethics reviews can’t just live with legal or compliance teams. Developers need to own them too. For us, that meant adding a simple "Ethics Check" to our pull request template. Every time someone opened a PR, they had to answer three questions: Does this code access or store new user data? If yes, why is it necessary? Could this feature create a bias (e.g., favoring certain customers)? If yes, what’s the mitigation? Is this change visible to the user? If not, should it be? Does this code access or store new user data? If yes, why is it necessary? Could this feature create a bias (e.g., favoring certain customers)? If yes, what’s the mitigation? Is this change visible to the user? If not, should it be? These questions forced us to think before we shipped, and they caught issues we would have completely missed in a traditional security review. Step 4: Be Transparent - Even When It’s Uncomfortable Early on, we decided to add an “AI Transparency Notice” to every conversation handled by our assistant. At first, it felt like a weird over-explaining move - does anyone care if a chatbot is powered by GPT or rule-based logic? “AI Transparency Notice” Turns out that yes, they do. User feedback showed us that knowing an AI was involved changed expectations. Customers were more forgiving of weird responses when they knew it wasn’t a human. More importantly, they appreciated the honesty. That transparency earned us trust - and trust, it turns out, is one of the hardest things to build into an AI product. Step 5: Build for Audits - Because Someone Will Ask Questions Will One thing we didn’t anticipate was just how much investors, partners, and even potential customers would care about our AI’s decision-making process. “Explain ability” went from an optional academic feature to a dealbreaker in serious negotiations. To prepare, we built audit trails into every major decision the AI made - what data it accessed, what confidence scores it assigned, and why it chose one response over another. We even created a developer-facing “bias dashboard” that tracked how often certain types of users got escalated to a human rep. This wasn’t just ethical window dressing - it saved us during multiple sales calls. When a prospect asked, “How do you ensure the AI isn’t biased against non-English speakers?” we could show, in real time, how our bias detection system worked. That transparency turned ethical practices into a competitive advantage. Final Thoughts: Ethics is a Culture, Not a Checklist If there’s one thing I’d tell every team building with AI in 2025, it’s this: You can’t bolt ethics onto a product after launch**.** It has to be part of your engineering culture from day one. We didn’t get everything right. There were moments when we prioritized speed over principles, and we had to walk back bad decisions. But by treating ethical design as a core part of our development process - not a side quest for legal - we built a product we’re proud of. If you’re a developer, product manager, or founder wrestling with similar questions, my advice is simple: Talk about ethics early, often, and out loud. Your users (and your future self) will thank you. Talk about ethics early, often, and out loud.

Stories, News & Mentions about HAL

Categories

Trending Topics

Stories, News & Mentions about HAL

Hal’s Stories on HackerNoon

Hal's News & Mentions

Categories

Trending Topics