Could you name a more contentious question in the British culinary landscape than “what constitutes the best way to roast a potato?” ()Actually, don’t answer that. Someone is bound to immediately say the word “scones”,

How to make the perfect roast potatoes, from the chunk size to the chuff

Plant-based fast food chain Neat Burger has announced the launch of its first ever steak product: a sandwich in collaboration with Chunk Foods. The Chunk Steak Sandwich is available to buy at the NOLITA location in New York City. It features a sliced ...

Vegan Chain Neat Burger Launches Chunk Foods Steak Sandwich

The Yukon government is on track to post a $3.6-million surplus this fiscal year, according to the territory's fall economic update. That's down from more than $48 million this past spring, when ...

Fires, mine closure, public sector spending carve chunk out of Yukon surplus

Two U.S. defense officials have confirmed to Fox News that the U.S. intercepted two one-way attack drones targeting Iraq's al-Asad air base where American troops are located. The incident happened ...

US military intercepts 2 attack drones targeting Iraq air base where American troops are located

The new foodservice collaborations are a direct result of Chunk Foods’ partnership with the Talk of the Town Restaurant Group, enabling the company to reach a broader consumer base with its cholesterol-free Chunk steak, made from fermented soy and wheat.

Chunk Foods’ Plant-Based Steak Expands Menu Presence in Florida Steakhouses and Seafood Restaurants

A drive against ticketless travel revealed that college-goers formed a sizeable chunk of those taking a free ride, particularly in the Tambaram-Chengalpattu route of Chennai Division. Officials ...

Students form sizeable chunk of ticketless travellers in Chennai suburban trains

Parts of Western Hills Viaduct will be closed Wednesday and Thursday for emergency repairs after chunks of concrete fell from the top to the bottom deck of the 91-year-old structure. The middle ...

Falling 'chunks of concrete' prompt closing of parts of Western Hills Viaduct

It was much larger than the basketball-sized chunk that fell near the court's front entrance in 2005. The weight of the marble that fell is unknown, but the Georgia marble used in the court's four ...

Supreme Court avoided disaster when a chunk of marble fell in a courtyard used by justices

It was much larger than the basketball-sized chunk that fell near the court’s front entrance in 2005. The weight of the marble that fell is unknown, but the Georgia marble used in the court’s ...

Chunk Base

chunk-base

Breaking Down Jamba: How Mixing Attention and State Spaces Makes a Smarter LLM

How to Import CSV Data Into PostgreSQL Using Spring Boot Batch

What is custom transformation logic? Index-as-a-service (or RAG-as-service), tends to package a predesigned service and expose two endpoints to users - one to configure the source, and an API to read from the index. Many predefined pipelines for unstructured documents do this. The requirements are fairly simple: parse PDFs, perform some chunking and embedding, and dump into vector stores. This works well if your requirements are simple and primarily focused on document parsing. We've talked to many developers across various verticals that require data indexing, and being able to customize logic is essential for high-quality data retrieval. For example: Basic choices for pipeline components

which parser for different files?
how to chunk files (documents with structure normally have different optimal chunking strategies)?
which embedding model? which vector database?



What should the pipeline do?

Is it a simple text embedding"?
Is it building a knowledge graph?
Should it perform simple summarization for each source for retrieval without chunking?



What additional work is needed to improve pipeline quality?

Do we need deduplication?
Do we need to look up different sources to enrich our data?
Do we need to reconcile and align multiple documents? Basic choices for pipeline components

which parser for different files?
how to chunk files (documents with structure normally have different optimal chunking strategies)?
which embedding model? which vector database? Basic choices for pipeline components which parser for different files?
how to chunk files (documents with structure normally have different optimal chunking strategies)?
which embedding model? which vector database? which parser for different files? how to chunk files (documents with structure normally have different optimal chunking strategies)? which embedding model? which vector database? What should the pipeline do?

Is it a simple text embedding"?
Is it building a knowledge graph?
Should it perform simple summarization for each source for retrieval without chunking? What should the pipeline do? Is it a simple text embedding"?
Is it building a knowledge graph?
Should it perform simple summarization for each source for retrieval without chunking? Is it a simple text embedding"? Is it building a knowledge graph? Should it perform simple summarization for each source for retrieval without chunking? What additional work is needed to improve pipeline quality?

Do we need deduplication?
Do we need to look up different sources to enrich our data?
Do we need to reconcile and align multiple documents? What additional work is needed to improve pipeline quality? Do we need deduplication?
Do we need to look up different sources to enrich our data?
Do we need to reconcile and align multiple documents? Do we need deduplication? Do we need to look up different sources to enrich our data? Do we need to reconcile and align multiple documents? Here we'll walk through some examples of the topology of index pipelines, and we can explore more in the future! Basic embedding​ ​ In this example, we do the following: In this example, we do the following: Read from sources, for example, a list of PDFs
For each source file, parse it to markdown with a PDF parser. There are lots of choices out there: Llama Parse, Unstructured IO, Gemini, DeepSeek, etc.
Chunk all the markdown files. This is a way to break text into smaller chunks, or units, to help organize and process information. There are many options here: flat chunks, hierarchical chunks, secondary chunks, and many publications in this area. There are also special chunking strategies for different verticals - for code, tools like Tree-sitter can help parse and chunk based on syntax. Normally the best choice is tied to your document structure and requirements.
Perform embedding for each chunk. There are lots of great choices: Voyage, models like OpenAI etc.
Collect the embeddings in vector stores. The embedding is normally attached with metadata, for example, which file this embedding belongs to, etc. There are many great choices for vector stores: Chromadb, Milvus, Pinecone and many databases now support vector indexing, for example PostgreSQL (pgvector) and MongoDB. Read from sources, for example, a list of PDFs For each source file, parse it to markdown with a PDF parser. There are lots of choices out there: Llama Parse, Unstructured IO, Gemini, DeepSeek, etc. Llama Parse Unstructured IO Gemini DeepSeek Chunk all the markdown files. This is a way to break text into smaller chunks, or units, to help organize and process information. There are many options here: flat chunks, hierarchical chunks, secondary chunks, and many publications in this area. There are also special chunking strategies for different verticals - for code, tools like Tree-sitter can help parse and chunk based on syntax. Normally the best choice is tied to your document structure and requirements. Tree-sitter Perform embedding for each chunk. There are lots of great choices: Voyage, models like OpenAI etc. Voyage OpenAI Collect the embeddings in vector stores. The embedding is normally attached with metadata, for example, which file this embedding belongs to, etc. There are many great choices for vector stores: Chromadb, Milvus, Pinecone and many databases now support vector indexing, for example PostgreSQL (pgvector) and MongoDB. Chromadb Milvus Pinecone A combination of TF-IDF and vector search​ ​ Anthropic has published a great article about Contextual Retrieval that suggests the combination of vector-based search and TF-IDF. Contextual Retrieval The way to think about a data flow for the pipeline is: In addition to prepare the vector embedding as basic embedding example above, after the source data parsing, we can do the following: For each document, extract keywords from it with its frequency.
Across all documents, group by keywords and sum up their frequencies, putting into an internal storage.
For each document each keyword, calculate the TF-IDF score using two inputs: frequency in the current document, and the sum of frequency across all documents. Store keyword and along with TF-IDF score keyword (if above a certain threshold) to a keyword index. For each document, extract keywords from it with its frequency. Across all documents, group by keywords and sum up their frequencies, putting into an internal storage. For each document each keyword, calculate the TF-IDF score using two inputs: frequency in the current document, and the sum of frequency across all documents. Store keyword and along with TF-IDF score keyword (if above a certain threshold) to a keyword index. On query time, we can query both the vector index and the keyword index, and combine results from both. Simple Data Lookup/Enrichment example​ ​ Sometimes, you want to enrich your data with metadata looked up from other sources. For example, if we want to create index on diagnostic reports, which uses ICD-10 (International Classification of Diseases Version 10) codes to describe diseases, we can have a pipeline like this: ICD-10 In this example, we do the following: On the first path, build a ICD-10 dictionary by

For each ICD-10 description document, convert it to markdown with a PDF parser.
Split into items.
For each item, extract the ICD-10 code and description, and collect into a storage.



On the second path, for each report

Parse it to markdown with a PDF parser.
Split into items.
For each item, lookup the ICD-10 dictionary prepared above and enrich the item with the descritions for ICD-10 codes. On the first path, build a ICD-10 dictionary by

For each ICD-10 description document, convert it to markdown with a PDF parser.
Split into items.
For each item, extract the ICD-10 code and description, and collect into a storage. On the first path, build a ICD-10 dictionary by For each ICD-10 description document, convert it to markdown with a PDF parser.
Split into items.
For each item, extract the ICD-10 code and description, and collect into a storage. For each ICD-10 description document, convert it to markdown with a PDF parser. Split into items. For each item, extract the ICD-10 code and description, and collect into a storage. On the second path, for each report

Parse it to markdown with a PDF parser.
Split into items.
For each item, lookup the ICD-10 dictionary prepared above and enrich the item with the descritions for ICD-10 codes. On the second path, for each report Parse it to markdown with a PDF parser.
Split into items.
For each item, lookup the ICD-10 dictionary prepared above and enrich the item with the descritions for ICD-10 codes. Parse it to markdown with a PDF parser. Split into items. For each item, lookup the ICD-10 dictionary prepared above and enrich the item with the descritions for ICD-10 codes. Now we have a vector index, built based on diagnostic reports enriched with ICD-10 descriptions. CocoIndex is an open-source engine that supports both custom transformation logic and incremental updates specialized for data indexing.  It supports various builtins to simplify your needs to customize data pipeline for AI.   ❤️ CocoIndex is Open Sourced under Apache License 2.0. If you like our work, please support us with a Github Star ⭐ at https://github.com/cocoindex-io/cocoindex. CocoIndex is an open-source engine that supports both custom transformation logic and incremental updates specialized for data indexing.  It supports various builtins to simplify your needs to customize data pipeline for AI.   ❤️ CocoIndex is Open Sourced under Apache License 2.0. If you like our work, please support us with a Github Star ⭐ at https://github.com/cocoindex-io/cocoindex. CocoIndex Apache License 2.0 https://github.com/cocoindex-io/cocoindex

How to Design Customizable Data Indexing Pipelines

Building a Hybrid RAG Agent with Neo4j Graphs and Milvus Vector Search

ዲቃላ RAG ወኪል በNeo4j Graphs እና Milvus Vector ፍለጋ መገንባት

Neo4j গ্রাফ এবং মিলভাস ভেক্টর অনুসন্ধান সহ একটি হাইব্রিড RAG এজেন্ট তৈরি করা

Construcció d'un agent RAG híbrid amb Neo4j Graphs i Milvus Vector Search

Creación de un agente RAG híbrido con gráficos Neo4j y búsqueda vectorial Milvus

Construíndo un axente RAG híbrido con Neo4j Graphs e Milvus Vector Search

Bati yon Ajan Hybrid RAG ak graf Neo4j ak rechèch vektè Milvus

Neo4j グラフと Milvus ベクトル検索を使用したハイブリッド RAG エージェントの構築

Dhisidda Wakiilka Isku-dhafka ah ee RAG oo wata Sawirro Neo4j ah iyo Raadinta Milvus Vector

Прављење хибридног РАГ агента са Нео4ј графовима и Милвус векторском претрагом

Neo4j வரைபடங்கள் மற்றும் மில்வஸ் வெக்டர் தேடலுடன் ஒரு கலப்பின RAG முகவரை உருவாக்குதல்

Сохтани агенти гибридии RAG бо графикҳои Neo4j ва ҷустуҷӯи Milvus

Neo4j Grafikleri ve Milvus Vektör Araması ile Hibrit Bir RAG Aracısı Oluşturma

This post is about my on performance techniques used to make Uber mobile web using as fast as possible. learning React **Motivation**It’s been a year since Flipkart Lite was launched and few months since Housing Go was launched and I was always fascinated with the idea of how mobile web is a future and I wanted to give a try. First I needed an app on which I can implement the perf techniques, and Uber had just recently launched their app with new design and it looked promising so I decided to clone the app using React. It took me some time to build the basic implementation of the app, I have used for the map and used to create a path from the source and destination along with the https://github.com/uber/react-map-gl svg-overlay html-overlay. Below is the gif of the app with basic interaction. Now that I had the basic app to work on, I started to improve the performance of it. I have used to check the performance of the web app in each stage. Chrome Lighthouse initially load time looked like this. To improve the above stats I have used following techniques **_Code Splitting — reduces load time from 19sec to 4sec_**First thing I did was used techniques to divide the app into various chunks based on the route and load what is needed for that particular route. webpack code splitting This I did by using the api of where I require the component only when the route is requested. getComponent react-router I also extracted the vendor code using in webpack. CommonChunkPlugin With this I have reduced the load time from 19secs to 4secs. Performance metrics using code splitting **_2. Server side rendering — reduces load time from 4sec to 921ms_**I then implemented SSR by rendering the initial route on the server and passing on to the client. I used Express for this on the backend and used api of the match react-router Thanks to SSR, now the load time is 921ms. Performance metrics using SSR **_reduces load time from 921ms to 546ms_**Then I decided to compress all the static files, this I did this by using in the webpack 3. Compressed static assets — CompressionPlugin and to serve the compressed file from the server which falls back to uncompressed if required file is not found. express-static-gzip yippee I saved almost 400ms. Good job, Narendra! Performance metrics after compressing the assets **_4. Caching — helped load time of repeat visits from ~500ms to ~300ms_**Now that i had improved the performance of my web app from 19seconds to 546ms, I wanted to cache static assets so the repeat visits are much faster. I did this by using for browsers which support service workers sw-toolbox and headers for browsers which don’t support. cache-control By doing this I improved the repeat visit by approximately 200ms. Repeat visit without service worker Repeat visit with service worker **5. Preload and then load**I have used in the tag and used for the browser which doesn’t support preload.At the end of the body I load the application JS in a regular tag.For more about preload and prefetch please visit <link rel="preload" ... as="script"> head prefetch <script> https://css-tricks.com/prefetching-preloading-prebrowsing/ Finally tested with Google PageSpeed and this is the result Scores from PageSpeed Insights This has been a good learning for me, I know I can optimize more and will keep exploring. Performance improvement is an ongoing process this is just a benchmark to what I have achieved. Give a try with your app and let me know your story. Github: Live Demo: or https://github.com/narendrashetty/uber-mobile-web https://uberweb.herokuapp.com/ https://uberweb-v2.herokuapp.com/ Please give a visit to the demo on your mobile browser and share your inputs with me. (Since I hosted it on heroku, it’s goes down when it has no visits. Don’t lose patience if it doesn’t load at first :P Such an irony). Maintaining Performance is like a chess, one wrong move and it undos lot of hardwork done. Thanks for reading, If you liked this article, click “Recommend” or write a response below. You can connect with me on Twitter . @narendra_shetty

Stories, News & Mentions about CHUNK-BASE

Categories

Trending Topics

Stories, News & Mentions about CHUNK-BASE

Chunk-base’s Stories on HackerNoon

Chunk-base's News & Mentions

Categories

Trending Topics