Weav AI

weavai

The TechBeat: Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management (4/13/2024)

The TechBeat: Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management (4/11/2024)

Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management

ডাটা ম্যানেজমেন্টে কৌশলগত বর্ধন অর্জনের জন্য MinIO S3 বাকেটের সাহায্যে ওয়েভিয়েট ব্যাক আপ করা হচ্ছে

Sichern von Weaviate mit MinIO S3-Buckets zur strategischen Verbesserung des Datenmanagements

Realizar una copia de seguridad de Weaviate con MinIO S3 Buckets para lograr una mejora estratégica en la gestión de datos

Sauvegarde de Weaviate avec les buckets MinIO S3 pour obtenir une amélioration stratégique de la gestion des données

डेटा प्रबंधन में रणनीतिक वृद्धि हासिल करने के लिए मिनियो एस3 बकेट के साथ वेविएट का बैकअप लेना

MinIO S3 バケットを使用して Weaviate をバックアップし、データ管理の戦略的な強化を実現

데이터 관리의 전략적 향상을 위해 MinIO S3 버킷으로 Weaviate 백업

Fazendo backup do Weaviate com buckets MinIO S3 para obter aprimoramento estratégico no gerenciamento de dados

Резервное копирование Weaviate с помощью сегментов MinIO S3 для достижения стратегического улучшения управления данными

Veri Yönetiminde Stratejik İyileştirme Sağlamak için Weaviate'i MinIO S3 Kovalarıyla Yedekleme

Sao lưu Weaviate bằng nhóm MinIO S3 để đạt được cải tiến chiến lược trong quản lý dữ liệu

使用 MinIO S3 Buckets 备份 Weaviate，实现数据管理的战略增强

Scientific American, Volume XLIII., No. 25, December 18, 1880, by Various, is part of the HackerNoon Books Series. You can jump to any chapter in this book . GLASS SPINNING AND WEAVING. here GLASS SPINNING AND WEAVING Quite recently a Pittsburg glass firm has succeeded, to a notable degree, in producing glass threads of sufficient fineness and elasticity to permit of their being woven into fabrics of novel character and quality. Their success is such as to warrant the assumption that garments of pure glass, glistening and imperishable, are among the possibilities of the near future. The spinning of glass threads of extreme fineness is not a new process, but, as carried on at present by the firm in question—Messrs. Atterbury & Co.—possesses considerable interest. From a quality of glass similar to that from which table ware is made, rods of glass averaging half an inch in diameter are drawn to any desired length and of various colors. These rods are then so placed that the flame of two gas burners is blown against that end of the rod pointed toward the large "spinning" wheel. The latter is 81/2 feet in diameter, and turns at the rate of 300 revolutions per minute. The flames, having played upon the end of the glass cylinder until a melting heat is attained, a thread of glass is drawn from the rod and affixed to the periphery of the wheel, whose face is about 12 inches wide. Motion is then communicated, and the crystal thread is drawn from between the gas jets and wrapped upon the wheel at the rate of about 7,500 feet per minute. A higher speed results in a finer filament of glass, and vice versa. During its passage from the flame to the wheel, a distance of five or six feet, the thread has become cooled, and yet its elasticity is preserved to a notable degree. The next step in the process consists in the removal of the layers of threads from the wheel. This is easily accomplished, and after being cut to the desired lengths, the filaments are woven in a loom somewhat similar to that used in weaving silken goods. Until within the past few weeks only the woof of the fabric was of glass, but at present both warp and woof are in crystal. Samples of this cloth have been forwarded to New York and to Chicago, and the manufacturers claim to be able to duplicate in colors, texture, etc., any garments sent them. A tablecloth of glass recently completed shines with a satiny, opalescent luster by day, and under gaslight shows remarkable beauty. Imitation plumes, in opal, ruby, pale green, and other hues, are also constructed of these threads, and are wonderfully pretty. The chief obstacle yet to surmount seems to lie in the manipulation of these threads, which are so fine that a bunch containing 250 is not so thick as an average sized knitting needle, and which do not possess the tractability of threads of silk or cotton. [The foregoing information is furnished by a correspondent in Pittsburg. A sample of the goods mentioned, a tablecloth of glass, is now on exhibition in this city. The weaving of such heavy fabrics of glass for ornamental purposes and for curiosities is no new thing; nor, in our estimation, does comparative success in such experiments warrant the enthusiastic claims of the Pittsburg manufacturers touching the adaptability of glass for wearing apparel. Unless it is in their power to change the nature of glass absolutely and radically, it does not seem possible for them so to overcome the ultimate brittleness of the separate fibers as to make the fabric fit to be brought in contact with the skin. The woven stuff may be relatively tough and flexible; but unless the entire fabric can be made of one unbreakable fiber the touch of the free ends, be they never so fine, must be anything but pleasant or beneficial, if one can judge by the finest filaments of glass spun hitherto. Besides, in weaving and wearing the goods, a certain amount of fiber dust must be produced as in the case of all other textile material. When the softest of vegetable fibers are employed the air charged with their fragments is hurtful to the lungs; still more injurious must be the spiculæ of spun glass. However, although the manufacturers are likely to be disappointed in their expectation of finding in glass a cheap and available substitute for linen, cotton, and silk in dress goods, it is quite possible that a wide range of useful application may be found for their new fabric.] About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Various (2007). Scientific American, Volume XLIII., No. 25, December 18, 1880. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/21081/pg21081-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

GLASS SPINNING AND WEAVING

Africa is a dynamic tapestry, and a new thread is being weaved into it that has the potential to alter the continent's whole social and economic landscape. This strand is blockchain technology, a disruptive innovation that might drastically alter Africa's future. Over 1.3 billion people call Africa home, and the continent is teeming with untapped potential and a hunger for development. Still, it's a continent where conventional financial institutions often fail to meet the needs of the people and where international commerce is still hampered by bureaucratic red tape. Herein lies blockchain's tremendous transformational potential. Blockchain's potential to solve many of Africa's most urgent problems lies in its decentralised, transparent, and secure nature. Envision a future where millions of people are able to use banking services without difficulty. Where migrant workers no longer have to pay expensive fees to send money home, and where farmers in remote areas may get loans without opening a bank account. Blockchain technology has the potential to usher in a new age of frictionless international commerce, lowering transaction costs and accelerating Europe's economic unification. Smooth, efficient commerce that supports regional development and prosperity without the need to wait days for transactions to settle and without punishing tariffs. But the blockchain's impact will extend well beyond the financial sector. It has the potential to revolutionise administration, instruction, medical treatment, and other fields digitally. Africa, a land of opportunity, is at the epicentre of this change, where the disruptive technology of blockchain is being welcomed with open arms. The voyage has only just begun, yet it already carries the promise of a future in which Africa is a technological powerhouse. Venom's Vision: A Blockchain-empowered Africa Standing Tall in the Digital Future In the pulsing heart of Africa, an awe-inspiring metamorphosis is underway. The pioneering blockchain company, Venom, fresh off the heels of a riveting testnet launch and hackathon, is joining forces with the Kenyan government. Their mission? To birth a state-of-the-art blockchain hub in Kenya, marking a significant milestone in the continent's digital journey. : Vested Interest Disclosure The author is an independent contributor publishing via our . Be it through direct compensation, media partnerships, or networking, the author has a vested interest in the company/ies mentioned in this story. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR brand-as-author program The importance of this endeavor cannot be overstated. Africa, home to 1.3 billion people and an economy that's projected to reach $4.8 trillion by 2024 to a report by African Development Bank, stands on the precipice of an extraordinary leap forward in its financial landscape. Blockchain technology, with its decentralization and transparency, presents a golden opportunity for this vibrant continent. according Venom, renowned for its technological prowess, is driven by a vision to empower Africa, bridging the gap between the age-old financial systems and the dynamic world of Web3. Venom has signed an with Kenyan government for setting a blockchain hub. This blockchain hub is not just a technological venture, it's a human-centric initiative poised to revolutionize the way people live, trade, and interact. MOU Imagine a world where the economic barriers crumble, where cross-border trade is as fluid as the mighty Zambezi. to the United Nations Economic Commission for Africa, intra-African trade stood at just 16% in 2020, compared to 68% in Europe. Venom's vision, powered by blockchain, could see this figure soar, unlocking untapped potential and stimulating economic growth on a monumental scale. According Beyond the realm of finance, Venom's initiative is set to equip African nations with the resources they need to build robust digital infrastructures. A World Bank report highlights that only 28% of Africa's population has internet access as of 2021, a figure that could dramatically change with the establishment of more such blockchain hubs. The transformation wouldn't just be digital; it would be social, cultural, and economic - a comprehensive metamorphosis for the continent. The creation of this blockchain hub symbolizes a pivotal moment in Africa's history. Here, in the vibrant intersection of tradition and innovation, Venom and Kenya are crafting a blueprint for a future where the continent's digital potential is as rich and vast as its cultural heritage. The road to this future may be long, but with each step, they are drawing Africa closer to a reality where its technological prowess is globally recognized and revered. Don’t forget to like and share the story! Image credits: AI.

Venom's Blockchain Breakthrough: Weaving a New Digital Dawn for Africa

El avance de la cadena de bloques de Venom: tejiendo un nuevo amanecer digital para África

La percée de la blockchain de Venom : Tisser une nouvelle aube numérique pour l'Afrique

विष की ब्लॉकचेन सफलता: अफ्रीका के लिए एक नया डिजिटल डॉन बुन रहा है

ヴェノムのブロックチェーンの躍進: アフリカに新たなデジタルの夜明けを紡ぐ

Avanço Blockchain de Venom: Tecendo um Novo Amanhecer Digital para a África

Đột phá chuỗi khối của Venom: Dệt nên bình minh kỹ thuật số mới cho châu Phi

Venom的区块链突破：为非洲编织新的数字黎明

What is a schema, why you need one and how to define one to store your own data. is a vector-native database: data is stored as vectors in a high-dimensional space. In order to store and retrieve vector data, Weaviate uses a data schema which is unique per instance. Once you have started up a Weaviate instance (via docker-compose, Kubernetes or using the Weaviate Cluster Service), you need to create a data schema before you can add and query data. Weaviate Like most other databases, a defines the structure described in a formal language. It is a sort of blueprint of how the data is structured in a database. The data schema you need to define for Weaviate is relatively simple. It consists of classes and properties definitions and how they are related to each other. Due to Weaviate’s vectorization modules, the classes and properties you add will be automatically placed in context. database schema This article is not only a tutorial for creating your own schema, but also explains what a schema is and why you need one. Ultimately, understanding what you’re creating a schema for will help you in making your own. What is a Weaviate data schema and why do you need one? RDF-inspired Just like most databases, Weaviate needs a data schema. Weaviate uses a class-property structure, inspired by the . RDF Schemas have classes with properties and relations, and are used as to represent knowledge. Born out of this RDF-like perspective is the Weaviate schema with classes, properties and relations. Weaviate uses the data schema to place data objects into context and derive its meaning. A schema is thus an essential part for Weaviate to understand your data objects. A well-defined schema (read on to learn best practices for creating a schema) is key to meaningful insights of your data. RDF Schema ontology Graph-like data model You can make relations between data objects in Weaviate. You can see the data schema with cross references as a graph-like data model. But keep in mind that Weaviate is not a Graph database. Instead, Weaviate is a search engine that stores data as , with a data model. Typical graph database tasks like shortest path prediction are not part of Weaviate’s features. In Weaviate, data objects, which are nodes in the graph, get a vector position. vectors graph-like Data is thus stored in a space according to its meaning, and can still be connected to other data entities. Storing the (graph) data as vectors has huge advantages when it comes to automatic classification, prediction of data relations, semantic search and deriving insights. How to define a Weaviate data schema Class - property structure Data objects in Weaviate always belong to a . A class describes the data object in the form of a noun or a verb. Every class is defined in the schema, and can have one or more . Properties define the actual data values of the objects in your dataset. Properties have a and a . The name may contribute to the object’s semantics, depending on module-specific settings you can also define in the schema. A property’s dataType lists which data formats it accepts (e.g. , , etc). Graph-like relations between data objects are also defined as property dataType. class properties name dataType string int Schema design example Let’s make a data schema for an . This dataset consists of 150k wine reviews scraped from . It contains wine titles, descriptions, points (score 0-100 given by the reviewer), prices, designations, regions, provinces, countries, varieties, wineries and reviewers. To keep the example for this tutorial simple, I’m leaving out the regions and reviewers. Now, let’s see how we can design a good data schema. open wine review dataset from Kaggle WineEnthusiast An obvious first class to create is . The metadata list mentioned above can all be seen as properties of the class . However, some properties have values which are not unique and shared between data instances of Wine. For example, the a wine is from will be a country from a limited list of countries. Wine Wine country Similarly, there is a limited set of wine (e.g. Chardonnay, Pinot Noir, etc). The same holds for and . We decide to let these characteristics have their own class. The figure below shows which classes we have now. varieties provinces wineries Classes We now have five classes. All classes have a unique name. A class name always starts with a capital letter. A good class name is descriptive of the data objects that you will add to this class. If you want to use multiple words for a class or property name, concatenate the words according to the (for class names) or (for property names) . In addition to naming the classes, we can add a (a string for your own reference). The class, without properties and module-specific settings could be designed in json as follows: CamelCase camelCase rule description Wine { : , : , : []
} "class" "Wine" "description" "A wine that has been tasted and reviewed" "properties" Properties Leaving relations between the classes out for now, the class Wine has five properties, while the other four classes only have a unique name. Property names always start with a lowercase letter. Again, in addition to naming the properties, we can add a description. With in mind, we define the properties. We extend the Wine class with its properties as follows [1] [2]. the original dataset { : , : , : [
        { : , : , : [ ]
        },
        { : , : , : [ ]
        },
        { : , : , : [ ]
        },
        { : , : , : [ ]
        },
        { : , : , : [ ]
        }
    ]
} "class" "Wine" "description" "A wine that has been tasted and reviewed" "properties" "name" "title" "description" "The name of the wine" "dataType" "text" "name" "description" "description" "The review of the wine" "dataType" "text" "name" "designation" "description" "The vineyard within the winery where the grapes that made the wine are from" "dataType" "string" "name" "points" "description" "Points 0-100 given by the reviewer" "dataType" "int" "name" "price" "description" "The price of the wine in dollars" "dataType" "number" Cross-references As mentioned before, the four classes , , and are all related to . Thus, we can make cross-relations between those classes and . In other words, those four classes are properties of , described by a cross-reference. In the next figure it is pictured how those classes can be related. Note that all relations have a single direction; relations between data classes in Weaviate are not automatically bidirectional. That is because relations to other classes (or data objects) are described as property values of a class or data object. Variety Province Country Winery Wine Wine Wine Assuming we have the classes , , , and without references in our schema, we can extend them by adding properties with references. For example, we can extend the class with a property with a cross-reference to . Note that the dataType of the property is now the class name : “Wine” “Variety” “Province” “Country” “Winery” “Province” "Country" “inCountry” “Country” { : , : , : [
        { : , : , : [ ]
        },
        { : , : , : [ ]
        }
    ]
} "class" "Province" "description" "A defined region in a country" "properties" "name" "name" "description" "The name of the province" "dataType" "string" "name" "inCountry" "description" "The country the province belongs to" "dataType" "Country" Advanced schema configuration For data classes, you need to define at least the name of the class in and at least one property. Additionally, you may add a description, the vector index type and configuration if other than default, the vectorizer (determines which vectorization module is used to vectorize data objects of this class) and its settings in case other than default, and other module-specific settings. For reference, check the documentation of or . “class” schema configuration module specific settings For class properties, defining the and are required. A description and module-specific settings are optional. Module-agnostic is the setting . If this boolean field is set to (default is ), the property and its value will be ignored in search and classification. This might come in handy if you explicitly don’t want a value without any semantic value (e.g. a person’s ID number) to influence your classification. “name” “dataType” “indexInverted” false true Upload data schema You can upload an entire data schema to a running Weaviate instance at once using the Python client. Alternatively, you can add classes one by one using other clients or with an HTTP request. Make sure classes exist before you use them in a reference property. Example of importing a full schema with the Python client to Weaviate running on . http://localhost:8080 weaviate

client = weaviate.Client( )

schema = { : [
        { : , : , : [
                { : , : , : [ ]
                }, { : , : , : [ ]
                }, { : , : , : [ ]
                }, { : , : , : [ ]
                }, { : , : , : [ ]
                }, { : [ ], : , : }, { : [ ], : , : }, { : [ ], : , : }, { : [ ], : , : }
            ]
        }, { : , : , : [
                { : [ ], : , : }
            ]
        }, { : , : , : [
                { : [ ], : , : }, { : [ ], : , : }
            ]
        }, { : , : , : [
                { : [ ], : , : }
            ]
        }, { : , : , : [
                { : [ ], : , : }, { : [ ], : , : }
            ]
        }
    ]
}

client.schema.create(schema) import "http://localhost:8080" "classes" "class" "Wine" "description" "A wine that has been tasted and reviewed" "properties" "name" "title" "description" "The name of the wine" "dataType" "text" "name" "description" "description" "The review of the wine" "dataType" "text" "name" "designation" "description" "The vineyard within the winery where the grapes that made the wine are from" "dataType" "string" "name" "points" "description" "Points 0-100 given by the reviewer" "dataType" "int" "name" "price" "description" "The price of the wine in dollars" "dataType" "number" "dataType" "Country" "description" "The country the wine is from" "name" "fromCountry" "dataType" "Province" "description" "The province or state that the wine is from" "name" "fromProvince" "dataType" "Variety" "description" "The type of grapes used to make the wine (ie Pinot Noir)" "name" "hasVariety" "dataType" "Winery" "description" "The winery that made the wine" "name" "fromWinery" "class" "Country" "description" "A country which produces wine" "properties" "dataType" "string" "description" "The name of the country" "name" "name" "class" "Province" "description" "A defined region or state of a country" "properties" "dataType" "string" "description" "The name of the country" "name" "name" "dataType" "Country" "description" "The country the province lies in" "name" "inCountry" "class" "Variety" "description" "The type of grapes a wine is made of" "properties" "dataType" "string" "description" "The name of the variety, which is the name of the grape" "name" "name" "class" "Winery" "description" "A wine producer" "properties" "dataType" "string" "description" "The name of the winery" "name" "name" "dataType" "Country" "description" "The country the province lies in" "name" "inCountry" Conclusion In this article, you learned what a Weaviate data schema is, why you need one and how to design one. The full example code can be found . Learn more about module specific schema configuration , and other property data types like geo-coordinates, phone numbers and dates . After designing a data schema, you can start to Weaviate! here here here uploading data [1] Note that is a property, while and are properties. values are indexed as one token, whereas values are indexed after applying tokenization. as string would be indexed as and also only match that in a GraphQL where filter, whereas as text it would be indexed as and also match the individual words. “designation” “string” “title” “description” “text” “string” “text” “jane.doe@foobar.com” “jane.doe@foobar.com” ['jane', 'doe', 'foobar', 'com'] [2] For more property dataType options, check . the documentation

What Is Weaviate And How To Create Data Schemas In It

Getting Started with the Weaviate Vector Search Engine

Weaviate is an open-source search engine with a build-in NLP model called the Contextionary. What makes Weaviate unique, is that it stores data in a vector space rather than a traditional row-column or graph structure, allowing you to search through data based on its meaning rather than keywords alone. In this article, I ( ) want to share the history of Weaviate, how the concept was born, and where we are heading towards in the near future. Bob van Luijt A World of Wonders called Natural Language Processing Somewhere early 2015 I was introduced to the concept of word embeddings through the publication of an article that contained a machine-learning algorithm to turn individual words into embeddings called . GloVe squarepants 0.27442 -0.25977 -0.17438 0.18573 0.6309 0.77326 -0.50925 -1.8926 0.72604 0.54436 -0.2705 1.1534 0.20972 1.2629 1.2796 -0.12663 0.58185 0.4805 -0.51054 0.026454 0.20253 0.32844 0.72568 1.23 0.90203 1.1607 -0.39605 0.80305 0.16373 0.053425 -0.65308 1.0607 -0.29405 0.42172 -0.45183 0.57945 0.20217 -1.3342 -0.71227 -0.6081 -0.3061 0.96214 -1.1097 -0.6499 -1.1147 0.4808 0.29857 -0.30444 1.3929 0.088861 # Example of an embedding If you are new to the world of word embeddings, a metaphor to understand them is in the form of a supermarket. The supermarket itself is a space in which products are stored based on the category they are in. Inside, you can find an apple by moving your shopping cart to the right coordinates in the fruit section and when you look around you, you’ll find similar products like oranges, limes, bananas, etcetera, and you also know that a cucumber will be closer by than washing powder. This is the same way a word embedding is structured. All the coordinates combined represent a multidimensional hyperspace (often around 300 dimensions) and words that have a similar meaning are more closely related to each other, like similar products in the store. Being able to represent words in a space gives you a superpower, because it allows you to calculate with language! Instead of creating algorithms to understand language, it is now possible to simply look up what is in the neighborhood in the space. How to semantically store data objects While working on software projects in my day-to-day life, I noticed that one of the most recurring challenges presented itself in naming and searching. How would we call certain objects and how could we find data that was structured in different ways? I fell in love with the semantic web but the challenge I saw there, was the need to have people agree on naming conventions and standards. This made me wonder, what if we wouldn’t have to agree on this any more? What if we could just store data and have the machine figure out the concept that your data represents? The validation of the concept was chunked up into three main sections that were validated one by one. Can one get more context around a word by moving through the hyperspace? If so; Can one keep semantic meaning by calculating a centroid of a group of words (e.g., “thanks for the sushi last week”)? If so; Can this be done fast without retraining the ML model? Finding more context around a word has to do with a concept called disambiguation. Take for example the word “apple”. In the hyperspace, if you look for other words in the neighborhood, you will find words related to apple the fruit (e.g., other fruits, juices, etcetera) but also Apple the company (iPhone, Macintosh, and other concepts related to the company). To validate if we could disambiguate the word “apple” the following simple step was taken. What if we looked for all the words that are in the neighborhood of the space in between “apple” and “fruit”? Turns out the results are way better! We can disambiguate by moving through the hyperspace while using individual words as beacons to navigate. In the next step, the goal was to validate if we could keep semantic meaning when storing a data object in the hyperspace by calculating the centroid using the individual words as beacons. We did that as follows, the title of this Vogue article: “Louis Vuitton’s New Capsule with League of Legends Brings French High Fashion to Online Gaming—and Vice Versa”. If we look up the vector positions for the individual words (i.e., Louis, Vuitton, new, capsule, etcetera). And place a new beacon in the center of the space of those vector positions, can we find the article by searching for “haute couture”? This turns out to work as well! Of course, through time, the centroid calculation algorithm in Weavite has become way more sophisticated, but the overall concept is still the same. By validating the above two assumptions, we knew that we could almost instantly store data objects in a semantic space rather than a more traditional row-column structure or graph. Allowing us to index data based on its meaning. Although we had validated the assumptions of the semantic concepts, it was not enough to create an actual semantic search engine, Weaviate also needed a data model to represent these results. Things Rather Than Strings In September 2017 I post about the overlap between the internet of things and the semantic web. IoT focusses on physical objects (i.e., “things”) and the semantic web focuses on the mapping of data objects that represent something (a product, transaction, car, person, etcetera) which on the highest level of abstraction, are also “things”. wrote a blog I wrote this article because, in January 2016, I was invited as part of the Google Developer Expert program in San Francisco to visit the . A conference where, back then, Google’s Weave and Brillo were introduced to the public. Ubiquity Conference Weave was the cloud application built around Brillo, but it piqued my interest because it focussed on “things”, how you defined them, and actions that you could execute around them. The very first iteration of Weaviate focussed on exactly this: “Could Weave be used to define other things than IoT devices like transactions, or cars, or any other things?” In 2017 Google deprecated Weave and renamed Brillo to Android Things but the concept for Weaviate stayed. From the get-go, I knew that the “things” in Weaviate should be connected to each other in graph format because I wanted to be able to represent the relationships between the data objects rather than flat, row-based information as simple and straightforward as possible. This search led to the RDF structure that is used on schema.org which functioned as an inspiration on how to represent Weaviate’s data objects. Weaviate is not perse RDF or schema.org based but definitely inspired by it. One of the most important upsides of this approach was that we could use GraphQL (the graph query language which was entering the software stage through Facebook open-sourcing it) to represent the data inside Weaviate. With the concept of realtime vectorization of data objects and RDF-like representation of Weaviate objects in GraphQL, all the ingredients to turn Weaviate into the search graph that it currently is, were present. The Birth of the Weaviate Search Graph By the end of 2018, I entered a startup accelerator in The Netherlands with Weaviate. A place where I had the time to build a team around Weaviate that could help to get the software to production level and create a business model around the open-source project (the startup became: SeMI Technologies, which is short for Semantic Machine Insights). When the team started, Weaviate was more a traditional graph where the semantic (NLP) element was a feature rather than the core architecture. But when we started to learn how powerful the out of the box semantic search was and the role that embeddings play in day-to-day software development (i.e., a lot of machine-learning models create embeddings as output) the team decided that we wanted to double down on the NLP part and vector storage, creating a unique open-source project which could be used as a semantic search engine, the Weaviate Search Graph was born. Today, people from the SeMI team are working on API design, core implementation, and , and many other things related to Weaviate. tools libraries How people use it today One of the coolest things about an open-source community and users of the software is to see how people use it and what trends you one can see emerge around implementations. The core features of Weaviate are the semantic search element and the semantic classification which are used in a variety of ways and industries. Examples of implementations include: classification of invoices to categories, searching through documents for specific concepts rather than keywords, site search, product knowledge graphs, and many other things. The Future Weaviate will stay fully open source for the community to use, this year we will launch a significant amount of new features (search functions, improved classification, semantic metadata around objects and many more things), an HNSW-based vector index, the Weaviate Cloud service and Weaviate Console (a renewed graphical user interface on top of Weaviate). We would love to hear from you! How do you use and what features would you like to see in the near future? You can leave your ideas on Github, and while you’re there, please don’t forget to give us a star 😉🙏 Weaviate By - Co-Founder & CEO at Bob van Luijt SeMI Technologies

The History of the Weaviate Vector Search Engine

A vector search for the masses needs an intuitive interface Vector-Searching enables a large spectrum of use cases which are impossible with traditional full-text search engines. A pure full-text search tries to match occurrences of terms in a set of documents. At the most, simple distances - such as Levenshtein distances between words - can be computed. on the other hand - a vector search engine - ranks each object by how close it is to another. As a vector searches, Weaviate isn’t limited to text objects. Vector searches also work with music, images, video clips, voice recordings and many more. For simplicity in this article we’re looking at vectors representing text objects. For example in the vector space, the term “Apple iPhone” is much closer to “smartphone” than it is to “hospital supplies”. Weaviate One of the challenges around vector searching is that data is often not present in vectors, but rather in text form. is a vector search engine with a graph-like data model which also allows for higher-level Machine-Learning applications, such as classifications. At the same time, Weaviate’s interface is easy to use and does not require any specific Machine Learning experience from the user. Weaviate High abstraction of underlying ML system for ease of use Weaviate’s user interface allows for insertion of “regular” text-based objects with a predefined schema. In addition to text properties, Weaviate also supports numerical fields, as well as special fields, such as phone numbers or geo coordinates. Additionally, because Weaviate has a graph-like data model, cross-references between data objects can be created. However, the part where vectors are created from those ordinary data objects is under the hood. On each import Weaviate builds a vector representation of an object, so it can assign a position in the multidimensional space to the object. This way “the company Apple Inc.” will be stored in close proximity to “the iPhone”, without the user explicitly making this connection between the two data items. Vector creation at real time thanks to the Contextionary A common association with machine learning models is a long training time. A goal in developing Weaviate, however, has been that it should not feel any different from using a traditional search engine. As a consequence, a long training period to create vectors after data input was never a possibility for us. Imported data objects must become available and searchable in real time or near real time. To achieve real-time vector creation, Weaviate makes use of a tool called the “Contextionary”. The Contextionary is a pre-trained vector-space which contains vectors for nearly all words used in a specific language. Thus, the Contextionary is language-specific, but we are also evaluating cross-language support for future releases. When a text object is imported, the Contextionary acts as a lookup tool to find the new and unique vector position of an object. For example let’s vectorize the following text object in real-time: “I am looking to buy a new smartphone, maybe an iPhone or an Android device”. The most primitive approach is split this string at each space and look up the vector position for each word in the Contextionary. The individual vectors can then be combined to form a new vector: function meanVector(inputVectors) { sums = Array(dimensions(inputVectors)) i = ; i < dimensions(inputVectors); i ++ { j = ; j < inputVectors.length(); j ++ {  
      sums[i] += inputVectors[j][i]
    } 
  } sums. (sum => sum/inputVectors.length())
}

inputParts = input.splitOnNonAlphaNumericCharacters()
vectors = inputParts. (word => getVectorForWord(word))
combinedVector = meanVector(vectors) // takes an array of vectors and outputs a single mean vector // sum up each dimension in a single “sum” vector new for 0 for 0 // divide sum by amount of vectors to get mean value // for each dimension return map map However, if each term was weighed equally this vector position would skew a lot into the “noise center”. The noise center is where stop and filler words, such as “I”, “am”, “looking”, etc. reside in the vector space. A traditional search engine would apply methods such as to determine how meaningful a word is - the Contextionary allows for something similar. tf-idf When vectorizing we’re not only interested in how unique individual terms are in relation to other documents, but also in relation to the pre-trained language model. Our goal is to place the above text snippet close to areas such as “technology”, “phone”, etc - even if no data objects in this category exist yet. So, to work out which terms are relevant we don’t just look at their relevance in relation to other objects, but also to the vector space itself. In more simpler terms this means that very common words in the training set for the Contextionary, such as “I, “am”, “looking”, etc. will always be assigned a lower weight in classification. Rarer terms, such as “smartphone” will receive a higher weight. As a result, each term influences the final position to a different degree, instead of just building the mean position we build the weighted mean. The two most influential terms in the above snippet are “iPhone” and “Android”, but the word “buy” also conveyed some meaning - i.e. the intention to purchase something. Therefore the final position of this dataobject will be close in the vector space to concepts such as “smartphone” and “technology”, but also moved slightly towards “buying” or “shopping”. Since the vector-space is a multi-dimensional space it is possible to move along multiple axes at the same time. Imagine a very simplified vector space that has only two dimensions: The x-axis represents “smartphone” on one end and “groceries” on the other. The y-axis represents “shopping” on one one and “dining” on the other. It is thus possible to move the object close to “shopping” on the y-axis without altering the position on the x-axis which should clearly be in the corner which represents “smartphone”. The Contextionary space can have upwards of 300 or 600 dimensions, thus allowing for a lot of context to be added to a single term. By using the Contextionary as a look-up tool to vectorize a previously unseen data object, Weaviate is able to determine a meaningful vector position in real-time without retraining the underlying machine-learning model. Where does the vector come from at query time? An intuitive interface for importing is only half the battle, at query time we also need a vector position to compare to the targets. Luckily the mechanism that we have used to vectorize an object at import time can be used to vectorize the search query in the same manner. Let’s assume the user searches for “smartphone” in expectation to find the data object we imported previously. Note that the term “smartphone” was never present in the original object, so a pure full-text search would fail at this point and not deliver any results. In Weaviate, however, we can vectorize the search term by looking up the individual words in the Contextionary. Once we have obtained a meaningful vector position of the query, we can do a simple distance comparison, such as cosine similarity, between the source and target objects. If there is no data object, which is even closer to the term “smartphone” than the one we imported previously, our example object will be returned at position one. Extending the Contextionary The Contextionary is trained on common knowledge sources, such as Wikipedia, Wiktionary, as well as website texts available through the . This provides it with a large set of words which are sufficient to cover most cases. Common Crawl However, you might be using very industry-specific terms to convey the most important meaning in a text corpus. Then the Contextionary has simply never learned these in the pre-training phase. To overcome this, the Contextionary can be extended - in real time without retraining - in a similar fashion to how new data objects are vectorized. As an example let’s look at a new fictional smartphone, the “classicphone” produced by “ACME Inc.” which the Contextionary is not yet aware of. Rather than waiting for a new release after the existence of the “classicphone” has become common knowledge, we can extend the current Contextionary with this industry-specific term. To do so, we can describe it with words that are already present in the Contextionary. This process is similar to how a person would explain a previously unknown concept to another person - using words they already know. So for example an explanation of the “classicphone” could be “A smartphone by ACME inc with a large library of classical music tracks”. Similarly, we can use this explanation to extend the Contextionary: curl /v1/c11y/extensions \
  -H -d $WEAVIATE_ORIGIN 'Content-Type: application/json' '{
    "concept": "classicphone",
    "definition": "smartphone by ACME Inc. with classical music",
    "weight": 1
  }' From now on, each data object containing the term “classicphone” will be moved towards this newly created concept during its vectorization phase. HNSW for ultra-large scale vector searches At SeMI, we are convinced that vector-searches will play an even bigger role in the future. We are currently looking into the latest findings of academia, such as - commonly referred to as HNSW graphs - and Random Projection Forests to equip Weaviate for ultra-large scale vector search cases. In the meantime, check out Weaviate on and our . Hierarchical Navigable Small World Graphs GitHub website By - Co-Founder & CTO at Etienne Dilocker SeMI Technologies

Weaviate Is A Search Engine For Vector Embeddings

Any kind of data storage architecture needs an API. You want your users, and their applications, to access and interact with their data. And no matter how complicated the database architecture itself is, you want this interaction to happen as intuitively as possible. Choosing a good API, its design and development, is a crucial but time-consuming process, especially if you want to develop one in an ongoing software development project. uses the API query language . GraphQL enables efficient development and provides high user experience (UX) for data interaction. Weaviate GraphQL In this article we explain how the use of GraphQL leverages the UX of Weaviate, and how we approach the design of this API. What is Weaviate and why does it need an API? is an Search Graph: for understandable knowledge representation, enabling semantic search and automatic classification. Weaviate does not only store data, but also its (automatically derived) context, consisting of linguistic information and relations to other concepts. The result is a network of knowledge, or a graph of data. Weaviate open-source One core question is: How do we interact with this new type of data storage? Interacting with big data, enriched with contextual information, might sound even more overwhelming than interacting with a traditional, relational database. Data needs to be added, retrieved and manipulated, all controlled by the user but enabled by the underlying database interface. Here's where APIs jump in. Because of Weaviate's graph-based architecture, an alternative to traditional RESTful APIs was what we were looking for. What is GraphQL, and why does Weaviate use it? is an API query language, which was open-sourced by Facebook in 2012 and currently maintained by the Linux Foundation. The core strength of GraphQL is that the client is fully in control of what data is returned. This means that over-fetching is never a problem and the need for multiple requests is removed. In addition, using GraphQL allows fast API development and easy API evolvement over time, because of its powerful development tools and single evolving versioning. GraphQL All in all, many reasons to adopt GraphQL for the graph-like structure of Weaviate. GraphQL seems the perfect solution for intuitive database interaction and efficient development. Weaviate still uses traditional RESTful endpoints (using ) to add data, but the main interaction for data consumption goes via GraphQL.) OpenAPI/Swagger GraphQL still follows the same constraints as REST APIs, but data is organized into a graph using only one interface. Thus instead of endpoints, the API is organized in terms of types and fields. The fields of interest are specified by the client, chosen from a schema defined by the server. This requires a well formulated schema. Let's dive into how we approach the design of Weaviate's GraphQL API. Human-Centered API Design So, how does our GraphQL design, development and implementation cycle look like? We approach this very carefully, since we are talking about a design for end-users, made by developers, two sides that differ in many ways. Instead of a data-driven approach, we took a user point of view. Humans tend to talk in interactions; we ask questions when we want to retrieve something, and use verbs or even commands if we want someone or something to do something for us. So why not also apply the same behaviour to interaction with data? So try to think about what you would want to get out of the database in different use cases (as a user), and how this could be done most efficiently. What naturally follows is the words that are frequently used in these interactions. We used these words in the design of our GraphQL schema. Let's look at an example. Consider you (digitally) store supermarket items in Weaviate, and now want to know which items are low in storage. This could be achieved by the following query: {
  Get {
    Things {
      Product(
        where: {
          path: [ ], 
          operator: LessThan, 
          valueInt: }) {
        name
        inStock
      }
    }
  }
} "inStock" 100 Which may return: { : { : { : { : [
          { : , : },
          { : , : },
          { : , : },
          { : , : }
        ]
      }
    }
  }
} "data" "Get" "Things" "Product" "name" "Baguette" "inStock" 8 "name" "Banana" "inStock" 46 "name" "Pineapple" "inStock" 17 "name" "Toilet paper" "inStock" 38 Here's how this query is structured. "Get" and "Things" are high level schema items that we defined. "Get" is a verb that we use when we want to retrieve something. Then, Weaviate makes the distinction between "Things" and "Actions". What comes next is defined by the data schema of the user. This supermarket use case is storing "Products", with at least the properties "name", and "inStock". The user might also store properties like "barcode" and "expirationDate", but chose not to retrieve those in this request. The goal of this query is to Get all the Products where the Stock is less than 100. See how the query is formed? The following query shows the automatic contextualization of concepts in Weaviate. Let's say we are only interested in the fruit items that are low in stock, but we never explicitly stored the supermarket products by category like 'fruits', 'dairy' and 'household'. We can then make the following query, where we want to Get all the Products where the Stock is less than 100, exploring only fruits. Now, only the Banana and Pineapple will be returned because these items are most related to fruits. If we are looking for more results, then Baguette will be shown earlier in the list than Toilet paper, because it is semantically (and in a real supermarket) closer to fruits.) {
  Get {
    Things {
      Product(
        where: {
          path: [ ], 
          operator: LessThan, 
          valueInt: },
        explore: {
          concepts: [ ],
          certainty: }
      ) {
        name
        inStock
      }
    }
  }
} "inStock" 100 "fruit" 0.7 Iterative Prototyping and Feedback In the end, developers and UX designers need to collaborate in this requirement engineering process. The challenge is to match the user needs with technical possibilities, while supporting all features the underlying database has to offer. For the GraphQL API design of Weaviate, an iterative approach was taken. Quick prototypes of new ideas are made all the time, which are passed in a circle of UX designers, core developers and end-users, continuously gathering and implementing constructive feedback. Only when everyone is happy with the proposal (thus not only the developers), a final 'pass' is made to the core developers to implement it in the software. In conclusion With Weaviate, we want to allow the end-user to be focused on their data, and thus not spend too much effort on thinking about how to actually access and interact with it. Only then we can ensure high user experience, and open doors to new insights. With GraphQL, users can count on predictable, fast and stable results, because they are in control of the data interaction. Using our human-centered, iterative design process, with a continuous prototyping and feedback loop, we enable efficient and effective data interaction. Click to learn more about the complete GraphQL structure. here By - Community Solution Engineer at Laura Ham SeMI Technologies

How Weaviate's GraphQL API was designed

Stories, News & Mentions about WEAVAI

Weavai’s Stories on HackerNoon