Streamlining Data Debugging for Game Engine Development with Algolia and Rust

We invited our friends at Starschema to write about an example of using Algolia in combination with Rust. They came up with a very unique use case that we hope will inspire you to use Algolia in different ways. Get a beverage and enjoy this article by Senior Developer Gyula László! The data-driven nature of most contemporary software means that debugging also happens on two different levels: debugging your code and debugging the data as used by your code. While modern IDEs offer a plethora of code debugging tools, they generally contain only very basic data-debugging facilities (the most commonly used is displaying the contents of variables from the current scope). IDEs cannot offer higher-level data-debugging services, because they do not know anything about the higher-level structure of your data. Table of Contents Example scenario What do we want? Algolia Sending data to Algolia Where do we send it and how? Authentication The hosts Creating the request Reality check Rust pieces Marking the data for serialization Sending the data Pre-serialization Wrapping up into a neat API Adding individual object ids Exposing to C/C++ Sample Use-case Making the date searchable The search UI Potential further steps Async data transfer Retry logic Less string manipulation. Example scenario The example we’ll be using envisions a small game engine dev team working as part of a larger team developing content for the game engine (in Rust). The challenge is to debug cases coming from QA. If we assume that unit testing catches the most straightforward bugs (like typos, basic logic problems, etc), these QA issues generally stem from the fact that the data created by the content teams are vastly different and much more complex than scenarios tested by the unit tests. Debugging generally involves large amounts of logs and post-run collation of data created during the run. One solution to ease this pain as used by CD Project Red, and explained in the ‘The Witcher 3: Optimizing Content Pipelines for Open-World Games’ GDC talk, is to have a searchable database for every entity used during build time. https://www.youtube.com/watch?v=p8CMYD_5gE8&embedable=true This is a great solution for product-wide issues (and was generally aimed at content developers), but may not be the best fit for our “small team focused on a core code development” scenario: They require cooperation from all other teams (and product-wide build system modifications) The DB requires maintenance “All of the data” is generally overkill for these debug sessions (sorting through will eat up more time) Debugging by modifying the data for individual test runs is not supported (as the whole DB assumes a single, consistent set of input data) What do we want? The idea of having a database feels good, but we have some more requirements: We want to build the database from runtime data (not build time) We want multiple, parallel databases (as the team can be working on a number of issues) Entity data is freeform — as we may change structure during debugging, we do not want to specify schemas in advance We do not want to waste time maintaining the database We want fast search capabilities * * Some notes on “fast search”: one of the most common tasks of tracking things down is to match addresses or ids across multiple entities, and we do not want to run multiple SQL queries per entity type, as it is slow and demotivating to copy-paste the same id to 9 different queries, and to then copy-paste the results into a single text editor window for checking. Algolia When looking at , we see that it meets our DB criteria: Algolia Support for multiple databases Data is freeform No maintenance needed Has really good search capabilities Can build simple, custom UIs for searching With this in mind, we can create a list of what we need to code to get data to Algolia: We want an easy way to mark up data for Algolia serialization (we can use JSON serialization for this purpose) Create a library that can send serialized data to Algolia (preferably with a single, simple call) Enable easy configuration of the library Sending data to Algolia Where do we send it and how? Algolia does provide on how to use the , however, none of the currently available libraries work for Rust (no C or C++ version to wrap either) so we’ll have to make use of the API clients automation repository which is used to generate the clients themselves. documentation REST API As a baseline, let’s look at the JavaScript client example provided: // for the default version import { algoliasearch } from 'algoliasearch'; // you can also import the lite version, with search only methods // import { liteClient } from 'algoliasearch/lite'; const client = algoliasearch(' ', ' '); // The records retrieved by any of your data sources const recordsFromDataSource = [ { name: 'Tom Cruise' }, { name: 'Scarlett Johansson' }, ]; // Here we construct the request to be sent to Algolia with the `batch` method const requests: BatchOperation[] = recordsFromDataSource.map((record) => { return { // `batch` allows you to do many Algolia operations, but here we want to index our record. action: 'addObject', body: record, }; }); const { taskID } = await client.batch({ indexName: ' ', batchWriteParams: { requests, }, }); // Wait for indexing to be finished await client.waitForTask({ indexName: ' ', taskID }); console.log('Ready to search!'); So the process looks fairly straightforward: Provide authentication for Algolia using the and APP_ID API_KEY Pack up the data that we’d like to send (see ) this spec partial Send the request to Algolia We can skip waiting for indexing to be done (as we are in core code we’d like to avoid long waits) One downside of using the REST API directly is that Algolia does not offer an if you are not using an official API Client. SLA Authentication To figure out how authentication is handled let’s trace the constructor call: algoliasearch(...) In we see that the default auth option is and is hard-coded — we’ll use this clients/algoliasearch-client-javascript/packages/algoliasearch/builds/browser.ts WithinQueryParameters Later we see that in is what does the actual job of building most of the object (including the parts we care about) createSearchClient() clients/algoliasearch-client-javascript/packages/client-search/src/searchClient.ts algoliaclient This creates the authentication using in : createAuth() clients/algoliasearch-client-javascript/packages/client-common/src/createAuth.ts const credentials = { 'x-algolia-api-key': apiKey, 'x-algolia-application-id': appId, }; return { headers(): Headers { return authMode === 'WithinHeaders' ? credentials : {}; }, queryParameters(): QueryParameters { return authMode === 'WithinQueryParameters' ? credentials : {}; }, }; This clearly shows that we’ll have to add and to the query parameters for authentication. x-algolia-api-key x-algolia-application-id The hosts The host list to target is coming from the function in , and we see that it has , , and hosts: getDefaultHosts() searchClient.ts read write readwrite function getDefaultHosts(appId: string): Host[] { return ( [ { url: `${appId}-dsn.algolia.net`, accept: 'read', protocol: 'https', }, { url: `${appId}.algolia.net`, accept: 'write', protocol: 'https', }, ] as Host[] ).concat( shuffle([ { url: `${appId}-1.algolianet.com`, accept: 'readWrite', protocol: 'https', }, { url: `${appId}-2.algolianet.com`, accept: 'readWrite', protocol: 'https', }, { url: `${appId}-3.algolianet.com`, accept: 'readWrite', protocol: 'https', }, ]) ); } When checking the Transporter method, we can see that the hosts are used for requests, and the hosts are used for everything else. request() read GET write As debugging is not mission-critical, we’ll ignore the retry logic used by fully-fledged Algolia clients. Creating the request We can also see the request structure in : searchClient.ts .batch() batch( { indexName, batchWriteParams }: ...) { // ... const requestPath = '/1/indexes/{indexName}/batch'.replace( '{indexName}', encodeURIComponent(indexName)); // ... const headers: Headers = {}; const queryParameters: QueryParameters = {}; const request: Request = { method: 'POST', path: requestPath, queryParameters, headers, data: batchWriteParams, }; return transporter.request(request, requestOptions); }, We can see that the path for the request will be , and the request looks like a standard request. /1/indexes/ /batch POST When tracing the call that initializes this we see that it adds a query parameter with value. createTransporter() x-algolia-agent Algolia for JavaScript ( ) Reality check With most of the details figured out, we want to create a quick test using the official Algolia Client SDK for Javascript to check if our assumptions about sending the data are correct. Using the , it’s easy to put together a simple HTML file that issues a batch upload request to Algolia: Javascript Client Documentation "> // Connect and authenticate with your Algolia app const client = algoliasearch(' ', ' ') // Create a new index and add a record const index = client.initIndex(' ') const record = { objectID: 1, name: 'test_record' } index.saveObject(record).wait() // Search the index and print the results index .search('test_record') .then(({ hits }) => console.log(hits[0])) When running this in the browser we can see that it’s working, however, one difference we do see is that the and authentication keys are sent in headers, not in the query string. x-algolia-api-key x-algolia-application-id We’ll take note of this and use it instead of passing them as query parameters. Another noteworthy bit is that the content type of the request (despite having a JSON body) is , not (as one would guess by default) or (as the default content type for Transport). We take this as a hint to ignore content type until we encounter problems. application/x-www-form-urlencoded application/json text/plain Rust pieces Marking the data for serialization For handling the serialization we’re going to use the library. SerDe SerDe is a framework for serializing and deserializing Rust data structures efficiently and generically. This requires our structures to implement the trait (for most structures), and we only have to add a derive to their definition: Serialize Serialize // A vector of 3 floats. // // The serialized form will look like `{"x":14.1,"y": 86.2,"z":14.3}` #[derive(Serialize)] struct Float3 { x: f32, y: f32, z: f32, } One extra bit we saw in the sample code for the JavaScript Client SDK is the field. To quote from the Algolia indexing documentation: objectID Every object (record) in an index eventually requires a unique ID, called the objectID. You can create the objectID yourself and send it when indexing. If you don’t send an objectID, Algolia generates it for you. Whether sent or generated, once you add a record, it has a unique identifier called objectID. For our application automatically generated unique ids would be ideal, but in reality, when sending data to the batch ingestion endpoint, all objects must have an objectID field and we can use SerDe to rename/transform/rearrange the fields. // create a string key from the integer and add a prefix for unique id fn particle_emitter_object_id (id: &i32, s: S) -> Result where S: Serializer, { s.serialize_str(format!("particle_emitter_{}", id)) } // An example entity that a customized objectID for serialization // The `rename_all` is used to transform underscored_name_format to camelCaseNameFormat #[derive(Serialize, Deserialized, Debug)] #[serde(rename_all = "camelCase")] struct ParticleEmitterEntity { // serialize this field as `objectID`, but transform it using the `particle_emitter_object_id` field #[serde(rename = "objectID", serialize_with="particle_emitter_object_id")] id: i32, // nested objects are automatically serialzied root_position : Float3, // ... } Sending the data We’ll use the Request crate for our HTTP transport: [dependencies] reqwest = { version = "0.11", features = ["blocking", "json"] } urlencoding = { version = "2.1" } use urlencoding; fn main() -> Result > { // The credentials data const APP_ID: &str = " "; const INDEX_NAME: &str = " "; const API_KEY: &str = " "; const ALGOLIA_AGENT: &str = "Algolia DataSender for Rust Dev (0.0.1)"; // build the URI for the batch let host = format!("{}.algolia.net", APP_ID.to_lowercase()); // The index name can have non-url-friendly characters let path = format!("/1/indexes/{}/batch", urlencoding::encode(INDEX_NAME)); let uri = format!("https://{}{}", host, path); let uri_with_client = format!("{}?x-algolia-agent={}", uri, ALGOLIA_AGENT); // The batch data needs to be in the proper batch request format. // // let data = .... // The `blocking` client waits for the request to complete let client = reqwest::blocking::Client::new(); let res = client .post(uri_with_client) .header("x-algolia-api-key", API_KEY) .header("x-algolia-application-id", APP_ID) .body(data) .send()?; Ok(()) } Pre-serialization Since our goal is to be able to send arbitrary types to Algolia, we do not really care about the underlying objects, just the data. This helps because we would have a very hard time managing object lifetimes in if we wanted to store actual objects in a buffer (and the type of magic that we’d have to do would be dense and impenetrable). The solution is to serialize objects into a shared buffer as soon as they arrive and work from the pre-serialized format when sending objects in batches. Rust Serde serialization is not suited to composing pre-serialized data, so we’ll do it the un-Rust way: we compose the data via string manipulation instead of type-safe serialization code. [dependencies] serde = { version = "1.0", features = ["derive"] } serde_json = { version = "1.0" } use serde::Serialize; // ... // The data we'll send for testing #[derive(Serialize, Debug)] struct Float3 { #[serde(rename="objectID")] object_id: String, x: f32, y: f32, z: f32, } // RECREATE THE REQUESTS STRUCTURE // ------------------------------- // {"requests":[{"action":"updateObject","body":{"objectID":1,"name":"test_record"}}]} // convert a pre-serialized JSON object into a request object for a batch request fn build_batch_request_entry(data: &String) -> String { format!("{{\\\\"action\\\\":\\\\"updateObject\\\\",\\\\"body\\\\":{}}}", data) } // wrap the individual requests into a batch fn wrap_batch_request_entry(rows: &Vec ) -> String { format!("{{\\\\"requests\\\\":[{}]}}", rows.join(",")) } // Create some raw data we'll send for testing let raw_data = vec![ serde_json::to_string(&Float3 { object_id: String::from("point1"), x: 0.1, y: 0.2, z: 0.3, }) .unwrap(), serde_json::to_string(&Float3 { object_id: String::from("point2"), x: 1.1, y: 1.2, z: 1.3, }) .unwrap(), ]; // transform the list of raw data objects to individual `updateObject` requests, // then collect and wrap them in the `requests` array for the batch Algolia request let data = wrap_batch_request_entry(&raw_data.iter().map(build_batch_request_entry).collect()); Using this pre-wrapped data we can combine them into a working example submitting data to Algolia indices which you can find in . example1.rs Wrapping up into a neat API Now that the transport and data encoding is ready, we can wrap this up into a neat API: // Allows sending serializable data to Algolia indices pub struct AlgoliaSender { /// ... } impl AlgoliaSender { // create a new sender with the given credentials and index pub fn new(app_id: &str, api_key: &str, index_name: &str) -> Self { } // adds a new serializable item to the list to be sent on the next send_items() call pub fn add_item (&mut self, v: &T) where T: Serialize {} // Send the items to the default index pub fn send_items(&mut self) {} // Sends items to the ingestion endpoint in a batch job to a specified index pub fn send_items_to_index(&mut self, index_name: &str) {} } Which then can be used like this: // create a sender let mut sender = AlgoliaSender::new(APP_ID, API_KEY, INDEX_NAME); // add some items to send sender.add_item(&Float3 { object_id: String::from("point3"), x: 0.1, y: 0.2, z: 0.3, }); sender.add_item(&Float3 { object_id: String::from("point4"), x: 1.1, y: 1.2, z: 1.3, }); // send the items in a request (so batch split points can be controlled) sender.send_items(); For the full code of this step please see . example2.rs Adding individual object ids Since we do not really want to add fields to our internal data structures, we have to add some extra logic to generate these for us. objectID First, let us create a generator for these IDs: // Generates unique IDs to use as ObjectIds. // // This implementation combines a prefix with an incremental index. pub struct IdGenerator { prefix: String, idx: i32, } impl IdGenerator { // Creates a new IdGenerator with the given prefix pub fn new(prefix: String) -> Self { IdGenerator { prefix: prefix, idx: 0 } } // Creates a new IdGenerator with the current epoch time as prefix pub fn new_from_time() -> Self { let now = SystemTime::now(); let prefix = format!("{:?}", now.duration_since(std::time::UNIX_EPOCH).unwrap_or_default()); Self::new(prefix) } // Returns a new integer id and increments the internal counter pub fn next_i32(&mut self) -> i32 { let idx = self.idx; self.idx += 1; idx } // Returns a concatenated string id and increments the internal counter pub fn next_string(&mut self) -> String { let next_id = self.next_i32(); format!("{}_{}", self.prefix.as_str(), next_id) } } Then we can do the ugly string manipulation of injecting a generated objectID into our already serialized objects. This assumes that everything we are sending is an object, not an array or a primitive value — with Algolia expecting objects this is just fine for us. We use to skip entries that are too short (which would be most likely invalid JSON). filter_map() // converts a pre-serialized JSON object into a request object for a batch request using DANGEROUS STRING MAGIC fn build_batch_request_entry(id: &String, data: &String) -> Option { if data.len() <= 2 { return None; } // Skip the opening curly brace and inject our objectID into the already serialized object let remaining_data:String = data.chars().skip(1).collect(); Some(format!("{{\\\\"action\\\\":\\\\"updateObject\\\\",\\\\"body\\\\":{{\\\\"objectID\\\\":\\\\"{}\\\\",{}}}", id, remaining_data)) } // Wrap all pre-serialized data with an object id and the `updateObject` request, then collect into a batch request let data = wrap_batch_request_entry( &self .data_buffer .iter() .filter_map(|entry: &String| -> Option { build_batch_request_entry(&self.id_generator.next_string(), entry) }) .collect(), ); The full code for this step is available in . example3.rs Exposing to C / C++ Since our library is separated into two distinct parts: serializing objects to and sending the data, we can actually expose our Rust library to C / C++ code for easy data ingestion from those languages. If they can provide JSON-serialized data then we can send it. JSON Our goal is to have an API with a constructor and destructor for the struct and functions that mimic the public API of the Rust side. The following is a first-draft, brute-force implementation of these functions: [dependencies] libc = "0.2" // We need to support adding pre-serialized data directly impl AlgoliaSender { //... // adds an already serialized value to be sent pub fn add_raw_item(&mut self, data: &str) { self.data_buffer.push(String::from(data)); } // ... } // C API // ----- // Constructor for the AlgoliaSender struct from C. // Returns nullptr if any of the parameters are empty. #[no_mangle] pub unsafe extern "C" fn algolia_sender_new(app_id: *const libc::c_char, api_key: *const libc::c_char, index_name: *const libc::c_char) -> *mut AlgoliaSender { // rustify all parameters let app_id_str: &str = std::ffi::CStr::from_ptr(app_id).to_str().unwrap_or(""); let api_key_str: &str = std::ffi::CStr::from_ptr(api_key).to_str().unwrap_or(""); let index_name_str: &str = std::ffi::CStr::from_ptr(index_name).to_str().unwrap_or(""); // check these parameters if app_id_str.is_empty() || api_key_str.is_empty() || index_name_str.is_empty() { return std::ptr::null_mut(); } // create a new sender and Box it, and use `Box::into_raw()` to get a pointer that outlives this function call let struct_instance = AlgoliaSender::new(app_id_str, api_key_str, index_name_str); let boxed = Box::new(struct_instance); Box::into_raw(boxed) } // Destructor for the AlgoliaSender struct from C #[no_mangle] pub unsafe extern "C" fn algolia_sender_destroy(struct_instance: *mut AlgoliaSender) { // let the Rust lifetime take over and destroy the instance after the function is done Box::from_raw(struct_instance); } // Adds an item to be sent to the target sender #[no_mangle] pub unsafe extern "C" fn algolia_sender_add_item(a: *mut AlgoliaSender, data: *const libc::c_char) { // convert data let data_str = match std::ffi::CStr::from_ptr(data).to_str() { Err(_) => return, Ok(s) => String::from(s), }; // attempt to add it match a.as_mut() { None => return, Some(sender) => sender.add_raw_item(data_str), }; } // Trigger the sending of items #[no_mangle] pub unsafe extern "C" fn algolia_sender_send_items(a: *mut AlgoliaSender) { // attempt to send the items match a.as_mut() { None => return, Some(sender) => sender.send_items(), }; } The full source code for this step is available in . example4.rs Sample use-case Our example is a fairly simple one: The content team is creating a map by placing objects and by writing rules for procedural content generation We want to keep a searchable database of objects (including procedural ones) all We want to be able to find objects by any property and display the known properties of that object in the browser From the browser search, we want to be able to quickly go to that object in the internal editor tool for evaluation (using a link with a custom protocol) The data structure that we’d like to send over looks like the following: #[derive(Serialize)] pub enum EntityCategory { StaticEntity, ... } #[derive(Serialize)] pub struct PlacedEntity { // The root position of the object -- we'll use these coordinates to create the link to open the editor root: Float3, // we'll use this as an example of adding selectable categories to the search UI. category: EntityCategory, // other properties // ... } The category enum is stored as an integer under the hood, but thankfully by default, SerDe represents Enums as “externally tagged”, so they should arrive at Algolia as Strings instead of integers. We’ll use this category attribute to add a “category selector” on the UI. To be able to keep track of both static and procedural objects, we can use the library to send object data to Algolia after object creation. For this example, we assume that a is used for placing a single object and is called after a batch of objects is added. ObjectContainer::place_entity() ObjectContainer::placement_done() // ... impl ObjectContainer { pub fn new() -> Self { Self { //... algolia_sender: AlgoliaSender::new( /* ... */), } } // ... pub fn place_entity(&mut self, e: &PlacedEntity) { // ... self.algolia_sender.add_item(e); // ... } // ... pub fn placement_done(&mut self) { // ... self.algolia_sender.send_items(); // ... } } Marking the data as searchable To set up our data for the Search UI, we must tell Algolia which attributes to make searchable for better results, as per the documentation: When you create an index, all attributes from all your records are searchable by default. Having all attributes searchable by default lets you perform searches right from the start without having to configure anything. Yet, if you want to make your search more relevant and remove the noise, you just want to set meaningful attributes as searchable. For our use case, this “search all” behavior is fine, but if we want a more fine-grained approach we can either use an existing Algolia API Client or the Algolia Dashboard ( ). Note: nested attributes (sub-objects) cannot be added from the Dashboard, only the API The search UI Algolia InstantSearch provides a very easy way to construct custom UIs for browsers that allow us to search our data. We’re simply going to copy the and modify it to fit our needs. InstantSearch getting started guide We can keep most of the layout and HTML: as in the demo --> Categories as the in the demo --> tags as in the demo --> The Javascript is also almost the same as the demo, except for setting up the Category list instead of the Brands list and changing the displayed hits to contain our attributes. The only unusual bit here is the customization of the hits to include a custom URL that opens an external editor as a demonstration of Algolia interacting with external applications: Open in editor /* global instantsearch algoliasearch */ const search = // ...same as the demo search.addWidgets([ instantsearch.widgets.searchBox({ container: '#searchbox', }), instantsearch.widgets.clearRefinements({ container: '#clear-refinements', }), // categories instead of brands instantsearch.widgets.refinementList({ container: '#category-list', attribute: 'category', }), // customize the hit display instantsearch.widgets.hits({ container: '#hits', templates: { item: ` Open in editor Category {{#helpers.highlight}}{ "attribute": "category" }{{/helpers.highlight}} `, }, }), instantsearch.widgets.pagination({ container: '#pagination', }), ]); search.start(); And with only a few lines of modification, our basic Search UI is ready: we can search for objects, filter them by category, and open the objects in an external editor: Potential further steps Async sending of data The create provides async support for Reqwest and can be used to implement async sending of the data. Using an async wrapper complicates calling the exported functions from C / C++ and can mess with the CPU core allocation of other running code. Tokio Retry logic All Algolia clients implement retrying failed requests (and retrying on different target servers). After moving the data transfer to async processing it is relatively easy to add the retry logic as a separate layer. Less string manipulation While we had fun -ing and concatenating Strings, it would be nicer if at least the objectID generation was more Rust-like: an trait and a nice macro would be a better solution for most use cases, as they allow control and repeatability, but keep the flexibility. format! objectID derive pub trait objectID { fn object_id(&self) -> String; } #[derive(Serialize, objectID)] #[object_id(prefix="something", field="id")] struct Something { id: i32, } impl objectID for Other { // .... } Also published here.