How to Achieve Schema and Validation in NoSQL Using Ottoman and Couchbase

With Ottoman, you declare schema in your code. Although Couchbase has no schema enforcement for your documents, most applications need some level of schema even in NoSQL. We will explore how to achieve schema and validation in NoSQL using Ottoman and Couchbase. It’s important to validate that documents meet certain requirements before persisting. Although Ottoman creates an abstraction over the Couchbase SDK, the benefits outweigh the drawbacks. A developer creates a lot of logic around creating and updating documents, writing pre/post lifecycle, working with data structures, and validation. NoSQL Database and Schema Design An ODM serves a similar role in NoSQL as it does in a relational database, but with additional benefits. Couchbase doesn’t enforce validation as it is schema-flexible. Ottoman can perform certain checks are being made as your applications persist data. We can warn and error against unwanted data types and formats for individual fields by defining schema and models for various document types. Your server application code is where a great place to apply business logic and validation. Ottoman’s goal is to provide a better development experience along with giving you control over schema and validation while using Couchbase with Node. We want to give developers a reliable tool to build systems that are easy to design, maintain, and scale. Couchbase is a NoSQL Database Couchbase Server is a schema-less document database, categorized as a NoSQL datastore, it’s not the best description as Couchbase uses a variant of SQL for querying called . Just because a schema is not strictly enforced in NoSQL databases like Couchbase does not mean you should not enforce it. N1QL With Couchbase, you get the benefits of enterprise-level scaling, clustered nodes, and the ability to store and retrieve data in a JSON format. If you are familiar with Mongoose, an ODM for MongoDB, you will feel pretty comfortable with Ottoman as they have many overlapping features because they both are made for NodeJS and used to model and persist data to a JSON document-oriented key-value database. Document vs Relational Database As we explore we will explore Couchbase and NoSQL Database and Schema Design, we will first look at how a document data structure differs from a relational database design, in the example below, you will see a side-by-side comparison of data that represents a Hotel. On the left, we have a document that can store phone numbers in an array allowing us to store multiple phone numbers for a single hotel. To do this in a relational database you would most certainly need a new table and to maintain a relationship between the two using primary keys. For more info on document modeling, check out the resources I have put together in a blog post: JSON Data Modeling Guide In Ottoman, we have many constructs to help define schema and models at the application level. Let’s go over some of the most important terms you need to know in Ottoman. Types As seen in pink in the image above, document properties with the name type in Couchbase are near equivalent to tables in a relational database. They can help to group different types of JSON documents together for indexing purposes. When using Secondary Indexes in Couchbase, we are able to index on any key in the documents. If only 100 of 10,000 documents in your database use a type of ‘hotel’ and you normally want to search for hotels based on city or state, then you might want to build a that only needs to search through those hundred documents where city or state equal a certain value. This is much more efficient than using, for instance, a Primary Index. Composite Secondary Index ! Learn more about indexing in Couchbase Collections Similar to types in Couchbase 6.x (latest Couchbase major version at the time of writing) and not shown in the illustration above, collections will be favored in Couchbase 7 ( ). In the case you were to use collections, you would simply not have a property on each document and instead have the document assigned to a collection. already in beta ‘type’ ‘hotel’ Documents Comparable to rows of data in a relational database. Traditional RDBMS systems will reference related documents from other tables as seen in the illustration above. You can also do this with a JSON document, however; it is suggested to include that information as an embedded document when possible, as we see with the hotel document’s phone number property which is an array of phone numbers. Albeit a very simple example of this, think if you had an address property that itself was another JSON object with many properties, you may think this needs to be given its own document, but nesting that information in the parent document in most cases is fine. Fields Also known as attributes, are similar to columns in a relational database and with Ottoman, you can create field-level requirements as part of your schema. Schema While Couchbase is schema-less or schema-flexible, we can still enforce structure at the application level for our documents. Model A constructor method that takes a schema and creates an instance of a document equivalent to a single record in a relational database. This document instance can be constructed and then persisted to Couchbase by Ottoman . using the save() method Getting Started Let’s get started creating a demo application that we can use to get familiar with Ottoman as an object document mapper. Couchbase Installation Before we get started, let’s set up Couchbase. You can choose from one of the following options (we are using option #1 for this article): Install Couchbase Server using Docker Download Couchbase specific for your OS from the Couchbase Website Let’s navigate through some of the basics of Ottoman by implementing a model that represents data for a simplified airline example keeping in tradition with Couchbase’s Travel-Sample dataset. I am using Visual Studio Code, NodeJS v12.14, and NPM 6.14.8, so you will need to have Node.js installed on your machine. next, we will create a blank project and get started writing some code. Initializing our Project with NPM Create a directory, initialize our project, install Ottoman JS and open in VS Code mkdir intro-ottoman && && npm init -y && npm i ottoman && touch createAirline.js && code . cd $_ Open the terminal in your editor of choice, I have added a command at the end that will open in VS Code. Connection to Couchbase with Ottoman We will start working out of the file under the project root and add the following (based on the default configuration): ./createAirline.js { Ottoman, model, Schema } = ( ) ottoman = Ottoman({ : }) ottoman.connect({ : , : , : , : }) const require 'ottoman' const new collectionName '_default' connectionString 'couchbase://localhost' bucketName 'travel' username 'Administrator' password 'password' Together this imports the ottoman package and specifies the default collection (Couchbase Server 6.x style) Ottoman Schema and Models Models are fancy constructors compiled from Schema definitions. An instance of a model is called a document. Models in Ottoman help you to easily create, read, update, and delete documents in your Couchbase database. Creating an Ottoman model comprises of a few things: Defining a Document Schema A schema defines document properties through an object where the key name corresponds to the property name in the collection. airlineSchema = Schema({ : , : , : }) const new callsign String country String name String Here we define three properties within our schema, all of type String. By specifying a type for each of our model properties, maps to an internal validator that will be triggered when the model is saved to the database and fail if the data type of the value is not of type String. (callsign, country, name) The following Schema Types are permitted: String Number Array Boolean Date EmbedType MixedType ReferenceTypes Defining a Document Model We need to call the model constructor on the Ottoman instance and pass it the name of the collection and a reference to the schema definition. Airline = ottoman.model( , airlineSchema) const 'Airline' When you call the function it creates a copy of the schema and compiles the model for you. model() Let’s also give the a phone number property. We can add a validation function that will ensure that the value is a valid phone number. Replace the section with these three blocks of code: airlineSchema airlineSchema regx = (value && !value.match(regx)) { ( ) } } addValidators({ : phoneValidator }) airlineSchema = Schema({ : , : , : , : [{ : , : }] }) const /^($[0-9]{3}$|[0-9]{3}-)[0-9]{3}-[0-9]{4}$/ if throw new Error `Phone Number is not valid` ${value} phone const new callsign String country String name String phone type String validator 'phone' In the example above, I show you how to create a custom validator, we just happen to be using a regular expression as the check-in our validator, just understand that you could have any logic inside one of these validators and in a moment I will show you a nice trick to reduce our code down considering we are using regular expression for our matching of the phone numbers. Defining Validators registered with Ottoman (as we have done here with the method) will be called once for every value our document’s property has in the array. If the property did not have an array and instead just a single String value, the validator would only be run once. For this reason, I print out the problematic phone number if the validation fails. Validators ottoman.addValidators() There is however an easier way to validate any document properties value so long as the check you are performing uses a regular expression. The can take a and as an argument, so we can reduce our code down to: ValidatorOption regexp message regx = airlineSchema = Schema({ : , : , : , : [{ : , : { : regx, : } }] }) const /^($[0-9]{3}$|[0-9]{3}-)[0-9]{3}-[0-9]{4}$/ const new callsign String country String name String phone type String validator regexp message 'phone invalid' As you can see, we can do everything we were doing before inline when creating a new Schema. But don't let this keep you from understanding how to create a custom validator, sometimes we need additional logic and that is why the first example is still something worth knowing. Basic Document Operations in NoSQL Most of the basic operations are covered in our Ottoman V2 documentation, we will cover some of that here, but feel free to dive into our and let us know if there is something that you cannot find or don’t understand. Ottoman V2 (alpha) docs Create Documents Considering the code that we already went over above that creates a schema, model, and validators. Saving a model and persisting it to the database is quite easy. Let’s create a new Airline model using our Schema and then save/persist it to the database. cbAirlines = Airline({ : , : , : , : [ , ] }) saveDocument = () => { { result = cbAirlines.save() .log(result) } (error) { error } } ottoman.start() .then( () => { saveDocument() .then( process.exit( )) .catch( .log(error)) }) // Constructing our document const new callsign 'CBA' country 'United States' name 'Couchbase Airlines' phone '321-321-3210' '321-123-1234' // Persist the Couchbase Airlines document to Couchbase Server const async try const await console catch throw // Ensure that all indexes exist on the server // Next, let's save our document and print a success message async => () 0 ( ) => error console You may be wondering why we don't just call the saveDocument() function on its own. Instead, we call it after the is finished. The start method is a shortcut to run and ensureIndexes. All you have to know for now is that this method makes sure that the proper Ottoman-related indexes have been created on Couchbase and this is important to ensure we can run things like the method and use tools like the which we will get to at the end of the article. ottoman.start() ensureCollections find() QueryBuilder At this point, if you were to run all of the code we have written using Node, our document would be saved to the database: node createAirline.js The result from this operation: _Model { callsign: , country: , name: , phone: [ , ], id: , _type: } 'CBA' 'United States' 'Couchbase Airlines' '321-321-3210' '321-123-1234' '2384568f-f1e9-446e-97d1-cad697c40e76' 'Airline' The following fields are returned: Callsign, country, and name fields are all String, the most basic value we could have in a document. The id field is auto-generated by Couchbase and is a unique key. The ID value is what you will use in any case to find a document with Ottoman in methods like or The phone field is represented by an array and contains valid phone numbers. The field can help us to organize our documents like a table does in a relational database, . findByID removeByID _type in Couchbase 7, we can use collections and scopes Validation Errors If we enter an invalid phone number and run node createAirline.js this file again, we would get an error message: ValidationError: Phone Number 321-321-32xx is not valid TIP: You can define your connection, schema, and models in separate files, export, and use them in other files. Create a new file named and move our schema and model definition to it: airline-schema-model.js { model, Schema } = ( ) regx = airlineSchema = Schema({ : , : , : , : [{ : , : { : regx, : }}] }) Airline = model( , airlineSchema) exports.airlineSchema = airlineSchema; exports.Airline = Airline; const require 'ottoman' const /^($[0-9]{3}$|[0-9]{3}-)[0-9]{3}-[0-9]{4}$/ const new callsign String country String name String phone type String validator regexp message 'phone invalid' // Compile our model using our schema const 'Airline' Now we can create a few new files, , , and removeAirline.js and populate each file with the following: findAirline.js updateAirline.js { Ottoman } = ( ) ottoman = Ottoman({ : }); ottoman.connect({ : , : , : , : }); { Airline } = ( ) const require 'ottoman' const new collectionName '_default' connectionString 'couchbase://localhost' bucketName 'travel' username 'Administrator' password 'password' const require './airline-schema-and-model' This will help us to separate some of our code so we are not repeating it in each file and as we go over each of the CRUD operations we can just add some code to each file and our schema and model will already be imported. Find Documents Let’s try to retrieve the record we saved to the database earlier. The model class exposes several static and instance methods to perform operations on the database. We will now try to find the record that we created previously using the find method and pass the callsign as the search term. Let’s create a new file named findAirline.js and we can add the following code: findDocument = () => { { Airline.find({ : { : } }) .then( .log(result.rows)); } (error) { error } } ottoman.start() .then( () => { findDocument() .then( process.exit( )) .catch( .log(error)) }) // Find the Couchbase Airline document by Callsign from Couchbase Server const async try callsign $like 'CBA' ( ) => result console catch throw async => () 0 ( ) => error console Find document result: Query Result: [ _Model { _type: , callsign: , country: , name: , phone: [ , ], id: , } 'Airline' 'CBA' 'United States' 'Couchbase Airlines' '321-321-3210' '321-123-1234' '971045ac-39d8-4e72-8c93-fdaac69aae31' Update Documents Let’s modify the record above by finding it using the callsign, which we can assume that callsign will be a unique field in our data, then we can update the document all in a single operation. findDocumentAndUpdate = () => { newDocument = { : , : , : , : [ , ] } { result = Airline.findOneAndUpdate( { : { : } }, newDocument, { : } ) .log(result) } (error) { error } } ottoman.start() .then( () => { findDocumentAndUpdate() .then( process.exit( )) .catch( .log(error)) }) // Update the Couchbase Airline document by Callsign from Couchbase Server const async const callsign 'CBSA' country 'United States' name 'Couchbase Airways' phone '321-321-3210' '321-123-1234' try let await callsign $like 'CBA' new true console catch throw async => () 0 ( ) => error console Find document and update result: _Model { _type: , callsign: , country: , id: , name: , phone: [ , ] } 'Airline' 'CBSA' 'United States' '971045ac-39d8-4e72-8c93-fdaac69aae31' 'Couchbase Airways' '321-321-3210' '321-123-1234' Remove Documents Ottoman has several methods that deal with removing documents: , and . Considering the many examples we have had so far, each of these should be very easy to understand how to use, so we will just provide a simple example here to show how to remove a document that we have already found using the method. remove removeById removeMany find() removeDocument = () => { { Airline.removeById( ) .then( .log(result)) } (error) { error } } // Remove the Couchbase Airline document by ID from Couchbase Server const async try await '60e3f517-6a2a-41fe-be45-97081181d675' ( ) => result console catch throw Remove document result is a simple , used to track changes in Couchbase documents. cas value { cas: CbCas { : } } '0' Middleware We have already seen our middleware in action, our validator that we initially created can take advantage of middleware using functions that run at specific stages of a pipeline passing control during the execution of asynchronous functions. Available Hooks validate save update remove Example of Middleware (a.k.a. ) pre and post hooks Let’s try an example by simply generating a log in the console before and after the creation (save) of a document, I’m going to create a new file called createWithHooks.js and most of the code will look familiar except I have added pre and post hooks that will just report to us the document name pre-save and document id post-save: { Ottoman } = ( ) ottoman = Ottoman({ : }); ottoman.connect({ : , : , : , : }); { Airline, airlineSchema } = ( ) pluginLog = { airlineSchema.pre( , (doc) => .log( ) ) airlineSchema.post( , (doc) => .log( ) ) }; airlineSchema.plugin(pluginLog) cbAirlines = Airline({ : , : , : , : [ , ] }) saveDocument = () => { { result = cbAirlines.save() .log(result) } (error) { error } } ottoman.start() .then( () => { saveDocument() .then( process.exit( )) .catch( .log(error)) }) const require 'ottoman' const new collectionName '_default' connectionString 'couchbase://localhost' bucketName 'travel' username 'Administrator' password 'password' const require './airline-schema-and-model' // Plugins and Hooks are middleware, think lifecycle hooks! const ( ) => airlineSchema 'save' console `Doc: about to be saved` ${doc.name} 'save' console `Doc: has been saved` ${doc.id} // Our plugin must be registered before the model creation // Constructing our document const new callsign 'UNITED' country 'United States' name 'United Airlines' phone '321-321-3210' '321-123-1234' const async try // pre and post hooks will run const await console catch throw async => () 0 ( ) => error console Save document result: Doc: United Airlines about to be saved Doc: 1316488a-98ba-4dbb-b0d7-ea6001a0bf57 has been saved _Model { callsign: , country: , name: , phone: [ , ], id: , _type: } 'UNITED' 'United States' 'United Airlines' '321-321-3210' '321-123-1234' '1316488a-98ba-4dbb-b0d7-ea6001a0bf57' 'Airline' We got our messages before and after the save. With validation, you can ensure certain document property values meet your criteria. Tapping into the lifecycle when the document is saved, updated, and removed also helped us gain a handle on Ottoman middleware! Query Building Ottoman has a very rich API that handles many complex operations supported by Couchbase and N1QL. Our query builder behind the scenes creates your N1QL statements for you. When using Query Builder you have three options of which mode to use. Using Parameters Access Functions or Using Parameters and Access Functions In the next three examples, I’ll do the same thing using each of the three different QueryBuilder modes (params, access functions, and mixed-mode). Each example will: Select name and country from Airline Where the country value is “United States” and LIMIT our results to 10 Let’s first create a new file named: findWithQueryBuilder.js, and add the following code: { Ottoman, Query } = ( ) ottoman = Ottoman({ : }); ottoman.connect({ : , : , : , : }); executeQuery = (query) => { { result = ottoman.query(query) .log( , result) } (error) { error } } generateQuery() .then( { executeQuery(query) .then( process.exit( )) }) .catch( .log(error)) const require 'ottoman' const new collectionName '_default' connectionString 'couchbase://localhost' bucketName 'travel' username 'Administrator' password 'password' /* Replace with QueryBuilder Example */ const async try const await console 'Query Result: ' catch throw ( ) => query => () 0 ( ) => error console This file has a comment in the middle that says: . We can just copy any of the following examples in this section into that area of the file. “Replace with QueryBuilder Example” Parameters generateQuery = () => { { params = { : [ { : }, { : } ], : { : [ { : { : }}, { : { : }} ] }, : } query = Query(params, ).build() .log( , query) query } (error) { error } } const async try const select $field 'name' $field 'country' where $and country $eq 'United States' _type $eq 'Airline' limit 10 const new '`travel`' console 'Query Generated: ' return catch throw Access Functions generateQuery = () => { { query = Query({}, ) .select([ { : }, { : } ]) .where({ : [ { : { : }}, { : { : }} ]}) .limit( ) .build() .log( , query) query } (error) { error } } const async try const new '`travel`' $field 'name' $field 'country' $and country $eq 'United States' _type $eq 'Airline' 10 console 'Query Generated: ' return catch throw Mixed Mode Once you have created a generateQuery() function using one of the mixed modes above, you would then need to asynchronously call and executeQuery and the code for that like I said will work with many flavors of the above code: generateQuery A result from any of the three modes above: Query Generated: SELECT name,country FROM default:`travel` WHERE (country= AND _type= ) LIMIT 10 Query Result: { meta: { requestId: , clientContextId: , status: , signature: { country: , name: }, profile: undefined, metrics: { elapsedTime: 6.219, executionTime: 5.9619, sortCount: undefined, resultCount: 2, resultSize: 106, mutationCount: undefined, errorCount: undefined, warningCount: undefined } }, rows: [ { country: , name: }, { country: , name: } ] } "United States" "Airline" '1514fa20-755e-49b3-bbfa-4ed75a1a40ee' '0334862c79e727f8' 'success' 'json' 'json' 'United States' 'United Airlines' 'United States' 'Jet Blue Airlines' Resources GitHub Repository for Ottoman Documentation for Ottoman V2 Ottoman Package on NPM Intro to Ottoman with Couchbase source A JSON Data Modeling Guide A Better Developer Experience with OttomanJS Couchbase NodeJS SDK Official Couchbase Docker Images Conclusion We’ve taken a guided tour through Ottoman gaining an understanding of many concepts. Schema, models, middleware, plugins, hooks, and query building. We explored how to connect to Couchbase in Ottoman. Defined schema and models. Touched on various fundamentals in NoSQL database schema and design. Finally, we walked through some of the most useful CRUD operations. All an effort to get you up to speed creating, reading, updating, and deleting documents in Couchbase via Ottoman. Give Feedback and Contribute I hope that this article has demystified why and how to use Ottoman and ODM for Couchbase. As shown you can use Ottoman for NoSQL database schema design, validation, and reduction of boiler-plate. Writing CRUD operations are made simple and aid in rapid development. If you have any questions about Ottoman, want to help contribute to this open-source project, or would just like to say hello, my name is Eric Bishard and I am the Developer Advocate here at Couchbase focusing on the Node.js and JavaScript developer experience, my DM’s are always open at . Twitter/@httpJunkie