Recently, I wrote an article about using which generated quite a bit of interest—So I would like to use this opportunity to follow up with a practical example. REST over WebSockets A while ago, I created a basic sample app to demonstrate an approach to building single-page real-time apps; see . The sample app demonstrates several key features that one might like to have in a real-time single page app including: sc-sample-inventory Authentication (via JWT tokens) Access control using back-end middleware Reactive data binding Real-time REST-like (CRUD) interface Real-time updates with support for lists (pagination, sorting, filtering) The sample app was written with Google’s Polymer framework, SocketCluster and RethinkDB but it could have been written with any other framework/database (I’m planning to do something similar for React/Redux). It borrows ideas from REST but takes the level of granularity one step further by applying modifications to individual fields within a resource instead of the entire resource at once — for the purpose of this article, we’ll call this technique “real-time CRUD”. If you don’t want to read about all the reasoning behind this project, feel free to skip the “Background” section below and go straight to the “How real-time CRUD works” section. Background One of the core features of REST is that it provides front-end clients with an interface to interact with data in a consistent, granular way. The finest level of granularity typically offered by REST is a ‘resource’ which in database-land is equivalent to document (in NoSQL) or a table row (in relational DBs). Resource-level granularity has served us pretty well for HTTP APIs in the past because the typical workflow on the front-end has mostly consisted of submitting entire forms (with all fields of a resource at the same time). Essentially, what this means is that if two users try to modify the same resource at the “same time” (e.g. using the same form), then the last HTTP POST request to hit the server will completely overwrite the previous one — So far, this has been the standard, accepted practice when building web apps because most users have become accustomed to the idea of dealing with outdated data (I.e. they know that they have to “refresh the page” to see the latest data). When it comes to building modern single-page apps, however, this approach has started to lose its appeal and a lot of UX-sensitive developers have started using third-party real-time data-sync services like Firebase and Parse to synchronise their data between their front-end and back-end. This allows multiple users to collaboratively modify the same resource at the same time while being aware of each other’s changes. Unfortunately, as I’ve mentioned in other articles in the past, the data-sync approach is often an unnecessary third-wheel for your system and it does not mitigate potential data conflicts entirely; it reduces flexibility and adds complexity to your app. Instead of doing expensive diffing and implementing conflict resolution handlers for everything, why not solve this problem by side-stepping the potential for conflicts entirely? In the same way that REST provides isolation between individual resources, we can take the idea one step further by isolating fields/properties from each other within the same resource. To make sense of this; imagine that you have a form like this which represents a resource in your database: In standard (not real-time) REST/HTTP, as a user, you simply wouldn’t know if someone else was trying to edit the same resource at the same time— You might find out later if you come back to that resource and notice that your changes have been overwritten with old data. With real-time CRUD, because changes happen in real-time one field at a time, other concurrent users can see your changes in their browsers as they are made and therefore won’t overwrite them. This approach doesn’t support Google Docs-style collaborative text editing by default (E.g. for large text-areas) but it’s suitable for 99% of cases (if you consider the form above, you wouldn’t benefit at all from allowing multiple users to edit individual input fields at the same time). For those rare cases that do require collaborative text-editing on a large body of text, you can use a transport-agnostic plugin like (or one of its more recent ). unintentionally OT.js forks How real-time CRUD works On the front-end, data can be represented either as a collection ( ) or a field ( ). A collection is essentially an array of empty placeholder resources with only an ID field (e.g. ) and a field holds values for an individual property of a resource (associated with a resource ID). You can automatically attach objects to each placeholder within an and they will automatically populate the placeholder with the appropriate field values from the server (more on that later). sc-collection sc-field [{id:”14443ea2-b553-4195-9220-bce456c8281b"}, ...] sc-field sc-collection On the back-end, the current implementation is built with SocketCluster/Node.js and RethinkDB, to use it, you can simply attach the module like : sc-crud-rethink this var crud = scCrudRethink.attach(worker, crudOptions); You will need to pass a object which defines the schema of your CRUD data like . crudOptions this Fields As a user, if another user makes a change to a specific field of a resource that you are also using, it will be updated in real-time on your front-end as well. Conversely, if you make a change to your model by invoking a on it: sc-field save operation targetModel.save(); other interested users will see the change in real-time. To construct an on the front end, you need to do it like : sc-field this <sc-fieldid="product-name"resource-type="Product"resource-id="{{productId}}"resource-field="name"resource-value="{{productName}}"></sc-field> <sc-fieldid="product-qty"resource-type="Product"resource-id="{{productId}}"resource-field="qty"resource-value="{{productQty}}"></sc-field> <sc-fieldid="product-price"resource-type="Product"resource-id="{{productId}}"resource-field="price"resource-value="{{productPrice}}"></sc-field> <sc-fieldid="product-desc"resource-type="Product"resource-id="{{productId}}"resource-field="desc"resource-value="{{productDesc}}"></sc-field> In Polymer, by default, data can flow either in or out of components so tag attributes can be used either as an input to the component or to capture output from it. In this case, the attribute causes data to flow out into the specified property of the parent component ( , , , ) — This output property can then be used as an input to other front-end component and it will update those other components in real-time. All other attributes of the model component are used as input to the and tell it which resource and field to hook up to on the back end. resource-value productName productQty productPrice productDesc resource-value sc-field sc-field Collections You can add a new resource to an by invoking the on it: sc-collection create operation var newProduct = {name: this.newProductName,category: this.categoryId}; // productListModel is our sc-collection model.productListModel.create(newProduct, function(err, newId) {if (err) {// TODO: Handle error} else {self.newProductName = '';}}); Just like with , all model updates will be propagated to the database and to all interested users in real-time. sc-field As mentioned before, a collection is just a list of empty placeholder objects with an ID property. To fill-out the collection with data on the front end, you need to attach components to it. To do this, you just need to iterate over each item in the and bind a property of the item/placeholder to an for the same resource. See : sc-field sc-collection sc-field this <sc-collectionid="category-products"disable-realtime="{{!realtime}}"resource-type="Product"resource-value="{{categoryProducts}}"resource-view="categoryView"resource-view-params="{{categoryViewParams}}"resource-page-offset="{{pageOffsetStart}}"resource-page-size="{{pageSize}}"resource-count="{{itemCount}}"></sc-collection> <template is="dom-repeat" items="{{categoryProducts}}" filter="hasIdFilter" observe="id"> <sc-field id$="{{item.id}}-qty" resource-type="Product" resource-id="{{item.id}}" resource-field="qty" resource-value="{{item.qty}}"> </sc-field> <sc-fieldid$="{{item.id}}-price"resource-type="Product"resource-id="{{item.id}}"resource-field="price"resource-value="{{item.price}}"></sc-field> <sc-field id$="{{item.id}}-name" resource-type="Product" resource-id="{{item.id}}" resource-field="name" resource-value="{{item.name}}"> </sc-field> </template> ^ Here we’re iterating over each placeholder ( ) in the sc-collection and binding each to a matching property on that (see the attribute on components above — Remember, the is an output attribute and will set item.xxxx to whatever value the gets from the server). item sc-field item resource-value sc-field resource-value sc-field Transformations of collections Transformations (or sorting and filtering in REST-speak) are done using views that are defined on the back end (as part of your object)—Views are defined under resource definitions like (in the following snippet, we are only exposing one view which provides a sorted list of Category resources): crudOptions this alphabeticalView var crudOptions = {defaultPageSize: 5,schema: {Category: {fields: {id: type.string(),name: type.string(),desc: type.string().optional()}, // --- BEGIN VIEW DECLARATION --- // Here we define a view for the 'Category' sc-collection // ordered alphabetically. // We use RethinkDB's ReQL to do the transformation. views: { **alphabeticalView**: { transform: function(fullTableQuery, r) { return fullTableQuery.orderBy(r.asc('name')); } } }, // --- END VIEW DECLARATION --- // This is an auth filter for the 'Category' resource to // decide which CRUD read/write operations to block/allow. // DO NOT CONFUSE THIS WITH COLLECTION TRANSFORMATION. filters: { pre: mustBeLoggedIn } }, // ... },// ...} Then, on the front end, you can specify what view of the Category collection to use with the attribute of the component like : resource-view sc-collection this <sc-collectionid="categories"resource-type="Category"resource-value="{{categories}}"resource-view=" "resource-view-params="null"></sc-collection> alphabeticalView ^ If you don’t provide a , sc-crud-rethink will fetch the raw (non-transformed) collection. resource-view Sometimes sorting isn’t enough, you also want to filter/exclude specific items from your views (e.g. maybe you only want to show products that belong to a specific category), you can do this with your transform functions by providing a more advanced database query like (note that we are using RethinkDB’s ReQL query language): this Product: {fields: {id: type.string(),name: type.string(),qty: type.number().integer().optional(),price: type.number().optional(),desc: type.string().optional(),// The category ID field is used as a parameter to transform// the collection for the categoryView defined below. : type.string()},views: { : {// Declare the fields from the Product model which// are required by the transform function. : [' '],transform: function(fullTableQuery, r, productFields) {// Because we declared the category field above, it is// available in here inside productFields.// This allows us to tranform the Product collection// based on a specific category ID.// Only include products that belong to the category ID// provided by the client as part of this view.return fullTableQuery.filter(r.row('category').eq( )).orderBy(r.asc('qty'))}}},filters: {pre: mustBeLoggedIn,post: postFilter}} category categoryView paramFields category productFields.category ^ The property from the code above will hold the category ID which was provided by the attribute on the front end like : productFields.category resource-view-params this <sc-collectionid="category-products"resource-type="Product"resource-value="{{categoryProducts}}"resource-view=" " ="{{ }}"resource-page-offset="{{pageOffsetStart}}"resource-page-size="{{pageSize}}"resource-count="{{itemCount}}"></sc-collection> categoryView resource-view-params categoryViewParams The value of the attribute ( in our case) must be a plain JavaScript Object whose properties exactly match those that were specified in in the definition on the back end, so on the front end, we’re computing it like : resource-view-params categoryViewParams paramFields categoryView this computeCategoryViewParams: function (categoryId) {return { : categoryId};} category Whatever ID we provide as on the front end will be used as the value in inside the transform function on the back end (which we use as an argument to our ReQL query). This part (as shown earlier): category productFields return fullTableQuery.filter(r.row('category').eq( )) productFields.category The is the ReQL query which fetches the entire table/collection — ReQL allows us to easily extend this query with additional filters and sorting rules so we can filter our products based on their category IDs as shown above. fullTableQuery Authorization We don’t want to allow just any user to read/write from/to any resource using our real-time CRUD interface so we need to add some access control rules. With SocketCluster, you can do this via custom middleware like . this Note that in the example above, we are being very broad with our rules, any logged in user can read or modify any data in our database. You can read the values from the auth token to make more fine-grained decisions. Notes While sc-crud-rethink, sc-field and sc-collection modules/components have been well tested during development, I can’t guarantee that they have been used in production so proceed with caution. If anything is not clear, feel free to mention it in the . I will keep updating this article over time to improve it. SocketCluster channel on Gitter
