Vadim Popov

@Vadim.Popov

Applications for Tarantool 1.7. Part 1: Stored procedures

August 1st 2017

Original article available at https://habrahabr.ru/company/mailru/blog/334266/

Hi there! In this article, I’d like to share my experience creating applications for Tarantool 1.7. It’s the first in a series of tutorials that might be useful both to those who’ve already decided to give Tarantool a try and to those who are still looking for a solution to streamline their projects.

The whole series will cover an existing Tarantool application, and this tutorial will touch upon such topics as installing Tarantool, storing and accessing data, and writing efficient stored procedures.

Tarantool is a NoSQL database that stores data in RAM or on disk (depending on the storage engine) and ensures persistence via a well-thought-out mechanism called a write-ahead log (WAL). Tarantool boasts a built-in LuaJIT (just-in-time) compiler that allows executing Lua code. You can also create stored procedures in C.

Why write your own Tarantool applications

There are two main reasons:

  1. This speeds up your service. Storage-side data processing reduces data traffic, and bundling several requests into a stored procedure helps minimize network latency.
  2. Ready applications can be reused. Tarantool ecosystem is actively evolving, with people creating more and more open-source Tarantool applications. With time, some of them become part of Tarantool itself. Such packages shorten development time for new services.

Naturally enough, this approach has its downside. Tarantool can’t take full advantage of a multi-core CPU, so if you strive to make your service scalable, you’ll need to shard your database and design an appropriate project architecture. On the bright side, as the number of requests grows, the workload becomes easy to scale.

Now I’m going to walk you through the creation of a Tarantool application that implements an API for registering and authenticating users. It offers the following capabilities:

  • registration and authentication via email in two steps: creating an account and confirming the registration and setting the password;
  • registration with social network credentials (FB, VK, Google+);
  • password recovery.

For an example of a stored procedure for Tarantool, we’ll take a look at the first step of email registration — getting a confirmation code. To make it more interactive, you can check this GitHub page and follow along.

Let’s go!

Installing Tarantool

You can find detailed installation instructions for different operating systems on this site. For example, to install Tarantool on Ubuntu, you’ll need to run the following commands in the console:

curl http://download.tarantool.org/tarantool/1.7/gpgkey | sudo apt-key add -
release=`lsb_release -c -s`
sudo apt-get -y install apt-transport-https
sudo rm -f /etc/apt/sources.list.d/*tarantool*.list
sudo tee /etc/apt/sources.list.d/tarantool_1_7.list <<- EOF
deb http://download.tarantool.org/tarantool/1.7/ubuntu/ $release main
deb-src http://download.tarantool.org/tarantool/1.7/ubuntu/ $release main
EOF
sudo apt-get update
sudo apt-get -y install tarantool

Let’s check the installation has been successful by typing tarantool and entering the interactive administrator console.

$ tarantool
version 1.7.3–202-gfe0a67c
type ‘help’ for interactive help
tarantool>

This is where you can try your hand at Lua programming. If you’re not too familiar with Lua, here’s a short tutorial to get you started.

Registration via email

It’s time to take it one step further and write our first script that creates a space that will be holding all the users. A space is analogous to a data storage table. Data itself is stored in tuples (arrays holding records). Each space must have one primary index and can have several secondary ones. Indexes can be defined on a single or multiple fields. Below is a space scheme for our authentication service:

As you can see from the image, we’re using indexes of two types: HASH and TREE. A HASH index allows finding tuples by a fully matching primary key and must be unique. A TREE index supports non-unique keys, enables searches by the first part of a composite index, and lets us streamline key sorting, since key values within an index are ordered.

The session space holds a special key (session_secret) used for signing session cookies. Storing session keys allows logging users out on the service side, if necessary. A session also has an optional link to the social space. It’s necessary for validating the sessions of those users who log in with social network credentials (we’re checking the validity of the stored OAuth2 token).

Before we start writing the application itself, it’s worth taking a look at the structure of the project:

tarantool-authman
├── authman
│ ├── model
│ │ ├── password.lua
│ │ ├── password_token.lua
│ │ ├── session.lua
│ │ ├── social.lua
│ │ └── user.lua
│ ├── utils
│ │ ├── http.lua
│ │ └── utils.lua
│ ├── db.lua
│ ├── error.lua
│ ├── init.lua
│ ├── response.lua
│ └── validator.lua
└── test
├── case
│ ├── auth.lua
│ └── registration.lua
├── authman.test.lua
└── config.lua

Paths specified in the package.path variable are used for importing Lua packages. In our case, packages are imported relatively to the current directory, that is tarantool-authman. However, if necessary, import paths can easily be extended as follows:

-- Prepending a new path with the highest priority
package.path = “/some/other/path/?.lua;” .. package.path

Before creating the first space, let’s put all the needed constants into separate models. We need to give each space and each index a name. It’s also necessary to specify the order of fields in a tuple. For example, here’s what the authman/model/user.lua model looks like:

-- Our package is a Lua table
local user = {}
-- The package has the only function — model — that returns a table
-- with the model’s fields and methods

-- The function receives configurations in the form of a Lua table
function user.model(config)
local model = {}
  -- Space and index names
model.SPACE_NAME = ‘auth_user’
model.PRIMARY_INDEX = ‘primary’
model.EMAIL_INDEX = ‘email_index’
  -- Assigning numbers to tuple fields
-- Lua uses one-based (!) indexing
model.ID = 1
model.EMAIL = 2
model.TYPE = 3
model.IS_ACTIVE = 4
  -- User types: registered via email or with social network
-- credentials

model.COMMON_TYPE = 1
model.SOCIAL_TYPE = 2
  return model
end
-- Returning the package
return user

When handling users, we’ll need two indexes: unique by id and non-unique by email address — when two different users register with social network credentials, they may be assigned the same email address, or even no email address at all. As for the users who registered the regular way, our application will make sure their email addresses are unique.

The authman/db.lua package contains a method for creating spaces:

local db = {}
-- Importing the package and calling the model function
-- The config parameter is assigned a nil (empty) value
local user = require(‘authman.model.user’).model()
-- The db package’s method for creating spaces and indexes
function db.create_database()
  local user_space = box.schema.space.create(user.SPACE_NAME, {
if_not_exists = true
})
user_space:create_index(user.PRIMARY_INDEX, {
type = ‘hash’,
parts = {user.ID, ‘string’},
if_not_exists = true
})
user_space:create_index(user.EMAIL_INDEX, {
type = ‘tree’,
unique = false,
parts = {user.EMAIL, ‘string’, user.TYPE, ‘unsigned’},
if_not_exists = true
})
end
return db

UUID will serve as a user id, and we’ll be using a HASH index for full-match searches. The index for searches by email will be two-part: (user.EMAIL, ‘string’) — user’s email address, (user.TYPE, ‘unsigned’) — user type. As a reminder, the types have been defined in the model a bit earlier. A composite index enables searches not only by all the fields, but also by the first part of the index; therefore, we can search by email address only (without the user type).

Let’s enter the interactive administrator console and try to use the authman/db.lua package.

$ tarantool
version 1.7.3–202-gfe0a67c
type ‘help’ for interactive help
tarantool> db = require(‘authman.db’)
tarantool> box.cfg({listen=3331})
tarantool> db.create_database()

Great, we’ve just created the first space! One thing to keep in mind here: before calling box.schema.space.create, you need to configure and run the server via the box.cfg method. Now we can perform some simple actions within the space we created:

-- Creating users
tarantool> box.space.auth_user:insert({‘user_id_1’, ‘exaple_1@mail.ru’, 1})
— -
- [‘user_id_1’, ‘exaple_1@mail.ru’, 1]

tarantool> box.space.auth_user:insert({‘user_id_2’, ‘exaple_2@mail.ru’, 1})
— -
- [‘user_id_2’, ‘exaple_2@mail.ru’, 1]

-- Getting a Lua table (array) with all the users
tarantool> box.space.auth_user:select()
— -
- — [‘user_id_2’, ‘exaple_2@mail.ru’, 1]
— [‘user_id_1’, ‘exaple_1@mail.ru’, 1]
-- Getting a user by the primary key
tarantool> box.space.auth_user:get({‘user_id_1’})
— -
- [‘user_id_1’, ‘exaple_1@mail.ru’, 1]
-- Getting a user by the composite key
tarantool> box.space.auth_user.index.email_index:select({‘exaple_2@mail.ru’, 1})
— -
- — [‘user_id_2’, ‘exaple_2@mail.ru’, 1]
-- Changing the data in the second field
tarantool> box.space.auth_user:update(‘user_id_1’, {{‘=’, 2, ‘new_email@mail.ru’}, })
— -
- [‘user_id_1’, ‘new_email@mail.ru’, 1]

Unique indexes restrict the insertion of non-unique values. If you need to create some records that may already be in a space, use the upsert (update/insert) operation. You can find the full list of available methods in the official documentation.

Let’s extend the user model with a capability to register users:

function model.get_space()
return box.space[model.SPACE_NAME]
end
  function model.get_by_email(email, type)
if validator.not_empty_string(email) then
return model.get_space().index[model.EMAIL_INDEX]:select({email, type})[1]
end
end
  -- Creating a user
-- Fields that are not part of the unique index are not mandatory
function model.create(user_tuple)
local user_id = uuid.str()
local email = validator.string(user_tuple[model.EMAIL]) and
user_tuple[model.EMAIL] or ‘’
return model.get_space():insert{
user_id,
email,
user_tuple[model.TYPE],
user_tuple[model.IS_ACTIVE],
user_tuple[model.PROFILE]
}
end
  -- Generating a confirmation code sent via email and used for
-- account activation

-- Usually, this code is embedded into a link as a GET parameter
-- activation_secret — one of the configurable parameters when
-- initializing the application

function model.generate_activation_code(user_id)
return digest.md5_hex(string.format(‘%s.%s’,
config.activation_secret, user_id))
end

The code snippet below uses two standard Tarantool packages — uuid and digest — and one user-created package — validator. Before you can use them, they need to be imported:

-- Standard Tarantool packages
local digest = require(‘digest’)
local uuid = require(‘uuid’)
-- Our application’s package (handles data validation)
local validator = require(‘authman.validator’)

When declaring variables, we’re using the local operator that limits their scope to the current block. Otherwise, these variables will be global, which is what we need to avoid due to potential name collisions.

Now let’s create the main package — authman/init.lua — that will hold all the API methods:

local auth = {}
local response = require(‘authman.response’)
local error = require(‘authman.error’)
local validator = require(‘authman.validator’)
local db = require(‘authman.db’)
local utils = require(‘authman.utils.utils’)
-- The package returns the only function — api — that configures and
-- returns the application

function auth.api(config)
local api = {}
-- The validator package contains checks for various value types
-- This package sets the default values as well
config = validator.config(config)
  -- Importing the models for working with data
local user = require(‘authman.model.user’).model(config)
  -- Creating a space
db.create_database()
  -- The api method creates a non-active user with a specified email
-- address

function api.registration(email)
-- Preprocessing the email address — making it all lowercase
email = utils.lower(email)
    if not validator.email(email) then
return response.error(error.INVALID_PARAMS)
end
    -- Checking if there already exists a user with a given email
-- address

local user_tuple = user.get_by_email(email, user.COMMON_TYPE)
if user_tuple ~= nil then
if user_tuple[user.IS_ACTIVE] then
return response.error(error.USER_ALREADY_EXISTS)
else
local code = user.generate_activation_code(user_tuple[user.ID])
return response.ok(code)
end
end
    -- Writing data to the space
user_tuple = user.create({
[user.EMAIL] = email,
[user.TYPE] = user.COMMON_TYPE,
[user.IS_ACTIVE] = false,
})
    local code = user.generate_activation_code(user_tuple[user.ID])
return response.ok(code)
end
  return api
end
return auth

Great! Now users can create accounts.

tarantool> auth = require(‘authman’).api(config)
-- Using api to get a registration confirmation code
tarantool> ok, code = auth.registration(‘example@mail.ru’)
-- This code needs to be sent to a user’s email address so that they
-- can activate their account
tarantool> code
022c1ff1f0b171e51cb6c6e32aefd6ab

That’s it for now. The next article will be about using ready Tarantool packages, networking, and implementing OAuth2 in tarantool-authman.

Thanks for reading and stay tuned!

More by Vadim Popov

More Related Stories