Grokking Diesel

Diesel ( ) is an ORM (Object-relational mapper) and Query Builder written in Rust. It supports Postgresql, Mysql and SQLite. It makes use of Rust’s custom derive functionality to generate all the code you need in order to get all the power of Rust’s type system when interacting with a database. This means that you get compile-time validation of your code, thus eliminating possible runtime errors. http://diesel.rs Diesel is very powerful, but by just following the examples might still leave you scratching your head, asking questions like “where do these types come from?”, “what should I add here”, “how does this work?”. This article is meant to shed a little bit of light on the topic from the perspective of a intermediate-level Rust developer. There are many topics that this article will not cover, but the hope is that you will be able to “grok” enough of Diesel so that the rest becomes obvious and that you understand how it works, to the extent that new territory that you explore in this crate also becomes simpler to understand. First Steps You should first run through the on the official Diesel website. This article focusses on Postgresql, and by far the easiest way to get a Postgresql server running for your tests is by using Docker ( )... the following one-liner should do the trick: Getting Started guide www.docker.com docker run --rm --detach --name postgres --env POSTGRES_USER=username --env POSTGRES_PASSWORD=password --publish 127.0.0.1:5432:5432 postgres not This is for production use, the whole container will be removed when the container is stopped (use _docker stop postgres_ ), but it is super handy for the Getting Started guide. A word about the format of the rest of this article This article assumes that you have worked through the Getting Started guide; it deliberately glosses over a whole set of features. This is intentional and done in order to keep the article as concise as possible. Core Components And Community At its core Diesel consists of 4 main components: The crate, crate (the code generator) and the cli, which by now you should have been introduced to. diesel diesel_codegen diesel The last major component, and ‘secret weapon’ or ‘unofficial guide’ (according to me) is the the test suite in the official repo ( ). It served as my guide whenever I got stuck, although it will surely be superseded by documentation as the project continues to evolve. https://github.com/diesel-rs/diesel/tree/master/diesel_tests Environment When I started off with Diesel I was baffled by the macro in (more on this below), which in turn used . The crate is just a convenience library that allows you to put your in a hidden environment file, ala (dot env, get it?). infer_schema! schema.rs "dotenv:DATABASE_URL" dotenv DATABASE_URL .env It follows that you can also just specify in your environment, which is particularly handy when you run your Diesel-using Rust binary in a docker container ala . DATABASE_URL 12-Factor apps The Diesel cli tool also reads the file, and if it's not available you will need to define in the environment or pass it the parameter. .env DATABASE_URL --database-url Definitely check out the _examples/sqlite/getting_started_step_3/README.md_ file in order to learn how to configure the _DATABASE_URL_ for SQLite (it doesn't use a URI format). Basic Flow The core workflow for creating an app built with Diesel can be broken down as follows: Design A Schema This is very obvious to seasoned SQL veterans, luckily for the rest of us the cli's subcommand allows us to easily iterate on our design and even evolve it over time. It is, however, very useful to have a clear idea of what you want your database to look like up front, and it should be noted that Diesel only works with tables that have a primary key. diesel migration Create Migrations Follow the pattern in the Getting Started guide, and note that you can add more migrations at any time using the subcommand. Migrations will be run in order whenever you run . You can rerun the last migration by issuing and if you truly get stuck, you can run the following command, but please never do so on a production database, . diesel migration generate diesel migration run diesel migration redo diesel database reset When designing your tables you should put plurals of your table names. As an example, Diesel will take the model and search for the table . You can define custom table names, but knowing the Diesel developers' assumptions will probably spare you some confusion. User users Diesel will take Rust structs (which will probably describe single objects, or rows in your database table) and translate them into table names with an tucked at the end to "pluralise" it. For example, will be assumed to map to a table named . PascalCase snake_case s AFancyNamedObject a_fancy_named_objects Infer The Schema From The Database Diesel has the ability to inspect your actual database and infer a schema for use in Rust, this will be used to create the necessary DSL (domain specific language) that allows you to interact with your database in a safe, lightning fast, and strongly typed fashion. The Getting Started guide and examples use the macro. The major disadvantage of using this macro is, firstly, that you have idea how Diesel actually interprets your database's types (this matters for your models, which will be revealed later) and secondly it requires a bootstrapped database instance during compile time, which can be a bit of a pain when compiling as part of a pipeline that might not have your database and compiler toolchain available to each other. infer_schema! no It is recommended that you use of the subcommand and simply copy and paste the inferred schema into a file in your project (in the Getting Started guide this is but it could just as easily be pasted into or ). diesel print-schema schema.rs lib.rs main.rs What you will see is a macro similar to: table! table! {users {id -> Integer,name -> VarChar,}} You want to grab the datatypes ( and in this case) and immediately run over to and plug it into the doc search bar. This will take you straight to (also explore ) and allows you to inspect the implemented and traits for each type. To take one example, maps to in Rust. This is super useful when implementing your models, or wrestling with compile-time errors. Integer VarChar docs.rs/diesel diesel::types::Foo diesel::types ToSql FromSql Integer i32 Create Models As with the schema, you don’t to put your models in the file. It is recommended that you split your models out using the Rust modules system, and something like the following might assist, especially when dealing with lots of models. have models.rs models/users/mod.rsposts/mod.rs I initially struggled to split the “magic” used to drive Diesel from idiomatic Rust. It turns out Diesel is just good old familiar Rust once you know how it is structured and now how to deal with the generated code. An example model expressed as a Rust struct is as follows, plucked straight from the Getting Started guide: #[derive(Queryable)]pub struct Post {pub id: i32,pub title: String,pub body: String,pub published: bool,} Models are normal Rust structs that to map to your tables. This is usually the case for simple models, however, it is very important to note that structs, as the name implies, actually map to the you want to obtain from a SQL query. seem Queryable results A struct is a single object or row (partial or complete) that you want to retrieve from a SQL query. This object will be queryable from the table, or any query that returns the correct types, as discussed in the section below. Queryable User users Queryable Note that diesel’s SQL types don’t support unsigned integers in Postgresql (this is a Postgresql limitation), so it would be worthwhile looking at the appropriate in the documentation to see what your database supports. diesel::*::types Derives, aka “The Codegen Magic” Diesel’s code generator is primarily used to embue your Rust structs with SQL magic without you having to handcode a ton of functionality. It also turns the Rust type system into the magic ingredient that can be used to construct the fast and reliable SQL that gets transmitted to the database. In order to imbue your structs with the SQL goodness, you use Rust’s custom derive functionality, for example: #[derive(Queryable)]pub struct Post {pub id: i32,... A more complex example: I loved stumbling across the following in the tests… it truly made me smile at the genius of the authors, but it also made me scratch my head, “where on earth are these things used and how do they work?” #[derive(PartialEq, Eq, Debug, Clone, Queryable, Identifiable, Insertable, AsChangeset, Associations)]#[table_name = "users"]pub struct User {pub id: i32,pub name: String,pub hair_color: Option ,} First, a ‘pro tip’ from a novice, you should install the cargo subcommand : expand cargo install cargo-expand This allows you to run, the following command and actually what is generated (warning, this guide assumes that you know Rust, but the following might make even a seasoned Rust developer’s eyes water, so feel free to skip it or use as bedtime reading). see cargo expand The full glory of the generated DSL is revealed… of particular interest is the columns for each model. Parsing the output is left as an exercise for the bored reader, or the readers who like a challenge. Let’s look at the derives in more detail. PartialEq, Eq, Debug, Clone These are your standard Rust derives that most developers rely on when appropriate. You probably want at least if you like (or the shiny new ) for debugging. Debug println! eprintln! _#![deny(missing_debug_implementations)]_ is a very useful compiler directive as it will error on any structs that do not have _Debug_ implementations, especially useful if you are developing a library. Queryable The first thing that most developers usually want to do with their databases is query the data within it. In order to do this, you need to decorate your as follows: struct #[derive(Queryable)]struct User {id: i32,firstname: String,lastname: String,age: i32,} This will cause to generate the query DSL needed to deserialize a query result of type which maps to in Postgresql, more examples of this towards the end of the article. diesel_codegen (i32, String, String, i32) (Integer, Text, Text, Integer) In SQLite a date/time type can be deserialized to _String_ too. A primary key column is mandatory to work with Diesel, but is probably good practice anyway. How to use the DSL to construct queries is addressed later in the article. Insertable Unless you are coming from an existing system, you probably want to insert data into your database too. A typical “insertable” object would look like this: #[derive(Insertable)]#[table_name="users"]struct NewUser {firstname: String,lastname: String,age: i32,} Note that we have dropped the field as the SQL server will handle this for us (this might change for advanced use-cases). id Also note that we now explicitly name the table, , this is also needed for , and inferred for users AsChangeset Identifiable Identifiable Inevitably, you are going to use SQL joins to construct results from more than one table at a time. In order for the join to successfully resolve the exact object in your target table, this needs to be annotated as follows: struct struct User {id: i32,firstname: String,lastname: String,age: i32,} By default Diesel will assume your primary key is called , and if not, you can override it as follows: id #[derive(Identifiable)]#[primary_key(guid)]struct User {guid: i32,firstname: String,lastname: String,age: i32,} Associations The tables that you want to enrich with your data needs to be annotated as follows: Identifiable #[derive(Associations)]#[belongs_to(User)]struct ActiveUsers {id: i32,user_id: i32,last_active: NaiveDateTime} This will allow you to join the data to this table by looking up the user, defined by the field , corresponding to the field in the table. In the event where your foreign key field is not specified in the pattern , you will need to manually map it: User user_id id User type_id #[derive(Associations)]#[belongs_to(User, foreign_key="user_lookup_key")]struct ActiveUsers {id: i32,user_lookup_key: i32,last_active: NaiveDateTime,} AsChangeSet is used to updates, more details can be found in this work-in-progress guide: #[derive(AsChangeset)] https://github.com/diesel-rs/diesel/blob/master/guide_drafts/all-about-updates.md Using Diesel At this point, you are ready to actually something with Diesel, probably insert some data and make some queries. This is covered by multiple examples in the Getting Started guide. There are, however, some items that need to be unpacked to, hopefully, make the lightbulbs go on. do First and foremost, at this point, it is super important to underscore that you are now entering the Rust world. What I mean by this is that the codegen and magic bits of Diesel now take a backseat. normal This implies that when you look at example code, there is no more slight of hand. You need only trust the complier and take time to unpack the statements and expressions like you would normally do. You’ll find that you are calling normal methods, passing normal data-structures (or references to them) and that you can , debug, step, re-order and organise like you would with any other Rust code. It took me an unfortunately long time to comprehend this, but it is an important insight and will allow you to code without fear. println! Connecting to the database The example uses the function to wrap the connection into a convenient function call for re-use in the rest of the example code. postgresql pub fn establish_connection() -> PgConnection encapsulates the handle to for you, keep this object around or recreate it on demand. When needed, look at the crate to create a pool of database connections. PgConnection posgresql r2d2 Everything you execute using Diesel will depend on the connection object being available. Inserting data The guide specifies the following function in . Let's step through it: src/lib.rs pub fn create_post(conn: &PgConnection, title: &str, body: &str) -> Post {use schema::posts;let new_post = NewPost {title: title,body: body,}; diesel::insert(&new\_post).into(posts::table) .get\_result(conn) .expect("Error saving new post") } First note the reference to the connection . As is done for some of the other files in , a new could also be instantiated by calling something like: &PgConnection src/bin/ PgConnection let conn = establish_connection(); The next line, , stumped me for a long time, as this is using generated code. In the expression, we see the use of . use schema::posts; diesel::insert posts::table At this point, it might be a good idea to take the output of and have a look at it in an IDE of sorts. Try the following: cargo expand # Check out a copy of Diesel git clone cd diesel/examples/postgres/getting_started_step_3/ https://github.com/diesel-rs/diesel.git # Build the example (assuming your postgres instance is ready# and running; see the docker hint above) echo DATABASE_URL=postgres://username:password@localhost/diesel_demo > .envdiesel setupcargo build # Expand the code (assuming cargo-expand is installed) cargo expand --lib > expanded.rs If you search for under the output you will also see , and there you'll find a struct (empty, but with a bunch of Traits impl'ed). mod schema mod posts table Next we have a new object (which is ) called . Insertable new_post Lastly we have the actual Diesel method . If you consult the documentation you will see that the Diesel module has 5 functions, , , , and at its core. insert insert delete insert_default_values select update If you look at the function you'll see it accepts (in this case our new user, but it can also be a ) and returns an object, which has one method called which accepts your struct. You could also have called and avoided the use statement. insert records: T Vec IncompleteInsertStatement into() table into(::schema::posts::table) In this case is the name of module, due to the schema being generated in the file. schema your schema.rs Querying data Again, we’ll refer to a function in the guide, in this case the file: src/bin/show_posts.rs extern crate diesel_demo;extern crate diesel;use self::diesel_demo::*;use self::diesel_demo::models::*;use self::diesel::prelude::*; fn main() { use diesel\_demo::schema::posts::dsl::\*; let connection = establish\_connection(); let results = posts.filter(published.eq(true)) .limit(5) .load:: (&connection) .expect("Error loading posts"); println!("Displaying {} posts", results.len()); for post in results { println!("{}", post.title); println!("----------\\n"); println!("{}", post.body); } } Stepping through it from the top, first we import our own crate (the example specifies the name as ). The we import the Diesel crate. Cargo.toml crate diesel_demo gives us access to the function, gives us access to the actual structs that we defined in our files, in this case the struct. use self::diesel_demo::*; establish_connection() use self::diesel_demo::models::*; models.rs Post is a necessary import that bring a whole bunch of Diesel Traits and types into scope. It is needed for Diesel to work and beyond the scope of this article to dive into. use self::diesel::prelude::*; In the function is where we encounter the use of some magic again, specifically the line. When I first started using Diesel I was stumped between the difference of imports as used above when inserting code and dsl imports used here. main() use diesel_demo::schema::posts::dsl::*; schema::tablename::* schema::posts::dsl::* A look at the output of allows for some clarification: cargo expand pub mod dsl {pub use super::columns::*;pub use super::table as posts;} In short, brings the of your table into scope. Each column type that is generated for you has a collection of implemented on it. To rephrase this, the (domain specific language) allows us to use the in our table's names (as defined by the generated code) and apply logic to it in order to construct our SQL queries. (see ) schema::posts::dsl::* columns expression_methods dsl columns schema.rs http://docs.diesel.rs/diesel/expression_methods/global_expression_methods/trait.ExpressionMethods.html#method.desc Extra credit if you spotted the convenience import of into the dsl module (I think this is for convenience). table Building queries In SQL we construct a select expression to return values from our database, and in Diesel we use Rust’s type system to construct type-checked, safe versions of those. This is great for anyone who has ever struggled with the fragility of SQL queries, and its implied security risks… only valid SQL should compile successfully. The next expression reflects that we want to run the method on the table (conveniently also imported into our context by the statement). takes a constructed filter as its input. A filter is constructed by combining columns and their expression methods. posts.filter(published.eq(true)) filter posts use schema::posts::dsl::* filter To inspect the results of this you can rewrite the relevant code as: use diesel::debug_sql; let posts_with_sql = posts.filter(published.eq(true)).limit(5); println!("SelectStatement: {:#?}", posts_with_sql); let results = posts_with_sql.load:: (&connection).expect("Error loading posts"); If you look at the output to the terminal you will see a object is returned. You can also use to look at the SQL that would be generated (make use you import diesel using ). SelectStatement println!("SQL: {}", debug_sql!(posts_with_sql)); #[macro_use] extern crate diesel; The main takeaway from this section is that you first build up the relevant SQL query by using your and imported from the module, and that this is introspect-able by and . table columns dsl println! debug_sql You should familiarize yourself as much as possible with the different you can call on columns - it will get you well on your way to building the SQL queries you want to use in your applications. expression_methods Getting results The last item we’ll handle in this article is actually reading your results. In the binary this is achieved by the lines: show_posts.rs let results = ...omitted....load:: (&connection).expect("Error loading posts"); As can be seen we are calling the method of the struct that we inspected a bit closer above. The method is generic, so we need to give the compiler some hints as to what type we want to return. The parameter for the load function is a reference to (or borrow of) the object returned by . .load() SelectStatement .load() connection establish_connection returns a result object, which in the case of this executable we deal with by just unwrapping the result with the method. We then pass the result, if we were successful, to the binding. load expect() results See or in the docs for some alternatives to . As will be seen the methods return a which is just a type alias. diesel::prelude::LoadDsl diesel::prelude::FirstDsl load QueryResult Result ; In all the methods just mentioned, either return a single object ( ) or a vector of objects ( ). In other words, one row or multiple rows of the selected columns in the database table. QueryResult QueryResult > Wrestling with results If we wanted to get all the results back, we could have used the code: let results :Vec = posts.load(&connection).expect("Error loading posts"); and are equivalent. That is, they both return a result ( ). Note that the code was reformatted to illustrate the return-type hint given to the compiler in the binding, . load get_results Vec QueryResult let results: Vec = ... Lastly, sometimes you will use a method or method (not illustrated in this article, but do check out the 'unofficial guide' aka mentioned above) which will return rows that do not map directly to the fields in your models. Use a to collect your results from your method in that case. This may look like: select join diesel_tests tuple load let results: Vec = posts.load(&connection).expect("Error loading posts"); Above, we expressed the model or as a of its constituent parts. Post struct tuple What we didn’t cover and what’s next There is a lot that wasn’t covered in this article, even though it is quite a large volume of information to consume in its own right. As mentioned initially, the goal was to explain enough of Diesel and its structure so that it would become possible for interested developers to “grok” or understand it, and then help themselves. I hope you made it this far and that you enjoyed the journey, please send feedback and enjoy Rust and Diesel. Special thanks to Sean Griffin, the author of Diesel, for the fact-checking he did on this article.