An Open Source SDK and Serde Magic: My First Two Months as a Member of the Rust Community

I am nearly two months into my rust journey and I still feel like this most days:

Thus, I can definitely vouch for the stereotypical depiction of Rust’s learning curve:

Credit: https://matthias-endler.de/2017/go-vs-rust/

And of course the constant war against the borrow checker:

Nevertheless, I have certainly come a long way in the past two months. I have:

• Read most of the Rust Book (Second Edition)
• Read part of Programming Rust
• Listened to nearly all of The New Rustacean Podcast (a must)
• Written a simple CLI game
• Published my own crate
• Became an open source contributor

So why Rust? After all, using languages like Ruby, Go, or JavaScript, I could have written numerous full stack applications at this point. Why worry about types and ownership and all that nonsense?

** Clears throat **

Well in Rust, there are certain safety guarantees… yes, you know, the typical spiel.

That may have been enough to interest me in rust, but it was not the reason I decided to stick with it — ultimately what drew me in was the community.

My Experience With the Rust Community

Having finished a few chapters of the Rust Book and successfully written a (questionable) linked list, I found myself itching with the desire to begin a Rust project. It was going to be blockchain and use the coolest crates and hell it may even turn into something really big (all the typical over-excited, ambitious, unrealistic pre-side-project thoughts). Before diving into code, I decided to search around Github for some motivation. During this exploration, I stumbled upon the Rust Stellar-SDK project.

I decided to reach out and email the owner of the project, explaining my situation:

Hello,

I am currently an intern at Stellar and beginning to learn rust.

I was snooping around github and ran into your SDK which looks awesome and also looks like it is under active development.

I would love to know more!

Ultimately as I said, I am trying to learn rust — so I am looking for projects that would be fun to really dive into with people who would have the patience to work with rust newbies :)

Within just a few hours I received back a reply with:

• suggestions for places to start
• a full explanation of the project
• and a very warm welcome, offering to help me in my rust journey

Fast forward a month…

At this point, I had made 5 contributions, the project was up to 6 contributors, I had begun utilizing this SDK for another side project and I even got to meet one of my fellow contributors in person. Pretty cool eh?

A Story of Community: My Most Recent PR

The story my most recent PR highlights the helpfulness of the rust community.

In the process of using the 0.1.0 release of the SDK’s client library, I realized we were missing a field, specifically the memo field of the transaction resource. So, I opened an issue and took the lead to make the fix.

An Overview:

We are building a Stellar SDK written entirely in Rust. This SDK communicates with the Horizon API server that sits on top of the stellar-core client.

Each endpoint method from the SDK receives data back from the Horizon server. When the SDK receives data, it must deserialize this data into a rust data structure — for this we utilize the wonderful serde library. These data structures are resources in the SDK. Below is the transaction resource. As you can see, this resource has many fields, however it does not have a memo field.

My job, as outlined in the issue linked above, was to add the memo field.

#[derive(Deserialize, Debug, Clone)]pub struct Transaction {id: String,paging_token: String,hash: String,ledger: u32,created_at: DateTime<Utc>,source_account: String,#[serde(deserialize_with = "deserialize::from_str")]source_account_sequence: u64,fee_paid: i64,operation_count: u32,envelope_xdr: String,result_xdr: String,result_meta_xdr: String,fee_meta_xdr: String,// INSERT MEMO HERE}

So what is a memo?

As defined in the Stellar Docs:

Memo

optional The memo contains optional extra information. It is the responsibility of the client to interpret this value. Memos can be one of the following types:

• MEMO_TEXT : A string encoded using either ASCII or UTF-8, up to 28-bytes long.
• MEMO_ID : A 64 bit unsigned integer.
• MEMO_HASH : A 32 byte hash.
• MEMO_RETURN : A 32 byte hash intended to be interpreted as the hash of the transaction the sender is refunding.

The Initial Plan:

After discussions with a couple other members of the project, my first thought was to use an intermediate deserialization technique. See here for an example.

So, I created an intermediate transaction, a nearly complete clone of the transaction struct above, but with:

memo_type: String,memo: Option<String>

This seemed to be a logical solution since all memos have a type, but not every memo has a value. Unfortunately, of the memos that have a value, one type, memo id, is an integer while the rest are strings. Since rust is a strictly typed language, this complicates things.

So, to deal with the possibility of two memo values, an integer or a string, I decided to create yet another intermediate struct, an intermediate memo value. At this point, here is what I had:

1. Intermediate Transaction
2. Transaction Deserialization Function
3. Intermediate Memo
4. Memo Deserialization Function
5. Memo Struct
6. Transaction Struct

That is a lot of work to just add one field isn’t it? Kind of makes you wonder:

Refactoring Begins:

Once I got a working version (and proved it worked with a series of tests) I submitted a PR explaining this monstrosity and also listing places I especially needed help with:

Through conversations on our gitter and on the PR thread, I began refactoring and immediately cut some of the obvious bloat. However, even after a couple, rounds of refactoring I was still looking at a PR that added 200+ lines of code for the addition of only a single field (not including tests).

The Rust Community Saves the Day:

Even though this code was still ugly, I was proud of the code I had written — remember I am still very new to the rust programming language and I had written all this code myself. So, I decided to post in the r/rust “What’s everyone working on this week thread.” A few days later, the creator of the serde crate, David Tolnay, reviewed my PR.

Here is an approach that avoids duplicating all the fields of Transaction by representing memo as a flattened adjacently tagged enum containing data that is deserialized as an untagged enum.

#[derive(Deserialize, Debug)]struct Transaction {id: String,

/\* other fields \*/  

#\[serde(flatten)\]  
memo: Memo,  

}

#[derive(Deserialize, Debug)]#[serde(rename_all = "lowercase", tag = "memo_type", content = "memo")]enum Memo {Text(String),Id(i64),Hash(String),Return(String),None,}

WOW, incredible! With the above revisions, my PR went from:

1. Intermediate Transaction
2. Transaction Deserialization Function
3. Intermediate Memo
4. Memo Deserialization Function
5. Memo Struct
6. Transaction Struct

To simply:

1. Memo Struct
2. Transaction Struct

Essentially, using a couple cheeky serde tricks I was able to cut my PR down to less than 50 lines (not including tests).

Conclusion

TL;DR: the rust community is amazing! All the most wonderful parts of my journey can be attributed to the rust community:

• The Stellar SDK group who so willingly took me in and has the patience to help me out.
• David Tonlay who graciously offered support in refactoring my most recent Stellar SDK PR.
• The New Rustacean Podcast that I listen to on my daily commute to work (Chris Krycho if you are reading this, I’d love to get in on a second t-shirt order)
• The rust Discord which offers 24/7 help
• And so much more…

Sure, ownership is hard and I still don’t fully understand it. Sure, it often seems as if the borrow checker is purposely working against you. Sure, rustaceans may only be marginally as cool as gophers. Sure you could compromise and write that app in JS 3x as fast.

However, that all being said, the rust community is very welcoming and will be there for you when you’re ready to quit (or smash your computer as you get another lifetime error). If you haven’t reached out or found an exciting open source project to contribute to, I highly recommend you give it a try!

The greatness of a community is most accurately measured by the compassionate actions of its members.- Coretta Scott King

Disclaimer: While I do work on many open source Stellar projects as part of my job, I contribute to the Rust Stellar SDK 100% on my free time and never at the workplace (typically between the hours of 10PM — 12AM). I did not start this project and I am far from being the lead maintainer. I am simply a newbie looking to learn rust while contributing to a project I am excited about.