An interesting message popped up on my radar from Kevin Law on September 21st, 2016:
“You’d be a perfect fit for a company I’m advising if you’re extremely flexible on location”
Since I love working on early consumer products and traveling, it was only three weeks later that I touched down at the airport in Kampala.
The mission: Serve as interim CTO for 1 month and provide leadership to a small ride share engineering team, paving the way for future success.
(Note — there are several HD YouTube videos of Uganda at the end of this post if you’re like me and you like to watch videos.)
Boda Boda
A “boda boda” is an inexpensive motorcycle ride offered in African cities (for my San Franciscans, imagine a ride from Hayes Valley to the Marina for just $1 versus a $5 Uber carpool). It is estimated that 1,000,000 boda rides take place everyday in Kampala. While on a walk, upon reaching a sidewalk, you will find you are solicited by a boda driver within seconds. “Boss!” “Boda!?”
A regular boda ride involves hopping on the back of a boda driver’s low power motorcycle. He will take you on a harrowing ride, dangerously weaving through the chaotic gridlocked streets, sometimes crossing a median, often narrowly avoiding a collision or sideswiping a bigger vehicle, and sometimes just driving on the sidewalk. Passengers rarely have helmets, they sometimes double or triple-up on the back of a single bike, drivers are often unlicensed, and, at least in Kampala, there are very few traffic signals, street lights, or stop signs. At the end of the boda trip, you negotiate with your driver and you pay in cash.
A look at the streets of Kampala during the day. Video shot here.
Don’t let the above-mentioned video fool you- if it looks fairly organized in the video, one must only travel a block or so before you hit points which seem totally chaotic. Most intersections are very disorganized and seem extremely dangerous to a mzungu like me.
Drivers Want “SafeBoda”
The name of the company I consulted for is called SafeBoda. Its business hypothesis is that consumers may want a safer, organized, and vetted system for requesting a boda ride. SafeBoda drivers are screened, have a bright orange reflective vest, and have an extra helmet available for you to wear on the ride. The SafeBoda driver’s unfair advantage? - they get many more boda passengers than before they were a “SafeBoda” driver and therefore they make more money. SafeBoda drivers also have access to insurance, quality equipment, training in first-aid and road safety, and a sense of community that is rare in the industry.
Evidence of early traction is that with virtually no technology, SafeBoda was able to sign up over 1,300 Kampalan boda drivers who each pay a fee as part of their onboarding. SafeBoda was so successful at this that they stopped accepting new drivers in order to focus on passenger demand and product.
Before my arrival in Kampala, SafeBoda had a basic app in the Google Play Store with which around 1,000 rides per month were taking place.
Clip of my “SafeBoda” ride from my apartment to the office
Uganda’s Mobile Phone & Payment Technology
Android is king in Uganda with around 70% market share. iPhones are rare… perhaps a 3% share. There are also ‘feature phones’ which only allow calls, texts, and a basic interactive textual menu provided by the carriers (USSD) which I would estimate is used by the remaining 27% of users.
The major cellular carrier is called MTN. In addition to data plans and voice calls, MTN also provides a money transfer service called “Mobile Money” which is tied to your SIM card and used by around 85% of Uganda’s citizens.
A Ugandan can deposit his or her pay (which is usually disbursed as cash) at any one of thousands of MTN locations throughout the country. MTN Money locations are usually offered as an extension to a physical store front such as a market or street vendor.
After depositing cash at an MTN Money vendor, the balance on the user’s mobile interface instantly updates. With the updated balance on the mobile interface, the MTN Money balance can then be used to pay bills and transfer money to others at a cost of ~3% in fees on each transfer.
Clearly as part of a technology revamp, SafeBoda should include MTN Money as part of the payment solution. However, we wouldn’t be able to complete the integration within 10 days due to MTN’s stone aged SOAP-based API. Instead, we decide to use Stripe’s API platform which we can implement in 1 day and which the founders felt would help to validate whether or not credit card holders are willing to use the service. It is decided MTN integration will take place in a subsequent development cycle.
SafeBoda’s Legacy Tech
Upon my arrival, the 5 person engineering team had first created a PHP CodeIgniter / MySQL backend on 1 Digital Ocean server, but then decided to rewrite it using Node.js with Socket.io after they had encountered a series of problems related to connection issues and deliverability problems with the Android client.
On my arrival, some of the problems the local engineering team was grappling with were:
- When the client would miss a socket.io message, the app would strand
- The MySQL database was hard to migrate
- The code was peppered with large, hand rolled complex inline SQL queries
- The developers were actually testing against the production servers
There was a lot of opportunity to improve both the development process and the product.
Overhaul In 10 Days
I convinced the founders to let me drive a complete overhaul to their product architecture and process in 10 days flat — roughly 300 man hours split between just 3 engineers.
Having worked for over 2 years engineering at Lyft, I also know a thing or two about the best way to design almost anything related to ride share services. Scaleable product architecture and fast development process using technologies the team is already familiar with was right up my alley and my experience would allow us to leverage everything I had learned.
I wanted to use Python and Flask, but after interviewing the team, I was able to see that due to their lack of significant Python experience, there would be little chance we would have a working revamp in 10 days. I had to keep them on PHP… something that made sense for them since scaling wasn’t an issue yet and rapid prototyping and market validation were the main focus. We would generally build with an eye towards future scalability in case things explode out of the gate, but for now, fast development was priority #1.
With 2 of their existing devs, the three of us started a 10 day process in which we locked ourselves away in a room, worked almost until midnight every night, and emerged at the end of the 10 days with a new minimally viable system.
Below is an animated gif (pardon the low resolution) of the “Golden Path” of a ride request & boda driver pickup. This recording was made on the evening of our 10th day of development. This screen recording used the real API’s that were working on the 10th night.
A “rough cut” of the working app. All of the screens used real API’s behind the scenes on our new server architecture. Almost all of the values were dynamic and correctly pulling from the REST API. Obviously several ui features were not completed at this stage.
The following week, the Android interfaces were updated:
HD video of the golden path about 1 week after the animated gif was recorded.
How We Made It In 10 Days
When you rapidly build something, you start with an idea of what the features will be, what platforms it should run on, and who will do what. Given Android is the market leader, our mobile developer would build out all of the front end screens according to our product spec. We mapped out all of the screens on a whiteboard and then our Android dev was off to work.
The other developer and I would would hack out the backend architecture. We would create a new development process, infrastructure, product architecture, and API’s. We would document them, and hand them to the Android dev to implement on the 4th day.
Hosting on Heroku
Hosting this project on Heroku propelled us faster and further than I had originally imaged. Using Heroku pipelines with GitHub integration meant that once a pull request’s tests pass in Circle CI, it automatically deploys into our staging environment. I had never used Heroku pipelines before, but I think it’s amazingly easy to setup and very powerful.
Heroku’s awesome pipeline interface used for deploying to production
The best part is that as we open up pull requests, Heroku senses it and offers us an ability to launch a test app called a “review app” — you see it above in the left column. One click will launch a fully operational environment that you can connect the mobile app to in order to test an upcoming feature. Setting all of this up takes only 3–4 hours, which includes learning how it all works.
Dockerized Development
In production, we use mlab.com to host MongoDB version 3.2. Docker allowed us to launch a local Docker development database container with 1 command:
docker run — name mongodb -d mongo:3.2.10
(Note — some of the additional switches we use are omitted for brevity)
As for PHP, I was able to create a PHP dev environment container that closely mirrors Heroku’s environment. The devs can do a similar docker ‘run’ statement to launch a local development container through which they can work on their local PHP files, do live debugging with PHPStorm via both web and cli requests, and even test their local development using an Android phone across an ngrok tunnel.
PHP Application Framework
The closest thing PHP seems to have to Python’s Flask is called Lumen. Lumen gives you a basic routing system and is setup to support REST API development. It makes versioning your REST API very easy. It also supports adding “Middleware” which performs things like API authentication. One thing I personally like is the IOC injector that detects which objects your controllers would like to use and automatically injects them for you by default.
For Mongo access, we used PHP’s Doctrine ODM — an open source data mapper for MongoDB that we used back at Lyft and a project to which I have also been an open source contributor to.
Gameplan
On day 1, we mapped out our data model. We really only needed two key domain models: User & Ride. We then went to the whiteboard and defined our ride state transitions. The happy path looks something like this: ‘requested’ -> ‘accepted’ -> ‘arrived’ -> ‘pickedup’ -> ‘droppedoff’. There are two edge cases. One is that if our dispatcher fails to find you a ride, we set the state to ‘operatordispatch’ - a final state in which the mobile client simply displays the number to SafeBoda’s human powered dispatching call center. They’re able to manually setup dispatches to drivers who have feature phones.
The other case is when the passenger cancels — that’s a final state where the request ends. If the driver cancels, the dispatch loop just continues. Below are some of the actual whiteboards drawn in our kickoff on day #1.
Initial diagram of the ride initial state changes.
Some chicken scratch from that first meeting where I described the way the phones would poll the server and how we would structure the HTTP JSON server response.
From left to right: Charles, Kirk, and Jon (me).
RestAPI
We next sketched out a basic HTTP REST API that has methods such as:
POST /v1/rideCreate a ride request.
PUT /v1/userlocationUpdate a user’s current location.
In total, we have about 20 REST calls that were defined. We wrote the whole API spec up in an swagger.yml file that is rendered beautifully if viewed with http://editor.swagger.io/.
For the following 9 days, we hacked away. We didn’t attend any of the company meetings and we spent some of the time in my personal apartment so as to not be distracted by anything.
Service Oriented Architecture
As with most minimally viable apps, we created a tiny so-called ‘monolith’ app. I like to organize business logic into service classes — these become the foundation of what you would eventually ‘decompose’ later into application partitions, or what some people call “microservices”. In our app, we have the following PHP “service” classes which can easily be broken out into the future and implemented in any programming language into microservices:
-
CurrencyBehaviors responsible for converting between USD and UGX using the latest exchange rate.
-
DispatchBehaviors responsible for finding the nearest drivers, and powering the dispatching lifecycle.
-
EtaBehaviors used to determine the number of minutes it will take to travel on the road between two places
-
FacebookBehaviors used to authenticate a Facebook user and retrieve profile data.
-
MetricsBehaviors responsible for tracking analytics event information.
-
PaymentsBehaviors related to billing users using a payments gateway.
-
PricingBehaviors related to pricing rides.
-
RatingBehaviors related to casting a rating for a ride and then also updating a driver’s record to update their average rating.
-
RideBehaviors related to creating and updating a ride.
-
UserBehaviors related to user creation and update.
-
UserLocationBehaviors related to updating the user’s location.
Coping with Eventually Consistent Data
Due to the fact that MongoDB is not an ACID database, the application must be able to cope with database references which can be out of sync. In MongoDB, there are no transactions or rollbacks. To help overcome concurrent update contention and data loss, we use optimistic record locking by default on all updates.
We use Doctrine’s event hooks in order to ensure two records point at each other. For example, If I put user A in ride B, I will first update user A’s record to have ride_id = B’s id. Then, during the post update hook, I will next update ride B’s user_id to have user A’s id.
There are short moments when the database is therefore out of sync during normal operations. Further, the app might encounter a show stopping error before the 2nd update hook completes, leaving the database in a state where the user record indicates the user is in a ride, but the ride does not correctly point back at the user and this would be the source of trouble if your code is not setup defensively.
To cope with this, we must have protective ‘cleanup’ routines that check for situations like this. Before key business logic fires, we invoke a “cleanup” routine to ensure the data integrity is correct.
Note that we do not employ MongoDB’s “ref” feature — we manually store record id’s and resolve database references manually.
Also noteworthy is that when we update the user’s location on their user record, we perform the update without the default optimistic locking record strategy used throughout the rest of the software, or else we would see tons of optimistic lock exceptions for no good reason.
3rd Parties Tools We Used
- GeofenceAPI
- PagerDuty
- PaperTrail
- Mlab.com for MongoDB hosting
- Ec2 for Jenkins & S3 hosting
- RapidSSL for the API cert, LetsEncrypt for the Jenkins cert
- CircleCI
- Mixpanel
- InstaBug
Features & Hindsight
A surprising number of drivers in Uganda cannot read maps. Some clearly had trouble knowing left from right. The society is also highly cash based and most folks don’t have credit cards.
If I had known some of these things, I would have advocated for a slightly different MVP in which relied less on screens with maps on them and put more emphasis on calling the driver.
Conclusion
The developers are no longer testing against production machines. We ditched complex hand rolled inline SQL for a MongoDB solution which requires no production migrations. If the project takes off, MongoDB will allow us to add more shards and rebalance data. It’s now up for the market to decide if the product is the right one for Uganda.
A Personal Note
Living in Kampala for a month and working on a real startup again was awesome. I feel personally enlightened and the trip gave me a new perspective on the world. In San Francisco, we have a lot of millionaires & billionaires and I often feel like I’ve missed out on something that everybody else was able to achieve. However, my trip to Africa made me realize just how fortunate I am to be a self made engineer and what an achievement it is to even be able to live in San Francisco in the first place.
My 30-day contract has ended. I wish the founders the best of luck. Here are the current state of things:
- SafeBoda is seeking an on-site full-time CTO. If you’re up to the challenge, contact [email protected]
- The version of SafeBoda in the Android Play Store is still the legacy version of the app. They continue to make iterative improvements to the upcoming platform, and have a strategy to release the new platform in January. The work we accomplished currently lives in the beta channel.
Make me proud guys!
Contact:Twitter: @blockjonEmail: [email protected] me on the DoorDash API Platform Team and browse other available positions:https://www.doordash.com/careers/
Pictures And Videos From My Trip
KLM flies from SFO all the way to Kampala’s international airport with layovers in Europe
My former office: San Francisco’s luxury “China Basin” building. My Kampala office? This inexpensive building. Photo taken here.
HD clip of a ride down a street in Kampala shot from an Uber.
A look at some Rhinos which were going to sleep for the day
World’s most powerful waterfall
Heading towards the center of Murchison National Park — home to thousands of wild animals
Giraffees in the road in Murchison National Park
Elephant hanging out at the water’s edge on a hot day. Hippos in the water.
Walking from my apartment to the coffee shop. Filmed here.
A Ugandan lunch: Rice + G-nut sauce, sweet potatoes, cabbages, and greens.
Wiping out in a Jinja class 4 rapid (me in front left)
A sophisticated fashion event
Buying a breakfast sandwich “Rolex” on the side of the road in a slum. Video shot here.
Eating tilapia at the shore of Lake Victoria. Video shot here.
Inside the ultra-modern Acacia Mall food court
Buying a random rolex on the side of the road heading from Jinja to Kampala