The original reasoning for the move was purely cost related, however the performance improvement was huge. Our company decided to officialy move into the “Cockroach Age”, and by that it sadly meant moving a lot of our server-side implementation out of Heroku. Heroku made things simple, that’s for sure, but we had a ton of Amazon credits and were feeling a bit taken advantage of by paying more and getting less all for pure laziness. I can’t say it was easy. I literally looked at dozens of articles, all providing partial information or outdated/wrong information. I was thinking to myself “this is weird, why is there no simple and clear guide to tell us small and poor startups how to move away from Heroku”. So after a lot of guides, stack overflow error solving, AWS architect 1 on 1, and some sheer frustration on the brink of giving up, I am here providing you a full guide on how to do this scary move. 1. Elastic Beanstalk (EB) Most of us have already used or are aware of EC2 instances, but when using a dedicated EC2 instance for an entire backend, it becomes harder to scale when workloads change. Therefore I chose to use Elastic Beanstalk, which is an easy to use service for deploying and scaling web applications, and moreover it is free. Here is how to set it up for Flask ( : I use the EB CLI interface, so you will need to download it prior: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html ) Go to the folder where your project lies In the command line type: eb init Choose your region, create a new application, enter its name, and select its platform (Python in our case). Choose the version (most likely 2.7), say yes to setting up SSH, and either select an existing key pair you already had set up in AWS or create a new one. Now we have finished setting up the directory for EB, and we can create our running environment. Type in the command line: eb create Congrats! you have just created your first EB environment. EB has now created an EC2 instance and attached a storage volume to it. You can now go into your AWS EB dashboard and see your newly made environment. EB is pretty smart, and as you see it was simple to set up. It zips and uploads your folder to AWS, and constructs the server based on its knowledge of Python environments. It will look at your requirements.txt file to know what libraries to download, and will try to find your WSGI configuration to start up the server. You can go into the configuration tab and change its scaling and networking. You can also see the enviroment’s health in the dashboard or fetch logs if you need to have more intel of what’s going on. Of course my execution brought with it a handful of errors, and the key thing to focus on is a creature they call “.ebextensions”. This is a folder you need to add to your project which consists of .config files. These files basically allow you to configure your environment while it is being set up. #my example.configpackages:yum:git: []libffi-devel: []libjpeg-turbo-devel: [] container_commands:01_wsgipass:command: 'echo "WSGIPassAuthorization On" >> ../wsgi.conf'AddGlobalWSGIGroupAccess:command: "if ! grep -q 'WSGIApplicationGroup %{GLOBAL}' ../wsgi.conf ; then echo 'WSGIApplicationGroup %{GLOBAL}' >> ../wsgi.conf; fi;" option_settings:"aws:elasticbeanstalk:container:python":WSGIPath: application.pyNumProcesses: 3NumThreads: 20"aws:elasticbeanstalk:application:environment":EXAMPLE_ENV_VAR: env_var_value In my case I was using the command to install some cryptography libraries and git on the machine. I used to add all my environment variables. Most importantly I had a few WSGI issues I needed to fix. WSGIPath to tell EB what file to launch my app from. to allow basic HTTP authentication headers to not be truncated when passed to the server, and due to issues I had related to this article: yum aws:elasticbeanstalk:application:environment 01_wsgipass AddGlobalWSGIGroupAccess http://djm.io/deploying-scipy-into-aws-elastic-beanstalk/ Alright! now after you have added your .ebextensions folder, just go back to the CLI and issue the command , and that basically re-deploys your server with the new changes. eb deploy 2. Workers We had a few more things to figure out, as we also had a worker process running our longer tasks, or tasks we didn’t want to have block our server. We used a Redis Queue and a separate dyno on Heroku to produce our current worker configuration. Luckily, Elastic Beanstalk also comes with a Worker environment that is pretty simple to set up. What you need to do is follow the same steps as setting up the web environment, only this time choose the existing application you have created and when prompted choose to create a worker environment. The worker environment will prompt you about an SQS creation, which we will talk about now. The way worker environments work in EB is by communicating with the web env using SQS. SQS is basically a simple queue service that you can send information to. In this case, our web env can send a message to the SQS queue and the worker thread has a daemon installed that listens to that queue and fetches the message when it arrives, does what is needed and returns a 200 (ok) status back. If it doesn’t return a 200, the message gets sent to a dead queue which can then be used if these failed requests need to be re-issued. What needs to be set up to make this work: Install (AWS SDK for Python) on your flask backend and create a method that sends an SQS message to the desired queue that was set up for the worker env . In my case, I sent an SQS message consisting of two parameters, a function name, and an argument list. boto (you will probably need to set up the proper AWS permissions for your server to allow you to send SQS messages) def send_message_to_sqs( ):# Put the message in the queuem = boto.sqs.message.RawMessage()m.set_body(json.dumps( ))status = q.write(m)print status data data send_message_to_sqs({'func':'save_to_s3_and_update_user', 'args':{'user_id':str(usr.id)}}) b. Have an endpoint in your worker env for it to have the SQS messages be sent to. You configure the endpoint name in the Worker env configuration. c. The method of that endpoint in my worker env basically looked at the incoming message, based on the function name understood what function to call, and gave it the argument list. At the end if everything was successful, it just returned a 200 (ok) status. Here is an example and Amazon provides as an example. web environment worker environment If you were able to reach this point, you most likely now have configured a basic Elastic Beanstalk web app with a worker thread that interacts with it. Hurray! 3. MongoDB We used one of the Heroku add-ons (mLab) for our MongoDB database. It makes things super simple, and they offer you full support, do the setting up, and maintain the database for you. However, we decided to give AWS a fair chance with this as well. This is a pretty long and elaborate guide, so I am just going to link you to the best guide I found on this: _EC2 instances can be configured with either ephemeral storage or persistent storage using the Elastic Block Store (EBS…_docs.mongodb.org Amazon EC2 Just a few pointers if you are having trouble or receiving errors (like we did): Make sure you have a security group that opens ports 22 and 27017. You will need to SSH to your machine so you need port 22, and 27017 is for your MongoDB connection. Edit /etc/mongod.conf and look for . If it is pointing to the localhost, you won’t be able to access it from an outside ip. Either give it the ip of the machine that needs to access it, or comment out that command entirely. bind_ip Make sure you add a user (or the right role that fits your needs) to your MongoDB users for the database you are interacting with. You do this by connecting to your mongo shell (SSHing then typing or doing a remote command from your local machine), and then issuing a command. The role I gave my new user is: dbOwner mongo mongo db.createUser() “roles” : [ {“role” : “dbOwner”, “db” : “my_database_name”} ] If you connect to your database without any user, you might be in read-only mode. Make sure you are connecting to the database with the right username and password on your flask backend. 4. Add-ons Heroku had a lot of neat add ons, specifically in my case I found a lot of use for the UI logging, and periodic database backups. No one said you can’t do that on EB too! Papertrail for logging _AWS Elastic Beanstalk is an easy way to quickly deploy and manage applications in the Amazon Web Services cloud…_help.papertrailapp.com AWS Elastic Beanstalk Basically as simple as adding another config file to your .ebextensions and configuring it properly :) Periodic backups for your MongoDB _node-mongodb-s3-backup - A node.js package that makes syncing a MongoDB database to S3 simple._github.com theycallmeswift/node-mongodb-s3-backup Simple and easy to use Node package that backups your database to S3 every day. Final Words I hope this guide was able to help you on journey of moving from Heroku to AWS, or at least showed you my personal tackling of it. Since our move, we are definitely more cost effective, have better response times for our app, and are pretty satisfied :) your is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories
