Part 1 — the infrastructure: an introduction to the challenge and an overview to the general project infrastructure Introduction Let me share with you, reader, this adventure with and a variety of development tools and programming languages.What’s this, you’re wondering?This is the first part of a journey I’ve made for learning something new and of course doing something useful.Actually, that’s called . You got the point. Buxfer hacking Journey This journey took me through the following milestones: Part 1 (this part): Introduction Part 2: Cleaner Part 3: Collector Part 4: Goxfer Part 5: Conclusions But, wait, what’s Buxfer? Buxfer helps you see all your accounts at one place, understand where your money goes, reduce unwanted spending, and save for future goals. In other words, it’s an online storage service for your financial data with cool charts, grids and very useful features, like budgets, forecasts and reminders. I discovered it last year because previously I was saving my transactions inside Excel files or using programs like .All the solutions I’ve tried did not satisfy me and I was always looking for something better.To be franklin, at the beginning my research was focused on free softwares; then I started looking also for premium solutions and that’s when Buxfer showed up. GnuCash Why Buxfer? The of my research were: requirements , so I can use it with any device with just a browser an online responsive web application , so I can put/get data to/from it web APIs , so I can see in real time what’s going on automagic charts, grids, forecasts , so I can dispose my data not only in a hierarchical form but also flat tags Of course, it’s not the only one and maybe it’s not the best one: franklin, I don’t know and I don’t care. Buxfer it’s the best for me and my needs.Anyway, one thing that attracted me during the research was the team (you can find it inside the ): a Software Engineer, a Designer and 3 Investorts.Woah.So few. And they did it. They did it well.So, why don’t give them a chance? Buxfer satisfies those requirements. About Page The challenge So, the goal is clear but the problem is that my data has been built and rebuilt many times, with different structures.Let’s take a look, year per year: Load all of my data on Buxfer and start analysing it. 2014: Excel files 2015: different Excel files 2016: GnuCash files As you can see, I was so smart to change the data structure year after year, breaking the rule of “backwards compatibility”. Very clever, huh?Now, regrets apart, let’s face it: I’ve got a lot of different data I need to put it online. Getting Started If you have a dream, you can spend a lifetime studying, planning, and getting ready for it. What you should be doing is getting started. Drew Houston Ok, first of all, what do I need to do to achieve my goal? choose the technology stack setup the development environment the data in a common format export define the data model the data ( ) clean part 2 the data ( ) collect part 3 use to put the data online ( ) Buxfer APIs part 4 draw the conclusions ( ) part 5 Technology stack I’m very good with but you know what? I use it every day, so, . it’s kinda fun and very useful for data manipulation: check!Oh, and don’t forget along with because I want to have everything isolated inside the project.I’ll need something fast, that can perform HTTP calls in parallel, because I want to load all the data in just few seconds (unfortunately, you can just add a transaction per time with Buxfer APIs): what about ?It’s fast, with native support for parallelism and it has got explicit types: I need it!Along with Python and GoLang, I’ll need also and , the package managers. Well, actually glide is not the official one for GoLang but it’s really good. So, what kind of language I’m going to use? Javascript just no Python virtualenv Python GoLang pip glide **Where to store data?**An important point is the data storage.I don’t need schema nor relations between tables: I just need something to fill with transactions.So, I think I’m going with a document-oriented database. ? Yes, why not? I don’t have special requirements for storing and I do know , so, go for it! MongoDB MongoDB Well, I don’t want to install that version of Python, neither that version of GoLang, nor that version of MongoDB on my operating system.I need containers. .Just Docker?Nope, let’s use to facilitate the setting-up/tearing-down process. What else? Docker FTW docker-compose Environment setup I need the worktable ready so when the rough time comes I’ll be prepared to face it with all of those shining tools listed above. The very first thing to setup is the project folder: .├── LICENSE├── README.md├── go└── python and folders will keep the project source code.Now, it’s time to setup Docker and docker-compose, so let’s add the file and the docker folder: go python docker-compose.yml .├── LICENSE├── README.md├── docker-compose.yml├── go└── python For now, docker-compose.yml is empty but it will be filled with services later.Instead, let’s define the structure inside each sub-projects (go and python): docker docker├── Dockerfile└── entrypoint.sh I’ve just added some docker definitions for each language (Python and GoLang), so I’ll put there dependencies and requirements for each service.docker-compose will use those Dockerfile to build all the images and the entrypoints will be used just to pass commands to the containers.Good, for now, that’s enough.The project folder structure looks like this: .├── LICENSE├── README.md├── docker-compose.yml├── go│ └── docker│ ├── Dockerfile│ └── entrypoint.sh└── python└── docker├── Dockerfile└── entrypoint.sh Ok, now it’s time to draft the docker-compose.yml file!What I need is: a service to the Python container setup a service to the GoLang container setup a service for each program ( ) cleaner, collector, goxfer a service for the DB a for the DB global volume It should look like this: Exactly!setup-python and setup-golang will be used just once to build the images.cleaner, collector and goxfer form the very core of this project: the last two depend on mongodb service.mongodb, well, it’s the db.db_mongo is a global volume to store the database data. I’m going to define each services and Dockerfiles later in this journey.For now, that’s enough! Data export Ok, this is easy.Every program, like Excel and GnuCash, usually has an that allows you to get data in a common format. , that’s it.With GnuCash you can also get a JSON of your data but I don’t want to parse different formats.Let’s use CSV.By the way, there’re a lot of libraries for parsing CSV: for instance, Python has got a for it. export procedure CSV native support Data model Another important part of this journey is to define a common model to use inside each program (cleaner, collector and goxfer): Model {description: String,amount: Float,tags: String[],account: String,date: String} Buxfer needs those information, like description, tags, account, etc.So, what I’m going to do is to convert the following data into that model: 04/06/2016,Abbigliamento,maglietta,"5,00" This CSV row can be viewed like this: <date>,<tag>,<description>,<amount> But wait: where’s the account?Actually, the account is hardcoded inside the file name, like : when I read a CSV file I know if it has to be marked as an expense or income. expenses.csv End of part 1 I know, I know, this part was tricky but I needed it to draw the whole picture and to define the structure of this project.Before starting, it’s always a good thing writing the “blueprint” of what you’re going to do.Now, that I’ve got the project skeleton, I’m able to start developing the main functionalities. If you enjoyed this article don’t forget to share it!See you in ! Part 2: Cleaner Spoiler Source code is already available here: https://github.com/wilk/from-csv-to-buxfer
