Flatfile’s vision is to enable seamless data transactions between every organization in the world

Flatfile

flatfile

It requires an ample amount of time, effort, and resources to produce great, user-friendly documentation. Good documentation is fundamental to the way engineers get started with a product and ultimately use it in the long run, which makes it essential to both user retention and growth. Building our documentation at was a task we therefore took very seriously. Courier We originally created our documentation for a very small, specific audience with specific uses for Courier. Over time, however, our user base, their use cases, and our product itself has grown dramatically. To cover our bases, we needed to improve and expand our documentation and this time, we wanted to make it scalable and with a focus on a great user experience. We decided to use to do so, which allowed our engineers to collaborate and update our documentation more efficiently. Here’s how we built our documentation and what we learned in the process that might be useful to you. Docusaurus Identifying challenges with our documentation process When we first started Courier, we just wanted that explained how to use Courier as a product. So, we found a tool that could help with that: ReadMe. documentation ReadMe gave us enough features to get started with our documentation. We could create and update our documentation when needed and without much effort, but we couldn’t collaborate as much as we needed and could only save one draft at a time. This was a major challenge because anyone could easily overwrite another’s work if they were working on the same file at the same time. ReadMe also had a flat file structure which was inconvenient to use. There was also no real integration into their CLI, so we had issues with document versioning and collaboration. Ultimately, this meant that docs were in the approval process for an unnecessary length of time and not everyone was able to make suggestions for improvement as they came up. Git So some years and a few significant product updates down the line, we knew that it was time for a major documentation update. ReadMe was easy to get started with and served us well for our first couple of years, but we needed something that addressed our more specific pain points. Prerequisites for a documentation tool So we knew we needed something new. The next step was to decide which solutions were necessary to address our pain points for the next version of our documentation. The team came up with a list of requirements for documentation tools we would use in the future. Limitless Collaboration A happy challenge with a growing company is the need to scale almost every aspect of it as the team and product grow. As Courier’s technical team grows, we have more subject matter experts to provide input on documentation, which is good news for the quality of docs. However, more SMEs mean that we need more people to be able to collaborate on a draft at a time. Since the team will only continue to grow from here, limitless collaboration was one of the basic necessities for our future docs tool(s). Versioning A growing company comes with the inevitability of a growing product. As new versions of the product are released, there will always be an overlap time when some of our users are using an older version of the product and some are on the newest version. Our documentation needs to cater to both of these groups of users, which means to make sure everyone has the correct information, versioning is an important docs feature. For example, we need to keep the documentation of our API 1.0.0 version separate from the one for API 2.0.0 and ensure that it is possible to update and switch between versions quickly and easily. Version was also important to allow for the option to separate and store drafts away from the official documentation so that the team would be comfortable working without the pressure of a breaking change. Link Validation One of the worst things that can happen to user experience in product documentation is for a user to click on a link and get a 404 error. It makes it more difficult to keep users in the docs and following instructions, which can ultimately affect how many users successfully get started with the product and/or are able to troubleshoot on their own. A docs tool that would include link validation for internal links would go a long way toward avoiding damaging errors and UX issues. Localization We have customers from different ethnic groups all around the world who speak a diverse set of languages. To cater to a global audience, users need to be able to read our documentation and get the help they need, regardless of the language they speak. Localizing our documentation like link validation, improves user retention and UX. Docusaurus matched our list With our list in place, we were able to begin researching to find the right new tool. After some research, we discovered that , which is built on React, allowed for limitless collaboration, versioning, link validation, and localization. Docusaurus With Docusaurus, we built our docs as markdowns and exported them as a static site. We created reusable custom components and now have complete control over how we build our documentation. The integration is also particularly useful, as it allows any JavaScript code to be embedded into the .md file. MDX With a few lines of code, we could tweak our theme to suit our brand colors or customize any part of our documentation by the components we wanted. We also could use plugins to extend the functionality we got from Docusaurus. An example is the plugin for our search feature that allows customers to find relevant documentation. swizzling I also built our custom API explorer myself, which enables developers to try out our API methods directly from our docs. I built it as an embedded React component placed in the MDX file. Because we host our documentation on , we could utilize Vercel Functions to proxy the API Explorer requests to the Courier API through our docs backend endpoint. Vercel With Docusaurus, we have been able to build our documentation quickly and easily. We can now make updates by simply cloning the repo on , branching off our main branch, implementing the changes, and creating a pull request (PR). Once someone has reviewed and approved the PR, it’s merged with the main branch, which deploys to Vercel and updates our site with the changes. GitHub All we need to contribute to our documentation is a GitHub account and an integrated development environment (IDE). This helps us keep track of every maintenance task for our documentation. What’s next for our documentation? There are a few more goals we have for our documentation. We have yet to utilize Docusaurus’s localization feature, so that’s in the works. We have also recently made our documentation to welcome external contributors. This way, the user can take ownership of the updates they want and help themselves and others who would benefit from their contribution. open-source Until then, learn to use Courier via our . We also content that’ll help you get . documentation and tutorials live stream started and continue with Courier Also Published here

How We Built Our Software Documentation On Docusaurus

During the height of the business intelligence (BI) craze earlier in my career, I worked with an internal reporting team to expose data for (ETL) processes that leveraged data structures inspired by . It was a new and exciting time in my life to understand how to optimize data for reporting and analysis. Honestly, the schema looked upside down to me, based on my experience with transaction-driven designs. extract, transform, and load Ralph Kimball In the end, there were many moving parts and even some dependencies for the existence of a flat file to make sure everything worked properly. The reports ran quickly, but one key factor always bothered me: I was always looking at yesterday’s data. Due to the increase in size, velocity, and types of data being produced in modern tech stacks, real-time data has become the holy grail for enabling up-to-date and proactive analysis and data-driven business decisions. When we use the common batch style approach to data ingestion, the data is always stale, and it can’t answer the pressing questions. To enable these real-time use cases, the data must be available in the storage layer and processing layers currently implemented at an organization. For many years—and this is still true today—transactional (OLTP) systems like Oracle, Postgres, and SQLServer are at the core of most businesses, and they’re central to downstream analytics and processing. This work is typically done by a team of data engineers who spend months building pipelines that pull and transform data from those OLTP systems into analytics data lakes and warehouses (OLAP). Over the past decade, organizations of all sizes have spent billions of dollars with the goal of improving their capacity to process data and extract business value from it. However, only of organizations believe they have achieved the vision of becoming a data-driven company. One of their biggest hurdles to achieving the “Data Utopia” state is gaining the ability to build reliable data pipelines that extract data from OLTP systems while ensuring data consistency and ease of scale as the data grows. 26.5% This made me wonder if there was a better way to accomplish the same results without a hefty overnight process. My first thought was to explore the change data capture (CDC) pattern to meet those needs. Why Change Data Capture (CDC) Matters Extracting data from OLTP systems can be cumbersome, for reasons that include: Unlike SaaS tools, enterprise databases don’t have flexible APIs that you can leverage to react to data changes. Building and maintaining pipelines to extract data from OLTP systems, especially as a business scale, is complex and costly to ensure data consistency. On the other hand, CDC is a software design pattern that determines and tracks database updates as they occur. This allows third-party consumers to react to database changes, vastly simplifying the process of retrieving data changes in the OLTP system. CDC events can be sourced through several different approaches: Time-based: uses the required timestamp column as a source for data Checksum: leverages checksum on tables in source/target, using differences as a source for data Log-based: reads database logs as a source for data For this publication, we will focus on the log approach. Log-based CDC is commonly accepted as the best option for real-time use cases because of its lightweight approach to producing data changes. Time-based and checksum approaches require more overhead and CPU, often using queries against the database to find the newest changes to a dataset. CDC using log-based reads can shortcut these expensive and time-consuming queries, capturing changes by going straight to the source: the logs. Using the log approach, database writes are handled via transaction logs, often referred to as changelogs. CDC solutions listen to those same logs and will generate events where established conditions apply. By using the logs, the CDC service gains access to the same data without waiting for the transaction to commit and then making a request to the database for that data. By implementing a CDC solution, third-party data consumers have quicker access to the database changes. Additionally, data consumers do not require access to the actual database since they are only interested in updates in those changelogs. Not only does CDC allow near-instant data replication, but it also has minimal impact on the performance of production databases. Because CDC only accesses the logs and doesn’t pose any security threat, the IT teams love it too. The unique characteristics of CDC made it widely adopted by many tech leaders that real-time use cases are key to their business models (e.g., , , ). Shopify Uber CapitalOne To understand CDC more, let’s consider a use case related to the financial industry. Detecting Fraud Using CDC Fraudulent transactions are a major concern for the financial industry. In fact, noted credit card fraud increased 44.7% in 2020 over prior-year values. this report Let’s assume WorldWide Bank, a fictional financial institution, wants to improve the speed at which they recognize fraud. Their current approach relies on an ETL process to query against their Oracle database and generate reports. They would like to explore using in order to process against the Oracle database changelogs to generate events that will feed their Databricks Delta Lake prototype. Like many organizations, WorldWide Bank has a variety of transactional systems, including a large subset of data in an Oracle database. They have a growing need to provide more machine learning (ML) capabilities as well as combine various datasets in an easy-to-use environment for analysis, which has led to their adoption of Databricks for a data lake. Arcion CDC The relational data model of Oracle, along with the hassle of bringing other ancillary data sources into Oracle and the lack of support to run ML directly on Oracle, makes Databricks an easy choice. But, this approach leads to another challenge: how to source the Oracle transactions into Databricks in a timely manner. To improve the speed at which they can detect fraud, it is critical to their operation to be able to source CDC events from Oracle quickly, depositing them into their Databricks data lake for further processing and detection. Regarding their decision to use Arcion CDC, why might WorldWideBank choose Arcion and not another option, even open source? The answers are simple. Debezium, for example, is an open-source option, but it requires the installation and running of Zookeeper, Kakfa, and MySQL—all before you can even begin to use or install the tool. This alone would require a large team of engineers to simply install and maintain the tech stack. While other tools—like Fivetran’s HVR for CDC—might also offer free trials, Arcion has two free options to choose from: a 30-day trial with a download or a 14-day trial on the cloud. This is convenient for testing, so it made Arcion an easy choice when selecting a CDC solution. At a high level, the Arcion CDC architecture can be illustrated as follows: If this approach goes well, WorldWide Bank will expand the design to replace other source databases in Oracle, Microsoft SQL Server, and MySQL. Prototyping Arcion CDC In order to simulate the WorldWide Bank scenario, let’s assume inbound transactions are being written to an Oracle database table called TRANSACTIONS, which has the following design: An example record in the TRANSACTIONS table would appear as shown below using JSON format: {
   "id" : "0d0d6c35-80d0-4303-b9d6-50f0af04beb3",
   "post_date" : 1501828517,
   "trans_date" : 1501828517,
   "type" : 1017,
   "card_number" : "4532394417264260",
   "account_number" : "001-234-56789",
   "amount" : 27.67,
   "vendor_number" : 20150307,
   "vendor_location_latitude" : 42.3465698,
   "vendor_location_longitude" : -71.0895284,
   "vendor_name" : "Berklee College of Music Bookstore" 
} In order to leverage the power of Databricks and Delta Lake for fraud prevention, we need to set up a CDC service to act as middleware between the two systems. The TRANSACTIONS table design and data in Oracle need to exist as a Bronze (Ingestion) layer within Delta Lake. Once the existing data has been ingested, all future transactions will leverage the CDC stream included in the Arcion solution. ** **Getting started with CDC begins with establishing a free and obtaining the connection/login information for the Oracle database and Databricks system. Arcion Cloud account After gathering this information, I started a new replication in the Arcion Cloud application and defined the replication and write modes between the source and target database: In this case, I selected the replication mode. This lets me replicate a full snapshot of the database and enables real-time operations for all future updates. I chose the write mode option to fully replace any existing database design and data on the ta Full Replacing rget host. Next, I established a connection to the Oracle database, similar to what is shown below: We can validate the connection by using the Test Connection button. Once ready, the button moves the progress to the next step. Continue to Destination On the Destination screen, I can add a connection to the Databricks service and select it as the target for the replication being created: Using the button, I was able to proceed ahead to the filter screen shown below: Continue to Filter I selected the source database and tables on the Filter screen. Once ready, all I had to do was click the button to complete the following tasks: Start Replication Remove any existing databases in Databricks that match the current replication request Establish new databases in the Databricks environment Perform a snapshot replication of all existing data that matches the filters provided in the replication request Establish a CDC stream between Oracle’s changelogs and Databricks for all future transactions. I completed all of this with zero code. Once the replication started, I could monitor the process in the Arcion Cloud user interface: As noted above, the snapshot will be performed first, then any future updates will arrive via the CDC stream. Once completed, the user interface is updated as shown below: At this point, the Bronze (Ingestion) layer of Delta Lake is ready for use. The ingested data can be transformed into Silver (Redefined Tables) and Gold (Feature/Agg Datastore) layers, allowing us to leverage the benefits of Delta Lake functionality to detect and mitigate fraud. Going forward, the expectation will be that all data from the Oracle TRANSACTIONS table will automatically find its way into Databricks. We have one final step of our CDC ingestion into the data lake. Prior to running the ML fraud model for prediction output, we would run pre-built dbt tests or Great Expectations checkpoints to ensure we have received new records since our last fraud detection run. This would give WorldWide Bank confidence that full end-to-end CDC from Oracle is flowing into Databricks as expected. I find it very impressive that I can establish this level of design by merely following a series of steps within the Arcion Cloud user interface without writing a single line of program code. For the WorldWide Bank use case, establishing similar replications for their other databases should be just as easy, allowing all fraud detection to be processed in one single system. Conclusion Since 2021, I have been trying to live by the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality which extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester Arcion provides CDC functionality without reinventing the wheel. Their platform allows feature teams to utilize a no-code approach to establishing source and target data sources, plus everything needed to route inbound database transactions to a secondary source before they are written to the database. In addition, Arcion can be deployed in multiple ways. Financial or health care enterprises, for example, maybe have security or compliance requirements. They can deploy self-hosted Arcion on-prem or in a virtual private cloud. For companies without the DevOps resources to manage their own deployments, fully-managed Arcion Cloud would be a good option. Furthermore, the ease at which snapshot and CDC streaming replications can be established allows for additional data sources to leverage the same processing logic found in the Databricks Delta Lake implementation. This ultimately provides a “don’t repeat yourself” (DRY) approach by allowing one source of logic to focus on fraud detection. The Arcion team is certainly adhering to my personal mission statement and allowing their customers to remain focused on meeting corporate priorities. If I were working with that same BI team that I mentioned in the introduction, I would certainly recommend exploring Arcion Cloud to meet their needs. This would provide the opportunity for real-time data rather than relying on day-old data to make decisions. Have a really great day!

How To Use Change Data Capture for Fraud Detection

Creating a Hiring Strategy Why should someone work at your company? Every company answers this question differently. Here are some valid strategies for hiring: “ ” - We’ll pay the median rate in your area, but we’re investing heavily in making this the place for you to do your best work — the best work culture. (early days New Relic) “ ” - We’re working in a compelling technical area, which attracts a lot of talented people, so you can work with a great team and solve interesting problems. Plus we’re making a great people-focused work environment. (Gremlin during my tenure there, also early days New Relic) “ - We’re making Disneyland for engineers, lots of perks and top pay, in a university-like environment. (early Google strategy). “ - We’ll pay really well for you to work your ass off, and in return you get a chance to work at a scale you won’t otherwise.” (Amazon today) “ - We won’t pay you the most, but you’ll get to spend time each week working on open-source software.” (a local software consultancy my friend ran for a while) In practice, when you haven’t done the work to articulate why someone should work there, your hiring strategy looks like this: “ Work super hard at a stressful startup for low pay and equity that is probably worthless.” You need to create your hiring strategy. Ask yourself, how are you going to stand out as a place people will want to work? Like any strategy, you should consider what your unique strengths and weaknesses are. Some dimensions you can flex or emphasize to attract candidates: Remote Versus In-Person: Having a strong stance on this will attract people who care about either of those options. Perks You are Willing to Offer: Consider offering allowances for teleconferencing (high quality cameras and microphones, for example). , with a designer helping you renovate your workspace. You can offer top of the line developer machines, or quality monitors. These are things that also help your team work more effectively together, so it’s offering a benefit that serves two purposes! If I were making my own startup nowadays, I’d focus on top-tier videoconferencing equipment (DSLRs and external microphones) as a main perk. Flatfile offers an office makeover Freedoms You’ll Offer: You can offer a . You can craft a narrative around . Or something else entirely. 4 day workweek offering no deadlines and no meetings Transparency on Salary and Equity: Research . There are a few things you can do to make your position stand out for candidates from diverse backgrounds: list the salaries (and equity!) for the positions you’re posting, and emphasize how you’ll make the workplace fair and equitable for all employees. Share your philosophy around how salary and equity are determined. Being more upfront about this than competitors will make you more desirable and competitive. shows diverse teams produce better business outcomes Start by , and then tell people you’re doing it. How bizarre is it that your hiring manager is expected to advocate for and win the trust of their team member once they’re hired, but they’re expected to negotiate against their best interests during the hiring process? Starting off the relationship without having to negotiate salary can be a huge relief for candidates. implementing pay equity A Haven for People From Diverse Backgrounds: The experience for many people in tech companies is far from ideal. Many people leave our field because they’re treated so poorly. Most companies haven’t invested the time and energy to build a work culture that retains and delights people who aren’t overrepresented in tech. Impact People can Have on the World: Consider ways your company helps employees connect their work to something meaningful or interesting to them. Are there ways you can emphasize this more? People at Patagonia are given the option to volunteer once a week, paid by the company. Working at a nonprofit can have direct impact. For most companies, you can think of the value of what you produce in a way that people will find meaningful. Think about how that ties into your narrative for why to join your company. : Attributes of the Team What is the team like, and how can you use that to attract the type of people you’d like to come to the company? For example, you might be good at hiring people that are collaborative, and not assholes. You might be in a place with a lot of creative thinkers. As you look through these, come up with a one sentence description of why people should join your company. You’ll know you have a real strategy if it is rational for a certain type of person to select your company over all the other companies out there. This post is originally from my . You can find a permalink for it . course on engineering leadership here Did I miss anything? Please share your favorite ideas with me, I’d love to hear them!

Rethinking the Hiring Process: How to to Stand Out and Delight Your Candidates

If You Have the Right App for It TagSpaces’ new Markdown editor marks it down as best note-taking app, hands down, no contest. If that sounds like an exaggerated claim, read on, I might even convince you. the If you have not read , you probably should: You might learn more about why this software is really good for your online health (see also: privacy and data ownership), but if you don’t like jumping across reviews, the short version goes as follows: my (last) review TagSpaces is an open-core, offline, no-backend, flat-file, privacy-focused, and privacy-respecting file organizer and editor tool that lets you keep 100% of your data 100% safe, private, and . There is no cloud server, no registration, no ever-changing privacy policy to accept — only a cross-platform, easy-to-use, continually improving software that lets you deal with your data in any way you like, even outside of the app (so no vendor lock-in either). yours As I have already (and often), I am no fan of Cloud-basing everything just for the sake of appearing modern, especially when it comes to both personal and professional note-taking. There are few advantages and many caveats of doing all things online, especially when it comes to keeping your notes and digital things organized, and TagSpaces delivers a very well-trimmed solution to this problem, which has only just got a lot better. mentioned elsewhere The new feature that prompted this review is, in reality, only new for me; since I was too lazy to update and been happily using v3.3, the feature that took me by quite some surprise has been out since v3.11. (Current version code is 4.1). So I have been missing out on quite a lot, as it turns out. A Brand New Markdown Editor Suddenly Appears I love using Markdown for my notes. It’s clean, easy to read, easy to write, easy to parse in code. So, besides all the privacy and freedom and the ability to use it the way I like (see my previously linked articles for more on this), a good note-taking app would ideally use Markdown. This is not the biggest requirement but certainly is a nice-to-have. The old Markdown editor in TagSpaces was my only real gripe with the software, yet I kept using it because of its superior tagging and organizational capabilities. I read Markdown well enough, so I did not need to see the formatted text (yes, it also has an HTML-based RichText editor, but I never cared much for those), but when I (finally) updated my copy, I was very pleasantly surprised. TagSpaces now has a that . And it’s quite easy to use, even for those who don’t like to look at the code. real Markdown editor actually works well The Best Features Are Those You Don’t Even Notice The editor is based on the framework, an awesome open-source Markdown library, which got tailored to fit TagSpaces perfectly. Milkdown The editor is WYSIWYG; when you type Markdown, it gets immediately turned into nicely formatted text. This is a big advantage for me, since however well I might read the markup, this is still easier to look at, and the formatting itself (the actual Markdown) gets preserved when I copy/paste text to or from the editor. I have, of course, used editors like that before. I write in Typora, for example, which has a nice editor, but many limitations beyond that, they recently switched to a paid model, which kind of hurts my senses after having used it for free. (I get it, money must be made, but, for example, TagSpaces only charges for professional use cases, and the base application is free, which is a much nicer way to both give to the community and earn some well-deserved money for your hard work as a developer. Dropping a paid-for update into the Linux repo was really a d*$# move, I have to say this much. Bait and switch much, Typora?) short fiction and So, while it is not the first WYSIWYG Markdown editor I tried, it fits nicely into its new environment and works really well for me, and fits the way I use the app. Formatting Text the Easy Way The editor offers a number of ways to format your text, all of which are unobtrusive and simple enough to use. Besides what I normally do (typing the Markdown straight in), you have the option to highlight some text, and choose some basic formatting from a popup. This is limited to , , strikethrough, code, and , and that seems to be kind of enough for simple note-taking, although I really do miss the ability to add highlights. You know, like . bold italics links ==this== There might be plenty of arguments for or against using this syntax, but I don’t care for any of them. To me, it’s very useful, and I’d . (You might think it’s a futile attempt, but I know the devs read my stuff, and my opinions even influenced the development before, so there is hope yet…) love to see this in a future update of TagSpaces Highlights aside, this simplest way of formatting text inline is complemented by slash commands that offer block-level formatting options. This is a feature that is quite new to me in a Markdown editor, but one I really got into quite fast. It’s kind of the same you’d see in, e.g., Notion, another online-first note-taking app that, although looking nice, is as far from my needs as most of its cloud-based competitors. So the editor has no formatting toolbar for now, and although this feature already exists in the Milkdown framework, and TagSpaces might introduce it later, I am personally not missing it. Toolbars make rapid note-taking harder on the computer when you have to constantly click away from the text just to format it or insert something. Enter the Slash Command TagSpaces’ slash commands, on the other hand, will let you stay right in the text, and even if you do not know the markup, you can just choose from a pop-up list right on the spot, then continue typing. Currently, slash commands support the following block-level formatting options: Headings to headings: A compromise between having a full list and a too-long pop-up list. I do not love it, but I can learn to live with it. H1 H3 All Types of Lists One limitation is the usage of multi-leveled lists. This is supported by the editor, but to use it, you must switch to Markdown view (only accessible from the Floating Action Button (or FAB, in short) in the bottom-right corner, I have not found out the key-combination yet), and add four full spaces the bullet point you want to indent. before Neither the slash commands, nor typing Markdown offers this option currently in the WYSIWYG editor, because there is no way to insert anything before a bullet point, and pressing the Tab key has a different binding in the software. But if there was no room for improvement, things would soon get stale, so here’s an area to improve, dear developers. It should not be very hard to do either, so there is hope. The greatest upside is the truly interactive task list, where you can click on tasks to complete them. It will not strike out the text, but being able to mark them easily is a huge advantage to any list addict. Images Adding images is fairly simple; with no file-system browser yet, you have to enter or paste image URLs or file links manually. Although has internal file linking (this is an affiliate link, I get no commission from sales, just sayin’), those file links were never meant for the Markdown editor to render them; those are for referencing only. Still, you can enter a local file path to have your images displayed, or just the filename, if it’s in the same folder. TagSpaces PRO not Quotes Selecting “Quote” will drop you into a block-quote block, with the standard Markdown syntax for block-quote blocks. You stay inside the block after pressing Enter, so you can easily do multiple lines and paragraphs. You can get out of the block by pressing Enter twice. Tables This is the coolest feature yet. Inserting a table gets you a simple but very easily customizable standard 3x3 Markdown table. The real feature here is how you can interact with it. Pressing the slightly hard-to-find-at-first but definitely unobtrusive little square in the upper-left corner ( ) will give you options to interact with the whole table. You can insert a column before ( ), or after ( ), and a row at the bottom ( ), set alignment globally ( ), or delete the whole thing ( ). 1 2 3 4 5 6 When you want to interact on a single column, you must press the column header and choose from the slightly more limited list. (The same, sans adding a new row.) Pressing the row headers gives you an even more limited menu: You can add a new row above or below the current one or delete it. Setting alignments on rows makes no sense in a Markdown table context (you know what I mean if you are familiar with the syntax), so that option is missing. Code Fences You can also add standard code fences, where you have the option of marking the language in a dropdown list. TagSpaces will not highlight code syntax for you, but it’s also not the point of the software; IDEs do that better anyway. What’s Missing from Slash Commands Quite a lot, actually. None of the usual inline formatting options are present like the aforementioned , striked, and you cannot add links here either, but that is intended behavior; slash commands are meant for block-level formatting. bold italic So, instead of being an alternative to the popup formatting toolbar, slash commands rather complement its use by offering all things missing from the popup. This certainly keeps the list shorter and the items probably easier to use, but whether it complicates the use of the application in the long run, or makes everything so much easier, will depend on your personal preference. As mentioned above, I type in markdown straight, so it does not affect me either way; thus, I also cannot form a meaningful opinion on this. You could probably let me know in the comments; I’d be interested to learn how others see and use this feature. However, this is a feature of Milkdown itself (the framework TagSpaces uses for its editor), so TagSpaces developers will not be able to change it, whatever we, mere mortals, think about it. Other Nice Features I Will Never Use There are also two nice features that I never use, but others seem to be quite keen on Math Expressions and Diagrams. Expressing Math The former seems to have been requested quite a few times on TagSpaces’ issue tracker and also their public suggestions/requests board, and so its implementation became another example of the developer’s responsiveness. In the new editor, you can do math expressions now. Unfortunately, I am probably the least qualified person on the internet to evaluate anything remotely connected to math or its expressions, but after some searching, I’ve found an example of math used in markdown online, and it turned out to render quite nicely. I will not pretend to know what it means, but it looks very math-ish: $\sum_{n=1}^{10} n^2$ becomes Currently, I find one limitation. If you type in the expression as you go, it will be immediately rendered, as you close it with the dollar symbol, behaving like any other formatting; but when you paste in an expression, found, e.g., on the internet (like I did), some symbols get “escaped” with a backslash character, and the expression does not render, until you fix this in the code view. The solution is to “paste as is,” usually meaning pressing Shift+Ctrl+V. I tried this in TagSpaces, and I managed to escape the escape symbol, meaning the expression was rendered expressively. Human-Readable to Even More Human-Readable: Enter the Diagram Diagrams in Markdown are a thing I was curious about for a while but never had any time, energy, or even reason to look into. So now that it’s one of the major new features of the editor I am reviewing, it’s also a good opportunity to continue not looking into diagrams because I am busy writing an article. Even so, I had a cursory glance, so you don’t have to. Diagrams in TagSpaces are based on the . Looking at their website, diagrams look like one could get used to using them, but not half as useful as creating them visually, with a dedicated tool. Without having any experience using these diagrams in the wild, I cannot comment on the usability, but they definitely have their place if you are restricted to Markdown and have to still visualize algorithms or classes or even just make charts. mermaid project So I tried one of the formulas from the mermaid website (the one you’d never be able to decipher easily), just to see what happens. So what happened was a very nicely rendered diagram on my screen. I am genuinely impressed. The script does not look very complicated and seems learnable, so if I ever need diagrams where I cannot draw them with some fancy visual tool, I know where to turn. Until then, I’ll probably remember, with great fondness, the time I made a diagram in Markdown and how pretty it turned out… To insert a diagram, you have to start a newly fenced codeblock, and give it the mermaid type. Since mermaid is not (currently) available from the slash-command originated codeblock’s dropdown list, you must type this in manually: ```mermaid That will turn into a special codeblock where you can type/paste your diagram and get pretty results like that above. To Conclude… This is it in a nutshell. TagSpaces stepped up in the league of note-taking apps by a huge stride with its new editor. So besides superior file-tagging and organization, note-taking, and writing, any kind of Markdown is now in the same league as some of the top players. Best thing? It’s free. Also Published Here

Markdown and Privacy-Focused Note Taking are the Perfect Match

Four company history lessons, and a forward looking statement: 2018 was the year of possibilities. 2019 was the year of issues. 2020 was the year of opportunities. 2021 will be the year of iterations. All in an effort to become . We’ve summarized Hacker Noon’s progress by revenue, growth, product, software expansion, and some final note. the best place for technologists to read, write and publish Revenue Hacker Noon doubled revenue for the 3rd year in a row in 2020. This is a YoY increase of +102%. This represents QoQ growth of + 21% and YoY growth of + 68%.  Our top sources of income were , , , , , and . Top Navigation Grants Brand-as-Author Noonies Newsletters Ad By Tag launched in Q4 2020 and is now booked through Q2 21. It allows sponsors to place their content next to content with relevant tags. Ad by tag niche marketing We started 2021 with revenue in January being at an all-time high for revenue generated in a single month for HackerNoon. To put it into perspective, the revenue we made in January was almost as much as the revenue we made in 2017 and 2018 - combined. , and more. These things can ebb and flow, i.e. when deals close and how buyer optimism flows, but it’s good to see quality companies committing to working with Hacker Noon early in the year. Much thanks to some of our new and bigger customers: Wix, LaunchDarkly, CouchbaseDB, ByBit, DeFi For You, Udacity, Particle, CryptoAdventure, Flatfile As top navigation and niche marketing inventory fills up, we’ll be putting more effort in publishing and republishing the corporate blog posts, press releases, and software updates of as many technology companies as possible. Later in the year, we’ll go to market with a subscription offering for brand publishing, dubbed Evergreen. Growth Our top source of traffic, Google, finally started to rise again, after a steady half-year decline that also hurt the company’s Alexa site ranking. This recent rise in Google traffic could be due to their algorithm changes, our infrastructure work, our edtiorial rules, our content quality, just dumb luck, or an executive at Google changing their mind about our site. We did see an 18% growth in unique pageviews from December to November and a 19% growth from Dec 2020 to January 2021. Editorial team highlights: Part-time editors hired: We added to our team in 2020, and they bring extensive knowledge and experience in the AI, machine learning, cyber security, gaming, and non-profit sectors. Going to expand into more junior editors to improve our rate of publishing. three new editors Writing competition: Starting with Wix, brands will be able to reward the best performing stories on a Hacker Noon tag. This is exciting because it drives all our key metrics: words written, time reading and money made Managed Accounts: We’ve experimented with commissioned content to increase yearly revenue from affiliate marketing sources and the early results are promising. We will be working with experts in the space to scale up these efforts this year. Additionally, while we aspire to offer visitors a sh*t ton of value without logging in, deeper relationships with our visitors is something we want to aspire toward. We are seeing a trend of more logged in users on , after introducing a new signup flow. app.hackernoon.com Product In Q4, we conducted two separate audits to review the process of software development projects. Vercel, the creators of NextJS, audited our codebase to help prioritize which projects and approaches will improve page performance. Additionally, we audited the quality of every function marked as shipped in 2020. The overall learning was that we should ship less new things and spend a higher percentage of our time improving what we do offer.  Nevertheless, we did still ship some new things, and have some cool things coming up this year. Content Management System projects shipped in 2020 Q4: - This new flow has been performing much better. It includes login/signup in addition to email, and educates the visitor on our value to readers, writers, and brands. New Signup Flow - The first are live and dynamically updating the curated story lists via Hacker Noon mentions and the Bing News API. Tech Company News Pages 913 pages - This is in partnership with Superpeer. We are exploring more ways to revenue share with the contributing writer. Paid Expert Video Calls Beta

State of the Noonion Q4: Ad By Tag, Tech Company News Pages, and More

This is the third article documenting what I’ve learned from a series of . 10 Trailhead Live video sessions on Modern App Development on Salesforce and Heroku In these articles I’m walking you through how to combine Salesforce with to build an “eCars” app—a sales and service application for a fictitious electric car company (“Pulsar”) that allows users to customize and buy cars, service techs to view live diagnostic info from the car, and more. In case you missed my first article, you can find the link to it below and start from the beginning. Otherwise, if you’re specifically looking for data modeling, you’re in the right place. Heroku (the first article and the ) can be read here second article here Just as a quick reminder: I’ve been following this Trailhead Live video series to brush up and stay current on the latest app development trends on these platforms that are key for my career and business. I’ll be sharing each step for building the app, what I’ve learned, and my thoughts from each session. These series reviews are both for my own edification as well as for others who might benefit from this content. The Trailhead Live sessions and schedule can be found here: https://trailhead.salesforce.com/live The Trailhead Live sessions I’m writing about can also be found at the links below: https://trailhead.salesforce.com/live/videos/a2r3k000001n2Jj/modern-app-development-on-salesforce https://www.youtube.com/playlist?list=PLgIMQe2PKPSK7myo5smEv2ZtHbnn7HyHI On the last episode… Last session, we walked through Heroku buildpacks and learned how to start building on Heroku with a particular web language. The buildpacks were nicely structured, with intuitive instructions and installation steps for getting Heroku and your local machine set up to develop on one of the supported languages. We also ran through the process of using the command line, VS Code, and specific Heroku git commands to deploy a sample Lightning Web Components (LWC) app to a new Heroku app environment. Not bad for under an hour’s work! In this episode, we’re going to get into the bones of any modern application: the data layer. One of the things I find funny about data modeling is that almost everybody does it in their regular work, but rarely do people think too deeply about it (unless they’re developers or DBAs). There’s also a really steep learning curve when you graduate from a flat file like Excel to a multi-table relational database like Access. Every client I talk to can describe, to a degree, what kind of data their business needs to track. But going beyond that—initiating the right conversations with clients, reading between the lines, and teasing out the best data model of their application—is somewhat of an art form. Data modeling also seems to be a bit of an underappreciated part of the application-building process. Speaking from experience, getting the data model right or wrong for an application can mean the difference between a smoothly performing app that generates useful, insightful data and an app that gets “spaghettified” with workarounds and code to make up for the data modeling missteps made at the outset. A lot harder when you’re already a few million rows down the data rabbit hole! Data Modeling in Salesforce Using Point-and-Click Methods As we dive into our data modeling for the eCars app, there are two things worth noting about Salesforce’s data modeling features: Standard Objects establish a data model right out of the gate for Salesforce applications. Custom Objects and Custom Fields provide the ability to rapidly extend and prototype additional data requirements for any application. They also touch on Big Objects and External Objects, but in my experience, these rarely come into play except for enterprise-level implementations that have large data sets and external data sources. For the most part, Standard and Custom Objects covers 80% or more of the use cases in my experience. One major Salesforce data-modeling feature that requires experience and attentiveness is its attached dependencies and automatically generated metadata components. Many “freebies” and metadata dependencies come attached (i.e. conveniently created for you) to the custom objects, fields, and relationships that one creates in the system. For example, when you create custom objects and fields, the system already allows you to set and handle some things conveniently like creating a tab, default list views, page layouts, reports, and the security model down to the field level. Creating a relationship field, such as a lookup or master-detail relationship, also conveniently renders the child object (i.e. the table on the “many” side of a one-to-many relationship) as a related list on the parent object’s record page so that navigation is easy and intuitive.  Some key things to know about the two different relationship fields are below and also demonstrates the kinds of “freebies” or metadata dependencies that are generated with each type (rollups, sharing rules, etc.). This all results in the ability to immediately interact with and prove out your data model in your app and make tweaks and adjustments quickly. I can’t count how many times this has come in handy when working with clients on the initial vision of their apps. Instead of having to create ERDs and work up the data model before putting pencil to paper, we can just go directly to prototyping in real-time. Plus, the Schema Builder conveniently draws out the ERD as the data model is built out. You’ll also see in the video that it’s possible to build out the data model visually directly on the Schema Builder! Pretty nice when the data model comes with these freebies as it’s being built out Building the eCars Data Model By using the point-and-click data modeling features of Salesforce, we should be able to build out the entire eCars data model in under 20 minutes. The screengrab from the Salesforce Schema Builder tool below shows us the different tables/objects that are part of the app, how they’re related to each other (they’re all Lookup relationships here, no Master-Detail relationships so far) and which fields are required or automatically generated upon record creation. A written interpretation of the data model’s relationships could be as follows: A car can have zero, one, or more Configurations. Each configuration can have zero, one, or more Car Options. A Vehicle Order can act as a parent and/or child for a Configuration and vice versa. A Lead can be related to zero, one, or more Configurations. (The data model implies that a Lead can be interested in multiple Configurations.) Understanding what limitations and implications a data model conveys is one of the key skills in app development I’ve grown to appreciate. Heroku Data Services Data modeling on Heroku, on the other hand, follows some of the more traditional tools you might expect with SQL databases. Still, there are several GUI tools to help as well, which I’ll mention later. Heroku also features several important add-ons for scalability. The four major data services we’ll focus on for the eCars app are: Heroku Postgres (primary relational database) Heroku Redis (in-memory key-value data store) Apache Kafka (distributed streaming platform) (Connect Salesforce Data and Heroku Postgres) Heroku Connect Let’s look at each of those to see how we’ll use them. For Heroku, while other 3rd-party databases are available in the Elements marketplace, the officially supported database of choice is and is appropriately the most popular Heroku add-on. Heroku, like Salesforce, allows you to create your data model on Postgres using both command-line methods such as in VS Code as well as through a UI via the Heroku application manager. Heroku Postgres I’ve also used pgAdmin to manage Postgres databases as well and have used it in conjunction with Heroku Postgres databases. I don’t know any people who purely live in the command-line when it comes to databases, but I’m willing to bet they’re pretty fast on the keys! For the eCars app, we’ll use Postgres to store our basic relational data—customer information, car information, etc. This is actually my first exposure to Redis and Kafka. However, after Julián explained the use cases for each, I can see how these are very important data services for highly scalable applications. Scalability seems to be more and more of a recurring theme for Heroku also. Modern applications have to run lightning fast as consumers have come to expect that kind of performance. And even though solid-state drives might improve performance, the reality is that on-disk databases are just not going to be as fast as in-memory data. So, to get high-volume apps running super fast, you can see why you need something like or Salesforce Platform Cache to store some of the higher-volume information that powers the app in fast, high-performance memory. Otherwise, the database would buckle under the number of requests. Heroku Redis On top of that, modern applications typically run on distributed cloud infrastructure and may be using multiple data sources which need a lot of real-time throughput. In front of the data sources are a collection of services, and we need a way to reliably and quickly pass messages among them. fits the bill for this requirement, especially in large enterprise applications. is an event-streaming platform that will allow our eCars app to potentially scale to very high loads in a distributed app architecture. It also makes sense that there isn’t a free tier for this service, as I can imagine the expense it takes just to run one Kafka cluster. Apache Kafka on Heroku Apache Kafka Finally, allows Salesforce data to connect with Heroku Postgres in a bi-directional manner with a quick set-up. This saves us from having to do a custom integration between Salesforce and Heroku Postgres, but usually the deciding factor is the cost. Using Heroku Connect will allow us to seamlessly and easily connect Salesforce data to Heroku Postgres in a bi-directional fashion in near real-time. Heroku Connect One thing I’d like to clarify (since it used to confuse me) is the difference between Heroku Connect and . While Heroku Connect works with Salesforce standard and custom objects, Heroku External Objects is strictly a uni-directional OData service, meant to allow Heroku Postgres tables to sync to Salesforce External Objects. The naming can get confusing, so it’s important to keep this nuance in mind. Heroku External Objects also comes with the Heroku Postgres add-on while Heroku Connect is a paid service. Heroku External Objects Concluding Thoughts That’s an overview of data modeling on both Salesforce and Heroku, and how we’ll get them to work together. As I mentioned previously, I’m of the opinion that getting the data model right is probably one of the most important things when it comes to designing applications. If it goes wrong, everything can turn into a futile exercise, from the way data is collected to what the business intelligence/analytics tools produce. I got introduced to this mode of thinking pretty early on, before I even started my career. A USC Marshall Business School instructor named taught a database class to business students like me. I still credit that class, its data-modeling lessons, and the bit of SQL it taught us, with my eventual pursuit of Salesforce as my career. Douglas Shook We did countless homework problems and quizzes where he would give the class a bunch of receipts and a report from a shoe store or something, and then ask us to draw an ERD for a possible database system that generated them. When I reflect on that experience, I definitely have some renewed appreciation for how Salesforce and Heroku make it simple to handle the data aspects of an app now. If you want to take a deeper dive into data modeling considerations, I recommend taking a look at this article that Mohith wrote on . Data Model Design Considerations When Building Salesforce Apps In the next article, we’re going to look at how we can customize the front-end UI components of Salesforce for our eCars app, all without writing any code. If you haven’t already joined the official Chatter group for this series, I certainly recommend you do so. That way, you can get the full value of the experience and also pose questions and start discussions with the group. Oftentimes, there are valuable discussions and additional references available there such as the slides from the presentation. https://sforce.co/34Ld7xz About me:  I’m an 11x certified Salesforce professional who’s been running my own Salesforce consultancy for several years. If you’re curious about my backstory on accidentally turning into a developer and even competing on stage on a quiz show at one of the Salesforce conventions, you can read I wrote for the Salesforce blog a few years ago. this article Also published at https://dev.to/jasonomnivo/data-modeling-in-salesforce-and-heroku-data-services-569e

Data Modeling in Salesforce and Heroku Data Services 

Automated Data Replication From AWS S3 To Microsoft Azure Storage Made Easy

Designing a data pipeline comes with its own set of problems. Take lambda architecture for example. In the batch layer, if data somewhere in the past is incorrect, you’d have to run the computation function on the whole (possibly terabytes large) dataset, the result of which would be absorbed in serving layer and are reflected. In the speed layer alone, we have the problem of data latency, data lag, data correctness, data completeness, and out of order data. Data might come later than expected, there might be an unusual delay from one stage of pipeline to the other stage of the pipeline ( ), it might be incomplete, some data might have gone missing or lost somewhere in the pipeline ( ). The data might have been corrupted at the source or at a particular place in the pipeline where a computation function is applied to it ( ), or data might not come in order, some packets arriving later than usual ( ). lag completeness correctness out of order The purpose of this post is to illustrate how to deal with data completeness, data lag and out of order data using a dictionary with expiring keys with a TTL attached to them, called The github repo can be found . dictttl. here Problem Statement The problem statement at hand is to make sure the data quality in the speed layer is good. This is done by tackling data lag, correctness, latency, completeness and out of order data. In this post i will be working with data completeness, lag and out of order data problems. Definition Data lag is defined as the time taken by data to go from one stage in the pipeline to another stage of the pipeline, not to be confused with latency, which means time taken by packet to go from source (for example a sensor) to the server. Data lag: If we have received all the chunks of data for a particular key (or sensor) in a given time frame, we say data is complete. Data Completeness: If the ordering of data is not strict, with packets having earlier timestamp arriving after packet with later timestamps, then it is an out of order problem. Out of order data: Sometimes data lag is too high, either because of plumbing of the pipeline, or simply because sometimes we have too much data in a particular window for a key, so running the computation function on it takes longer than usual. At times, computation functions drop some data, or all the packets are not picked up at the source, which causes data completeness problem. And finally, streaming tools such as kafka or sensors do not always send data in order. It is important to handle those scenarios as well. Approach The consequence of: lag is that the data packets arrive late, completeness problem is that all the data packets do not arrive, and out of order problem is that some packets arrive later than the others. Now, my current, really simplified version of data pipeline looks like this Streaming Tool > Big data tool > Storage tool For example, data would be arriving at kafka from a sensor, from there it will go to spark for aggregation and from there it will be stored in MongoDB. I can put a check at the big data tool stage to make sure i handle the mentioned problems. To handle lag, I can wait for say, n units of time to make sure all of them arrive. If they do not arrive within that time, I can 1) choose to discard them entirely or 2) log them that they arrived late, process them and update serving layer with the new results. In this post, I will choose to discard them. Also, these packets are called straggler packets . To handle the out of order problem, I can 1) choose to discard them entirely, or 2) take them in to account in any next window that they come in, process them and update my serving view. In this post, I will choose to discard them. To handle data completeness problem, I can use a metric based on the value of the sensor. For example, I can say that the sum of all the values for a sensor in a particular window should be at least m. If it is less than that, I would discard it. With this logic in mind, let’s design the simplified version of the pipeline. Design To design the pipeline, I will be using Kafka and Pandas. Kafka for streaming the data, and pandas for aggregating and quality checks. I have chosen not to store them in any database or flat file. To handle out of order and lag situations, I will wait for 5 seconds till all the data from the sensor arrives. If it does not, I will discard it and move on. To handle completeness problem, I will sum all the values for a sensor and say that it should be at least 7000. If it is less than that, I will discard that row (and use batch layer to compute the speed view). I will send data from a producer to a consumer, producer will send data regularly, except at two intervals. At one point it will send a custom value of 1 to demonstrate completeness problem, and at another point it will sleep for 7 seconds to demonstrate out of order and lag problem. The consumer will consume the data, by creating a tumbling window. I create a window of 5 seconds that will tumble from one to the another, and within a window I will collect all data points for a sensor, aggregate them (mean), and print them. I use a python library called dictttl for the purpose of handling lag and out of order data problems. In a nutshell, this is a dictionary that has keys that expire after a manually set time. Producer To generate data, I have used the following code, it can also be found at . github It is basically a command line program that taken in an input sensor and range(value_start, value_end), and it generates data based on those parameters, and sends to the topic. Also, sensor id is the same, and I am using defaultDict(list) variant of the dictttl, so the values would be appended to a list for a particular window, like this: sensor_1, { : (timestamp, [ data ])} e,g
{‘sensor_1’ : ( , [ , , , , , , , , , , , , , ]} key 16000000000 1 2 4 4 2 4 5 5 6 3 1 2 3 4 The lines: Makes sure that once a reaches 10000, it sends data value of 1, so that the consumer can check the sum of elements and discard it. It will be shown later. The lines: Makes sure that once a reaches 20000, it sends data but then halts for 7 seconds. This would be useful later. Consumer The consumer consumes the data and checks if the timestamp of the data is less than the starting point of the window. If it is less, it discards it, partially solving the out of order and late data problem (which could have arisen from the packets belonging to data from previous window). If it is in the window, it appends it to the expiring dict, which has a time to live of 5 seconds. The lines: Checks if the data is out of the window. First, it checks if the expiring dictionary is empty. If it is, it creates the next window and continues. The dictionary could be empty because it might have expired, as it happens because at one point in time the producer halted for 7 seconds, and the threshold for discarding data was 5 seconds. If it is not empty, it creates a data frame from the dictionary, and the following lines will check if the sum of all the values for the sensor are more than 7000. If not, i will print “drop the row”. The following line: Will make an aggregation and prints it. Result The result looks like the following: Note that the first 3 timestamps are sequential in order. At the fourth entry, length of dict is 0 because the dictionary expired, and the next timestamp has a difference of 5 seconds as compared to the previous timestamp. Also, the sum of values is less than 7000 in the fourth entry so it is dropped. Next three rows are again sequential. Summary In this post, I went through what kind of problems arise when dealing with streaming data, and how to tackle them. I used a library called dictttl which has expiring keys, and in those situations where we choose to discard data they are really useful, as we don’t have to manually search and delete those items. As seen above, we handled out of order and lag problem by creating a buffer to absorb data within a threshold and if any data arrives after the threshold, it is discarded. Again, this is a very simple example, and it assumes mostly ordered data. If a packet arrives ahead of time it will simply create the new window. But in cases where out of order problem is high, we can create a buffer for that too, we can say after 1% of out of order cases we will discard the values. We also saw how to handle data completeness problem, by taking sum of the values. If the sum is less than the threshold, we simply delete those items. I hope you enjoyed my post.

Data Pipelines and Expiring Dictionaries

When first announced a couple of years ago that they were releasing an Open Source Business Intelligence (BI) tool, it really caught my interest and I reached out to founder Nikhilesh Tiwari to find out more about what he was doing. I spent a little time with the product and really liked where it was going and was determined to do more of a deep dive in the future, and with their release of , that time is now. Helical Insight version 3.0 (HI) is a Java application, so the installation was pretty straight forward, download the JAR file, double click and run the installer and take the default settings. It’s worth noting that you can use the application either on-premise or in a hosted cloud environment, I didn’t check if you can use it on AWS, but I imagine you could. Helical Insight Now we’re installed and we have a login page with extensive support for Single Sign On (SSO) as part of the role management. Since HI is a browser-based application, the developers planned ahead to make sure that it can be white labeled and embedded into your own systems. This embedding extends to the designer and report creation interfaces as well, not just the report delivery. What I really liked out of the box was the variety of data sources and flat file options that were available to connect to, however, I didn’t see a method to connect to an API. In order to connect with certain data sources like , have to jump through some hoops and using something like which is some additional work and would have been nice to have available by default. That said, HI does provide tight integration with the JDBC4 driver which gets you connected directly to pretty much any database. ElasticSearch Dremio Once the data source connections are defined, then you need to create the metadata that describes all the joining conditions and relationships for the data, the metadata also allows you to embed SQL queries, a powerful feature. The metadata definitions allow for granular level security definitions, down to the table level/column level and actual data/row-level for user roles profiles and organizations. The security layer also allows you to write to create very complex security conditions as well. One lack I saw with the metadata definitions is the lack of ability to create any cube or hierarchies or be able to define data types, this would be a desirable feature and make reporting easier in a variety of places. Groovy security code Reports are generated from the created metadata, the ad-hoc reporting interface supports a wide variety of different charts which will fulfill most needs, such as table, pivot, maps, grouped charts, card, axis charts, pie, donut, sunburst, even treemaps. Certain charts will only work when a certain number of dimensions and measures are available on the canvas and if those conditions are not met those charts appear blank. The ability to disable a blank shart would be a good feature. While the charts have a significant number of customization scripts available and the “custom” column allows you to put your own SQL/match calculations right there, a UI interface for putting specific types of calculations would have been helpful for your typical business user. That said, HI has a really cool feature to allow you to add any kind of Javascript based chart you want to within the applications. The charts have placeholders to add CSS, HTML and Javascript (prefetch, post fetch, pre-execution scripts) which adds a lot of dynamicity and flexibility to what you are building. Similarly, HI also allows adding more database functions, aggregate functions, charts customization within the applications without any vendor dependence. There are other functions available, such as filters and a filter expression language to build advanced and complex filtering options. In design mode, the dashboard canvas allows you to drag-drop previously created reports and create a dashboard where you can add images and text components. You can right-click on any component, including a dashboard, report, images, text, input parameter or group, and you will see a lot of customization options that include background color, border, shadow, header, images as background, etc. Alignment options allow pixel perfect position and control of the components. Sticking with their overall philosophy of flexibility, HI allows you to add HTML, CSS, Javascript to each and every component via the right click option on that specific component. The dashboard canvas makes use of the concept of grouping, which allows you to logically link items and then use them as overlays, and within that overlay, you can define what is in the front and what is pushed to the back. Your typical BI product often only allows a dashboard to be created in a grid mode, which can be limiting, while the HI approach provides a high level of flexibility to create attractive dashboards. The current selection of input parameter fields are a slider, date range picker, drop down and searchable drop down. I’d like to see a few more added, such as a radio button as well as the option to dock or hide the input controls as they consume a fair amount of display space. I will say that implementing inter-panel communication is quite straight forward though. The API support in HI is pretty impressive with over 400 API presents. Any company looking for the ability to extensively customize for a white label, these APIs are a huge incentive as the company claims that every piece of the frontend functionality has an associated API, but I was only able to test a few, but they all worked nicely. Looking at the data access layer of HI, I see that they are hitting the DB directly using standard SQL queries, which means any query that generates a large result set is going to take a while. This can be mitigated if you are doing any prep on your data for BI work, which is often the case for larger companies. If HI were to make more direct use of in-memory storage and processing, it would see some dramatic speed improvements, but you also run into problems potentially where you are blowing off the amount of available RAM. I’m not sure how they solve this in the short term, but that would be a game changer for them. There is a tagline on the HI website that says “Instant BI”, but I couldn’t find anywhere on the site or the product roadmap where that is defined, so it’s a bit of a mystery. Pricing is also not mentioned on the website, but I asked about it and it was crazy cheap, basically, you are buying help/support and/or a hosted solution since you could use the Apache licensed “Community Edition” for free and do whatever you want that is in compliance with the license. Helical Insight CE I gotta say, this is a pretty nifty effort, it covers a lot of bases for a lot of people and is preferable to some products I’ve seen in the BI space. I’ve noted some things that I think could be done to the product or possibly be done better, but this is a really solid tool and worth looking at to see if it satisfies your needs before you jump into the deep end of the pool with a commercial product.

Deep Dive Into Open Source BI Tool Helical Insight

Big data is a big problem, at least getting anything useful out of it. there is about three quintillion (the next step up is sextillion or one zettabyte) bytes of data created and only about 20% of it is structured and available to easily process. Nearly all useful processing that is done relies on a philosophy that is little changed from the green bar reports we were generating during the night shift and handing out up till the turn of the century. The whole map/reduce process is overnight batch processing, you aren’t working on live data, you are working on a snapshot, which might be fine for some companies, but for others, they need to be able to make decisions on high-velocity inbound data in near/real time. Every day Your typical relational database uses a b-tree index and these are generally good for retrieving smaller subsets of data from large sets of data. They are notoriously slow when adding data as the indexes get populated, although fractal indexes like make the put process faster. This brings me to bitmap indexes, which are a radical departure from the type of indexing you are used to. They are incredibly fast and performing searches that have to combine queries using multiple columns that have low cardinality such as months of the year or shoe size as opposed to something like the distance you live from the North Pole, which has a massive number of variations. The issue with bitmap indexes has historically been that they haven’t scaled well, which brings me to . TokuDB Pilosa Pilosa is an open source bitmap indexing that is now maintained by , the same people that originally developed it while at and have gone through some organizational changes. Inspired by a conversation at a conference where the topic of bitmap indexes came up, there was a realization that this could solve a lot of their problems with massive amounts of high-velocity data that needed to be indexed and analyzed. The group spent the next several years building and perfecting their technology which resulted in 8 patents thus far (yes, it is still open source though). project Molecula Umbel Cassandra Using a data repository, cloud, and operating system agnostic approach, Pilosa acts as an additional indexing system that can be applied to an existing repository, including a flat file. Looking at the example below, if you imagine that the column headers represent Customer_ID, and the rows in Field A indicate product inventory, then each 1 represents the products purchased by each Customer_ID.. You will also notice that this is separated into shards, this is how Pilosa scales horizontally so effectively and was one of the huge technical hurdles that they solved with the project. These bitmaps are very small and are compressed in addition to that and do not need to be decompressed to read. Pilosa’s software can store a knowledge representation of underlying data in memory, making it orders of magnitude smaller and unimaginably fast. Keeping that agnostic approach in mind, you can actually have Pilosa index across data repositories to tie together disparate data sources and allow you to execute those very fast queries against all of them at once. The implications of that are really staggering when you think about it for a moment, with a performance that claims that a simple query can traverse more than 2 billion edges in one second on commodity cloud hardware, there is no reason you can’t do your critical analytics in real time. (This graph is from 2017 and there have been significant improvements since then) Some really clever applications of this technology is the ability to create a virtual data source from your underlying data and share it with a client or partner by giving them access to an API and now they can return index data, something like all XL t-shirts sold in August in Los Angeles but without giving them access to the underlying data. The implications here are tremendous, think about being able to aggregate out data from a lot of different companies in the same space into an anonymized repository that you could report on? This is another unique and powerful feature that I don’t recall seeing anywhere, Molecula refers to it as being the VMWare of data. Now, I will say that their documentation could use some beefing up, but you can get a learn about their schema language and their query language ( ) is not anything like you are used to, simple but different, and therein lies the challenge, (re)writing your code to take advantage of it. Pilosa isn’t terribly hard to install and setup, but you really want to think about what you are doing and plan it out so you are taking full advantage of it. The index rate is insanely fast, depending on the data density, you could do a billion records in a half hour because it isn’t making copies of the data. The issue is that you need to rearchitect your code to make use of it, they have available for Golang, Java, and Python. Having to redo your code isn’t necessarily a bad thing, but it is a time-consuming thing and a lot of people would love to see a magical drop-in replacement that makes all their code run faster. Molecula has set out to solve many of these challenges, with one focus being a plugin for Kafka so that you are only dealing with that pipe instead of a huge number of programs. here PQL client libraries This system is ripe for a partnership with a data visualization dashboard, it would make a great combination for a business to have a live dashboard into certain types of data that have high volatility. There is not really a point in using Pilosa until you are in the 4tb range at a minimum, so keep that in mind. If you want to learn more about bitmap indexes, I really enjoyed . this article

Pilosa: A Scalable High Performance Bitmap Database Index

Databases and Blockchains, The Difference Is In Their Purpose And Design

What is a graph database? Do you really need one, and if yes, how do you choose? That’s what it all comes down to. This month’s edition of the Year of the Graph newsletter is special. Apart from the usual hubbub, which is somewhat slower this time of year, this month the Year of the Graph offering expands to the Year of the Graph Report. Other than that, this edition features 10+1 items to make for the extended time period it covers, and features mostly educational resources on using Gremlin, Cosmos DB, DSE Graph, on graph data modeling, and combining graphs with machine learning. 1. What is a graph database? Do you really need one, and if yes, how do you choose? I’ve been often asked those questions, and they are not the type that can be answered easily. It takes both expertise and hard work to be able to do this. Good news, i’ve done this so you don’t have to. This work is more up to date and comprehensive than anything else out there, and at a fraction of the price. Special offer — the first 10 buyers get 33% off using code 10FIRST _What is a graph database? Do you really need one, and if yes, how do you choose? Do you really know the answers to…_yearofthegraph.xyz The Year of the Graph 2. All evidence points to the same direction — graph databases are poised for growth. Markets & Markets projects a Compound Annual Growth Rate (CAGR) of 24.0% to reach a market size of USD 2,409.1 million by 2023. _Northbrook, IL -- ( SBWIRE) -- 06/19/2018 -- Healthcare and life sciences vertical is expected to grow at the highest…_www.sbwire.com Graph Database Market Expected to Reach 2,409.1 Million USD by 2023 3. If you want to get an overview of graph databases before getting to read the Year of the Graph Report, you can start with David Bechberger’s presentation. For everything you can’t find the answer to there, well, you know what to do. 4. Care for some expert advice on graph data modeling? Ted Wilmes gives his Graph Tips and Tricks that go beyond DSE Graph. 5. Need some more tips and tricks, this time of using Gremlin? What Jayanta Mondal talks about applies beyond Cosmos DB as well. _Jayanta Mondal stops by to chat with Scott Hanselman about Gremlin, the traversal query language for Cosmos DB graph…_channel9.msdn.com Tips for using the Gremlin API with Azure Cosmos DB 6. How about time traveling with graphs? Again, the techniques Daniel Larkin-York elaborates can be applied beyond ArangoDB _Graph databases are often used to analyze relations within highly interconnected datasets. Social networks…_www.arangodb.com Time traveling with graph databases, such as NoSQL multi-model DB: ArangoDB 7. Wrapping up with tutorials, here is one on getting started with using GraphX on Spark to do graph analytics. One important point made here is that useful as it may be, this is NOT a graph database. _"Simplicity is prerequisite for reliability," by Edsger Dijkstra A graph is an apparent and easy-to-understand data…_dzone.com Distributed Graphs Processing With Spark GraphX - DZone Big Data 8. Let us not forget about query languages. With the call for a standardized query language for labeled property graphs still open in the community, TigerGraph’s Mingxi Wu weighs in with his opinion on what matters for graph query languages. _Just about every enterprise in the world makes use of a database, whether it's a parallel, distributed, flat file…_www.eweek.com What Enterprises Should Know About Selecting a Graph Query Language 9. DeepMind’s Marta Garnelo has been working on combining symbolic methods for AI with machine learning. Although the former have been overshadowed by the latter, Garnelo argues they should be able to complement each other, and sees a role for RDF in this. 10. People in Octavian have also been working on the intersection of graph and machine learning, and they just released a dataset that can be used for question answering. _We’re excited to announce the release of a new dataset. CLEVR graph aims to help further research into machine…_medium.com CLEVR graph: A dataset for graph based reasoning 11. Here is how the DIG team built their Domain Insight Graph to help counter trafficking using RDF: _Human traffickers don't play by the rules. They use elaborate encryption and often post sex ads once-changing phone…_publications.computer.org AI tool "DIG" exposes sex rings and decodes open and dark web domains. Would you like to receive the latest Year of the Graph Newsletter in your inbox each month? Easy — just signup below. Have some news you think should be featured in an upcoming newsletter? Easy too — drop me a line here . Post Tagged with , , , , , , , , , , , , , , , , , AI AllegroGraph AnzoGraph ArangoDB AWS CosmosDB DSE Graph GraphDB GraphX Gremlin Machine Learning MarkLogic Neo4j Neptune Report Spark Tigergraph Originally published at linkeddataorchestration.com on July 19, 2018.

The Year of the Graph Newsletter: July 2018

Prologue This is a Big data story told in a different style. Why? When we tell stories, we only connect things that are regular, familiar, and easier on our minds. If facts do not match, we overlook. What happens when we look at all possibilities and ask how decisions connect to each other? A great example of this is this excerpt from Tolestory’s War and Peace about Napoleon . The essay raises many questions. It is easy to ignore. If you did not, if you accept the cognitive dissonance, and if you think, you will understand Napoleon much better. Following is Big data looked through the same lens. I am an not opponent of Big data; rather I am a believer. Hope this will help you understand Big data deeper just like it did help me. Act 1: Google doesn’t like Databases All analytics were done with databases. All data were stored in databases. It was a well-known anti-pattern to store data worthy of a database in a flat file. Google did not like databases. Maybe they tried it, and it did not work. Maybe it did work, but the company that can make it work ask for too much money. Maybe they knew it does not work. End of the day, they used files. Being Google, they build a big ( distributed) file system. Of course, they wrote a new shiny distributed file system. Google wanted to query their big files. Many geniuses got together, and for once, came up with the most simple solution. Where the solution came from is not clear. Maybe they took the map and reduce operations from functional programming, throw away other operators. Then mash these two and create a new operator. Maybe they took MPI, threw away all 100+ dominant operators and kept two. Maybe one of genius dreamt it up. We do not know. They created the most straightforward solution to a complicated problem. This is unheard of and not worthy of such geniuses. They did another thing no one has done. They told everyone about it. While making much money, Google told everyone about part of the secret sauce. Why? We do not know. Maybe they were thinking about increasing human knowledge. Maybe they wanted others to think in the way they did so they can easily hire others. Maybe, they wanted the world to know that they do serious stuff. Maybe they were so far ahead, and sure others can’t catch up. Maybe they did know MapReduce does not work in the long run so want to send their competition in a wild goose chase. In this world of bluff and double-bluff, Who knows. Act 2: New Dreams However, all hell broke loose. People felt like when Prometheus brought fire from heaven. Well yes, we do not know how it felt then, but I am sure it was something like this. Few people got together and implemented it in opensource. Yahoo, the competitor to Google, help fund some of it, Hadoop is born. People did not have use cases like Google. They did not have data like Google. Most did not have data to fill even a MySQL database. Yet everybody loved MapReduce. They dreamed up about a lot of data and created Big Data.They dreamed how one can collect data about the world, make sense of it., and change the world. Then they counted words with it. Some dig in and found some data that few gigabytes big but others couldn’t even do that. So they dream of when they have a lot of data. Others figured scientists were handling big data for a long time calling it scientific computing. Everyone marveled at what scientists were doing all along. Now it is much easier to get research grants, so scientists did not mind either. Now we really have big data ( which we had all along). Act 3: I hate you .. sorry I love you SQL Since Google had beef with databases, someone figured the problem is SQL. They created a new kind of storage and called it NoSQL. Soon they figured they need a way to query there storage. Whatever they did, queries look like SQL. So they change the name to Not Only SQL. Mike Stonebraker, ten years before receiving his Turing award, spoke out. He told in his humble and spear-like prose that “Guys all you do is counting and grouping. SQL can do all this and more. Just make SQL work with your glorified big files”. Of course, nobody listened. Academics and Investors went crazy and threw their brains and money into Hadoop. Act 4: Continuing the legacy Soon comes Spark, which beat Hadoop 10–20X performance. Bye, bye Hadoop. Wait wait, what happens to all investor money to build Hadoop companies. They got together and huddled Spark, so tightly. Now it is hard to tell where Hadoop ends and Sparks starts. They explained both are MapReduce technologies ( although MapReduce is less than 1/10 of Spark does) and Spark is the future. Everyone is happy. Meanwhile, Google dropped Hadoop but did not bother to tell us. To be fair, they talked about all technologies they build instead but did not help put the 2+2 together. It is not like many were paying attention. Thanks to Spark, Machine Learning (ML) takes off big. Soon data science is born. Years of old ML research comes back, new things were found, improvements made, new techniques discovered or rediscovered. Soon it turns out Spark does not work that well with deep learning. Google and others had to create new techniques. This did not matter. Most data is small. So we can do the data science with R and Python in a single machine and be mysterious about how we can run it in scale. GAFA(Big four tech companies) kept running machine learning in large scale and told us about it. That is enough to keep mystery and aura going. Also, GAFA hiring everyone who can do machine learning also helped. Act 5: Show me the money All this is said and done; money is in the enterprise. They already had data warehouses and BI. Big data goes there and fold it in. Well, I did not say replace. Sometime BI and data warehouses were just folded in and counted as analytics. Sometimes, upgrading the current product took you from old technology to new technology. Sometimes old technology is replaced.Meanwhile, SQL and NoSQL databases are merging. NoSQL databases are supporting full SQL or coming close. SQL databases are supporting NoSQL features. Someone should have listened to Mike, but he has grown tired of saying “I told you so.” He does not say anything. Act 6: Big data has it all Now big data/ analytics/ AI has it all. Huge markets, use cases, customers, investors. All dreams have came through. All is not well. It is tough to find people who can build these systems. It is even harder to find architects who can think it through and make it usable. Almost no one thinks about usability yet, and we are just waking up to problems like bias. However, some systems are up, somebody must be making use of it, and somebody is getting some benefits. Who knows? Big companies that were renamed as Big data companies are not growing. All that promised growth must have gone to Blockchain. Open source companies are growing 50%, but they are too small. In current rate, they might catch up in about 10–20 years. Act 7: Onward Ho Everyone is busy. There are AI, and singularity is coming, and robots are coming. What will happen when they take over our jobs? Bots are in raging too. Most bots break within two sentences. Among the rest, most are not usable, and it is much easier to use a UI to get the same thing done. Some turn into a shame within hours. However, this is details, who has time to read the details. Who cares whether big data works or not? There might still be time to make it work. Maybe hard work is done in already. Programmers have been trained. The new generation is being thought. More and more is asking for analytics. We can get the end to end stories right, get usability right, and get the tools right. It can work. Keep up the good work; I will check back when I am bored with blockchain. Hope this was useful. If you enjoyed this post you might also like my other posts and . Mastering the 4 Balancing Acts in Microservices Architecture Introduction to Anomaly Detection

History of Big Data: A Technical Comedy

Last year has been a very exciting year for devs like us. A whole lot of promising up-and-coming have made their place in the . technologies spotlight Maybe even so much that it’s hard to keep up. To stay sane I guess we all have our favorites to focus on . My team and I are big fans of everything related to the , & . JAMstack headless CMSs static site generators So let’s keep moving this way and pushing new boundaries with some of the great tools we’ve explored in the last few months. In this post I’ll show you how to: Set up as a headless CMS. Grav CMS Install React-powered static site generator . Gatsby Create a source plugin to query the API with . GraphQL The result? A small demo shop powered by . Snipcart Sounds heavy? Bear with me, it’s gonna be fun. Let’s start by quickly introducing these tools. What is Grav CMS? Simply put, . It actually won the CMS Critic award for “Best open source CMS” in 2016. Grav is a modern open-source flat-file CMS In this case, I’ll use it solely for content management, decoupling the backend from the frontend, or, as we like to call it, as a . headless CMS Why bother doing so you may ask? Mainly because it’s a great way to mix the backend capacities of the CMS with some of the amazing frontend JS frameworks out there (React in our case). For more info, we’ve already covered all the benefits of working with headless CMS in another blog post . I wanted to try Grav this time because of its nice and extensible admin panel which makes it a great choice to manage a simple website. It’s also a very easy to install, easy to learn that is a great alternative to heavier systems like WordPress or Drupal. flat file CMS React static site with Gatsby.. and GraphQL After I’ve got my decoupled CMS running, I’ll go on strapping the static site generator on top of it. Gatsby Gatsby is the most popular static site generator powered by React. I’ve a little bit, but this time I’ll try to push it even further. already played with it Mostly because I think it’s awesome and deserving of the exploration. Here I want to use one of Gatsby’s great features which allows us to bring our own data from databases, SaaS services, APIs and, as in our current case, headless CMSs. This data is pushed directly into our pages using GraphQL. In our first blog post on we stated that GraphQL is probably more than just a trendy tool and that it’s here to stay. this new technology It really seems to be the case as it’s probably going . to get even bigger in 2018 It’s nice to see it getting implemented with great tools such as Gatsby and I’m really pumped to give it another try today. Okay, this is gonna be a long one, so without further ado, let’s get to work! Grav as headless CMS tutorial Grav installation Grav is very easy to install. I used to do so. Simply open a terminal and type this command: composer composer create-project getgrav/grav snipcart-grav-headless Then, we want to start a web server, I chose to go with the built-in PHP development web server. . Note that I am on Windows cd snipcart-grav-headless php -S localhost:8000 .\system\router.php This will start a web server on your machine with Grav running. You can open in a browser and you should see it in action. http://localhost:8000 Grav CMS as headless setup We now have a working Grav instance. The first thing we’ll do is to add their awesome admin panel. One thing I really like about Grav is that everything is a plugin, even their admin panel. ;) We’ll install it via their own package manager. php .\bin\gpm install admin Once the installation is completed, open your browser to . You will have a small form to fill to create your initial user, then you'll be logged in. http://localhost:8000/admin We’ll also add another plugin, this plugin will be responsible to serve the content we’ll add in Grav in JSON. We’ll then be able to use this data in our Gatsby application, later in the tutorial. To be honest, I don’t think Grav CMS is widely used this way yet. I’ve got a hard time finding a plugin that does the job and that had a bit of documentation. I started by looking at from the Grav team, but, it wasn't exporting all the data I needed. This plugin is really intended to generate a feed file for RSS readers and things like that. [feed](https://github.com/getgrav/grav-plugin-feed) The one I took is . Although it does not seem to be maintained anymore, it does the trick. It generates a JSON file containing everything I need to use Grav as a headless CMS. [page-as-data](https://github.com/btopro/page-as-data) Defining products in Grav The last step will be to define our product . A blueprint in Grav allows you to define the structure of the content. In our case, we’ll create one that will define what is a product with Snipcart required options. blueprint For this step, I prefer to open the folder in VS Code (or any other editor) and edit it manually. You’ll need to go to your themes folder. In my case, I kept the default one as I won’t really use Grav to display anything. So I had to create the file in: . /users/themes/antimatter/blueprints This new blueprint will give us a new “content type” in Grav CMS, with some default fields that will keep product information within Grav. Adding content in Grav We now have everything needed to make Grav work as we expect. We’ll add some content. We’ll start by creating a new page which is going to be the parent of each product we’ll add. Log into the Grav admin panel, the URL should be: [http://localhost:8000/admin](http://localhost:8000/admin.) . Go to the section, and click on the button in the top right corner and create a new page: Pages Add We can keep the default Page template. As we won’t use Grav to render the content it doesn’t really matter. However, we’ll need to instruct the page about its content. Open the page in mode to get access to the and paste this: Expert frontmatter Once the page is created and configured, we’ll define the products. Click on the Add button again: Make sure to select the correct parent page and template. Once the page is generated, you’ll get a form, which uses the fields we defined in our blueprint earlier in the post. You can set the product attributes there and hit the Save button. . Make sure you are in Normal mode, not Expert Now that we have come content, we can open this URL in the browser: [http://localhost:8000/products?return-as=json](http://localhost:8000/products?return-as=json.) . The plugin will catch this request and return a JSON object containing our data. This is exactly what we're going to use as the data source in our Gatsby project. page-as-data Gatsby setup using GraphQL schema Installing Gatsby Our first move is to install Gatsby. is well done and you should refer to it if you want more information. Their documentation Make sure you have the classic NodeJS stack installed and have access to and then open a command line. npm We’ll install the package, this package will expose a couple of commands related to Gatsby. gatsby-cli npm install --global gatsby-cli Then, we’ll scaffold our new project: gatsby new gatsby-site This will clone the necessary files and install the default starter. Creating a source plugin to query our API In order to get access to our data in Gatsby, we’ll need to craft a source plugin. The plugin allows to “source” data into nodes, a concept built into Gatsby. These nodes will generate a GraphQL schema that will be able to query in our components. I have to say, this is a very nice approach. I love the fact that I can use GraphQL in Gatsby in such a fun and simple way. Start by creating a folder at the root of your workspace. You could also work with a standalone NPM package if you'd prefer, but I decided to use the simplest way for this example. plugins My tree looks like this (note that I ran in the folder to create the file): npm init package.json You can consider as the of the package. This is where the plugin code will start. gatsby-node.js index.js Our plugin will basically make an HTTP call to our endpoint we've built with Grav. Then, it will use the response payload and constructs the nodes that will be used by Gatsby to generate our GraphQL schema. /products We’re going to use two packages for this. The first one being , a very well known HTTP client in the Node community. [axios](https://github.com/axios/axios) Then, which will be very useful to generate nodes. A node in Gatsby contains, amongst other things, some internal data that must be hashed. Using the helpers will save us a lot of code. [gatsby-node-helpers](https://github.com/angeloashmore/gatsby-node-helpers) npm install --save axios gatsby-node-helpers Our Gatsby plugin must export a function named: , this function will call our API and create nodes. sourceNodes Add a file named in your project if you did not already, and add this code in it: gatsby-node.js I think the code is quite straightforward and simple so I won’t explain it step by step. Once it’s done, we’ll have to let Gatsby know about our plugin. Open file. This file should be in your workspace root. In the array, add . gatsby-config.js plugins 'grav-headless-plugin' My config file looks like this: You can then launch your Gatsby development server: gatsby develop --port 8001 Gatsby comes with a pre-configured . GraphiQL It’s a very clean GraphQL client that enables you to execute queries directly in your browser. Navigate to in your browser. http://localhost:8001/___graphql You can then run this query: This will return a JSON object: This data comes from Grav and is now available on your Gatsby site! Awesome isn’t it? Generating the site The first thing we’ll do is to add Snipcart dependencies. To do so you’ll need a Snipcart account (forever free in test mode). Open file and add these lines in the component: layouts/index.js <Helmet> This will add required files to your site and Snipcart will work when we’ll need it. :) As we want to create a static site, we’ll need to make a page per product. Luckily, Gatsby has it covered and exposes some methods we can use to do so. Basically, we’ll query all of our products via GraphQL, then we’ll generate a page dynamically for each product using a generic component. To do so, open file at the root of your workspace (not the one from the plugin we made). This is where we'll generate the pages. gatsby-node.js We want to run this GraphQL query and create a page for each result: In the file, we can export a function that will create these dynamic pages. The function is named . gatsby-node.js createPages exports.createPages = async ({graphql, boundActionCreators}) => { } Copy the two lines above. Notice that we inject that will be used to query our datastore and that will be used to build the page. graphql boundActionCreators We’ll first get our products: With these results, we’ll create the pages automatically. For this, we’ll need to require two dependencies, which comes with Node and to deal with Windows file system correctly. path slash You’ll have to install this one via npm. npm install slash --save-dev Include these two dependencies in the top of your file: const slash = require('slash') const path = require('path') Now we’ll use the method available in to generate the pages: createPage boundActionCreators As you can see, we get all the products, then for each of them we create a page. We pass the product ID to the context, which will be available in the component data. Product My whole file looks like this: gatsby-node.js Speaking of component, this is what we'll do next. This component will be used to render each product page. Product Create a file named in your folder. product.js src/components This file will contain the Snipcart buy button and general information about the product. I won’t dig too much into React and styling things, but here’s my component code: Note that we use GraphQL there too to fetch the product by its ID. The value comes from the value we set in the while we were in file. id context gatsby-node.js I used CSS modules to manage my components CSS. I suggest you read this excellent tutorial on the matter. We also imported the component. You'll want to use it to add link between pages generated by Gatsby. Link Once your site is loaded, React router will take care of the routing and it will create a very fluid experience. No full reloads, only speed! If you hit this page (make sure that you have your Gatsby dev server running with ): you should see an awesome product page. gatsby develop http://localhost:8001/products/hat-hat Notice the link? This is intended to redirect to our products page that does not exist yet. Back to products Let’s craft a page for it, in the folder, create a new file named : pages products.js As we did earlier, we again use GraphQL to query our products. Products will then be available in component object and we'll be able to render each of the products. data If you go to you should now have an awesome product listing page. :) http://localhost:8001/products Live demo & GitHub repo See the open source repo on GitHub . See live demo here ! Closing thoughts I had a blast doing this tutorial! I enjoyed the simplicity of Grav, it took me only a couple of minutes to get it running. The hardest part was to find the plugins I was going to use it as a headless CMS and to be honest, the plugin I used in the demo is kind of limited. I think Grav might be a good solution for a very basic scenario, but is not as advanced as other options we looked at in previous posts such as Directus or GraphCMS. That being said, I’m well aware that it is not the way most people use Grav. ;) For the frontend part with Gatsby, it’s been a real delight to use once again. This really is an awesome piece of software. The documentation is on point and it’s very well executed. The GraphQL implementation makes the whole experience even more special. I haven’t touched to React a lot, only with a while ago. Still, Gatsby was easy to learn and made me want to dig deeper into React. a few tutorials If you’ve enjoyed this post, please take a second to share it on Twitter . Got comments, questions? Hit the section below! I originally published this on and shared it in our . the Snipcart Blog newsletter

Build a Gatsby & GraphQL App on Top of a Headless CMS

After seeing numerous Dockerfiles, repositories, too many blog posts to count and a whole lot of medium, I decided that I wanted to have an open discussion on how folks manage their headless browser work. Especially now that there’s new libraries out there (headless Chrome, puppeteer and chromeless), I feel we’ve been given a second chance at these in a better way. GitHub using tools This is something I’ve personally spent many days and months working on at as I knew that getting it right was the best chance for success for both it and its users (especially folks just wanting so to speak). First and foremost I considered what I, as a consumer would want, but then story progressed from there to “how did we even end up here?” browserless a way to run headless “jobs” How we got here Not so long ago developers were left with few choices in terms of browser-based automation. PhantomJS, nightmare, and a handful of others ruled the realm and few questioned whether or not one of the big three (Google, Mozilla or Microsoft) would release a headless variant of their browsers. As great as those browsers are, there’s one thing most of them kind of suck at: working well on linux. This achilles heel is actually what PhantomJS and a few others got somewhat right: installing and working right away. As a matter of fact, you could pretty reliably install phantom and deploy an app in a matter of seconds with it. The issue was, and still is, that that’s what most folks did. actually It was this mentality and means of deployment that got phantom more flack than it deserved. In most application environments you wouldn’t deploy a database alongside your server’s app, nor would you interact with it directly. Instead you’d instead operate it separately (even if it is physically located on the same box) and talk with it over some socket connection. But more often than not developers didn’t do this. Instead they’d just invoke phantom, run some work, and call it a day. But what happens when your app starts getting hammered with requests? What happens when you want to massively parallelize work? Back to the Future I want to think back to the database as an example of how we should be doing distributed and parallelized browser work. Databases have all these cool things that come with them: connection pools, sockets, and are pretty easy to install and run. I’d say that, without much hesitation, that databases have a decent service layer around their functionality. That is: they have a strict protocol with how and when they can be interacted with. To think about it in a different way, imagine this: writing and saving data manually on each request to a flat file. This is essentially what we’re asking headless browsers to do by the nature of how we’re interacting with them. Of course they’re going to consume a ton of resources and be flaky. Of course they are going to crash and be generally unavailable. You’re explicitly running a massive binary alongside your core application, what do you expect? There’s a certain class problems that are going to be tough or impossible to solve any way you spin them. You can only do so much to limit the amount of resources Chrome will consume, and you can only parallelize in so much as you have resources available to do so. But you still use things like queues and load-balancing to make things more scalable. However, it’s hard to load-balance a binary when there’s no service boundary. can Where we should be tries to solve some of these problems mostly by treating a web browser like a database, more or less. In a database you can’t use fancy workflows like lambdas or functions-as-a-service otherwise your data would disappear after each invocation. Similarly, in a web browser, an invocation could last minutes or longer, which makes stateless lambdas tough. You also need a lot of assets for a browser to operate properly, which breaks a lot of limitations inside the serverless-type workflows. If you don’t believe me, go sort any headless library by date and witness folks fighting to get this working. I think it’s a novel way to do browser work, but it’s somewhat flawed fundamentally and fairly brittle. The fact is: if you want to remain flexible on how you interact with a browser, lambdas might be the right choice**.** Browserless not I’d like to suggest that, instead, you treat the browser like any other service with some small differences. Instead of relying on stateless HTTP, you’ll have to let WebSockets run the show (which is how remote Chrome library works at some level). This means that your headless service will need to proxy HTTP and WebSockets as well as babysit Chrome and concurrent connections. Sound like a pain? It kind of is. But you get pretty much everything you’d need as a consequence: every A clear path to scaling your app/tests. Easy to load balance and parallelize. Ability to change your headless jobs without deploying a new version. Since you are driving remotely you’ll hardly ever need to deploy. Sandboxing Chrome on its own infrastructure so it won’t bring everything down. Easily interchange running scripts locally versus a headless farm. In browserless this is a single line of code to toggle between. You also can build some really cool tooling alongside this. For instance, browserless comes bundled with a nifty debugger that allows you to test and see what the browser is up to during execution. The browserless debugger in action What are you doing with Chrome? I’m very curious to hear how you operate and run headless work in your application. Are you still deploying it alongside your app? Running it in lambdas? Not convinced on my position? I ask that you consider giving a look over as it solves a lot of problems I experienced personally. There’s also even specifically so you can hit the ground running on your next project. browserless hosted plans I’ve setup

Rethinking headless browser automation with Chrome

(for those linked in here from out of the blue, Floathub is sort of a one-way “Nest for Boats”; a hardware device that monitors a vessel’s systems and sends data up to an online interface ) As we make the final preparations for our , many of the folks we’ve press-ganged into acting as beta testers have been asking us about FloatHub’s history. Rather than dribbling out snippets of answers here and there, we’ve attempted to set down a relatively brief narrative of how we got here (in a technical sense). KickStarter launch The original impetus was a definite itch; after the umpteenth time arriving at our boat and discovering some minor disaster related to flat batteries, water over the floorboards, etc., etc, we thought, how hard could it be to hack a solution to this problem? . Just solder-up some sensors to an old 2g cell phone and push readings to a server somewhere. Flat file data storage, a few grep scripts, some log rotations, and you’re done. Shouldn’t take more than an afternoon. :-) There has to be an affordable way to monitor what’s happening on a boat when we’re not onboard Initially, we had no intention of trying to create a commercial product. We just wanted to solve our own problem. Maybe create a few extra copies for friends in the same situation Hardware Choices After the first few largely forgettable experimental failures, we made some basic hardware decisions. We wanted to go with a true microcontroller (as opposed to a more complicated Raspberry Pi or other higher level system). The main reason was power usage, since boats often have to make their own. But we were also attracted to a really _simple_ design goal for the device; ideally it would just sit on a boat for months and years, with no-one ever having to intervene and update it, install a new driver, and so on. To the extent that we wanted to do anything clever with the data, we were pretty sure the right idea was to do that out on the network, after it had already been pushed off the boat. Plus there is something hard to define but extremely satisfying about writing low-level microcontroller code; you are in charge of just about _everything_. Working just a compiler above bare wires with memory measured in kilobytes is such a refreshing change from most other kinds of software development. Since we wanted to talk to other onboard systems (depth sounders, wind speed, etc.) and since most historic data systems on boats use a 4800 baud protocol that is a very close cousin of RS-232/RS-422, we knew we’d want multiple serial ports. Accordingly, we settled on the for the prototyping platform (it has 4 serial ports). We also wanted to be able to directly measure analog voltages to be able to report on battery levels, charging systems, and pump voltages (to see when and for how long pumps were running). Again we stayed very simple, and started designing a custom Arduino shield that included straightforward composed of simple resistors. We also made sure our board design passed through all header connections so that we could stack more shields on top. Adding an embedded GPS was an absolute breeze (just supply power and make a serial connection to any of the plentiful GPS chipsets, e.g. the ubiquitous family). That gave us not only position, but date and time info as well. We also added a that measured temperature and barometric pressure. Arduino Mega voltage dividers Neo 6 Bosch chip So with an off-the-shelf Mega and a half-breadboard/half-shield prototype, we had a basic system that could monitor 9 voltages, talk to other onboard systems, know where it was in space & time, and take some ambient weather measurements . FloatHub Hardware v0.0.1 (~2012) In the first real working iteration of the hardware, we used a cellular shield that interacted with the Mega on serial port 2 and basically used a Hayes Modem command set. With that, we could periodically do an AT-CALL (which would get a GRPS-based TCP/IP socket connection to a host and port) and then AT-SEND to send a packet to the host. {HOME} {STRING OF DATA} So the next challenge was what should that actually be composed of? {STRING OF DATA} A Protocol On the server side, we could, for all intents and purposes, do anything we wanted in terms of data types, markup, handshaking, transmission frequency, security, encryption, and so on. The challenge was the device side, where our little microcontroller took a non-trivial number of clock cycles just to assemble characters into a string. Sending them took longer still. And if the connection was lost, we had to worry about somehow storing the data in a tiny amount of EEPROM and trying to keep phoning home in the meantime. As far as marking up the data, the most obvious choice was the NMEA protocol that many marine instruments already use, and essentially all GPS chipsets speak. This would have the advantage of making the device really, _really_ simple. It would just be a gateway that sent NMEA up over a cellular link. The main disadvantage is that NMEA is, in this context, idiotically verbose; a GPS chip just keeps spitting out location data on a NMEA link at 4800 baud even if it remains entirely motionless. And since NMEA is really a local, hard-wired serial protocol, it has absolutely no notion of identity or security. A second natural choice was a then emerging JSON-based standard for onboard systems called . This had several great advantages, not the least of which is that it is truly free and open. Unfortunately, while trying to mark up JSON in a minuscule memory footprint is just barely possible, it is quite difficult and very limiting in how many pieces of data you can handle (the right place for SignalK in the FloatHub world is as an output from the server, something we are currently actively working on). SignalK And so, of course, we rolled our own protocol. Details are , but the gist of it is that we needed two message types, and both of them had to be nice and compact. First, we needed a sample-in-time type of message that showed the current state of all the systems (battery voltages, water depth, etc.). We also needed a point-in-time message that could be sent when an important event had just occurred (e.g. a pump turned on at precisely 11:23:06 UTC). Then we had to have some convention to explain which device the message was coming from (i.e. a unique identifier). On top of all of that, we needed some form of encryption, as otherwise we would be passing plaintext data over TCP/IP (including location information). A full SSL-style stack of infrastructure would have overwhelmed our little Megas, but we did have enough spare program space to squeeze in an routine. available AES-CBC Iterating Hardware We then spent something close to _3 years_ of otherwise idle time in the evenings and on weekends iterating on the hardware. We’re still far from true circuit design ninjas, but gradually progressed from thru-hole, hand soldered components to mostly surface mount. All of the layout and design was done in . We tried a few different places for printed circuit board (PCB) prototype manufacturing, but eventually settled on as they could do both the PCB creation and in very small batches at reasonable rates. Eagle Seeed component assembly Hardware Rev P3.A3 (2013) While our custom shield progressed, we used a few different Arduino WiFi and Cellular shields on the top of the physical stack for communications. We needed an enclosure that could encompass this triple-stacked PCB sandwich, have a cutout where the screw terminals would allow hookups for voltage testing, mounting points for GPS, WiFi, and Cellular antennas, and so on. We did spend a fair bit of time hunting for a ready-made case we could use off the shelf, but never found one that really worked. So, just to ratchet up the time commitment to this “easy hack” solution, we started fiddling with CAD/CAM software. We knew we could 3-D print a case if we could manage to wrap our brains around modeling software. We started out with , which we absolutely loved. It’s really more of a programming language than a traditional GUI-based CAD/CAM tool, so we were able to keep thinking like software developers when using it. Data and directives are in the “source code” (e.g. draw a box , take away the volume of a cylinder ), which is then run through a “compiler” (i.e. OpenSCAD), to create “object” (i.e. files that can be sent to a 3-D printer). We acquired a and started churning out enclosure revisions. As our volume of prototype units went up, we initially printed them with , although recently have been doing more and more with . OpenSCAD here there Printrbot simple Shapeways Voodoo Manufacturing Hardware Rev P4/P5 (2014) At some point in 2015, we had really settled on the physical design and thought we were “done” from an enclosure standpoint. Still only half serious about ever moving forward with a real commercial product, we started reading about . We’d naively assumed that any file that could be 3-D printed would also be just fine for tooling molds. Couple of problems with that: 1) at the time, OpenSCAD was not great with the IGES or STEP files used by most tooling systems, and 2) a little something called “draft”. We designed our enclosure with simple right angles just about everywhere. Turns out that does not work very well for parts that will come out of a mold. You need angles that taper (at least a little) so the part can be popped out after it’s cooled. This notion of is probably taught in the first month of any CAD/CAM or engineering course, but it was news to us. Since we were already having format issues with OpenSCAD and we were going to have to redesign the whole enclosure anyway, we switched to for our CAD/CAM work. After more months of spare hours learning the tool and then using it to re-design the case, we finally had something that the automated design checker at said could be injected molded without too much difficulty. injection molding draft FreeCAD ProtoLabs A Server Process With the device electronics and physical specification coming along nicely, we turned back to the server side of our home grown data protocol. This was a nice project that had a very well defined scope: listen for an incoming connection receive a string of characters from the device, terminated by a newline acknowledge reception by sending a short response string (and generally closing the connection) given the unique identifier in the initial part of the string, attempt to decrypt it (AES-CBC) On successful decryption, parse out data elements and stuff them into permanent storage (a database) We basically used a Python TCPSocketServer with a threading mixin, running on some fairly small AWS EC2 instances. Each job is very discreet. When the string arrives, there’s a defined section near the beginning that includes a unique identifier. Looking this up in a server side database, we grab the relevant AES key for that “account” (the device’s unique identifier) and attempt decryption. If that succeeds, then we have plaintext data to parse out of the string and store in the relevant table(s) for this “account”. The only thing we did here which even verged on clever was to realize that the streaming data for each “account” could be stored in its own tablespace (no device data is directly reliant on any other device’s data). We could use a central database to look up “account” information and include in that a pointer to device-specific host/database/tables where message data could be stored. Although each transmission from the device is quite small, they can send up to 120 per hour, and over the days, weeks, and months, the set can grow to multiple GBs per device. Having the data sharded in this way makes it cleaner to work with. For example, deleting an “account” is just a simple table drop, not a careful select on some enormous über-table of data. And we can scale in a relatively straightforward way by just horizontally adding more database instances. The Amazing ESP8266 Somewhere in 2015, we noticed that we were starting to use the word “account” a lot, implying that we really had started to think of this as something we might try to sell commercially. The biggest remaining technical barrier was then device configuration. Up until that point, we had been making changes in device identifiers, AES keys, network settings, and other parameters largely by editing source code, recompiling, and then re-flashing the Mega. Perfectly adequate for proving out the core functionality, but far from viable as a “user experience”. We experimented a little bit with a very simple serial protocol where you could set parameters via a terminal program, but even that was not terribly user friendly. We also experimented with bluetooth and an app, where we could hide the serial line protocol behind a phone/tablet GUI. The ESP-8266 WiFi Microcontroller Around the same time, everyone and their uncle started writing about a new WiFi-centric microcontroller called the . This chip has a whole TCP/IP stack onboard, can do both access point and regular station mode (and even both at the same time), was quickly supported within the , and is ridiculously cheap (less than $10, even in a full breakout-board configuration). We started experimenting and realized we could add one to FloatHub and have it serve two roles: 1) replace shield-based WiFi solutions for sending date off the device, and 2) run a minuscule web-server right onboard that would let end users set configuration values. ESP8266 Arduino environment Compared to the modern web development world of complex application server stacks and API’s to API’s, writing a whole microcontroller-based http server is starkly clear cut. Much like our previous code for the Mega, you’re dealing with tiny executable sizes, and pretty severe memory limitations. For example, we needed cookies for our device logon, so we had to actually come up with a small-as-possible C struct to represent them. Graphic elements in the interface had to be byte-coded into the executable, as there is no traditional file system to serve them from. The ESP8266 is also _rock_ solid as an upstream data channel. As long as it can see it’s designated WiFi connection, it will stay connected, and the interactions over loosing a connection and then asking it to re-establish one are very straightforward and reliable. The fact that it can do Access Point and regular Station mode at the same time means FloatHub effectively has two WiFi interfaces; a “private” one with it’s own 192.168.4.* address space, and a “public” one where it connects with the outside world. Our only grumble is that since these two modes share the actual same physical hardware, they have to be at the same WiFi frequency. So if the upstream connection has to move to a new frequency to connect, it must drag along the private connection to that new frequency as well, effectively bumping off clients and forcing them to re-connect. But it’s a very small price to pay, and a joy to be able to fiddle with device settings with any browser that happens to be handy. Since we needed the ESP8266 on all FloatHub devices (even the cellular version) in order to do device configuration, we realized we should also add an option to broadcast NMEA over WiFi. That is, have a little server process in parallel with the http server that would push NMEA data to any connected client. This is really a NMEA to WiFi gateway, pushing any valid data generated internally by the onboard sensors or seen coming in from other devices (via serial). There are lots of marine applications for phones, tablets, and computers that can make a connect to these sorts of servers and then display vessel location, depth, wind speed, etc. Originally this was an almost accidental by-product of the fact that we needed some way to easily configure a FloatHub. Now it is easily the favorite feature among our beta testers. Online Site & Web Interface The last major piece of the, “is this actually something we could sell?” question was to make a with an interface to the data. We’re really more server side developers than front-end types, but where there’s a will, there’s a way. functioning website FloatHub’s Web Interface Our core framework is , which we run with . is the public facing web server (and handles https with letsencrypt certificates). Our Flask-based user accounts, e-mail based verification, etc. is all fairly vanilla stuff. We set up for payments and subscriptions. There is a main user account databases on an RDS instance, and then the flow of device data is pulled in according to the shard setup we mentioned earlier. Flask gunicorn Nginx Stripe The actual vessel monitoring interface is a simple “one page application”, in that it shows a bunch of data and updates automatically as new data becomes available. Nothing really novel in how we implement that; we use a JQuery approach to call the data API every 30 seconds and see if new data has arrived. If it has, we update the gauges and displays accordingly. You’d _never_ want to use it to actually navigate with, but for a quick summary of what’s up with your boat or for following along with a vessel on a voyage, it’s quite serviceable. getSON() For the gauges, we looked at a fair number of different packages. There are a _lot_ of them out there. We settled on by Mikhail Stadnyk, because it has wide browser support, is under active development, and works as advertised. We haven’t really done full justice to what’s possible with Canvas Gauges yet, and plan to add more detail and nicer polish over time. Canvas Gauges Charts from the Web Interface To display historical data (and update it live), we went with (based, in turn, on ). C3 makes it dead easy to create attractive, interactive charts. Because it’s D3-based, there’s a fairly rich set of code to draw on, including JSON methods where you can just ask for some data from the API and the code will automatically draw the relevant graph once the data comes back. There are some great examples of this and many other features of C3 in the . C3 D3 documentation Off to Launch That pretty much sums it up. Of course we’ve skipped over lots of detail, but we’ve covered the major items in getting from there to here. At the time of putting these thoughts and notes together, we have just a few days left before our . We’re optimistic, as we think FloatHub is a really useful product and service and our beta testing group has been giving reasonably positive feedback. But whether FloatHub enjoys successful adoption or sinks into insignificance, it seemed like the right time to try and get some of this history down. KickStarter launch From that one little itch, we got to spend time exploring circuit layout, microcontrollers, protocol design, WiFi chipsets, embedded servers, CAD/CAM modeling, injection molding (draft!), component sourcing, database design, server processes, Flask, front-end libraries, Javascript/JQuery, and responsive data interfaces. Please let us know your thoughts or comments here or via email at . info@floathub.com

From Itch to Launch

So you’ve been doing REST APIs for the past 5 years. Or perhaps you’ve been optimizing searches for your company’s gigantic database. Maybe writing the embedded software for a microwave oven? It’s been a while since you were rocking some Prototype.js to do proper OOP in the browser. And now you’ve decided it’s time to get up to speed with your frontend skills. You take a look at the landscape and it looks like . this Of course you’re not looking for Waldo. You’re looking for 25 random guys and don’t even know what their names are. This feeling of being overwhelmed is so common in the community that the term “JavaScript fatigue” actually exists. When you have time for some comedy about the subject, reflects the phenomenon brilliantly. JavaScript this post But you don’t have the time for that now. You’re in a giant maze, and you need a map. So I made a map. A little disclaimer first: This is a cheat-sheet to get you up and running fast without having to make too many decisions by yourself. Basically I will be laying out a set of tools that just work together for general-purpose frontend . This will get you comfortable with the environment and will save you a few headaches. Once you’re done with these topics, you’ll be confident enough to adjust the stack to your needs. development Structure of the map I will divide the map into problems that you’ll need to tackle. For each problem, I will: Describe the problem or the need for a tool Decide which tool you will use to solve the problem Explain why I chose that tool Give a few alternatives Package management Need to organize your project and your dependencies. Problem: NPM and Yarn Solution: NPM is pretty much the de-facto package manager. Yarn runs on top of NPM but optimizes dependency resolution and keeps a lock file of the exact version of your libraries (use it in tandem with NPM’s semantic versioning, they’re not exclusive, they complement each other). Reason: None that I know of. Alternatives: JavaScript flavor ECMAScript 5 (aka old-school JavaScript) sucks. Problem: ES6 Solution: It’s the future JavaScript but you can use it right now. Incorporates many useful features that have been available to other programming languages for a long time. Interesting new features: arrow functions, module import/export capabilities, de-structuring, template strings, let and const, generators, promises. If you’re a Python coder you’ll feel at home. Reason: TypeScript, CoffeeScript, PureScript, Elm Alternatives: Transpiling Many browsers that are still massively in use don’t implement ES6. You need a program that translates (transpiles) your modern ES6 into equivalent, well-supported ES5. Problem: babel Solution: Works perfectly and it’s pretty much the de-facto standard. Transpiles server-side. Reason: Traceur Alternatives: You will use babel-loader, a Webpack loader (more on that later on). You’ll need transpiling if you plan to use any of the other JavaScript flavors as well. Notes: Linting There’s a zillion ways of writing JavaScript and consistency is hard to achieve. Some bugs can be prevented with a linter. Problem: ESLint Solution: Great code insight and very configurable. The airbnb preset is all you need to get up and running. Really helps you get used to the new syntax. Reason: JSLint Alternatives: Bundling You are no longer using a flat file or sequence of files. Dependencies need to be resolved and loaded properly. Problem: Webpack Solution: Highly configurable. Can load all sorts of dependencies and assets. It’s pluggable. It’s pretty much the de-facto bundler for React projects. Reason: Browserify Alternatives: Can be a little hard to configure at first. Disadvantages: You’ll want to spend some time really understanding how this guy works. You should also learn about babel-loader, style-loader, css-loader, file-loader, url-loader. Notes: Testing Your app is fragile. It will fall apart. You need tests. Problem: Jest. Solution: Batteries included, , can detect tests affected by your changes and run only those, works nicely with monorepos, comes by default with CRA, backed by Facebook. Reason: snapshot testing Jasmine, Mocha, Tape. Alternatives: UI framework / state management This is one of the big ones. SPAs have grown more and more complex. Mutable state is particularly troublesome. Problem: React and Redux Solution: Mind-blowing paradigm shift, breaks a lot of dogmas as old as the web and does it amazingly. Better separation of concerns than traditional approach: instead of separating by technology (HTML/CSS/JavaScript) you break things up by their functionality (cohesive components). Your UI is a pure function of your state. Reasons for using React: If your app is non-trivial, you need a tool to manage the state (otherwise you’ll be doing gymnastics for your components to talk to each other, learn vanilla inter-component communication first to experience the limitations). Every tutorial on the web will walk you through the confusing, abstract Flux pattern and all implementations that there have ever been. Save yourself a decent amount of time and go straight to Redux. It implements the pattern in a very simple manner. Even Facebook uses this. Extra awesomeness: reload and keep application state, time travel, testability. Reasons for using Redux: Angular2, Vue.js. Alternatives: You might feel an urge to pry your eyes out with a rusty spoon the first time you see JSX code. Resist the temptation to find a forum and yell in outrage. This is just cognitive dissonance caused by years of indoctrination. Turns out mixing HTML, JavaScript and CSS in a single file is super awesome. Believe me! — Achievement unlocked for using two lame references in a single bullet. Warning: DOM manipulation and animations Guess what? You’ll still need occasional quickfixes where you’ll have to target selectors and perform operations directly on DOM nodes. Problem: Plain ES6 or jQuery. Solution: Yes, jQuery is still alive and well. React and jQuery aren’t mutually exclusive. Although, be aware that you should be able to do most of what you need with vanilla React (and ). Adding jQuery will also increase your bundle’s footprint slightly. I’d say that using jQuery on top of React is a smell and you should avoid it whenever possible. If you hit a certain corner case that you can’t figure out with just React + ES6 features, or if you’re dealing with some annoying cross-browser quirk, jQuery might save the day. Reason: querySelector Dojo (does that still even exist?). Alternatives: Styling Now that you have proper modules, you want them to be self-contained, reusable pieces of software that you can move around. Component styles should be as portable as the components themselves. Problem: CSS modules. Solution: As much as I love inline styles (and use them extensively), I must admit that they’re rather limited. Yes, it’s totally OK to use inline styles in React, but you can’t target pseudo-class selectors (like ) with them, which is a deal-breaker in many cases. Reason: :hover Inline styles. What I particularly like about inline styles in React is that they allow you to treat styles as regular JavaScript objects, which lets you process them programatically. Also, they live in the same file as your component, which makes them super easy to maintain. Some people still advocate for SASS/SCSS/Less. These languages imply an extra build step and aren’t as portable as CSS modules/inline styles but are as powerful as they’ve ever been. Alternatives: About Boilerplates Boilerplate projects like can lift some of the burden for the problems described above. When using a boilerplate, you still need to understand what’s going on under the hood — otherwise, you’ll never truly have ownership of your build. Create React App That’s it! You now have a metric shit-ton of stuff to study, but at least you won’t need to spend so much time doing research. Do you think I missed something? Did I drop the ball somewhere? Leave a comment or reach me on twitter . @bug_factory Corrected link to the post about JavaScript fatigue (thanks, !). Added notes about how other languages also need transpiling (thanks, !). Added Jest to the list of alternative testing tools (thanks, and !). Added section about styling (thanks, !). Thank you all for sharing and sending in your feedback! EDIT 1: Chris Woods Ignacio Rossi Tiago Pina Chris Khoo Tinkerer Improved section about jQuery (thanks, !) EDIT 2: Eldar Shamukhamedov Replaced Mocha with Jest in testing section. EDIT 3: Added note about boilerplates. EDIT 4: is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising & sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

A Map To Modern JavaScript Development (2017)

The Way We Live Now by Anthony Trollope is part of the HackerNoon Books Series. You can jump to any chapter in this book . WHERE "THE WILD ASSES QUENCH THEIR THIRST." here WHERE "THE WILD ASSES QUENCH THEIR THIRST." We must now go back a little in our story,—about three weeks,—in order that the reader may be told how affairs were progressing at the Beargarden. That establishment had received a terrible blow in the defection of Herr Vossner. It was not only that he had robbed the club, and robbed every member of the club who had ventured to have personal dealings with him. Although a bad feeling in regard to him was no doubt engendered in the minds of those who had suffered deeply, it was not that alone which cast an almost funereal gloom over the club. The sorrow was in this,—that with Herr Vossner all their comforts had gone. Of course Herr Vossner had been a thief. That no doubt had been known to them from the beginning. A man does not consent to be called out of bed at all hours in the morning to arrange the gambling accounts of young gentlemen without being a thief. No one concerned with Herr Vossner had supposed him to be an honest man. But then as a thief he had been so comfortable that his absence was regretted with a tenderness almost amounting to love even by those who had suffered most severely from his rapacity. Dolly Longestaffe had been robbed more outrageously than any other member of the club, and yet Dolly Longestaffe had said since the departure of the purveyor that London was not worth living in now that Herr Vossner was gone. In a week the Beargarden collapsed,—as Germany would collapse for a period if Herr Vossner's great compatriot were suddenly to remove himself from the scene; but as Germany would strive to live even without Bismarck, so did the club make its new efforts. But here the parallel must cease. Germany no doubt would at last succeed, but the Beargarden had received a blow from which it seemed that there was no recovery. At first it was proposed that three men should be appointed as trustees,—trustees for paying Vossner's debts, trustees for borrowing more money, trustees for the satisfaction of the landlord who was beginning to be anxious as to his future rent. At a certain very triumphant general meeting of the club it was determined that such a plan should be arranged, and the members assembled were unanimous. It was at first thought that there might be a little jealousy as to the trusteeship. The club was so popular and the authority conveyed by the position would be so great, that A, B, and C might feel aggrieved at seeing so much power conferred on D, E, and F. When at the meeting above mentioned one or two names were suggested, the final choice was postponed, as a matter of detail to be arranged privately, rather from this consideration than with any idea that there might be a difficulty in finding adequate persons. But even the leading members of the Beargarden hesitated when the proposition was submitted to them with all its honours and all its responsibilities. Lord Nidderdale declared from the beginning that he would have nothing to do with it,—pleading his poverty openly. Beauchamp Beauclerk was of opinion that he himself did not frequent the club often enough. Mr. Lupton professed his inability as a man of business. Lord Grasslough pleaded his father. The club from the first had been sure of Dolly Longestaffe's services;—for were not Dolly's pecuniary affairs now in process of satisfactory arrangement, and was it not known by all men that his courage never failed him in regard to money? But even he declined. "I have spoken to Squercum," he said to the Committee, "and Squercum won't hear of it. Squercum has made inquiries and he thinks the club very shaky." When one of the Committee made a remark as to Mr. Squercum which was not complimentary,—insinuated indeed that Squercum without injustice might be consigned to the infernal deities,—Dolly took the matter up warmly. "That's all very well for you, Grasslough; but if you knew the comfort of having a fellow who could keep you straight without preaching sermons at you you wouldn't despise Squercum. I've tried to go alone and I find that does not answer. Squercum's my coach, and I mean to stick pretty close to him." Then it came to pass that the triumphant project as to the trustees fell to the ground, although Squercum himself advised that the difficulty might be lessened if three gentlemen could be selected who lived well before the world and yet had nothing to lose. Whereupon Dolly suggested Miles Grendall. But the Committee shook its heads, not thinking it possible that the club could be re-established on a basis of three Miles Grendalls. Then dreadful rumours were heard. The Beargarden must surely be abandoned. "It is such a pity," said Nidderdale, "because there never has been anything like it." "Smoke all over the house!" said Dolly. "No horrid nonsense about closing," said Grasslough, "and no infernal old fogies wearing out the carpets and paying for nothing." "Not a vestige of propriety, or any beastly rules to be kept! That's what I liked," said Nidderdale. "It's an old story," said Mr. Lupton, "that if you put a man into Paradise he'll make it too hot to hold him. That's what you've done here." "What we ought to do," said Dolly, who was pervaded by a sense of his own good fortune in regard to Squercum, "is to get some fellow like Vossner, and make him tell us how much he wants to steal above his regular pay. Then we could subscribe that among us. I really think that might be done. Squercum would find a fellow, no doubt." But Mr. Lupton was of opinion that the new Vossner might perhaps not know, when thus consulted, the extent of his own cupidity. One day, before the Whitstable marriage, when it was understood that the club would actually be closed on the 12th August unless some new heaven-inspired idea might be forthcoming for its salvation, Nidderdale, Grasslough, and Dolly were hanging about the hall and the steps, and drinking sherry and bitters preparatory to dinner, when Sir Felix Carbury came round the neighbouring corner and, in a creeping, hesitating fashion, entered the hall door. He had nearly recovered from his wounds, though he still wore a bit of court plaster on his upper lip, and had not yet learned to look or to speak as though he had not had two of his front teeth knocked out. He had heard little or nothing of what had been done at the Beargarden since Vossner's defection. It was now a month since he had been seen at the club. His thrashing had been the wonder of perhaps half nine days, but latterly his existence had been almost forgotten. Now, with difficulty, he had summoned courage to go down to his old haunt, so completely had he been cowed by the latter circumstances of his life; but he had determined that he would pluck up his courage, and talk to his old associates as though no evil thing had befallen him. He had still money enough to pay for his dinner and to begin a small rubber of whist. If fortune should go against him he might glide into I. O. U.'s;—as others had done before, so much to his cost. "By George, here's Carbury!" said Dolly. Lord Grasslough whistled, turned his back, and walked up-stairs; but Nidderdale and Dolly consented to have their hands shaken by the stranger. "Thought you were out of town," said Nidderdale. "Haven't seen you for the last ever so long." "I have been out of town," said Felix,—lying; "down in Suffolk. But I'm back now. How are things going on here?" "They're not going at all;—they're gone," said Dolly. "Everything is smashed," said Nidderdale. "We shall all have to pay, I don't know how much." "Wasn't Vossner ever caught?" asked the baronet. "Caught!" ejaculated Dolly. "No;—but he has caught us. I don't know that there has ever been much idea of catching Vossner. We close altogether next Monday, and the furniture is to be gone to law for. Flatfleece says it belongs to him under what he calls a deed of sale. Indeed, everything that everybody has seems to belong to Flatfleece. He's always in and out of the club, and has got the key of the cellar." "That don't matter," said Nidderdale, "as Vossner took care that there shouldn't be any wine." "He's got most of the forks and spoons, and only lets us use what we have as a favour." "I suppose one can get a dinner here?" "Yes; to-day you can, and perhaps to-morrow." "Isn't there any playing?" asked Felix with dismay. "I haven't seen a card this fortnight," said Dolly. "There hasn't been anybody to play. Everything has gone to the dogs. There has been the affair of Melmotte, you know;—though, I suppose, you do know all about that." "Of course I know he poisoned himself." "Of course that had effect," said Dolly, continuing his history. "Though why fellows shouldn't play cards because another fellow like that takes poison, I can't understand. Last year the only day I managed to get down in February, the hounds didn't come because some old cove had died. What harm could our hunting have done him? I call that rot." "Melmotte's death was rather awful," said Nidderdale. "Not half so awful as having nothing to amuse one. And now they say the girl is going to be married to Fisker. I don't know how you and Nidderdale like that. I never went in for her myself. Squercum never seemed to see it." "Poor dear!" said Nidderdale. "She's welcome for me, and I dare say she couldn't do better with herself. I was very fond of her;—I'll be shot if I wasn't." "And Carbury too, I suppose," said Dolly. "No; I wasn't. If I'd really been fond of her I suppose it would have come off. I should have had her safe enough to America, if I'd cared about it." This was Sir Felix's view of the matter. "Come into the smoking-room, Dolly," said Nidderdale. "I can stand most things, and I try to stand everything; but, by George, that fellow is such a cad that I cannot stand him. You and I are bad enough,—but I don't think we're so heartless as Carbury." "I don't think I'm heartless at all," said Dolly. "I'm good-natured to everybody that is good-natured to me,—and to a great many people who ain't. I'm going all the way down to Caversham next week to see my sister married, though I hate the place and hate marriages, and if I was to be hung for it I couldn't say a word to the fellow who is going to be my brother-in-law. But I do agree about Carbury. It's very hard to be good-natured to him." But, in the teeth of these adverse opinions Sir Felix managed to get his dinner-table close to theirs and to tell them at dinner something of his future prospects. He was going to travel and see the world. He had, according to his own account, completely run through London life and found that it was all barren. "In life I've rung all changes through, Run every pleasure down, 'Midst each excess of folly too, And lived with half the town." Sir Felix did not exactly quote the old song, probably having never heard the words. But that was the burden of his present story. It was his determination to seek new scenes, and in search of them to travel over the greater part of the known world. "How jolly for you!" said Dolly. "It will be a change, you know." "No end of a change. Is any one going with you?" "Well;—yes. I've got a travelling companion;—a very pleasant fellow, who knows a lot, and will be able to coach me up in things. There's a deal to be learned by going abroad, you know." "A sort of a tutor," said Nidderdale. "A parson, I suppose," said Dolly. "Well;—he is a clergyman. Who told you?" "It's only my inventive genius. Well;—yes; I should say that would be nice,—travelling about Europe with a clergyman. I shouldn't get enough advantage out of it to make it pay, but I fancy it will just suit you." "It's an expensive sort of thing;—isn't it?" asked Nidderdale. "Well;—it does cost something. But I've got so sick of this kind of life;—and then that railway Board coming to an end, and the club smashing up, and—" "Marie Melmotte marrying Fisker," suggested Dolly. "That too, if you will. But I want a change, and a change I mean to have. I've seen this side of things, and now I'll have a look at the other." "Didn't you have a row in the street with some one the other day?" This question was asked very abruptly by Lord Grasslough, who, though he was sitting near them, had not yet joined in the conversation, and who had not before addressed a word to Sir Felix. "We heard something about it, but we never got the right story." Nidderdale glanced across the table at Dolly, and Dolly whistled. Grasslough looked at the man he addressed as one does look when one expects an answer. Mr. Lupton, with whom Grasslough was dining, also sat expectant. Dolly and Nidderdale were both silent. It was the fear of this that had kept Sir Felix away from the club. Grasslough, as he had told himself, was just the fellow to ask such a question,—ill-natured, insolent, and obtrusive. But the question demanded an answer of some kind. "Yes," said he; "a fellow attacked me in the street, coming behind me when I had a girl with me. He didn't get much the best of it though." "Oh;—didn't he?" said Grasslough. "I think, upon the whole, you know, you're right about going abroad." "What business is it of yours?" asked the baronet. "Well;—as the club is being broken up, I don't know that it is very much the business of any of us." "I was speaking to my friends, Lord Nidderdale and Mr. Longestaffe, and not to you." "I quite appreciate the advantage of the distinction," said Lord Grasslough, "and am sorry for Lord Nidderdale and Mr. Longestaffe." "What do you mean by that?" said Sir Felix, rising from his chair. His present opponent was not horrible to him as had been John Crumb, as men in clubs do not now often knock each others' heads or draw swords one upon another. "Don't let's have a quarrel here," said Mr. Lupton. "I shall leave the room if you do." "If we must break up, let us break up in peace and quietness," said Nidderdale. "Of course, if there is to be a fight, I'm good to go out with anybody," said Dolly. "When there's any beastly thing to be done, I've always got to do it. But don't you think that kind of thing is a little slow?" "Who began it?" said Sir Felix, sitting down again. Whereupon Lord Grasslough, who had finished his dinner, walked out of the room. "That fellow is always wanting to quarrel." "There's one comfort, you know," said Dolly. "It wants two men to make a quarrel." "Yes; it does," said Sir Felix, taking this as a friendly observation; "and I'm not going to be fool enough to be one of them." "Oh, yes, I meant it fast enough," said Grasslough afterwards up in the card-room. The other men who had been together had quickly followed him, leaving Sir Felix alone, and they had collected themselves there not with the hope of play, but thinking that they would be less interrupted than in the smoking-room. "I don't suppose we shall ever any of us be here again, and as he did come in I thought I would tell him my mind." "What's the use of taking such a lot of trouble?" said Dolly. "Of course he's a bad fellow. Most fellows are bad fellows in one way or another." "But he's bad all round," said the bitter enemy. "And so this is to be the end of the Beargarden," said Lord Nidderdale with a peculiar melancholy. "Dear old place! I always felt it was too good to last. I fancy it doesn't do to make things too easy;—one has to pay so uncommon dear for them! And then, you know, when you've got things easy, then they get rowdy;—and, by George, before you know where you are, you find yourself among a lot of blackguards. If one wants to keep one's self straight, one has to work hard at it, one way or the other. I suppose it all comes from the fall of Adam." "If Solomon, Solon, and the Archbishop of Canterbury were rolled into one, they couldn't have spoken with more wisdom," said Mr. Lupton. "Live and learn," continued the young lord. "I don't think anybody has liked the Beargarden so much as I have, but I shall never try this kind of thing again. I shall begin reading blue books to-morrow, and shall dine at the Carlton. Next session I shan't miss a day in the House, and I'll bet anybody a fiver that I make a speech before Easter. I shall take to claret at 20 a dozen, and shall go about London on the top of an omnibus." s. "How about getting married?" asked Dolly. "Oh;—that must be as it comes. That's the governor's affair. None of you fellows will believe me, but, upon my word, I liked that girl; and I'd 've stuck to her at last,—only that there are some things a fellow can't do. He was such a thundering scoundrel!" After a while Sir Felix followed them up-stairs, and entered the room as though nothing unpleasant had happened below. "We can make up a rubber;—can't we?" said he. "I should say not," said Nidderdale. "I shall not play," said Mr. Lupton. "There isn't a pack of cards in the house," said Dolly. Lord Grasslough didn't condescend to say a word. Sir Felix sat down with his cigar in his mouth, and the others continued to smoke in silence. "I wonder what has become of Miles Grendall," asked Sir Felix. But no one made any answer, and they smoked on in silence. "He hasn't paid me a shilling yet of the money he owes me." Still there was not a word. "And I don't suppose he ever will." There was another pause. "He is the biggest scoundrel I ever met," said Sir Felix. "I know one as big," said Lord Grasslough,—"or, at any rate, as little." There was another pause of a minute, and then Sir Felix left the room muttering something as to the stupidity of having no cards;—and so brought to an end his connection with his associates of the Beargarden. From that time forth he was never more seen by them,—or, if seen, was never known. The other men remained there till well on into the night, although there was not the excitement of any special amusement to attract them. It was felt by them all that this was the end of the Beargarden, and, with a melancholy seriousness befitting the occasion, they whispered sad things in low voices, consoling themselves simply with tobacco. "I never felt so much like crying in my life," said Dolly, as he asked for a glass of brandy-and-water at about midnight. "Good-night, old fellows; good-bye. I'm going down to Caversham, and I shouldn't wonder if I didn't drown myself." How Mr. Flatfleece went to law, and tried to sell the furniture, and threatened everybody, and at last singled out poor Dolly Longestaffe as his special victim; and how Dolly Longestaffe, by the aid of Mr. Squercum, utterly confounded Mr. Flatfleece, and brought that ingenious but unfortunate man, with his wife and small family, to absolute ruin, the reader will hardly expect to have told to him in detail in this chronicle. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Anthony Trollope (2004). The Way We Live Now. Urbana, Illinois: Project Gutenberg. Retrieved October 2022 https://www.gutenberg.org/cache/epub/5231/pg5231-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Stories, News & Mentions about FLATFILE

Flatfile’s Stories on HackerNoon