Flatfile’s vision is to enable seamless data transactions between every organization in the world

Flatfile

It requires an ample amount of time, effort, and resources to produce great, user-friendly documentation. Good documentation is fundamental to the way engineers get started with a product and ultimately use it in the long run, which makes it essential to both user retention and growth. Building our documentation at was a task we therefore took very seriously. Courier We originally created our documentation for a very small, specific audience with specific uses for Courier. Over time, however, our user base, their use cases, and our product itself has grown dramatically. To cover our bases, we needed to improve and expand our documentation and this time, we wanted to make it scalable and with a focus on a great user experience. We decided to use to do so, which allowed our engineers to collaborate and update our documentation more efficiently. Here’s how we built our documentation and what we learned in the process that might be useful to you. Docusaurus Identifying challenges with our documentation process When we first started Courier, we just wanted that explained how to use Courier as a product. So, we found a tool that could help with that: ReadMe. documentation ReadMe gave us enough features to get started with our documentation. We could create and update our documentation when needed and without much effort, but we couldn’t collaborate as much as we needed and could only save one draft at a time. This was a major challenge because anyone could easily overwrite another’s work if they were working on the same file at the same time. ReadMe also had a flat file structure which was inconvenient to use. There was also no real integration into their CLI, so we had issues with document versioning and collaboration. Ultimately, this meant that docs were in the approval process for an unnecessary length of time and not everyone was able to make suggestions for improvement as they came up. Git So some years and a few significant product updates down the line, we knew that it was time for a major documentation update. ReadMe was easy to get started with and served us well for our first couple of years, but we needed something that addressed our more specific pain points. Prerequisites for a documentation tool So we knew we needed something new. The next step was to decide which solutions were necessary to address our pain points for the next version of our documentation. The team came up with a list of requirements for documentation tools we would use in the future. Limitless Collaboration A happy challenge with a growing company is the need to scale almost every aspect of it as the team and product grow. As Courier’s technical team grows, we have more subject matter experts to provide input on documentation, which is good news for the quality of docs. However, more SMEs mean that we need more people to be able to collaborate on a draft at a time. Since the team will only continue to grow from here, limitless collaboration was one of the basic necessities for our future docs tool(s). Versioning A growing company comes with the inevitability of a growing product. As new versions of the product are released, there will always be an overlap time when some of our users are using an older version of the product and some are on the newest version. Our documentation needs to cater to both of these groups of users, which means to make sure everyone has the correct information, versioning is an important docs feature. For example, we need to keep the documentation of our API 1.0.0 version separate from the one for API 2.0.0 and ensure that it is possible to update and switch between versions quickly and easily. Version was also important to allow for the option to separate and store drafts away from the official documentation so that the team would be comfortable working without the pressure of a breaking change. Link Validation One of the worst things that can happen to user experience in product documentation is for a user to click on a link and get a 404 error. It makes it more difficult to keep users in the docs and following instructions, which can ultimately affect how many users successfully get started with the product and/or are able to troubleshoot on their own. A docs tool that would include link validation for internal links would go a long way toward avoiding damaging errors and UX issues. Localization We have customers from different ethnic groups all around the world who speak a diverse set of languages. To cater to a global audience, users need to be able to read our documentation and get the help they need, regardless of the language they speak. Localizing our documentation like link validation, improves user retention and UX. Docusaurus matched our list With our list in place, we were able to begin researching to find the right new tool. After some research, we discovered that , which is built on React, allowed for limitless collaboration, versioning, link validation, and localization. Docusaurus With Docusaurus, we built our docs as markdowns and exported them as a static site. We created reusable custom components and now have complete control over how we build our documentation. The integration is also particularly useful, as it allows any JavaScript code to be embedded into the .md file. MDX With a few lines of code, we could tweak our theme to suit our brand colors or customize any part of our documentation by the components we wanted. We also could use plugins to extend the functionality we got from Docusaurus. An example is the plugin for our search feature that allows customers to find relevant documentation. swizzling I also built our custom API explorer myself, which enables developers to try out our API methods directly from our docs. I built it as an embedded React component placed in the MDX file. Because we host our documentation on , we could utilize Vercel Functions to proxy the API Explorer requests to the Courier API through our docs backend endpoint. Vercel With Docusaurus, we have been able to build our documentation quickly and easily. We can now make updates by simply cloning the repo on , branching off our main branch, implementing the changes, and creating a pull request (PR). Once someone has reviewed and approved the PR, it’s merged with the main branch, which deploys to Vercel and updates our site with the changes. GitHub All we need to contribute to our documentation is a GitHub account and an integrated development environment (IDE). This helps us keep track of every maintenance task for our documentation. What’s next for our documentation? There are a few more goals we have for our documentation. We have yet to utilize Docusaurus’s localization feature, so that’s in the works. We have also recently made our documentation to welcome external contributors. This way, the user can take ownership of the updates they want and help themselves and others who would benefit from their contribution. open-source Until then, learn to use Courier via our . We also content that’ll help you get . documentation and tutorials live stream started and continue with Courier Also Published here

How We Built Our Software Documentation On Docusaurus

During the height of the business intelligence (BI) craze earlier in my career, I worked with an internal reporting team to expose data for (ETL) processes that leveraged data structures inspired by . It was a new and exciting time in my life to understand how to optimize data for reporting and analysis. Honestly, the schema looked upside down to me, based on my experience with transaction-driven designs. extract, transform, and load Ralph Kimball In the end, there were many moving parts and even some dependencies for the existence of a flat file to make sure everything worked properly. The reports ran quickly, but one key factor always bothered me: I was always looking at yesterday’s data. Due to the increase in size, velocity, and types of data being produced in modern tech stacks, real-time data has become the holy grail for enabling up-to-date and proactive analysis and data-driven business decisions. When we use the common batch style approach to data ingestion, the data is always stale, and it can’t answer the pressing questions. To enable these real-time use cases, the data must be available in the storage layer and processing layers currently implemented at an organization. For many years—and this is still true today—transactional (OLTP) systems like Oracle, Postgres, and SQLServer are at the core of most businesses, and they’re central to downstream analytics and processing. This work is typically done by a team of data engineers who spend months building pipelines that pull and transform data from those OLTP systems into analytics data lakes and warehouses (OLAP). Over the past decade, organizations of all sizes have spent billions of dollars with the goal of improving their capacity to process data and extract business value from it. However, only of organizations believe they have achieved the vision of becoming a data-driven company. One of their biggest hurdles to achieving the “Data Utopia” state is gaining the ability to build reliable data pipelines that extract data from OLTP systems while ensuring data consistency and ease of scale as the data grows. 26.5% This made me wonder if there was a better way to accomplish the same results without a hefty overnight process. My first thought was to explore the change data capture (CDC) pattern to meet those needs. Why Change Data Capture (CDC) Matters Extracting data from OLTP systems can be cumbersome, for reasons that include: Unlike SaaS tools, enterprise databases don’t have flexible APIs that you can leverage to react to data changes. Building and maintaining pipelines to extract data from OLTP systems, especially as a business scale, is complex and costly to ensure data consistency. On the other hand, CDC is a software design pattern that determines and tracks database updates as they occur. This allows third-party consumers to react to database changes, vastly simplifying the process of retrieving data changes in the OLTP system. CDC events can be sourced through several different approaches: Time-based: uses the required timestamp column as a source for data Checksum: leverages checksum on tables in source/target, using differences as a source for data Log-based: reads database logs as a source for data For this publication, we will focus on the log approach. Log-based CDC is commonly accepted as the best option for real-time use cases because of its lightweight approach to producing data changes. Time-based and checksum approaches require more overhead and CPU, often using queries against the database to find the newest changes to a dataset. CDC using log-based reads can shortcut these expensive and time-consuming queries, capturing changes by going straight to the source: the logs. Using the log approach, database writes are handled via transaction logs, often referred to as changelogs. CDC solutions listen to those same logs and will generate events where established conditions apply. By using the logs, the CDC service gains access to the same data without waiting for the transaction to commit and then making a request to the database for that data. By implementing a CDC solution, third-party data consumers have quicker access to the database changes. Additionally, data consumers do not require access to the actual database since they are only interested in updates in those changelogs. Not only does CDC allow near-instant data replication, but it also has minimal impact on the performance of production databases. Because CDC only accesses the logs and doesn’t pose any security threat, the IT teams love it too. The unique characteristics of CDC made it widely adopted by many tech leaders that real-time use cases are key to their business models (e.g., , , ). Shopify Uber CapitalOne To understand CDC more, let’s consider a use case related to the financial industry. Detecting Fraud Using CDC Fraudulent transactions are a major concern for the financial industry. In fact, noted credit card fraud increased 44.7% in 2020 over prior-year values. this report Let’s assume WorldWide Bank, a fictional financial institution, wants to improve the speed at which they recognize fraud. Their current approach relies on an ETL process to query against their Oracle database and generate reports. They would like to explore using in order to process against the Oracle database changelogs to generate events that will feed their Databricks Delta Lake prototype. Like many organizations, WorldWide Bank has a variety of transactional systems, including a large subset of data in an Oracle database. They have a growing need to provide more machine learning (ML) capabilities as well as combine various datasets in an easy-to-use environment for analysis, which has led to their adoption of Databricks for a data lake. Arcion CDC The relational data model of Oracle, along with the hassle of bringing other ancillary data sources into Oracle and the lack of support to run ML directly on Oracle, makes Databricks an easy choice. But, this approach leads to another challenge: how to source the Oracle transactions into Databricks in a timely manner. To improve the speed at which they can detect fraud, it is critical to their operation to be able to source CDC events from Oracle quickly, depositing them into their Databricks data lake for further processing and detection. Regarding their decision to use Arcion CDC, why might WorldWideBank choose Arcion and not another option, even open source? The answers are simple. Debezium, for example, is an open-source option, but it requires the installation and running of Zookeeper, Kakfa, and MySQL—all before you can even begin to use or install the tool. This alone would require a large team of engineers to simply install and maintain the tech stack. While other tools—like Fivetran’s HVR for CDC—might also offer free trials, Arcion has two free options to choose from: a 30-day trial with a download or a 14-day trial on the cloud. This is convenient for testing, so it made Arcion an easy choice when selecting a CDC solution. At a high level, the Arcion CDC architecture can be illustrated as follows: If this approach goes well, WorldWide Bank will expand the design to replace other source databases in Oracle, Microsoft SQL Server, and MySQL. Prototyping Arcion CDC In order to simulate the WorldWide Bank scenario, let’s assume inbound transactions are being written to an Oracle database table called TRANSACTIONS, which has the following design: An example record in the TRANSACTIONS table would appear as shown below using JSON format: {
   "id" : "0d0d6c35-80d0-4303-b9d6-50f0af04beb3",
   "post_date" : 1501828517,
   "trans_date" : 1501828517,
   "type" : 1017,
   "card_number" : "4532394417264260",
   "account_number" : "001-234-56789",
   "amount" : 27.67,
   "vendor_number" : 20150307,
   "vendor_location_latitude" : 42.3465698,
   "vendor_location_longitude" : -71.0895284,
   "vendor_name" : "Berklee College of Music Bookstore" 
} In order to leverage the power of Databricks and Delta Lake for fraud prevention, we need to set up a CDC service to act as middleware between the two systems. The TRANSACTIONS table design and data in Oracle need to exist as a Bronze (Ingestion) layer within Delta Lake. Once the existing data has been ingested, all future transactions will leverage the CDC stream included in the Arcion solution. ** **Getting started with CDC begins with establishing a free and obtaining the connection/login information for the Oracle database and Databricks system. Arcion Cloud account After gathering this information, I started a new replication in the Arcion Cloud application and defined the replication and write modes between the source and target database: In this case, I selected the replication mode. This lets me replicate a full snapshot of the database and enables real-time operations for all future updates. I chose the write mode option to fully replace any existing database design and data on the ta Full Replacing rget host. Next, I established a connection to the Oracle database, similar to what is shown below: We can validate the connection by using the Test Connection button. Once ready, the button moves the progress to the next step. Continue to Destination On the Destination screen, I can add a connection to the Databricks service and select it as the target for the replication being created: Using the button, I was able to proceed ahead to the filter screen shown below: Continue to Filter I selected the source database and tables on the Filter screen. Once ready, all I had to do was click the button to complete the following tasks: Start Replication Remove any existing databases in Databricks that match the current replication request Establish new databases in the Databricks environment Perform a snapshot replication of all existing data that matches the filters provided in the replication request Establish a CDC stream between Oracle’s changelogs and Databricks for all future transactions. I completed all of this with zero code. Once the replication started, I could monitor the process in the Arcion Cloud user interface: As noted above, the snapshot will be performed first, then any future updates will arrive via the CDC stream. Once completed, the user interface is updated as shown below: At this point, the Bronze (Ingestion) layer of Delta Lake is ready for use. The ingested data can be transformed into Silver (Redefined Tables) and Gold (Feature/Agg Datastore) layers, allowing us to leverage the benefits of Delta Lake functionality to detect and mitigate fraud. Going forward, the expectation will be that all data from the Oracle TRANSACTIONS table will automatically find its way into Databricks. We have one final step of our CDC ingestion into the data lake. Prior to running the ML fraud model for prediction output, we would run pre-built dbt tests or Great Expectations checkpoints to ensure we have received new records since our last fraud detection run. This would give WorldWide Bank confidence that full end-to-end CDC from Oracle is flowing into Databricks as expected. I find it very impressive that I can establish this level of design by merely following a series of steps within the Arcion Cloud user interface without writing a single line of program code. For the WorldWide Bank use case, establishing similar replications for their other databases should be just as easy, allowing all fraud detection to be processed in one single system. Conclusion Since 2021, I have been trying to live by the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality which extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester Arcion provides CDC functionality without reinventing the wheel. Their platform allows feature teams to utilize a no-code approach to establishing source and target data sources, plus everything needed to route inbound database transactions to a secondary source before they are written to the database. In addition, Arcion can be deployed in multiple ways. Financial or health care enterprises, for example, maybe have security or compliance requirements. They can deploy self-hosted Arcion on-prem or in a virtual private cloud. For companies without the DevOps resources to manage their own deployments, fully-managed Arcion Cloud would be a good option. Furthermore, the ease at which snapshot and CDC streaming replications can be established allows for additional data sources to leverage the same processing logic found in the Databricks Delta Lake implementation. This ultimately provides a “don’t repeat yourself” (DRY) approach by allowing one source of logic to focus on fraud detection. The Arcion team is certainly adhering to my personal mission statement and allowing their customers to remain focused on meeting corporate priorities. If I were working with that same BI team that I mentioned in the introduction, I would certainly recommend exploring Arcion Cloud to meet their needs. This would provide the opportunity for real-time data rather than relying on day-old data to make decisions. Have a really great day!

How To Use Change Data Capture for Fraud Detection

Creating a Hiring Strategy Why should someone work at your company? Every company answers this question differently. Here are some valid strategies for hiring: “ ” - We’ll pay the median rate in your area, but we’re investing heavily in making this the place for you to do your best work — the best work culture. (early days New Relic) “ ” - We’re working in a compelling technical area, which attracts a lot of talented people, so you can work with a great team and solve interesting problems. Plus we’re making a great people-focused work environment. (Gremlin during my tenure there, also early days New Relic) “ - We’re making Disneyland for engineers, lots of perks and top pay, in a university-like environment. (early Google strategy). “ - We’ll pay really well for you to work your ass off, and in return you get a chance to work at a scale you won’t otherwise.” (Amazon today) “ - We won’t pay you the most, but you’ll get to spend time each week working on open-source software.” (a local software consultancy my friend ran for a while) In practice, when you haven’t done the work to articulate why someone should work there, your hiring strategy looks like this: “ Work super hard at a stressful startup for low pay and equity that is probably worthless.” You need to create your hiring strategy. Ask yourself, how are you going to stand out as a place people will want to work? Like any strategy, you should consider what your unique strengths and weaknesses are. Some dimensions you can flex or emphasize to attract candidates: Remote Versus In-Person: Having a strong stance on this will attract people who care about either of those options. Perks You are Willing to Offer: Consider offering allowances for teleconferencing (high quality cameras and microphones, for example). , with a designer helping you renovate your workspace. You can offer top of the line developer machines, or quality monitors. These are things that also help your team work more effectively together, so it’s offering a benefit that serves two purposes! If I were making my own startup nowadays, I’d focus on top-tier videoconferencing equipment (DSLRs and external microphones) as a main perk. Flatfile offers an office makeover Freedoms You’ll Offer: You can offer a . You can craft a narrative around . Or something else entirely. 4 day workweek offering no deadlines and no meetings Transparency on Salary and Equity: Research . There are a few things you can do to make your position stand out for candidates from diverse backgrounds: list the salaries (and equity!) for the positions you’re posting, and emphasize how you’ll make the workplace fair and equitable for all employees. Share your philosophy around how salary and equity are determined. Being more upfront about this than competitors will make you more desirable and competitive. shows diverse teams produce better business outcomes Start by , and then tell people you’re doing it. How bizarre is it that your hiring manager is expected to advocate for and win the trust of their team member once they’re hired, but they’re expected to negotiate against their best interests during the hiring process? Starting off the relationship without having to negotiate salary can be a huge relief for candidates. implementing pay equity A Haven for People From Diverse Backgrounds: The experience for many people in tech companies is far from ideal. Many people leave our field because they’re treated so poorly. Most companies haven’t invested the time and energy to build a work culture that retains and delights people who aren’t overrepresented in tech. Impact People can Have on the World: Consider ways your company helps employees connect their work to something meaningful or interesting to them. Are there ways you can emphasize this more? People at Patagonia are given the option to volunteer once a week, paid by the company. Working at a nonprofit can have direct impact. For most companies, you can think of the value of what you produce in a way that people will find meaningful. Think about how that ties into your narrative for why to join your company. : Attributes of the Team What is the team like, and how can you use that to attract the type of people you’d like to come to the company? For example, you might be good at hiring people that are collaborative, and not assholes. You might be in a place with a lot of creative thinkers. As you look through these, come up with a one sentence description of why people should join your company. You’ll know you have a real strategy if it is rational for a certain type of person to select your company over all the other companies out there. This post is originally from my . You can find a permalink for it . course on engineering leadership here Did I miss anything? Please share your favorite ideas with me, I’d love to hear them!

Rethinking the Hiring Process: How to to Stand Out and Delight Your Candidates

If You Have the Right App for It TagSpaces’ new Markdown editor marks it down as best note-taking app, hands down, no contest. If that sounds like an exaggerated claim, read on, I might even convince you. the If you have not read , you probably should: You might learn more about why this software is really good for your online health (see also: privacy and data ownership), but if you don’t like jumping across reviews, the short version goes as follows: my (last) review TagSpaces is an open-core, offline, no-backend, flat-file, privacy-focused, and privacy-respecting file organizer and editor tool that lets you keep 100% of your data 100% safe, private, and . There is no cloud server, no registration, no ever-changing privacy policy to accept — only a cross-platform, easy-to-use, continually improving software that lets you deal with your data in any way you like, even outside of the app (so no vendor lock-in either). yours As I have already (and often), I am no fan of Cloud-basing everything just for the sake of appearing modern, especially when it comes to both personal and professional note-taking. There are few advantages and many caveats of doing all things online, especially when it comes to keeping your notes and digital things organized, and TagSpaces delivers a very well-trimmed solution to this problem, which has only just got a lot better. mentioned elsewhere The new feature that prompted this review is, in reality, only new for me; since I was too lazy to update and been happily using v3.3, the feature that took me by quite some surprise has been out since v3.11. (Current version code is 4.1). So I have been missing out on quite a lot, as it turns out. A Brand New Markdown Editor Suddenly Appears I love using Markdown for my notes. It’s clean, easy to read, easy to write, easy to parse in code. So, besides all the privacy and freedom and the ability to use it the way I like (see my previously linked articles for more on this), a good note-taking app would ideally use Markdown. This is not the biggest requirement but certainly is a nice-to-have. The old Markdown editor in TagSpaces was my only real gripe with the software, yet I kept using it because of its superior tagging and organizational capabilities. I read Markdown well enough, so I did not need to see the formatted text (yes, it also has an HTML-based RichText editor, but I never cared much for those), but when I (finally) updated my copy, I was very pleasantly surprised. TagSpaces now has a that . And it’s quite easy to use, even for those who don’t like to look at the code. real Markdown editor actually works well The Best Features Are Those You Don’t Even Notice The editor is based on the framework, an awesome open-source Markdown library, which got tailored to fit TagSpaces perfectly. Milkdown The editor is WYSIWYG; when you type Markdown, it gets immediately turned into nicely formatted text. This is a big advantage for me, since however well I might read the markup, this is still easier to look at, and the formatting itself (the actual Markdown) gets preserved when I copy/paste text to or from the editor. I have, of course, used editors like that before. I write in Typora, for example, which has a nice editor, but many limitations beyond that, they recently switched to a paid model, which kind of hurts my senses after having used it for free. (I get it, money must be made, but, for example, TagSpaces only charges for professional use cases, and the base application is free, which is a much nicer way to both give to the community and earn some well-deserved money for your hard work as a developer. Dropping a paid-for update into the Linux repo was really a d*$# move, I have to say this much. Bait and switch much, Typora?) short fiction and So, while it is not the first WYSIWYG Markdown editor I tried, it fits nicely into its new environment and works really well for me, and fits the way I use the app. Formatting Text the Easy Way The editor offers a number of ways to format your text, all of which are unobtrusive and simple enough to use. Besides what I normally do (typing the Markdown straight in), you have the option to highlight some text, and choose some basic formatting from a popup. This is limited to , , strikethrough, code, and , and that seems to be kind of enough for simple note-taking, although I really do miss the ability to add highlights. You know, like . bold italics links ==this== There might be plenty of arguments for or against using this syntax, but I don’t care for any of them. To me, it’s very useful, and I’d . (You might think it’s a futile attempt, but I know the devs read my stuff, and my opinions even influenced the development before, so there is hope yet…) love to see this in a future update of TagSpaces Highlights aside, this simplest way of formatting text inline is complemented by slash commands that offer block-level formatting options. This is a feature that is quite new to me in a Markdown editor, but one I really got into quite fast. It’s kind of the same you’d see in, e.g., Notion, another online-first note-taking app that, although looking nice, is as far from my needs as most of its cloud-based competitors. So the editor has no formatting toolbar for now, and although this feature already exists in the Milkdown framework, and TagSpaces might introduce it later, I am personally not missing it. Toolbars make rapid note-taking harder on the computer when you have to constantly click away from the text just to format it or insert something. Enter the Slash Command TagSpaces’ slash commands, on the other hand, will let you stay right in the text, and even if you do not know the markup, you can just choose from a pop-up list right on the spot, then continue typing. Currently, slash commands support the following block-level formatting options: Headings to headings: A compromise between having a full list and a too-long pop-up list. I do not love it, but I can learn to live with it. H1 H3 All Types of Lists One limitation is the usage of multi-leveled lists. This is supported by the editor, but to use it, you must switch to Markdown view (only accessible from the Floating Action Button (or FAB, in short) in the bottom-right corner, I have not found out the key-combination yet), and add four full spaces the bullet point you want to indent. before Neither the slash commands, nor typing Markdown offers this option currently in the WYSIWYG editor, because there is no way to insert anything before a bullet point, and pressing the Tab key has a different binding in the software. But if there was no room for improvement, things would soon get stale, so here’s an area to improve, dear developers. It should not be very hard to do either, so there is hope. The greatest upside is the truly interactive task list, where you can click on tasks to complete them. It will not strike out the text, but being able to mark them easily is a huge advantage to any list addict. Images Adding images is fairly simple; with no file-system browser yet, you have to enter or paste image URLs or file links manually. Although has internal file linking (this is an affiliate link, I get no commission from sales, just sayin’), those file links were never meant for the Markdown editor to render them; those are for referencing only. Still, you can enter a local file path to have your images displayed, or just the filename, if it’s in the same folder. TagSpaces PRO not Quotes Selecting “Quote” will drop you into a block-quote block, with the standard Markdown syntax for block-quote blocks. You stay inside the block after pressing Enter, so you can easily do multiple lines and paragraphs. You can get out of the block by pressing Enter twice. Tables This is the coolest feature yet. Inserting a table gets you a simple but very easily customizable standard 3x3 Markdown table. The real feature here is how you can interact with it. Pressing the slightly hard-to-find-at-first but definitely unobtrusive little square in the upper-left corner ( ) will give you options to interact with the whole table. You can insert a column before ( ), or after ( ), and a row at the bottom ( ), set alignment globally ( ), or delete the whole thing ( ). 1 2 3 4 5 6 When you want to interact on a single column, you must press the column header and choose from the slightly more limited list. (The same, sans adding a new row.) Pressing the row headers gives you an even more limited menu: You can add a new row above or below the current one or delete it. Setting alignments on rows makes no sense in a Markdown table context (you know what I mean if you are familiar with the syntax), so that option is missing. Code Fences You can also add standard code fences, where you have the option of marking the language in a dropdown list. TagSpaces will not highlight code syntax for you, but it’s also not the point of the software; IDEs do that better anyway. What’s Missing from Slash Commands Quite a lot, actually. None of the usual inline formatting options are present like the aforementioned , striked, and you cannot add links here either, but that is intended behavior; slash commands are meant for block-level formatting. bold italic So, instead of being an alternative to the popup formatting toolbar, slash commands rather complement its use by offering all things missing from the popup. This certainly keeps the list shorter and the items probably easier to use, but whether it complicates the use of the application in the long run, or makes everything so much easier, will depend on your personal preference. As mentioned above, I type in markdown straight, so it does not affect me either way; thus, I also cannot form a meaningful opinion on this. You could probably let me know in the comments; I’d be interested to learn how others see and use this feature. However, this is a feature of Milkdown itself (the framework TagSpaces uses for its editor), so TagSpaces developers will not be able to change it, whatever we, mere mortals, think about it. Other Nice Features I Will Never Use There are also two nice features that I never use, but others seem to be quite keen on Math Expressions and Diagrams. Expressing Math The former seems to have been requested quite a few times on TagSpaces’ issue tracker and also their public suggestions/requests board, and so its implementation became another example of the developer’s responsiveness. In the new editor, you can do math expressions now. Unfortunately, I am probably the least qualified person on the internet to evaluate anything remotely connected to math or its expressions, but after some searching, I’ve found an example of math used in markdown online, and it turned out to render quite nicely. I will not pretend to know what it means, but it looks very math-ish: $\sum_{n=1}^{10} n^2$ becomes Currently, I find one limitation. If you type in the expression as you go, it will be immediately rendered, as you close it with the dollar symbol, behaving like any other formatting; but when you paste in an expression, found, e.g., on the internet (like I did), some symbols get “escaped” with a backslash character, and the expression does not render, until you fix this in the code view. The solution is to “paste as is,” usually meaning pressing Shift+Ctrl+V. I tried this in TagSpaces, and I managed to escape the escape symbol, meaning the expression was rendered expressively. Human-Readable to Even More Human-Readable: Enter the Diagram Diagrams in Markdown are a thing I was curious about for a while but never had any time, energy, or even reason to look into. So now that it’s one of the major new features of the editor I am reviewing, it’s also a good opportunity to continue not looking into diagrams because I am busy writing an article. Even so, I had a cursory glance, so you don’t have to. Diagrams in TagSpaces are based on the . Looking at their website, diagrams look like one could get used to using them, but not half as useful as creating them visually, with a dedicated tool. Without having any experience using these diagrams in the wild, I cannot comment on the usability, but they definitely have their place if you are restricted to Markdown and have to still visualize algorithms or classes or even just make charts. mermaid project So I tried one of the formulas from the mermaid website (the one you’d never be able to decipher easily), just to see what happens. So what happened was a very nicely rendered diagram on my screen. I am genuinely impressed. The script does not look very complicated and seems learnable, so if I ever need diagrams where I cannot draw them with some fancy visual tool, I know where to turn. Until then, I’ll probably remember, with great fondness, the time I made a diagram in Markdown and how pretty it turned out… To insert a diagram, you have to start a newly fenced codeblock, and give it the mermaid type. Since mermaid is not (currently) available from the slash-command originated codeblock’s dropdown list, you must type this in manually: ```mermaid That will turn into a special codeblock where you can type/paste your diagram and get pretty results like that above. To Conclude… This is it in a nutshell. TagSpaces stepped up in the league of note-taking apps by a huge stride with its new editor. So besides superior file-tagging and organization, note-taking, and writing, any kind of Markdown is now in the same league as some of the top players. Best thing? It’s free. Also Published Here

Markdown and Privacy-Focused Note Taking are the Perfect Match

Four company history lessons, and a forward looking statement: 2018 was the year of possibilities. 2019 was the year of issues. 2020 was the year of opportunities. 2021 will be the year of iterations. All in an effort to become . We’ve summarized Hacker Noon’s progress by revenue, growth, product, software expansion, and some final note. the best place for technologists to read, write and publish Revenue Hacker Noon doubled revenue for the 3rd year in a row in 2020. This is a YoY increase of +102%. This represents QoQ growth of + 21% and YoY growth of + 68%.  Our top sources of income were , , , , , and . Top Navigation Grants Brand-as-Author Noonies Newsletters Ad By Tag launched in Q4 2020 and is now booked through Q2 21. It allows sponsors to place their content next to content with relevant tags. Ad by tag niche marketing We started 2021 with revenue in January being at an all-time high for revenue generated in a single month for HackerNoon. To put it into perspective, the revenue we made in January was almost as much as the revenue we made in 2017 and 2018 - combined. , and more. These things can ebb and flow, i.e. when deals close and how buyer optimism flows, but it’s good to see quality companies committing to working with Hacker Noon early in the year. Much thanks to some of our new and bigger customers: Wix, LaunchDarkly, CouchbaseDB, ByBit, DeFi For You, Udacity, Particle, CryptoAdventure, Flatfile As top navigation and niche marketing inventory fills up, we’ll be putting more effort in publishing and republishing the corporate blog posts, press releases, and software updates of as many technology companies as possible. Later in the year, we’ll go to market with a subscription offering for brand publishing, dubbed Evergreen. Growth Our top source of traffic, Google, finally started to rise again, after a steady half-year decline that also hurt the company’s Alexa site ranking. This recent rise in Google traffic could be due to their algorithm changes, our infrastructure work, our edtiorial rules, our content quality, just dumb luck, or an executive at Google changing their mind about our site. We did see an 18% growth in unique pageviews from December to November and a 19% growth from Dec 2020 to January 2021. Editorial team highlights: Part-time editors hired: We added to our team in 2020, and they bring extensive knowledge and experience in the AI, machine learning, cyber security, gaming, and non-profit sectors. Going to expand into more junior editors to improve our rate of publishing. three new editors Writing competition: Starting with Wix, brands will be able to reward the best performing stories on a Hacker Noon tag. This is exciting because it drives all our key metrics: words written, time reading and money made Managed Accounts: We’ve experimented with commissioned content to increase yearly revenue from affiliate marketing sources and the early results are promising. We will be working with experts in the space to scale up these efforts this year. Additionally, while we aspire to offer visitors a sh*t ton of value without logging in, deeper relationships with our visitors is something we want to aspire toward. We are seeing a trend of more logged in users on , after introducing a new signup flow. app.hackernoon.com Product In Q4, we conducted two separate audits to review the process of software development projects. Vercel, the creators of NextJS, audited our codebase to help prioritize which projects and approaches will improve page performance. Additionally, we audited the quality of every function marked as shipped in 2020. The overall learning was that we should ship less new things and spend a higher percentage of our time improving what we do offer.  Nevertheless, we did still ship some new things, and have some cool things coming up this year. Content Management System projects shipped in 2020 Q4: - This new flow has been performing much better. It includes login/signup in addition to email, and educates the visitor on our value to readers, writers, and brands. New Signup Flow - The first are live and dynamically updating the curated story lists via Hacker Noon mentions and the Bing News API. Tech Company News Pages 913 pages - This is in partnership with Superpeer. We are exploring more ways to revenue share with the contributing writer. Paid Expert Video Calls Beta

State of the Noonion Q4: Ad By Tag, Tech Company News Pages, and More

This is the third article documenting what I’ve learned from a series of . 10 Trailhead Live video sessions on Modern App Development on Salesforce and Heroku In these articles I’m walking you through how to combine Salesforce with to build an “eCars” app—a sales and service application for a fictitious electric car company (“Pulsar”) that allows users to customize and buy cars, service techs to view live diagnostic info from the car, and more. In case you missed my first article, you can find the link to it below and start from the beginning. Otherwise, if you’re specifically looking for data modeling, you’re in the right place. Heroku (the first article and the ) can be read here second article here Just as a quick reminder: I’ve been following this Trailhead Live video series to brush up and stay current on the latest app development trends on these platforms that are key for my career and business. I’ll be sharing each step for building the app, what I’ve learned, and my thoughts from each session. These series reviews are both for my own edification as well as for others who might benefit from this content. The Trailhead Live sessions and schedule can be found here: https://trailhead.salesforce.com/live The Trailhead Live sessions I’m writing about can also be found at the links below: https://trailhead.salesforce.com/live/videos/a2r3k000001n2Jj/modern-app-development-on-salesforce https://www.youtube.com/playlist?list=PLgIMQe2PKPSK7myo5smEv2ZtHbnn7HyHI On the last episode… Last session, we walked through Heroku buildpacks and learned how to start building on Heroku with a particular web language. The buildpacks were nicely structured, with intuitive instructions and installation steps for getting Heroku and your local machine set up to develop on one of the supported languages. We also ran through the process of using the command line, VS Code, and specific Heroku git commands to deploy a sample Lightning Web Components (LWC) app to a new Heroku app environment. Not bad for under an hour’s work! In this episode, we’re going to get into the bones of any modern application: the data layer. One of the things I find funny about data modeling is that almost everybody does it in their regular work, but rarely do people think too deeply about it (unless they’re developers or DBAs). There’s also a really steep learning curve when you graduate from a flat file like Excel to a multi-table relational database like Access. Every client I talk to can describe, to a degree, what kind of data their business needs to track. But going beyond that—initiating the right conversations with clients, reading between the lines, and teasing out the best data model of their application—is somewhat of an art form. Data modeling also seems to be a bit of an underappreciated part of the application-building process. Speaking from experience, getting the data model right or wrong for an application can mean the difference between a smoothly performing app that generates useful, insightful data and an app that gets “spaghettified” with workarounds and code to make up for the data modeling missteps made at the outset. A lot harder when you’re already a few million rows down the data rabbit hole! Data Modeling in Salesforce Using Point-and-Click Methods As we dive into our data modeling for the eCars app, there are two things worth noting about Salesforce’s data modeling features: Standard Objects establish a data model right out of the gate for Salesforce applications. Custom Objects and Custom Fields provide the ability to rapidly extend and prototype additional data requirements for any application. They also touch on Big Objects and External Objects, but in my experience, these rarely come into play except for enterprise-level implementations that have large data sets and external data sources. For the most part, Standard and Custom Objects covers 80% or more of the use cases in my experience. One major Salesforce data-modeling feature that requires experience and attentiveness is its attached dependencies and automatically generated metadata components. Many “freebies” and metadata dependencies come attached (i.e. conveniently created for you) to the custom objects, fields, and relationships that one creates in the system. For example, when you create custom objects and fields, the system already allows you to set and handle some things conveniently like creating a tab, default list views, page layouts, reports, and the security model down to the field level. Creating a relationship field, such as a lookup or master-detail relationship, also conveniently renders the child object (i.e. the table on the “many” side of a one-to-many relationship) as a related list on the parent object’s record page so that navigation is easy and intuitive.  Some key things to know about the two different relationship fields are below and also demonstrates the kinds of “freebies” or metadata dependencies that are generated with each type (rollups, sharing rules, etc.). This all results in the ability to immediately interact with and prove out your data model in your app and make tweaks and adjustments quickly. I can’t count how many times this has come in handy when working with clients on the initial vision of their apps. Instead of having to create ERDs and work up the data model before putting pencil to paper, we can just go directly to prototyping in real-time. Plus, the Schema Builder conveniently draws out the ERD as the data model is built out. You’ll also see in the video that it’s possible to build out the data model visually directly on the Schema Builder! Pretty nice when the data model comes with these freebies as it’s being built out Building the eCars Data Model By using the point-and-click data modeling features of Salesforce, we should be able to build out the entire eCars data model in under 20 minutes. The screengrab from the Salesforce Schema Builder tool below shows us the different tables/objects that are part of the app, how they’re related to each other (they’re all Lookup relationships here, no Master-Detail relationships so far) and which fields are required or automatically generated upon record creation. A written interpretation of the data model’s relationships could be as follows: A car can have zero, one, or more Configurations. Each configuration can have zero, one, or more Car Options. A Vehicle Order can act as a parent and/or child for a Configuration and vice versa. A Lead can be related to zero, one, or more Configurations. (The data model implies that a Lead can be interested in multiple Configurations.) Understanding what limitations and implications a data model conveys is one of the key skills in app development I’ve grown to appreciate. Heroku Data Services Data modeling on Heroku, on the other hand, follows some of the more traditional tools you might expect with SQL databases. Still, there are several GUI tools to help as well, which I’ll mention later. Heroku also features several important add-ons for scalability. The four major data services we’ll focus on for the eCars app are: Heroku Postgres (primary relational database) Heroku Redis (in-memory key-value data store) Apache Kafka (distributed streaming platform) (Connect Salesforce Data and Heroku Postgres) Heroku Connect Let’s look at each of those to see how we’ll use them. For Heroku, while other 3rd-party databases are available in the Elements marketplace, the officially supported database of choice is and is appropriately the most popular Heroku add-on. Heroku, like Salesforce, allows you to create your data model on Postgres using both command-line methods such as in VS Code as well as through a UI via the Heroku application manager. Heroku Postgres I’ve also used pgAdmin to manage Postgres databases as well and have used it in conjunction with Heroku Postgres databases. I don’t know any people who purely live in the command-line when it comes to databases, but I’m willing to bet they’re pretty fast on the keys! For the eCars app, we’ll use Postgres to store our basic relational data—customer information, car information, etc. This is actually my first exposure to Redis and Kafka. However, after Julián explained the use cases for each, I can see how these are very important data services for highly scalable applications. Scalability seems to be more and more of a recurring theme for Heroku also. Modern applications have to run lightning fast as consumers have come to expect that kind of performance. And even though solid-state drives might improve performance, the reality is that on-disk databases are just not going to be as fast as in-memory data. So, to get high-volume apps running super fast, you can see why you need something like or Salesforce Platform Cache to store some of the higher-volume information that powers the app in fast, high-performance memory. Otherwise, the database would buckle under the number of requests. Heroku Redis On top of that, modern applications typically run on distributed cloud infrastructure and may be using multiple data sources which need a lot of real-time throughput. In front of the data sources are a collection of services, and we need a way to reliably and quickly pass messages among them. fits the bill for this requirement, especially in large enterprise applications. is an event-streaming platform that will allow our eCars app to potentially scale to very high loads in a distributed app architecture. It also makes sense that there isn’t a free tier for this service, as I can imagine the expense it takes just to run one Kafka cluster. Apache Kafka on Heroku Apache Kafka Finally, allows Salesforce data to connect with Heroku Postgres in a bi-directional manner with a quick set-up. This saves us from having to do a custom integration between Salesforce and Heroku Postgres, but usually the deciding factor is the cost. Using Heroku Connect will allow us to seamlessly and easily connect Salesforce data to Heroku Postgres in a bi-directional fashion in near real-time. Heroku Connect One thing I’d like to clarify (since it used to confuse me) is the difference between Heroku Connect and . While Heroku Connect works with Salesforce standard and custom objects, Heroku External Objects is strictly a uni-directional OData service, meant to allow Heroku Postgres tables to sync to Salesforce External Objects. The naming can get confusing, so it’s important to keep this nuance in mind. Heroku External Objects also comes with the Heroku Postgres add-on while Heroku Connect is a paid service. Heroku External Objects Concluding Thoughts That’s an overview of data modeling on both Salesforce and Heroku, and how we’ll get them to work together. As I mentioned previously, I’m of the opinion that getting the data model right is probably one of the most important things when it comes to designing applications. If it goes wrong, everything can turn into a futile exercise, from the way data is collected to what the business intelligence/analytics tools produce. I got introduced to this mode of thinking pretty early on, before I even started my career. A USC Marshall Business School instructor named taught a database class to business students like me. I still credit that class, its data-modeling lessons, and the bit of SQL it taught us, with my eventual pursuit of Salesforce as my career. Douglas Shook We did countless homework problems and quizzes where he would give the class a bunch of receipts and a report from a shoe store or something, and then ask us to draw an ERD for a possible database system that generated them. When I reflect on that experience, I definitely have some renewed appreciation for how Salesforce and Heroku make it simple to handle the data aspects of an app now. If you want to take a deeper dive into data modeling considerations, I recommend taking a look at this article that Mohith wrote on . Data Model Design Considerations When Building Salesforce Apps In the next article, we’re going to look at how we can customize the front-end UI components of Salesforce for our eCars app, all without writing any code. If you haven’t already joined the official Chatter group for this series, I certainly recommend you do so. That way, you can get the full value of the experience and also pose questions and start discussions with the group. Oftentimes, there are valuable discussions and additional references available there such as the slides from the presentation. https://sforce.co/34Ld7xz About me:  I’m an 11x certified Salesforce professional who’s been running my own Salesforce consultancy for several years. If you’re curious about my backstory on accidentally turning into a developer and even competing on stage on a quiz show at one of the Salesforce conventions, you can read I wrote for the Salesforce blog a few years ago. this article Also published at https://dev.to/jasonomnivo/data-modeling-in-salesforce-and-heroku-data-services-569e

Data Modeling in Salesforce and Heroku Data Services 

Automated Data Replication From AWS S3 To Microsoft Azure Storage Made Easy

Designing a data pipeline comes with its own set of problems. Take lambda architecture for example. In the batch layer, if data somewhere in the past is incorrect, you’d have to run the computation function on the whole (possibly terabytes large) dataset, the result of which would be absorbed in serving layer and are reflected. In the speed layer alone, we have the problem of data latency, data lag, data correctness, data completeness, and out of order data. Data might come later than expected, there might be an unusual delay from one stage of pipeline to the other stage of the pipeline ( ), it might be incomplete, some data might have gone missing or lost somewhere in the pipeline ( ). The data might have been corrupted at the source or at a particular place in the pipeline where a computation function is applied to it ( ), or data might not come in order, some packets arriving later than usual ( ). lag completeness correctness out of order The purpose of this post is to illustrate how to deal with data completeness, data lag and out of order data using a dictionary with expiring keys with a TTL attached to them, called The github repo can be found . dictttl. here Problem Statement The problem statement at hand is to make sure the data quality in the speed layer is good. This is done by tackling data lag, correctness, latency, completeness and out of order data. In this post i will be working with data completeness, lag and out of order data problems. Definition Data lag is defined as the time taken by data to go from one stage in the pipeline to another stage of the pipeline, not to be confused with latency, which means time taken by packet to go from source (for example a sensor) to the server. Data lag: If we have received all the chunks of data for a particular key (or sensor) in a given time frame, we say data is complete. Data Completeness: If the ordering of data is not strict, with packets having earlier timestamp arriving after packet with later timestamps, then it is an out of order problem. Out of order data: Sometimes data lag is too high, either because of plumbing of the pipeline, or simply because sometimes we have too much data in a particular window for a key, so running the computation function on it takes longer than usual. At times, computation functions drop some data, or all the packets are not picked up at the source, which causes data completeness problem. And finally, streaming tools such as kafka or sensors do not always send data in order. It is important to handle those scenarios as well. Approach The consequence of: lag is that the data packets arrive late, completeness problem is that all the data packets do not arrive, and out of order problem is that some packets arrive later than the others. Now, my current, really simplified version of data pipeline looks like this Streaming Tool > Big data tool > Storage tool For example, data would be arriving at kafka from a sensor, from there it will go to spark for aggregation and from there it will be stored in MongoDB. I can put a check at the big data tool stage to make sure i handle the mentioned problems. To handle lag, I can wait for say, n units of time to make sure all of them arrive. If they do not arrive within that time, I can 1) choose to discard them entirely or 2) log them that they arrived late, process them and update serving layer with the new results. In this post, I will choose to discard them. Also, these packets are called straggler packets . To handle the out of order problem, I can 1) choose to discard them entirely, or 2) take them in to account in any next window that they come in, process them and update my serving view. In this post, I will choose to discard them. To handle data completeness problem, I can use a metric based on the value of the sensor. For example, I can say that the sum of all the values for a sensor in a particular window should be at least m. If it is less than that, I would discard it. With this logic in mind, let’s design the simplified version of the pipeline. Design To design the pipeline, I will be using Kafka and Pandas. Kafka for streaming the data, and pandas for aggregating and quality checks. I have chosen not to store them in any database or flat file. To handle out of order and lag situations, I will wait for 5 seconds till all the data from the sensor arrives. If it does not, I will discard it and move on. To handle completeness problem, I will sum all the values for a sensor and say that it should be at least 7000. If it is less than that, I will discard that row (and use batch layer to compute the speed view). I will send data from a producer to a consumer, producer will send data regularly, except at two intervals. At one point it will send a custom value of 1 to demonstrate completeness problem, and at another point it will sleep for 7 seconds to demonstrate out of order and lag problem. The consumer will consume the data, by creating a tumbling window. I create a window of 5 seconds that will tumble from one to the another, and within a window I will collect all data points for a sensor, aggregate them (mean), and print them. I use a python library called dictttl for the purpose of handling lag and out of order data problems. In a nutshell, this is a dictionary that has keys that expire after a manually set time. Producer To generate data, I have used the following code, it can also be found at . github It is basically a command line program that taken in an input sensor and range(value_start, value_end), and it generates data based on those parameters, and sends to the topic. Also, sensor id is the same, and I am using defaultDict(list) variant of the dictttl, so the values would be appended to a list for a particular window, like this: sensor_1, { : (timestamp, [ data ])} e,g
{‘sensor_1’ : ( , [ , , , , , , , , , , , , , ]} key 16000000000 1 2 4 4 2 4 5 5 6 3 1 2 3 4 The lines: Makes sure that once a reaches 10000, it sends data value of 1, so that the consumer can check the sum of elements and discard it. It will be shown later. The lines: Makes sure that once a reaches 20000, it sends data but then halts for 7 seconds. This would be useful later. Consumer The consumer consumes the data and checks if the timestamp of the data is less than the starting point of the window. If it is less, it discards it, partially solving the out of order and late data problem (which could have arisen from the packets belonging to data from previous window). If it is in the window, it appends it to the expiring dict, which has a time to live of 5 seconds. The lines: Checks if the data is out of the window. First, it checks if the expiring dictionary is empty. If it is, it creates the next window and continues. The dictionary could be empty because it might have expired, as it happens because at one point in time the producer halted for 7 seconds, and the threshold for discarding data was 5 seconds. If it is not empty, it creates a data frame from the dictionary, and the following lines will check if the sum of all the values for the sensor are more than 7000. If not, i will print “drop the row”. The following line: Will make an aggregation and prints it. Result The result looks like the following: Note that the first 3 timestamps are sequential in order. At the fourth entry, length of dict is 0 because the dictionary expired, and the next timestamp has a difference of 5 seconds as compared to the previous timestamp. Also, the sum of values is less than 7000 in the fourth entry so it is dropped. Next three rows are again sequential. Summary In this post, I went through what kind of problems arise when dealing with streaming data, and how to tackle them. I used a library called dictttl which has expiring keys, and in those situations where we choose to discard data they are really useful, as we don’t have to manually search and delete those items. As seen above, we handled out of order and lag problem by creating a buffer to absorb data within a threshold and if any data arrives after the threshold, it is discarded. Again, this is a very simple example, and it assumes mostly ordered data. If a packet arrives ahead of time it will simply create the new window. But in cases where out of order problem is high, we can create a buffer for that too, we can say after 1% of out of order cases we will discard the values. We also saw how to handle data completeness problem, by taking sum of the values. If the sum is less than the threshold, we simply delete those items. I hope you enjoyed my post.

FLATFILE

flatfile.com

-- employees

Since n.d.

COMPANY RANKING

FLATFILE

EVERGREEN INDEX #3881

Representing this company? Click here to claim and customize this page!

Flatfile WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

FLATFILE

flatfile.com

-- employees

Since n.d.

COMPANY RANKING

FLATFILE

EVERGREEN INDEX #3881

RELATED COMPANIESRANK

STORIES ABOUT FLATFILE

Representing this company? Click here to claim and customize this page!

Flatfile WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!