Introducing HarperDB 4.2, a groundbreaking update that combines user-defined applications, real-time data streaming, and a lightning-fast database into a single package, expediting the development of enterprise-grade applications while

New HarperDB 4.2 - The Distributed Real-Time Application Platform That's As Easy To Use As It Is Powerful

Another customer focused on high throughput and low latency use cases is HarperDB, a distributed global application. The ability to dynamically route workloads based on geography through AWS Wavelength on Verizon will continue to allow HarperDB to improve ...

Verizon APIs leverage AWS Wavelength for latency-sensitive app … – Verizon

Today we’re going to talk about marketing technically complex products and services, and some of the challenges and opportunities they present. To help me discuss this topic, I’d like to welcome Jaxon Repp, Field CTO & Head of Marketing at HarperDB.

Ep 424: Marketing complex products and services with Jaxon Repp, Field CTO & Head of Marketing, HarperDB

HarperDB is a globally-distributed edge application platform. It reduces complexity, increases performance, and lowers costs by combining user-defined applications, a high-performance database, and an enterprise-grade streaming broker into a single package. The platform offers unlimited horizontal scale at the click of a button, and synchronizes data across all the nodes in a cluster in milliseconds. HarperDB simplifies the process of delivering applications and the data that drives them to the edge, which dramatically improves both the user experience and total cost of ownership for large-scale applications. Deploying HarperDB on global infrastructure enables a CDN-like solution for enterprise data and applications. With our flexible, component-based architecture, simple HTTP/S interface, and a high-performance single-model data store that accommodates any data structure, HarperDB scales with your application from proof of concept to production. Install and manage on your hardware, or have us host it with HarperDB Cloud.

HarperDB

HackerNoon Reporter: Please tell us briefly about your background. I started programming when I was 13 years old working for my uncle’s startup that did ISP billing in the 90s. I worked for Red Hat running infrastructure for global support services, and have founded a few startups and worked at others. What's your startup called? And in a sentence or two, what does it do? HarperDB is a globally distributed database focused on making developers' lives easier while providing capability for low latency applications. What is the origin story? Kyle Bernhardy, our CTO, and I have been solving big data and integration challenges for over a decade. We met at a company called Coresite where we were locked in a small conference room for a month working on a project. Since then he and I have worked together through 4 companies and along the way picked up several other amazing folks. We have continued to work on large scale and complex data projects. Ultimately we realized that the vast majority of the problems we were faced with in analytics, code, middleware, etc. were created by the database. Furthermore, we felt like databases were overly complex, and even though they are frequently used by developers, they were not developer friendly. We wanted to build a product with the flexibility of NoSQL and the analytical capability of SQL while being super easy to scale and incredibly developer friendly. Our goal was to enable a brand new developer to confidently build a globally distributed scalable application with HarperDB as the back-end. What do you love about your team, and why are you the ones to solve this problem? Our team is amazing, and the core team has worked together for a very long time. This has allowed us to build a really strong company culture. Everyone gets along extremely well and embodies our core values. We look for people who we can empower that are authentic, transparent, kind, and capable of taking accountability. I think we are the ones to solve the problem because we were determined and far too stupid to know what we were getting into. Building a database is hard. Much harder than we expected. We are not MIT grads, we didn’t take any classes from Andy Pavlo at CMU on databases, and that is exactly what makes our product unique and awesome. Kyle has a degree in exercise management and mine is in theology. Had we known what we were getting into we wouldn’t have done it. We also would have brought a mindset to the problem that wouldn’t have really added anything new to the space. While HarperDB started as a database it has really morphed into an edge development platform. We built the product we would want to use and I think that shows. If you weren’t building your startup, what would you be doing? Stand up comedy? At the moment, how do you measure success? What are your core metrics? We look at a variety of things. Obviously typical metrics like number of customers, quality of customers, ARR and MRR, but we are also looking at things like freemium adoption on npm and HarperDB Cloud, our ranking in db-engines month over month, and a number of other factors. What’s most exciting about your traction to date? Being featured on HackerNoon? Aside from that, watching developers build amazing products all over the world on top of HarperDB is my favorite part. Watching developers on social media find HarperDB and get excited by it keeps the whole team going. What technologies are you currently most excited about, and most worried about? And why? I am not really worried about any technology, I am of the mindset that all technological advancement is exciting. That said, what people do with it can obviously be scary, but that is a different matter. Specifically for HarperDB we are pretty excited about CRDTs and the capability that will be gained by adding that to HarperDB’s replication. I am also super excited about the release of HarperDB Custom Functions which will give developers the ability to build their entire stack on HarperDB! What drew you to get published on HackerNoon? What do you like most about our platform? Margo, our head of dev rel made me? Just kidding. First and foremost I like the font. I am a bit older and the font makes me feel at home! In addition, I think that the broad range of topics covered that encompass the entire startup journey from a technical perspective is interesting and important. What advice would you give to the 21-year-old version of yourself? You are not that smart. Be humble. Sales are just as important as product, because if you build an amazing product but can’t sell it you don’t have a company. Listen to feedback, no matter how smart you think you are, it doesn’t matter if the world wants something else. Meet people where they are, not where you want them to be. Being kind is easy and much more effective than being an asshole. Hiring cool people and teaching them what they need to know is more effective than hiring really talented Premadonna's. Vote for HarperDB as the . startup of the year from Denver, Colorado

The Importance Of Staying Humble And Strong Company Culture: Stephen Goldberg, HarperDB CEO

Learn how to use these two technologies together to build a REST API This article will teach you how to use Node.js, Fastify and HarperDB to build a course management system. This application will help you track the courses you are doing and the courses you plan to do. You will use the following technologies: Node.js HarperDB Fastify Introduction When it comes to Node.js and Fastify, people are familiar with these technologies. However, when it comes to databases, there is a new kid on the block! Recently, I took a break from the databases I usually use to play around with HarperDB. is a distributed database with NoSQL and SQL capabilities. The thing about HarperDB is that you perform all the CRUD database operations using one endpoint. HarperDB most fascinating only The most notable features of HarperDB are: using a single endpoint execute SQL queries on JSON data it removes the need for an ORM by returning the results as JSON arrays the ability to insert data via JSON, CSV or using SQL Does not that look interesting? Prerequisites Before going further, there are some prerequisites you should be aware of. The tutorial assumes basic knowledge of: Node.js and FastifyJS Databases JavaScript Knowing how to use a package manager like npm Additionally, you should have a Node version of 12.xx or higher for HarperDB to run. Lastly, you need to have a HarperDB account, which you can create . Do not worry; it's ! here free Set up the folder structure The first step of the tutorial is to set up the folder structure of the application. Create a new folder called and open it. You can do it with the following commands: course-management mkdir course-management
cd course-management The basic folder structure is ready. Of course, when you build each part of the application, you'll create more folders. Install dependencies At this point, you should be in the folder. Let's initialise the project with the following command: course-management npm init -y The flag generates the "package.json" automatically without you having to answer the usual questions about the project name, description and so on. If you open "package.json", you can see that all fields are autocompleted. -y Install the required packages The next step is to install the packages needed to build the application. You can install multiple packages in one line. Run the command below in your terminal: npm i fastify harperive dotenv --save The command installs three packages/dependencies: - FastifyJS is a new web framework, and it claims to be the fastest web framework in town. Fastify - It's the Node.js driver for HarperDB. Harperive - A module that loads environment variables from an .env file to process.env. Dotenv Configure the Fastify server Before proceeding to configure the server, you need to create a few files. The first step is to create the following files: touch app.js
touch .env The file contains the server configuration. On the other hand, the file stores your environment variables. Open the file and write the following code: app.js .env app.js app = ( )({ : }); ( ).config()

app.get( , (req, res) => {
    res.send({ : });
});

app.listen(process.env.PORT, (err, addr) => { (err) {
        app.log.error(err);
        process.exit( );
    }

    app.log.info( );
}); const require 'fastify' logger true require 'dotenv' '/' hello 'world' if 1 `Your server is listening on port 🧨` ${process.env.PORT} With the first line, you import the Fastify package and store it in a constant named . In the same line, you also enable the . After that, you import so you can use environment variables. app logger dotenv Lastly, you create a GET route that responds with JSON, and then you listen on the port specified in the file. .env Moving on, open the file and add the following lines: .env PORT= INSTANCE_URL=
INSTANCE_USERNAME=
INSTANCE_PASSWORD=
INSTANCE_SCHEMA= 3000 The fields starting with will hold information about the database. Now, it's time to create a HarperDB account if you do not have one already. INSTANCE Create and configure the database First of all, login into the . After you log in, click on the button, which creates a new HarperDB Cloud Instance. Figure 1 illustrates that! HarperDB Studio + Figure 1 After that, you will be asked to choose either a "New HarperDB Cloud Instance" or "Register User-Installed Instance", as shown in figure 2. Figure 2 Click on the button saying " ". As a result, you create a database instance in their cloud rather than hosting it yourself. After that, you need to enter: Create HarperDB Cloud Instance the name of the instance a username a password In figure 3 below, you can see my instance information. Figure 3 The next step is to choose your . For this tutorial, I advise you to choose the free options. You get the following specs for free: instance specifications 0.5GB RAM 1GB of storage and 100 IOPS Figure 4, below, illustrates that! However, if you plan to use the DB heavily, I recommend powering up the instance and choose better specifications! Figure 4 The last step is to confirm the instance details and then wait for the instance to spin up. After the instance is set up and initialised, click on the instance. For example, you can see my newly-created instance in figure 5 below. If you are accessing the instance for the first time, you might be prompted to enter the username and password you set previously. Figure 5 Your instance does not have any schemas or tables at the moment. As a result, you need to create a schema first! You can see that I chose as the name for my schema (figure 6). However, you can use any other name for your schema. mydb Once you chose a name for your schema, click on the green checkmark to save it! Figure 6 Lastly, you need to create tables. For the moment, you'll set up only one table called . Enter the name "courses" and provide a name as well. The "hash attribute" is used to uniquely identify each record. In simpler words, the "hash attribute" is the ID, which is unique for each record. Now, click on the green checkmark to save it. As shown in figure 7. courses hash_attribute Figure 7 Now, you are ready to use the database in the Node.js application! Set and config HarperDB in the app Go into the .env file and add the following lines: INSTANCE_URL=https: INSTANCE_USERNAME=admin
INSTANCE_PASSWORD=password
INSTANCE_SCHEMA=mydb //app-tutorial.harperdbcloud.com/ Of course, you need to change those details with yours! The above information is fictive, so it will not work if you try! Save the file, and let's move on! Now create two folders and a database file. You can do so by following these commands: mkdir src
mkdir src/db
touch src/db/db_config.js With the above commands, you create the "db" folder inside "src". Also, you create the file "db_config.js" that stores the configuration for the database. Open the newly-created file and write the following code: db_config.js harperive = ( ).Client; DB_CONFIG = { : process.env.INSTANCE_URL, : process.env.INSTANCE_USERNAME, : process.env.INSTANCE_PASSWORD, : process.env.INSTANCE_SCHEMA
}; client = harperive(DB_CONFIG); .exports = client; const require 'harperive' const harperHost username password schema const new module With the first line, you import the Node.js driver for HarperDB. After that, you create the database configuration with the details stored in the file. Lastly, you create a new client and export it. .env You'll use the client later, in the controller file! For now, let's move to the routes and controllers. Configure app endpoints The first step is to create two new folders in the folder. Create them as follows: src mkdir src/controllers
mkdir src/routes After that, you need to create two new JavaScript files in the newly-created folders. Create the files and as follows: course.js courseController.js touch src/routes/course.js
touch src/controllers/courseController.js You'll start by creating the routes. In this scenario, you can make use of the FastifyJS plugin architecture. So open the file from the "routes" folder, and write the following code: course.js courseController = ( ); {

} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts module There are three things happening: You import the controllers used for routes. You create a routes function. You export the newly-created routes function. In this method - routes - you configure the endpoints of your application. For each route, you specify the , the and the at minimum. HTTP method URL handler However, there are many other parameters. You can see the complete list here . With that being said, let's build the first endpoint that returns all the courses when accessed. You can do so by calling the method on the , as follows: route app courseController = ( ); { app.route({ : , : , : courseController.getCourses
    });
} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts // Get all courses method 'GET' url '/courses' handler module You build all the other routes/endpoints in a similar fashion. The only things that change are the method, URL and handler. In the Gist below, you can see all the endpoints. You have an endpoint for each CRUD operation. Application users can see all the courses, a specific course, add new courses, delete and edit existing courses. courseController = ( ); { app.route({ : , : , : courseController.getCourses
    }); app.route({ : , : , : courseController.getSpecificCourse
    }); app.route({ : , : , : courseController.addCourse
    }); app.route({ : , : , : courseController.editCourse
    }); app.route({ : , : , : courseController.deleteCourse
    });
} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts // Get all courses method 'GET' url '/courses' handler // Get a specific course method 'GET' url '/courses/:id' handler // Add a course method 'POST' url '/courses' handler // Edit a course method 'PUT' url '/courses/:id' handler // Delete a course method 'DELETE' url '/courses/:id' handler module : Each route/endpoint has a specific controller, as you can see. At the moment, they are just placeholders because there are no controllers. That means, if you try to access the routes they won't work! But that changes in the next step because you'll build the application logic now. Note Go to the in the root folder, and add the following line: One more thing before going further. app.js app.register( ( )); require './src/routes/course' The purpose of that line is to make the routes available in your application. If you do not add this line, your routes will not work. The final version of should look as follows: app.js app = ( )({ : }); ( ).config()
app.register( ( ));

app.listen(process.env.PORT, (err, addr) => { (err) {
        app.log.error(err);
        process.exit( );
    }

    app.log.info( );
}); const require 'fastify' logger true require 'dotenv' require './src/routes/course' if 1 `Your server is listening on port 🧨` ${process.env.PORT} Build application logic with HarperDB The application logic defines what happens when a user makes a request to the routes specified in the previous section. As an example, when someone accesses the route , they should see all the courses from the database. As a result, let's start with that route. /courses The code for all routes will be in . Thus, open the file and import the "Harperive" client at the top of the file: courseController.js courseController.js client = ( ); const require "../db/db_config" Now, this where the fun begins. To portray the capabilities of HarperDB, you'll use both SQL and NoSQL operations to manipulate data. Build the GET route In the same file, add the following code after importing the database client: getCourses = (req, res) => { allCourses = client.query( );

    res.send({ allCourses });
}; .exports = {
    getCourses,
} const async const await 'SELECT * FROM mydb.courses' module As you can see, the HarperDB Node.js driver allows you to run SQL queries on your database. The above query selects and returns all the courses from the database. After that, you use the object to send the data to the client. If you run the application and try to access the route, you won't get any data because the database is empty. response /courses Figure 8 Figure 8 illustrates a successful request. You can see that the "data" array is empty because the database is empty. However, we'll repeat the request once there is data in the database. Build the POST route The next step is to build the POST route so you can add data into your database. In this case, we'll use a NoSQL operation. Write the following code after the constant: getCourses addCourse = (req, res) => { allProperties = .keys(req.body).length; (allProperties !== ) {
        res.code( ).send({ : }); ;
    } { newCourse = client.insert({ : , : [
                { : req.body.name, : req.body.description, : req.body.author, : req.body.link
                }
            ]
        });

        res.send({ newCourse });
    } (error) {
        res.send({ error });
    }
}; .exports = {
    getCourses,
    addCourse,
} const async const Object if 4 400 error 'Some properties are missing! Add the name, description, author and link!' return try const await table 'courses' records name description author link catch module First of all, the part before the block makes sure the user provides all the fields. If the user tries to add a course without specifying the , , or , it will return an error. The user must always provide all four fields. try catch name description author link Then, in the block, you use the database client to insert the new course into the database. try catch - You need to specify the table (which in this case is ) and then the . courses records - The field is an array of objects, which means you can enter multiple courses in one go. However, for now, one is enough. records Provided the insertion is , it returns the newly-added course. You can see the route in action in figure 9 below. successful Figure 9 The operation was successful, and the new course was added to the database! Figure 10 shows the course in the database! Figure 10 GET a specific course Getting a specific course is similar to getting all courses, but with one addition. Now, you use a clause to select a specific course. WHERE You pass the course "ID" in the URL, and then the SQL query matches that with a record from the database. But first of all, , you might ask! There are two options ( ): how do you get the ID? you need courses in the database first 1. You can visit the route and get the ID of any course you want. http://localhost:3000/courses 2. Go to the HarperDB studio, click on the course, and you'll see all the details. At the top of the page, you'll see that it specifies which one is the ID - see figure 10 above. getSpecificCourse = (req, res) => { course = client.query( );

    res.send({ course });
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
} const async const await `SELECT * FROM mydb.courses WHERE id=" "` ${req.params.id} module After that, it returns the course. Figure 11 below illustrates what happens when you make a GET request to get a specific course! Figure 11 DELETE a specific course You can delete a course by providing the course ID in the URL. For example, if you make a request to , you delete the course with the specified ID. DELETE http://localhost:3000/courses/0eb95456-255b-44da-b3df-b90671ef0908 deleteCourse = (req, res) => { course = client.query( );

    res.send({ : , : course
     });
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
    deleteCourse
} const async const await `DELETE FROM mydb.courses WHERE id=" "` ${req.params.id} message 'Course deleted!' deleteCourse module Once the course is deleted, it sends a message confirming that and some more extra information. Figure 12 illustrates a "DELETE" request. Figure 12 Build the PUT route Lastly, this route allows you to update an existing course. All the code before the block makes sure the users only use the allowed fields - , , and . If the user tries to update a field that does not exist, the server will respond with an error. try catch name description author link To edit the course, you're using a NoSQL operation again. You call the method on the database client, then pass the table name and the records you want to update. update (courses - in this case) Also, it's essential to specify the in the records array. If you do not, the server does not know which record to update. id editCourse = (req, res) => { updates = .keys(req.body); allowedUpdates = [ , , , ]; isValidOperation = updates.every( allowedUpdates.includes(update)); (!isValidOperation) {
        res.code( ).send({ : }); ;
    } { updatedCourse = client.update({ : , : [
                { : req.params.id, : req.body.name, : req.body.description, : req.body.author, : req.body.link
                }
            ]
        });

        res.send({ updatedCourse });
    } (error) {
        res.send({ error });
    }
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
    deleteCourse,
    editCourse
} const async const Object const 'name' 'description' 'author' 'link' const ( ) => update if 400 error 'Not a valid operation! ' return try const await table 'courses' records id name description author link catch module If there is an error, the server sends back the error. Otherwise, it returns the updated course. The figure below shows how to make a PUT requests and what happens it's successful! Figure 13 That's all about routes and controllers! You can see the whole file in . I chose not to embed it because it is big and takes a significant space on the page! courseController.js this Gist Similarly, you can . To learn better, go through it, change stuff, break stuff, add things and so on! see the complete application on my GitHub How to start the application You can start the application by running node in the root folder. app.js However, if you clone the application from GitHub, you need to install the dependencies first. You can do so by running: npm install
node app.js Conclusion By this point, you should have a REST API that allows you to store and manipulate courses. The purpose of this tutorial was to get up-to-speed with the two technologies. As a result, other things could be added, such as: Authorization API rate limiting Adding a front-end and transform it into a full-stack application With that being said, if you like the tutorial and want part 2, where I build the front-end with Vue, let me know in the comments! Previously published at https://catalins.tech/build-a-rest-api-with-harperdb-and-fastifyjs

How to Create a RESTful API With HarperDB and FastifyJS

Many people learn or understand new things relative to things they already know. This makes sense; it’s probably a natural instinct. When it comes to products and technology, a lot of people ask “how are you different,” but different from what? You need some sort of baseline to start from, so you can say, “Similar to X, but different because of Y.” Because of this, comparisons, competitive analysis, and feature matrices are a great way to understand which technology solutions are right for you. So today, let’s do a comparison of three different database systems. The point of this article is not to determine which database is the best but to help uncover the factors to consider when selecting a database for your specific project. In most cases, it’s not that one database is better than the other, it’s that one is a better fit for a specific use case due to numerous factors. With MongoDB and PostgreSQL being two of the most popular tools out there, you may already know that there are tons of resources comparing the two. However, with HarperDB being a net new database, I thought it might be helpful to throw it in the mix to provide further clarity. Referring to my overview post: It’s important to understand things such as data type/structure, data volume, consistency, write & read frequency, hosting, cost, security, and integration constraints. That article provides a great high-level explanation across all different types of databases, but today we’ll get more specific. database architecture Similarities These technologies are all similar in that they are used to store data, but that simple concept is where the similarities end. However, while MongoDB and PostgreSQL are actually quite different from one another, HarperDB lies somewhere in the middle. As mentioned, there are numerous resources out there comparing MongoDB and PostgreSQL, which are both awesome databases. from Educative is one great place to start for understanding the differences between them. Therefore, to avoid redundancy, in this post I will focus a bit more on HarperDB compared to the two. This article Definitions is classified as a NoSQL database. It is document-oriented and uses JSON-like documents with optional schemas. MongoDB is a traditional RDBMS (relational database management system). Mainly used for relational data, it is object-oriented in nature. PostgreSQL is a geo-distributed database with a REST API and dynamic schema, that supports NoSQL and SQL including joins. (For example, you can ingest data via NoSQL JSON then immediately query it via SQL) HarperDB Differences Overview : PostgreSQL is a relational database handling more complex procedures, designs, and integrations. MongoDB is a NoSQL database often used for simpler, more unstructured data, great for app development. Ultimately, PostgreSQL enforces schema validations whereas Mongo does not. MongoDB vs. PostgreSQL : MongoDB is a document store that is great for unstructured data, whereas HarperDB offers full document store capability plus enterprise-grade ACID SQL. Benchmark tests found that HarperDB is 37 times faster than MongoDB at less than half the price. HarperDB also has a native REST API, supports SQL on JSON, and can be easier to use and manage. See the . HarperDB vs. MongoDB full benchmark here (Mongo is optimized for high scale writes, but not for reads. HarperDB's data storage algorithm written on top of LMDB enables both high scale reads and writes, resulting in high performance overall.) : HarperDB is more flexible than PostgreSQL, which is a great technology for complicated data or strict consistency, but HarperDB has simplified much of the work on the installation, configuration, and administration. HarperDB is allowing developers from relational backgrounds to use their existing knowledge with SQL with a database that also allows their team to use NoSQL from the same data model. HarperDB v. PostgreSQL Accessibility MongoDB, PostgreSQL, and HarperDB can each run anywhere; in the cloud, locally, data center, etc. (There is not a PostgreSQL cloud like there is with HarperDB Cloud and MongoDB Atlas, but some cloud providers offer PostgreSQL-as-a-service.) Under the hood MongoDB is mainly programmed in C, C++, and JavaScript language. PostgreSQL is mainly programmed in C. HarperDB is (LMDB is written in C, and Python is used in the install process). written in Node.js MongoDB uses BSON (binary JavaScript Object Notation) under the hood to make querying fast. This is then translated to JSON to be read when you open your collection. PostgreSQL ultimately employs SQL, a structured query language, to define, access, and manipulate the database. PostgreSQL also has a JSON datatype. HarperDB does not enforce data types; it currently stores all data attributes as strings, and data can be queried via SQL and/or NoSQL. (HarperDB is also working on enabling the ability for administrators to explicitly set attribute types (as numbers or strings) for performance tuning). Data Storage & Architecture MongoDB stores data as individual documents without regard to attributes, PostgreSQL stores data in traditional tables and rows, and HarperDB stores data in tables and rows/objects with all top-level attributes indexed by default. HarperDB has a unique data storage algorithm running on top of LMDB, this enables HarperDB to ingest JSON documents and relational data in a single product. As data comes in, HarperDB maps it to the data model; it's not a SQL engine or a NoSQL engine. Regardless of how data is ingested, data is stored according to the HarperDB data model and can be queried via SQL or NoSQL. MongoDB and HarperDB are more distributed architectures, whereas PostgreSQL might be considered a monolithic architecture. ACID Properties (atomic, consistent, isolated, & durable) HarperDB and PostgreSQL both have enterprise-grade ACID SQL transactions, meaning the validity of data is quite reliable. NoSQL databases like MongoDB usually adopt eventual consistency instead of ACID properties. (A study from May 2020 identified a bug that disputes claims that Mongo is ACID compliant, as MongoDB’s transactions are not fully isolated). HarperDB and PostgreSQL both follow the ANSI SQL standard. Schema & Tables With both MongoDB and HarperDB, using JSON allows you to change your schema flexibly without consequence. Documents can vary in terms of key/value pairs. Both MongoDB and HarperDB scale horizontally, which allows for speed. HarperDB has bidirectional table-level data replication. HarperDB uses a simple pub-sub model; data is replicated by publishing data to different “chat rooms” which different nodes subscribe to and are able to be distributed horizontally. PostgreSQL scales vertically (as it gets bigger, more space or more memory is needed), therefore it requires downtime to upgrade. With relational databases like PostgreSQL, altering your table is necessary to make any changes. The whole schema needs to be designed and configured at creation. You might be able to alter a table later on, but this may lead to database downtime and bugs in your application. PostgreSQL databases can use foreign keys which explicitly link data between tables and are used to keep the data normalized. Use Cases & Summary As stated in , because transactions in PostgreSQL follow ACID properties, it’s a good choice for industries such as fintech. When you absolutely need to control the state of your data, use a relational database like Postgres. Alternatively, if you only have unstructured data, or are working with big data, it might be a good idea to use the horizontal scaling approach with a tool like MongoDB. this article Use cases where HarperDB might be a better fit than existing systems include projects where you need SQL and NoSQL, rapid application development, hybrid cloud, integration, edge computing, and distributed computing. Our team will also be the first to tell you when HarperDB is not a good fit for your specific project. HarperDB is not recommended when you need full-text indexing, highly structured relational data, or strict consistency across systems. Both MongoDB and PostgreSQL are really great tools for certain use cases. They are loved by many and the team at HarperDB is thankful that they have paved the way for technologies like ours to exist. HarperDB was created to expand and blend the capabilities of awesome SQL, NoSQL, and products on the market to fill in the gaps and ensure that developers are continuously empowered to use the right tool for the job. NewSQL Sources https://www.educative.io/blog/mongodb-versus-postgresql-databases https://harperdb.io/ https://en.wikipedia.org/wiki/MongoDB https://en.wikipedia.org/wiki/PostgreSQL https://www.techshout.com/software/2020/19/mongodb-vs-postgresql-what-are-the-similarities-and-differences/ Also published at https://dev.to/harperdb/harperdb-vs-mongodb-vs-postgresql-49en

Which Database Is Right For You? HarperDB vs. MongoDB vs. PostgreSQL

Meet Stephen Goldberg Today, we hear from , the CEO of , Stephen began his career interning for Sequoia Software in the mid-’90s. He soon found success in the tech space, working as a CTO and CEO of multiple startups and in other roles at larger organizations like Red Hat. He is well-recognized as a leader in the industry. Stephen Goldberg HarperDB Denver’s Startup of the Year Winner. Q&A What does it mean for you to win this title? We are thrilled about winning Hackernoon's 2023 Startup of the Year! For us, community-driven awards are the most authentic and meaningful awards of all. It is further validation of what we have developed and continue to build upon as a company. We are grateful to all those who supported us in winning this award. What sets you apart from the competition? HarperDB was founded by developers who spent decades fighting the limitations of multi-system architectures. After exploring the logic behind systems like MongoDB, Kafka, Redis, and Node.js, it was apparent that combining the underlying systems into one distributed technology would have significant benefits from a performance, cost, and ease of development perspective.  From that realization, we created our unified system architecture that backs HarperDB. Aside from its performance advantage, I think HarperDB’s simplicity motivates companies to make the transition to our technology. We’ve seen distributed applications built in weeks with HarperDB that easily could have taken several quarters to plan and build with other technologies. Ultimately, HarperDB was a database built by developers for developers.  We built a product we would want to use. As our product continues to advance while we are focused on solving large enterprise problems, we still want to create a product that seamlessly integrates into a developer's workflow and is fun to build and deploy. What do you love about your team, and why are you the ones to solve this problem? Our team is made up of amazing people. One of our core values is authenticity, and what I love about our culture and our team is that people bring their authentic selves to HarperDB. We have a team of weirdos and misfit toys who are extremely passionate about innovation and solving complex problems.  We like working on hard things, and we love delighting developers and customers. Our team is made up of people who do not let existing paradigms and patterns stop them but will push through to create something new and incredible. If you weren’t building your startup, what would you be doing? Sleeping. We have been building HarperDB for almost seven years.  It has been an incredible journey, and we have had many amazing moments and a lot of hard ones. At the end of the day, everything that we have achieved has been due to the people who make up our team.  When we founded HarperDB, it was all about technology, and while that remains a huge part of what we do, I have learned that people matter more. If I weren’t running HarperDB, I would be doing something to help improve people's lives in meaningful ways. What that looks like exactly, I am not sure. At the moment, how do you measure success? What are your core metrics? From a business development perspective, proofs-of-concept (PoCs) are our number one metric. Although we also track npm and Docker downloads, PoCs are the key metric leading to business success. Since what we do is a first for the software development industry, people need to experience the benefits firsthand. We find that engineers tend to experience a light bulb moment with HarperDB once they fully internalize the breadth of value a unified system architecture can bring. After the light bulb moment, HarperDB tends to be part of nearly every deployment since building without it feels cumbersome and counterproductive. Tell us how your startup is changing the world. HarperDB changes the world on many levels. For , we simplify their workflow by giving them more capabilities with less effort. software engineers For , we empower innovation by providing them with systems that improve their agility and cost models. companies For , we make the technology they use substantially more performant, cutting down load times and delivering real-time experiences they did not have before. software users For the , we dramatically cut down on the infrastructure required to deliver software. Sometimes, by as much as 90%, making platforms that run on HarperDB more sustainable than systems that run on legacy alternatives. world How do you or your company intend to embrace the responsibility of this title in 2024? This title is a recognition that we are building a tool developers want to use. To embrace that, we will continue delivering features and performance improvements that help drive their software products forward. Today, nearly every feature we release is a direct response to requests from development teams worldwide. We will continue to serve them cutting-edge features that make their processes and products more performant. Furthermore, we want to continue to build a company where developers want to work.  We are successful due to the passion of the great technologists who have joined us on this journey, and we want to continue building a developer-led, techno-centric, authentic organization. What goals are you looking forward to accomplishing in 2024? 2023 was a breakthrough year for HarperDB in many ways. Our product gained significant traction with three releases that embedded data streaming capabilities deep into our technology. This was a welcome addition for many companies courting standalone data streaming solutions since they now had all the power they needed without the multi-year journey standalone streaming solutions required. 2024 so far has been customer and market-focused.  We have seen a strong reaction in the market to our product and tremendous validation, with many large organizations implementing mission-critical workloads on HarperDB.  We plan to spend 2024 growing our customer base while continuing to innovate. Which trend(s) are you most excited about in 2024? Share your reasons. For the last several years, people have been exploring what distributed computing really means. We have seen containers explode in their usage and other distributed deployment methodologies.  That said, that has created new paradigm shifts in how people think about application development, compute, deployment, security, and many other areas. We are seeing 2024 as the year where experimentation becomes a reality.  Organizations, large and small, are embracing new architectures that unlock the true power of distributed computing to improve customer experiences, do more with AI and ML, reduce cost, and build high-availability architectures. In a lot of ways, this leads to the democratization of technology.  It allows people to build bigger and better without being locked into a single cloud vendor.  I am most excited about the things people will build with this newfound freedom. 2023 had been another crazy year, especially in tech, with layoffs and the Generative AI takeover! Which trend are you most concerned about? Be as brief or as detailed as you like. We have had a pretty clear vision for where we wanted to take HarperDB from the beginning and what we wanted to build both as a product and as a company.  We try to stay focused on putting one foot in front of the other.  Ultimately, trends in the tech industry are cyclical, and there are booms and busts.  High valuations and low valuations.  Lots of hiring and lots of layoffs. Quite honestly, the trend that worries me the most is worrying about trends. People keep building great stuff.  Great people need jobs.  Good companies need funding.  Large companies need help solving big problems.  These things remain the same regardless of the current trends. I think the thing I worry about the most is people losing sight of the fact that this will all repeat itself. Share your biggest success so far and/or your biggest failure so far. Our biggest failure by far was that we were way too early for the market.  People thought we were CRAZY when we started HarperDB.  We were too early because we didn’t realize the problem we founded HarperDB to solve was not a common problem at that time, and only a handful of companies were experiencing it ten years ago. We should have done more research, talked to more companies, and paid closer attention to the market.  We were just so enamored with solving the problem that we didn’t do our homework.   We let passion get in the way of reason. Our biggest success has been our resilience as founders and as a team as a whole.  We are extremely proud of the team we have built and have had relatively little turnover in our time. We have also managed to stay alive and continue to build a tremendous product to reach the point where the market has caught up with our crazy ideas, which have become a commonplace everyday problem. We would love your feedback on HackerNoon as a tech publication! How has your experience been with us? We love HackerNoon! We have enjoyed using HackerNoon as a publication platform. In addition to our own articles, we see many independent creators writing HarperDB-related content for your site. It’s this robust community that has driven us to Hackernoon. Any words of wisdom you’d like to share with us? Be very careful who you take advice from.  We have gotten a lot more bad advice along the way than good advice.  It has cost us a lot of money and a lot of time.  We have gotten some really good advice as well.  Ultimately, if you're getting advice and following it feels inauthentic to who you are or the vision or values of your company, ignore it.  Trust your gut. This interview series celebrates all the phenomenal . Congratulations! Well-Deserved, Champion.The HackerNoon community is excited to learn more about you! Startups of the Year Winners Startups of The Year is HackerNoon’s Flagship community-driven event celebrating startups that survived and thrived in 2023. 30,000 startups across 4200+ cities and six continents participated this year to be crowned the best startup in their city. See our global winners announcement . here Companies that blog more get 67% more leads - Jumpstart your company’s blogging journey by purchasing our . Business Blogging Package

Meet HarperDB, Winner of the Startups of the Year in Denver

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, HarperDB has been nominated in HackerNoon's annual Startup of the Year awards in Denver, USA! Please vote for us here: https://hackernoon.com/startups/north-america/north-america-denver-co-usa Read more about us below to understand why we deserve your vote. Meet HarperDB HarperDB is a database and application development platform that is focused on performance and ease of use. With flexible user-defined APIs, a simple HTTP/S interface, and a high-performance single-model data store that accommodates both NoSQL and SQL workloads, HarperDB scales with your application from proof of concept to production. Install and manage on your hardware, or have us host it with . HarperDB Cloud Fun fact: HarperDB was named after our CEO's pup named Harper! We believe dogs symbolize loyalty, trust, and dependability, so we took these characteristics and built the brand around Harper and all of our other loyal employee pups. :) My Role I’ve worn a few different hats in my 4+ years at HarperDB, and currently am the Head of DevRel & Partnerships. This includes community engagement, content creation, and supporting our users to ensure that they have everything they need to be successful. I also manage our partnerships from a high level, coordinating training, events, timelines, etc., to make sure both parties are working towards their joint and individual strategic goals. I’m also the host of the and have written a . 😁 Select* podcast few articles How We're Disrupting/Improving the Software Dev/Data Management Industry HarperDB is disrupting the data management industry in a few ways. HarperDB reduces complexity, increases performance, and lowers costs by combining user-defined applications, a high-performance database, and an enterprise-grade message queue into a single package. The platform offers unlimited horizontal scale at the click of a button and synchronizes data across all the nodes in a cluster in milliseconds. HarperDB simplifies the process of delivering applications and the data that drives them to the edge, which dramatically improves both the user experience and total cost of ownership for large-scale applications. Deploying HarperDB on global infrastructure enables a CDN-like solution for enterprise data and applications. Standing Out From the Crowd - Streamlining the Journey to the Edge Even though edge computing is becoming mainstream, no one is talking about the complexity of deploying distributed applications and data to the edge. There are a lot of great articles about the edge being the future, and for the most part, edge companies are addressing this demand similar to how companies addressed the cloud. However, the edge is a fundamentally different problem. Once you consider the exponential complexity of building multiple data stores, deploying APIs that load balance across geo-distributed data stores, and the sneaky complexity of keeping everything in sync, there are very few companies that have actual solutions for the edge that won't stifle future innovation with mountains of system complexity. HarperDB is the only solution that actually decreases complexity while moving systems to the edge, and the only platform that gives innovation room to flourish while moving to the edge. Our Predictions/Thoughts on the Software Dev/Data Management Industry in 2023 2023 is poised to be a breakthrough year for Unified Backend technology in the Software Development Industry, heralding a monumental shift that will forever change the landscape of development. This transformative innovation blurs the line between edge and cloud, effectively commoditizing infrastructure. Unified Backend technology combines the finest elements of best-in-class database management, API server functionality, and data streaming capabilities, unifying their underlying systems at the core. Imagine a world where developers no longer grapple with the complexities of integrating disparate backend components, but instead embrace the power of a Unified Backend. In this new era, ease, scalability, and unprecedented performance take center stage. Applications seamlessly tap into a unified system processing data at the edge or sending real-time messages. The result is a paradigm shift, enabling faster development cycles, reduced latency, and lower infrastructure cost than ever before attainable. We predict that embracing the transformative power of Unified Backend technology is the key to forever changing the way software development is approached and executed, shaping the future of the industry. Why We Decided to Participate in HackerNoon’s Startup of the Year Awards We decided to participate in the awards because we love the HackerNoon community, and who doesn’t like a good ole’ friendly competition? Final Thoughts Thanks for reading! Please feel free to reach out with any questions/comments; we would love to hear from you. Vote for us today! https://hackernoon.com/startups/north-america/north-america-denver-co-usa

Startups of the Year 2023: Meet HarperDB - A Database and Application Development Platform

Suppose you are searching for information on a website. Let’s imagine a Twitter user writes about CRYPTOCURRENCY! What do you do? You can copy and paste the tweets about CRYPTOCURRENCY into your own file. But what if you want to retrieve massive volumes of information from Twitter? Such as vast quantities of information for your data science project? In this circumstance, copying and pasting won’t work! Then you will need to utilize Web Scraping. What is Web Scraping? The term “web scraping” refers to an automated process that can collect significant volumes of data from websites. The majority of this data is unstructured data that is stored in an HTML format. In order for this data to be utilized in a variety of applications, it must first be converted into structured data that is stored in a spreadsheet or a database. For many businesses, web scraping can be used to quickly and inexpensively gather data that can then be analyzed in a variety of ways such as news monitoring, sentiment analysis, email marketing, and others. Web scraping, the process of obtaining data from websites through automated means, can be carried out in a variety of different methods. Application Programming Interfaces (APIs) e.g from Twitter, StackOverflow & Google. Write code (e.g in the Python programming language). Online services from different providers e.g Octoparse. In this article, you will learn how to: How to execute web scraping on Twitter using the Python library. snsscrape How to store scraped data automatically in the database using . HarperDB How to share your data via API call by using Custom Function from . HarperDB So let’s get started. What is Snscrape? snscrape is a scraping tool for social networking services (SNS). It scrapes information like user profiles, hashtags, searches, and threads and returns the discovered items, e.g. the relevant posts. It was released on July 8, 2020, and it is capable of scraping data from a variety of platforms, including the following: Twitter Instagram Reddit Facebook Weibo Telegram Mastodon You can use snscrape by typing its command-line interface (CLI) commands into the command prompt/terminal. If you don’t feel comfortable using a terminal, you can use snscrape as a , but this is not yet documented. Python library On Twitter, it can scrape users, user profiles, hashtags, searches, tweets (single or surrounding thread), list posts, and trends. Note: What is HarperDB? is a lightning-fast and versatile platform for managing SQL and NoSQL data. You can put it to work for a wide variety of purposes, some of which include but are not limited to quick application development, distributed computing, edge computing, software as a service (SaaS), and many others. HarperDB HarperDB does not duplicate data, is fully indexed and can run on any device, from the edge to the cloud. Additionally, it may be used with any programming language, such as Javascript, Java, , and others. Python The following is a list of a few of the features that can be accessed with HarperDB: Allows JSON and CSV file insertions. Single endpoint API. Supports SQL queries for full CRUD operations. (Lambda-like application development platform with direct access to HarperDB’s core methods). Custom Functions Limited database configuration required. Math.js and GeoJSON are both supported. HarperDB has a built-in HTTP API, custom functions for user-defined endpoints, and a dynamic schema that can help you easily share your scraped data with your coworkers after storing them in a HarperDB cloud instance. HarperDB allows you to quickly download scraped data held in the HarperDB instance as a CSV file so that you can perform extra analysis before making a final choice. After being introduced to the tools (snscrape & harperDB) that you will use to automate the process of scraping data and saving it in the database. Then all you have to do is follow the steps that are described below Step 1: Create a HarperDB Account We will start by working on the HarperDB database first. You can visit and then click the navigation bar to see a link called “Start Free.” Click it in order to create your account. https://harperdb.io/ If you already have an account, use the following URL to sign in with your credentials. https://studio.harperdb.io/ Step 2: Create a HarperDB Cloud Instance After registration, you need to create a cloud instance to store and fetch your scraped data from Twitter. Click the Create New HarperDB Cloud Instance link to add a new instance to your account. You just need to follow all instructions provided by harperDB to create your cloud instance, such as: Note: Select Instance Types.Choose Cloud Provider. Add instance information. Select instance specification (RAM size, instance storage size, and instance region). Confirm and create a cloud instance. When the HarperDB Cloud Instance has been created successfully, you will see the status as OK for that particular instance, check the image below. Step 3: Configure the HarperDB Schema and Table To add the Twitter data that has been scraped into the database, you must first create a schema and a table. It only requires loading the HarperDB cloud instance you already created from the dashboard and creating the schema by giving it a name (like “data_scraping”). You then have to add a table (e.g ). Additionally, HarperDB will ask you to specify the hash attribute, which is equivalent to an ID number. “tweets” Step 4: Install the Required Packages You need to install the following package on your local machine. This is the we’ll use to implement different HarperDB API functions sucha as inserting data into to the cloud instance. It also provides wrappers for an object-oriented interface. (a) harper-sdk-python Python package pip install harperdb Snscrape requires Python 3.8 or higher. When you install snscrape, the dependencies for the Python package are automatically installed. (b) snscrape pin install snscrape Step 5:Import Important Packages The next step is to import Python packages to scrape data from Twitter and automatically store them on harperDB cloud instance. #import packages #snscrape import snscrape.modules.twitter as sntwitter # harperdb import harperdb import warnings # To ignore any warnings warnings.filterwarnings( "ignore" ) Step 6: Connect to HarperDB Cloud Instance You need to connect to the HarperDB cloud instance in order to insert scraped tweets into the table called . tweets Here you need to provide three parameters: Full URL of the HarperDB instance Your username Your password db = harperdb.HarperDB(url=URL, username=USERNAME, password=PASSWORD) db.describe_all() # connect to harperdb URL = "https://1-mlproject.harperdbcloud.com" USERNAME = "USERNAME" PASSWORD = "PASSWORD" # check if you are connected When you execute the above code, you will see output similar to that displayed below, indicating a successful connection to your HarperDB Cloud Instance. '__updatedtime__': 1660390877630,
   'hash_attribute': 'id',
   'id': 'd140645e-3af2-42d7-8594-2195826dabbc',
   'name': 'tweets',
   'residence': None,
   'schema': 'data_scraping',
   'attributes': [{'attribute': '__createdtime__'},
    {'attribute': '__updatedtime__'},
    {'attribute': 'id'}],
   'record_count': 0}}} {'data_scraping': {'tweets': {'__createdtime__': 1660390877630, Step 7:Create a Function to Record the Scrapped Tweets Using the from the harperdb-python package, the following function will insert the scraped tweets as data (in dictionary format) into the specified table.The insert function will receive three parameters: insert function SCHEMA name TABLE name data (scraped tweets) result = db.insert(SCHEMA, TABLE, [data]) # define a function to record scraped data into the table def record_tweets ( data ): #define the schema and table SCHEMA = "data_scraping" TABLE = "tweets" # insert data into the table return result Step 8:Scrape tweets by using snsscrape Now you can use T from snsscrape python package to scrap tweets with the particular search query. In this example, I will show you how to scrap 1,000 tweets about “ from 1st January 2022 to 13th August 2022. witterSearchScrapper method cryptocurrency” sntwitter.TwitterSearchScraper( data = { } result = record_tweets(data) #1 Using TwitterSearchScraper to scrape data and append tweets to list for i, tweet in enumerate ( 'crytocurrency since:2022-01-01 until:2022-08-13' ).get_items()): if i > 1000 : break #2 save data automatically to the HarperB cloud instance "user_name" : tweet.user.username, "content" : tweet.content, "lang" : tweet.lang, "url" : tweet.url, "source" : tweet.source # insert result into the HarperDB table As you can see from the code block above , harperDB will automatically store scraped data into the tweets table with the following attributes. (comment #2) Username Content Lang Url Source Step 10:View the Tweets Table If you open your HarperDB cloud instance, you will be able to see all records of your scraped data from Twitter. 🎉 You have successfully completed all required steps to automate the process of scraping data and saving it in the database. Congratulations What if you wish to share the scraped information with your colleagues? Custom Function provides a straightforward solution to this problem in HarperDB. What is a Custom Function? A Custom Function is a brand-new feature included in HarperDB’s 3.1+ release. You can use the feature to add your own to HarperDB. Custom functions are powered by Fastify, which is incredibly flexible and makes it simple to interact with your data by using HarperDB core methods. API endpoints You will learn how to use the HarperDB studio to create your very own custom function in this section. You can then use an API call to share the outcomes of your scraped data with your coworkers at the office. Here are the steps you need to follow:- The first step is to enable the Custom functions by clicking “ ” in your HarperDB Studio (it is not enabled by default). 1. Enable Custom Functions functions The next step is to create a project by specifying the name. For example It will also create setting files for the project including: 2. Create a Project tweets-api-v1. Routes folder File to add helper functions Static folder For this article, you will focus on the routes folder. Note: In this step, you will create the first route to fetch some data from the tweets table from the HarperDB Datastore. You also need to know that Route URLs are resolved in the following manner: 3. Define a Route [Instance URL]:[Custom Functions Port]/[Project Name]/[Route URL] It will include: Cloud Instance URL Custom Functions Port Project name you have created The route you have defined In the route file (example.js) from the function page, you will see some template code as an example. You need to replace that code with the following code: server.route({ request.body= { }; }
}); 'use strict' ; module .exports = async (server, { hdbCore, logger }) => { url : '/' , method : 'GET' , handler : ( request ) => { operation : 'sql' , sql : 'SELECT user_name,content,lang,url,source FROM data_scraping.tweets ORDER BY __createdtime__' return hdbCore.requestWithoutAuthentication(request); In the code above, the route is defined with the GET method and the handler function will send an SQL query to the database to get from the ordered by the /tweets-api-v1 user_name, content, lang, URL, and source tweets table __createdtime__ column. Finally, you can now use the route you have defined to get the data from the tweets table. Here you will send an API request by using the Python package. 4. Access data via API Endpoint requests r = requests.get(url = URL) data = r.json() #send an API request import requests # api-endpoint URL = "https://functions-1-mlproject.harperdbcloud.com/tweets-api-v1" # sending get request and saving the response as response object # extracting data in json format for experiment in data: print (experiment) Here is the sample output from the above code. {"user_name": "DailyCryptoTrad","content": "DXY forming a bullish bull flag on the daily - a break out of 106.6 will give crypto red days however if we fail below 105 will give crypto green days - Keep an eye on DXY #DXY #SPY #crypto #btc #eth #bitcoin #crytocurrency #cryptocurrencies https://t.co/AkF8Igf3Uc","lang": "en","url": "https://twitter.com/DailyCryptoTrad/status/1558211511461597188","source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>"},{"user_name": "Ariscrypto1970","content": "@scrypto_1977 @Epayme_uae #Saitama will go parabolic when it happens! This is the #WeAreSaitama and the world are waiting for. 🔥🔥🔥🚀🚀🚀🚀#crytocurrency #DeFi","lang": "en","url": "https://twitter.com/Ariscrypto1970/status/1558200674273345537","source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>"},{"user_name": "dan_nyeche","content": "Cryptocurrency market up 24Hrs. #Bitcoin #Dan_Trades #crytocurrency Emilokan Big Brother Modella FireBoy Giddyfia  GTBank President Obama #gayfish Gen Z Ethereum Chi Chi Obidatti2023 Sapa Lewandoski #HAPPYJAEMINDAY #Jalsa4K #GomoraMzanzi #SheggzOlu𓃵 #ViratKohli𓃵 https://t.co/QbU4ei3MGA","lang": "in","url": "https://twitter.com/dan_nyeche/status/1558188248362467329","source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>"}, With HarperDB, you can quickly and easily build API endpoints to share the scraped data with your team working on the same data science project. Note: Conclusion Congratulations 🎉, you have made it to the end of this article. You have learned: How to execute web scraping on Twitter using the Python library. snsscrape How to store scraped data automatically in the database using . HarperDB cloud instance How to create a from the HarperDB cloud instance to share your scraped data with your coworkers working on the project via an API endpoint. custom function If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter @Davis_McDavid.

How to Web Scrape Using Python, Snscrape & HarperDB

Adding authorization & authentication to an app is a common task for developers. When you're launching a new product, a startup, or a side project, it can feel like a mission. This tutorial will teach us how to make a music player app that uses SuperTokens for authentication and HarperDB for the backend. Some of the app's most essential features On Backend Side: Secure your API with almost significantly fewer and zero configurations. Secure your third-party API on the server side using SuperTokens. Session management is simple. On the Front-end side: For login, use SuperTokens' library component, which is available as an npm package. There's no need to develop any extra code for authentication. There's no need to maintain tokens on the UI side. You won't have to worry about managing token refresh calls on the UI side because SuperTokens will take care of it for you. TL;DR Here are the links to get to the source code or learn more about both products faster Source Code SuperTokens SuperTokens GitHub HarperDB What you'll discover How to add authentication using SuperTokens. How to secure 3rd party (HarperDB) endpoints. Let's talk about SuperTokens SuperTokens is an open-source authentication. They help you launch quickly so that you can focus on your core product offering. SuperTokens is 100% open source. Developers can own and manage their user's data. SuperTokens can be run on your premise for free and also has a generously hosted tier for those who don't want to manage it themselves. SuperTokens is a versatile authentication and authorization solution for your applications. Prerequisites This tutorial assumes the reader has the following: Node installed on their local development machine. You can verify if you do by running the command below in your terminal. node -v If otherwise, download from here A basic knowledge of HTML, CSS, JavaScript, and React Take a peek at how we developed our React app and used SuperTokens to authenticate it. A summary of the project's setup Yay! This section will create a React music player app and incorporate SuperTokens and HarperDB. Alternatively, you can utilize the start directory as your project root by cloning the . It includes the whole project setup that will get you started. GitHub repository How to Begin Using SuperTokens SuperTokens provides a variety of ready-to-use recipes from which to choose. You can see all the recipes. here We'll use the pre-built Recipe to access the , which will look something like this. ⬇ EmailPassword demo app The supertokens website has the documentation for this recipe. Click here Now, fork the GitHub repository Let's have a look at all of the project dependencies that were used to make this app Dependencies used:

"dependencies": {
        "axios": "^0.21.0",
        "body-parser": "^1.19.0",
        "cors": "^2.8.5",
        "express": "^4.17.1",
        "helmet": "^4.6.0",
        "morgan": "^1.10.0",
        "mysql": "^2.18.1",
        "npm-run-all": "^4.1.5",
        "react": "^17.0.1",
        "react-dom": "^17.0.1",
        "react-router-dom": "^5.2.0",
        "react-scripts": "4.0.0",
        "supertokens-auth-react": "^0.17.0",
        "supertokens-node": "^8.0.0",
        "web-vitals": "^0.2.4"
    },
    "devDependencies": {
        "@testing-library/jest-dom": "^5.12.0",
        "@testing-library/react": "^11.1.0",
        "@testing-library/user-event": "^12.6.2",
        "nodemon": "^2.0.12"
    }, Using the SuperTokens npm package in a React application The following use cases are demonstrated in this demo app. Login Sign up Logout Forgot password flow Session management & Calling APIs You can now install the project dependencies with npm: npm install Now it's time to put this demo app to work. npm run dev The example app will run on ( ), while the API server will run on ( ). http://localhost:3000 http://localhost:3001 On the frontend, authentication We use the supertokens-auth-react package on the front end to handle authentication (sending and receiving tokens). The Session is initialized in the app.js file: SuperTokens.init({
    appInfo: {
        appName: "Music Player", // TODO: Your app name
        apiDomain: getApiDomain(), // TODO: Change to your app's API domain
        websiteDomain: getWebsiteDomain(), // TODO: Change to your app's website domain
    },
    recipeList: [
        EmailPassword.init({
            emailVerificationFeature: {
                mode: "REQUIRED",
            },
        }),
        Session.init(),
    ],
}); To deliver the token to the server in an API request, we must include the following line. Session.addAxiosInterceptors(axios); The Supertokens-auth-react package will handle storing tokens on the client, transmitting tokens to the server, and updating tokens before they expire. Let's discuss the react component that we used to create the music player app In Visual Studio Code, the folder structure appears like this: 1- Music Container (MusicContainer.jsx) We manage all of the state-related stuff in this component and render all of the child components. Here, we call the songs endpoint exposed by API server.js to get the list of songs. import React, { useCallback, useState } from "react";
import SongList from "./SongList.jsx/SongList";
import Player from "./Player/Player";
import SongDetail from "./SongDetails/SongDetails";
import axios from "axios";
import { getApiDomain } from "../utils/utils";
import useAsync from "../Hooks/useAsync";

export default function MusicContainer() {
    const asyncCallback = useCallback(() => {
        return axios.get(`${getApiDomain()}/songs`);
    }, []);
    const { data } = useAsync(asyncCallback);
    const songs = data || [];
    const [selectedSong, setSelectedSong] = useState(0);

    return (
        <>
            <SongDetail selectedSongId={selectedSong} songs={songs} />
            <SongList selectedSong={selectedSong} songs={songs} selectedSongId={(id) => setSelectedSong(id)} />
            {songs.length > 0 && (
                <Player songs={songs} selectedSongId={selectedSong} selectSongById={(id) => setSelectedSong(id)} />
            )}
        </>
    );
} 2- Song List (SongList.jsx) All of the songs are rendered in this component. And the player component will play each of the songs in the panel when you click on them in the panel. import React from 'react'
import './SongList.css'
import logo from '../../playing.gif'

export default function SongList({songs, selectedSongId, selectedSong}) {
  return (
    <>
      <div className="header">
        <div className="track-number">#</div>
        <div className="track-title">Title</div>
        <div className="track-author">Author</div>
      </div>

      <div className="song-main">
        {' '}
        {songs.map((item, index) => (
          <div
            key={index}
            className={`song-list ${index === selectedSong ? 'active' : ''}`}
            onClick={() => selectedSongId(index)}
          >
            {index !== selectedSong ? (
              <div className="track-number">{index + 1}</div>
            ) : (
              <div className="index">
                <img alt="" src={logo} id="focused" className="small-icon" />
              </div>
            )}
            <div className="track-title">{item.name}</div>
            <div className="track-author">{item.author}</div>
          </div>
        ))}
      </div>
    </>
  )
} 3. Player (Player.jsx) The HTML5 audio element is used in this player component to play all of the songs in the app. import "./Player.css";
import { useCallback, useEffect, useRef, useState } from "react";
import { forwardsSvg, backwardsSvg, shuffleSvg } from "../svg";
import Progress from "../ProgressBar/ProgressBar";
import SongTime from "./SongTime";

export default function Player({ selectedSongId, songs, selectSongById }) {
    const [shuffled, setShuffled] = useState(false);
    const [currentTime, setCurrenTime] = useState(0);
    const [duration, setDuration] = useState(0);
    const [currentVolume, setCurrentVolume] = useState(100);
    const [playerState, setPlayerState] = useState(false);
    const audioRef = useRef();
    let intervalRef = useRef();
    let clicked = useRef(false);

    const spaceDownFunc = useCallback((event) => {
        if (event.keyCode === 32 && !clicked.current) {
            clicked.current = true;
            document.getElementsByClassName("main-control")[0].click();
        }
    }, []);
    const spaceUpFunc = useCallback((event) => {
        if (event.keyCode === 32 && clicked.current) {
            clicked.current = false;
        }
    }, []);

    useEffect(() => {
        document.addEventListener("keydown", spaceDownFunc);
        document.addEventListener("keyup", spaceUpFunc);
        return () => {
            clearInterval(intervalRef.current);
            document.removeEventListener("keydown", spaceDownFunc);
            document.removeEventListener("keyup", spaceUpFunc);
        };
    }, [spaceDownFunc, spaceUpFunc]);

    if (selectedSongId < 0 || selectedSongId > songs.length - 1) {
        selectSongById(0);
    }

    useEffect(() => {
        if (audioRef.current) {
            audioRef.current.volume = currentVolume / 500;
        }
    }, [currentVolume]);

    const onMusicPlay = (e) => {
        e.preventDefault();
        setPlayerState((prev) => !prev);
    };

    const onBackwardClick = () => {
        if (selectedSongId > 0) {
            selectSongById(selectedSongId - 1);
        }
    };
    const onForwardClick = () => {
        if (selectedSongId < songs.length - 1) {
            selectSongById(selectedSongId + 1);
        }
    };

    useEffect(() => {
        setPlayerState(true);
    }, [selectedSongId]);

    useEffect(() => {
        if (playerState) {
            audioRef.current.play();
        } else {
            audioRef.current.pause();
        }
    }, [playerState, selectedSongId]);

    return (
        <div id="player">
            <SongTime currentLocation={currentTime} duration={duration} />
            <div
                className="control"
                id={shuffled ? `active` : null}
                onClick={() => {
                    setShuffled(!shuffled);
                }}>
                {shuffleSvg}
            </div>
            <div className="control" onClick={onBackwardClick}>
                {backwardsSvg}
            </div>
            <div className="main-control control" onClick={onMusicPlay}>
                <i className={`fas fa-${playerState ? "pause" : "play"}-circle`}></i>
            </div>
            <div className="control" onClick={onForwardClick}>
                {forwardsSvg}
            </div>
            <Progress value={currentVolume} setVolume={(vol) => setCurrentVolume(vol)} />

            <audio
                id="main-track"
                controls
                src={songs[selectedSongId].url}
                preload="true"
                onEnded={() => {
                    selectSongById(shuffled ? Math.round(Math.random() * songs.length) : selectedSongId + 1);
                }}
                onLoadedMetadata={() => {
                    setDuration(audioRef.current.duration);
                    intervalRef.current = setInterval(() => {
                        if (audioRef.current) {
                            setCurrenTime(audioRef.current.currentTime);
                        } else {
                            clearInterval(intervalRef.current);
                        }
                    }, 1000);
                }}
                ref={audioRef}
                hidden>
                Your browser does not support the
                <code>audio</code> element.
            </audio>
        </div>
    );
} 4- Progress (Progress.jsx) The progress bar component is used to show the song's progress. import React from "react";
import "./ProgressBar.css";

export default class ProgressBar extends React.Component {
    constructor(props) {
        super(props);
        this.state = { showTooltip: false };
    }
    render() {
        return (
            <div className="progress">
                <input
                    type="range"
                    min="0"
                    max="100"
                    className="slider"
                    value={this.props.volume}
                    onChange={(e) => this.props.setVolume(e.target.value)}
                    onMouseEnter={() => this.setState({ showTooltip: true })}
                    onMouseLeave={() => this.setState({ showTooltip: false })}
                />
                {this.state.showTooltip && <span className="tooltip">{this.props.volume}</span>}
            </div>
        );
    }
} 5- Song Detail (SongDetail.jsx) The track title and album thumbnail image are displayed in this component. import React from 'react'
import './SongList.css'
import logo from '../../playing.gif'

export default function SongList({songs, selectedSongId, selectedSong}) {
  return (
    <>
      <div className="header">
        <div className="track-number">#</div>
        <div className="track-title">Title</div>
        <div className="track-author">Author</div>
      </div>

      <div className="song-main">
        {' '}
        {songs.map((item, index) => (
          <div
            key={index}
            className={`song-list ${index === selectedSong ? 'active' : ''}`}
            onClick={() => selectedSongId(index)}
          >
            {index !== selectedSong ? (
              <div className="track-number">{index + 1}</div>
            ) : (
              <div className="index">
                <img alt="" src={logo} id="focused" className="small-icon" />
              </div>
            )}
            <div className="track-title">{item.name}</div>
            <div className="track-author">{item.author}</div>
          </div>
        ))}
      </div>
    </>
  )
} Securing 3rd party APIs We're using an API server to use the Super tokens backend package for token management and session management. let supertokens = require("supertokens-node");
let Session = require("supertokens-node/recipe/session"); The supertokens node package must first be initialized: supertokens.init({
    framework: "express",
    supertokens: {
        // TODO: This is a core hosted for demo purposes. You can use this, but make sure to change it to your core instance URI eventually.
        connectionURI: "https://try.supertokens.io",
        apiKey: "<REQUIRED FOR MANAGED SERVICE, ELSE YOU CAN REMOVE THIS FIELD>",
    },
    appInfo: {
        appName: "SuperTokens Demo App", // TODO: Your app name
        apiDomain, // TODO: Change to your app's API domain
        websiteDomain, // TODO: Change to your app's website domain
    },
    recipeList: [EmailPassword.init(
        {
                override: {
                apis: (originalImplementation) => {
                    return {
                        ...originalImplementation,

                        signInPOST: async ({formFields, options}) => {
                                     let email = formFields.filter((f) => f.id === "email")[0].value;
                            let password = formFields.filter((f) => f.id === "password")[0].value;
                       
                           
                            // const res = await query(`select * from user where email='${email}'`)
                            if(userId[email]) {
                            let sessionHandles = await Session.getAllSessionHandlesForUser(userId[email]);
                            if(sessionHandles.length > 0) {
                                return {
                                    status: 'SESSION_ALREADY_EXISTS'
                                }
                            }
                        }
                            let response = await options.recipeImplementation.signIn({ email, password });
                            if (response.status === "WRONG_CREDENTIALS_ERROR") {
                                return response;
                            }
                            let user = response.user;
                            userId[email] = user.id;

                            await Session.createNewSession(options.res, user.id, {}, {});
                            // query(`insert into user (email, status) values ('${email}', 'ACTIVE')`)
                            return {
                                status: "OK",
                                user,
                            };
                        },
                       
                    }
                },
            }
    }
    ), Session.init(),
       
    ],
}); We exposed the Song endpoint to the react app to retrieve the music list. We're calling the HarperDB endpoint in this endpoint to receive a list of songs from DB. app.get("/songs", verifySession(), async (req, res) => {
    const resp = await axios.get('https://functions-custom-tyagi.harperdbcloud.com/ToDoApi/songs');
    res.send(resp.data);
}); As a result, the get method's second parameter, verifySession, does the validation(token, Session) for us. The super tokens make this method available. SuperTokens takes charge of token creation and session management when the user logs into the demo app. That was it for this blog. Today, I hope you learned something new, and if you did, please like and share it so that others can see it as well. Thank you for being a regular reader; you're a significant reason I've been able to share my life/work experiences with you. For the most up-to-date information, follow on Twitter. SuperTokens Follow me on Twitter at . TheAnkurTyagi If you want to understand more about SuperTokens, I recommend reading last year's blog article. ⬇ Everything about super tokens - an open-source alternative to Auth0, Firebase Auth, and AWS Cognito. First Published here

Implement Authentication & Security Using SuperTokens in Your React and HarperDB App

Introducing the newest innovation from HarperDB: . With the release of HarperDB 3.1, users are able to define their own API endpoints within HarperDB. What does that mean for you? . We’re collapsing the stack! HarperDB Custom Functions HarperDB grows from a distributed database to a distributed application development platform with integrated persistence - one that can serve as a single solution for all of your backend needs Alright, alright, what’s the big deal? Up until 3.1, in order to power an application, you would need to deploy and host your backend API code on additional servers, then have them call out to HarperDB for database needs. This is a pretty typical software stack, but at HarperDB, we’re far from typical. We’re constantly innovating and changing the game. Oh yeah, and it’s faster, significantly faster! Traditional architectures naturally introduce latency as data moves across multiple servers through a local network or potentially even the Internet. HarperDB is collapsing the stack onto a single server, which eliminates any and all network latency. Custom Functions enable developers to build their entire application backend in one place. This frees up headroom for achieving higher throughput from a single server. Capitalizing on HarperDB’s already powerful horizontal scalability means you can now distribute your APIs and your database to the edge. both For those familiar with modern cloud architectures, Custom Functions are just like AWS Lambda functions. For those familiar with relational databases, they’re like Stored Procedures. You define your logic and choose when to execute it. At a high level, it’s as simple as that! They’re low maintenance and easy to develop. You can develop HarperDB Custom Functions in the Studio or in your own IDE and Version Management System. HarperDB Custom Functions can be maintained like any other development project; in fact, the sample Custom Functions provided in the Studio are generated from our public GitHub repository. That means you can develop, maintain, and deploy your HarperDB Custom Functions code just like any other development project, so you don’t have to deviate from your existing development practices. That’s great news! HarperDB Studio Functions Editor What makes Custom Functions so powerful? They leverage the full power of and . HarperDB Custom Functions projects are effectively just Node.js projects, which means you can leverage the npm ecosystem, opening the doors to fast and efficient development. Node.js Fastify Fastify serves as the basis for the webserver, which means you can define and build a fully functional REST API with all the bells and whistles you’d expect. The key differentiator is that these Fastify routes have direct access to HarperDB core methods, bypassing the HarperDB API and, instead, interacting directly with HarperDB on the same machine. WebStorm IDE with HarperDB Functions Project By collapsing the stack, we deliver unparalleled performance and efficiency out of the box. Let’s take a look at some different ways Custom Functions can be used. I’m not going to cover everything here; in fact, I’m sure there are plenty of options that I haven’t even thought of. : Seamlessly connect third-party/external data with data stored in HarperDB within a single function. Integrate with third-party apps and APIs : Tightly integrate with third-party application providers to validate user requests within your API. Utilize third-party authentication : HarperDB is always adding features, but let’s say there’s a feature you need that’s missing. Build a HarperDB Custom Function to solve the problem. For example, if you need to enforce row-level security based on a user account, write a function! Define your own database functionality : Custom Functions can serve static content and serve as backend APIs, which means you can fully power a website or web app all with HarperDB. Serve a website These are just some of the ideas we’re kicking around with HarperDB Custom Functions. Since this will be the initial release of HarperDB Custom Functions, please let us know what else you’d like to see in future releases. You can submit your ideas to our feedback board here: . feedback.harperdb.io Also Published On: https://lzomedia.com/blog/harperdb-is-collapsing-the-stack-introducing-custom-functions/

HarperDB

harperdb.io

11-50 employees

Since 2017

COMPANY RANKING

HARPERDB

EVERGREEN INDEX #5278

HarperDB WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

HarperDB

harperdb.io

11-50 employees

Since 2017

COMPANY RANKING

HARPERDB

EVERGREEN INDEX #5278

RELATED COMPANIESRANK

STORIES ABOUT HARPERDB

HarperDB WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!