HarperDB is a globally-distributed edge application platform. It reduces complexity, increases performance, and lowers costs by combining user-defined applications, a high-performance database, and an enterprise-grade streaming broker into a single package. The platform offers unlimited horizontal scale at the click of a button, and synchronizes data across all the nodes in a cluster in milliseconds. HarperDB simplifies the process of delivering applications and the data that drives them to the edge, which dramatically improves both the user experience and total cost of ownership for large-scale applications. Deploying HarperDB on global infrastructure enables a CDN-like solution for enterprise data and applications. With our flexible, component-based architecture, simple HTTP/S interface, and a high-performance single-model data store that accommodates any data structure, HarperDB scales with your application from proof of concept to production. Install and manage on your hardware, or have us host it with HarperDB Cloud.

HarperDB

harperdb

Introducing HarperDB 4.2, a groundbreaking update that combines user-defined applications, real-time data streaming, and a lightning-fast database into a single package, expediting the development of enterprise-grade applications while

New HarperDB 4.2 - The Distributed Real-Time Application Platform That's As Easy To Use As It Is Powerful

Another customer focused on high throughput and low latency use cases is HarperDB, a distributed global application. The ability to dynamically route workloads based on geography through AWS Wavelength on Verizon will continue to allow HarperDB to improve ...

Verizon APIs leverage AWS Wavelength for latency-sensitive app … – Verizon

Today we’re going to talk about marketing technically complex products and services, and some of the challenges and opportunities they present. To help me discuss this topic, I’d like to welcome Jaxon Repp, Field CTO & Head of Marketing at HarperDB.

Ep 424: Marketing complex products and services with Jaxon Repp, Field CTO & Head of Marketing, HarperDB

Digital interfaces, displays, signage, and more are a fixture of our lives, and they aren’t going anywhere. In fact, as digital displays get more flexible and sustainable, they will continue to grow in usage, but at what impact to our health, energy ...

Ep 416: Digital displays and their impact on health and sustainability, with Tim O'Malley, E Ink

HackerNoon Reporter: Please tell us briefly about your background. I started programming when I was 13 years old working for my uncle’s startup that did ISP billing in the 90s. I worked for Red Hat running infrastructure for global support services, and have founded a few startups and worked at others. What's your startup called? And in a sentence or two, what does it do? HarperDB is a globally distributed database focused on making developers' lives easier while providing capability for low latency applications. What is the origin story? Kyle Bernhardy, our CTO, and I have been solving big data and integration challenges for over a decade. We met at a company called Coresite where we were locked in a small conference room for a month working on a project. Since then he and I have worked together through 4 companies and along the way picked up several other amazing folks. We have continued to work on large scale and complex data projects. Ultimately we realized that the vast majority of the problems we were faced with in analytics, code, middleware, etc. were created by the database. Furthermore, we felt like databases were overly complex, and even though they are frequently used by developers, they were not developer friendly. We wanted to build a product with the flexibility of NoSQL and the analytical capability of SQL while being super easy to scale and incredibly developer friendly. Our goal was to enable a brand new developer to confidently build a globally distributed scalable application with HarperDB as the back-end. What do you love about your team, and why are you the ones to solve this problem? Our team is amazing, and the core team has worked together for a very long time. This has allowed us to build a really strong company culture. Everyone gets along extremely well and embodies our core values. We look for people who we can empower that are authentic, transparent, kind, and capable of taking accountability. I think we are the ones to solve the problem because we were determined and far too stupid to know what we were getting into. Building a database is hard. Much harder than we expected. We are not MIT grads, we didn’t take any classes from Andy Pavlo at CMU on databases, and that is exactly what makes our product unique and awesome. Kyle has a degree in exercise management and mine is in theology. Had we known what we were getting into we wouldn’t have done it. We also would have brought a mindset to the problem that wouldn’t have really added anything new to the space. While HarperDB started as a database it has really morphed into an edge development platform. We built the product we would want to use and I think that shows. If you weren’t building your startup, what would you be doing? Stand up comedy? At the moment, how do you measure success? What are your core metrics? We look at a variety of things. Obviously typical metrics like number of customers, quality of customers, ARR and MRR, but we are also looking at things like freemium adoption on npm and HarperDB Cloud, our ranking in db-engines month over month, and a number of other factors. What’s most exciting about your traction to date? Being featured on HackerNoon? Aside from that, watching developers build amazing products all over the world on top of HarperDB is my favorite part. Watching developers on social media find HarperDB and get excited by it keeps the whole team going. What technologies are you currently most excited about, and most worried about? And why? I am not really worried about any technology, I am of the mindset that all technological advancement is exciting. That said, what people do with it can obviously be scary, but that is a different matter. Specifically for HarperDB we are pretty excited about CRDTs and the capability that will be gained by adding that to HarperDB’s replication. I am also super excited about the release of HarperDB Custom Functions which will give developers the ability to build their entire stack on HarperDB! What drew you to get published on HackerNoon? What do you like most about our platform? Margo, our head of dev rel made me? Just kidding. First and foremost I like the font. I am a bit older and the font makes me feel at home! In addition, I think that the broad range of topics covered that encompass the entire startup journey from a technical perspective is interesting and important. What advice would you give to the 21-year-old version of yourself? You are not that smart. Be humble. Sales are just as important as product, because if you build an amazing product but can’t sell it you don’t have a company. Listen to feedback, no matter how smart you think you are, it doesn’t matter if the world wants something else. Meet people where they are, not where you want them to be. Being kind is easy and much more effective than being an asshole. Hiring cool people and teaching them what they need to know is more effective than hiring really talented Premadonna's. Vote for HarperDB as the . startup of the year from Denver, Colorado

The Importance Of Staying Humble And Strong Company Culture: Stephen Goldberg, HarperDB CEO

Learn how to use these two technologies together to build a REST API This article will teach you how to use Node.js, Fastify and HarperDB to build a course management system. This application will help you track the courses you are doing and the courses you plan to do. You will use the following technologies: Node.js HarperDB Fastify Introduction When it comes to Node.js and Fastify, people are familiar with these technologies. However, when it comes to databases, there is a new kid on the block! Recently, I took a break from the databases I usually use to play around with HarperDB. is a distributed database with NoSQL and SQL capabilities. The thing about HarperDB is that you perform all the CRUD database operations using one endpoint. HarperDB most fascinating only The most notable features of HarperDB are: using a single endpoint execute SQL queries on JSON data it removes the need for an ORM by returning the results as JSON arrays the ability to insert data via JSON, CSV or using SQL Does not that look interesting? Prerequisites Before going further, there are some prerequisites you should be aware of. The tutorial assumes basic knowledge of: Node.js and FastifyJS Databases JavaScript Knowing how to use a package manager like npm Additionally, you should have a Node version of 12.xx or higher for HarperDB to run. Lastly, you need to have a HarperDB account, which you can create . Do not worry; it's ! here free Set up the folder structure The first step of the tutorial is to set up the folder structure of the application. Create a new folder called and open it. You can do it with the following commands: course-management mkdir course-management
cd course-management The basic folder structure is ready. Of course, when you build each part of the application, you'll create more folders. Install dependencies At this point, you should be in the folder. Let's initialise the project with the following command: course-management npm init -y The flag generates the "package.json" automatically without you having to answer the usual questions about the project name, description and so on. If you open "package.json", you can see that all fields are autocompleted. -y Install the required packages The next step is to install the packages needed to build the application. You can install multiple packages in one line. Run the command below in your terminal: npm i fastify harperive dotenv --save The command installs three packages/dependencies: - FastifyJS is a new web framework, and it claims to be the fastest web framework in town. Fastify - It's the Node.js driver for HarperDB. Harperive - A module that loads environment variables from an .env file to process.env. Dotenv Configure the Fastify server Before proceeding to configure the server, you need to create a few files. The first step is to create the following files: touch app.js
touch .env The file contains the server configuration. On the other hand, the file stores your environment variables. Open the file and write the following code: app.js .env app.js app = ( )({ : }); ( ).config()

app.get( , (req, res) => {
    res.send({ : });
});

app.listen(process.env.PORT, (err, addr) => { (err) {
        app.log.error(err);
        process.exit( );
    }

    app.log.info( );
}); const require 'fastify' logger true require 'dotenv' '/' hello 'world' if 1 `Your server is listening on port 🧨` ${process.env.PORT} With the first line, you import the Fastify package and store it in a constant named . In the same line, you also enable the . After that, you import so you can use environment variables. app logger dotenv Lastly, you create a GET route that responds with JSON, and then you listen on the port specified in the file. .env Moving on, open the file and add the following lines: .env PORT= INSTANCE_URL=
INSTANCE_USERNAME=
INSTANCE_PASSWORD=
INSTANCE_SCHEMA= 3000 The fields starting with will hold information about the database. Now, it's time to create a HarperDB account if you do not have one already. INSTANCE Create and configure the database First of all, login into the . After you log in, click on the button, which creates a new HarperDB Cloud Instance. Figure 1 illustrates that! HarperDB Studio + Figure 1 After that, you will be asked to choose either a "New HarperDB Cloud Instance" or "Register User-Installed Instance", as shown in figure 2. Figure 2 Click on the button saying " ". As a result, you create a database instance in their cloud rather than hosting it yourself. After that, you need to enter: Create HarperDB Cloud Instance the name of the instance a username a password In figure 3 below, you can see my instance information. Figure 3 The next step is to choose your . For this tutorial, I advise you to choose the free options. You get the following specs for free: instance specifications 0.5GB RAM 1GB of storage and 100 IOPS Figure 4, below, illustrates that! However, if you plan to use the DB heavily, I recommend powering up the instance and choose better specifications! Figure 4 The last step is to confirm the instance details and then wait for the instance to spin up. After the instance is set up and initialised, click on the instance. For example, you can see my newly-created instance in figure 5 below. If you are accessing the instance for the first time, you might be prompted to enter the username and password you set previously. Figure 5 Your instance does not have any schemas or tables at the moment. As a result, you need to create a schema first! You can see that I chose as the name for my schema (figure 6). However, you can use any other name for your schema. mydb Once you chose a name for your schema, click on the green checkmark to save it! Figure 6 Lastly, you need to create tables. For the moment, you'll set up only one table called . Enter the name "courses" and provide a name as well. The "hash attribute" is used to uniquely identify each record. In simpler words, the "hash attribute" is the ID, which is unique for each record. Now, click on the green checkmark to save it. As shown in figure 7. courses hash_attribute Figure 7 Now, you are ready to use the database in the Node.js application! Set and config HarperDB in the app Go into the .env file and add the following lines: INSTANCE_URL=https: INSTANCE_USERNAME=admin
INSTANCE_PASSWORD=password
INSTANCE_SCHEMA=mydb //app-tutorial.harperdbcloud.com/ Of course, you need to change those details with yours! The above information is fictive, so it will not work if you try! Save the file, and let's move on! Now create two folders and a database file. You can do so by following these commands: mkdir src
mkdir src/db
touch src/db/db_config.js With the above commands, you create the "db" folder inside "src". Also, you create the file "db_config.js" that stores the configuration for the database. Open the newly-created file and write the following code: db_config.js harperive = ( ).Client; DB_CONFIG = { : process.env.INSTANCE_URL, : process.env.INSTANCE_USERNAME, : process.env.INSTANCE_PASSWORD, : process.env.INSTANCE_SCHEMA
}; client = harperive(DB_CONFIG); .exports = client; const require 'harperive' const harperHost username password schema const new module With the first line, you import the Node.js driver for HarperDB. After that, you create the database configuration with the details stored in the file. Lastly, you create a new client and export it. .env You'll use the client later, in the controller file! For now, let's move to the routes and controllers. Configure app endpoints The first step is to create two new folders in the folder. Create them as follows: src mkdir src/controllers
mkdir src/routes After that, you need to create two new JavaScript files in the newly-created folders. Create the files and as follows: course.js courseController.js touch src/routes/course.js
touch src/controllers/courseController.js You'll start by creating the routes. In this scenario, you can make use of the FastifyJS plugin architecture. So open the file from the "routes" folder, and write the following code: course.js courseController = ( ); {

} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts module There are three things happening: You import the controllers used for routes. You create a routes function. You export the newly-created routes function. In this method - routes - you configure the endpoints of your application. For each route, you specify the , the and the at minimum. HTTP method URL handler However, there are many other parameters. You can see the complete list here . With that being said, let's build the first endpoint that returns all the courses when accessed. You can do so by calling the method on the , as follows: route app courseController = ( ); { app.route({ : , : , : courseController.getCourses
    });
} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts // Get all courses method 'GET' url '/courses' handler module You build all the other routes/endpoints in a similar fashion. The only things that change are the method, URL and handler. In the Gist below, you can see all the endpoints. You have an endpoint for each CRUD operation. Application users can see all the courses, a specific course, add new courses, delete and edit existing courses. courseController = ( ); { app.route({ : , : , : courseController.getCourses
    }); app.route({ : , : , : courseController.getSpecificCourse
    }); app.route({ : , : , : courseController.addCourse
    }); app.route({ : , : , : courseController.editCourse
    }); app.route({ : , : , : courseController.deleteCourse
    });
} .exports = routes; const require '../controllers/courseController' async ( ) function routes app, opts // Get all courses method 'GET' url '/courses' handler // Get a specific course method 'GET' url '/courses/:id' handler // Add a course method 'POST' url '/courses' handler // Edit a course method 'PUT' url '/courses/:id' handler // Delete a course method 'DELETE' url '/courses/:id' handler module : Each route/endpoint has a specific controller, as you can see. At the moment, they are just placeholders because there are no controllers. That means, if you try to access the routes they won't work! But that changes in the next step because you'll build the application logic now. Note Go to the in the root folder, and add the following line: One more thing before going further. app.js app.register( ( )); require './src/routes/course' The purpose of that line is to make the routes available in your application. If you do not add this line, your routes will not work. The final version of should look as follows: app.js app = ( )({ : }); ( ).config()
app.register( ( ));

app.listen(process.env.PORT, (err, addr) => { (err) {
        app.log.error(err);
        process.exit( );
    }

    app.log.info( );
}); const require 'fastify' logger true require 'dotenv' require './src/routes/course' if 1 `Your server is listening on port 🧨` ${process.env.PORT} Build application logic with HarperDB The application logic defines what happens when a user makes a request to the routes specified in the previous section. As an example, when someone accesses the route , they should see all the courses from the database. As a result, let's start with that route. /courses The code for all routes will be in . Thus, open the file and import the "Harperive" client at the top of the file: courseController.js courseController.js client = ( ); const require "../db/db_config" Now, this where the fun begins. To portray the capabilities of HarperDB, you'll use both SQL and NoSQL operations to manipulate data. Build the GET route In the same file, add the following code after importing the database client: getCourses = (req, res) => { allCourses = client.query( );

    res.send({ allCourses });
}; .exports = {
    getCourses,
} const async const await 'SELECT * FROM mydb.courses' module As you can see, the HarperDB Node.js driver allows you to run SQL queries on your database. The above query selects and returns all the courses from the database. After that, you use the object to send the data to the client. If you run the application and try to access the route, you won't get any data because the database is empty. response /courses Figure 8 Figure 8 illustrates a successful request. You can see that the "data" array is empty because the database is empty. However, we'll repeat the request once there is data in the database. Build the POST route The next step is to build the POST route so you can add data into your database. In this case, we'll use a NoSQL operation. Write the following code after the constant: getCourses addCourse = (req, res) => { allProperties = .keys(req.body).length; (allProperties !== ) {
        res.code( ).send({ : }); ;
    } { newCourse = client.insert({ : , : [
                { : req.body.name, : req.body.description, : req.body.author, : req.body.link
                }
            ]
        });

        res.send({ newCourse });
    } (error) {
        res.send({ error });
    }
}; .exports = {
    getCourses,
    addCourse,
} const async const Object if 4 400 error 'Some properties are missing! Add the name, description, author and link!' return try const await table 'courses' records name description author link catch module First of all, the part before the block makes sure the user provides all the fields. If the user tries to add a course without specifying the , , or , it will return an error. The user must always provide all four fields. try catch name description author link Then, in the block, you use the database client to insert the new course into the database. try catch - You need to specify the table (which in this case is ) and then the . courses records - The field is an array of objects, which means you can enter multiple courses in one go. However, for now, one is enough. records Provided the insertion is , it returns the newly-added course. You can see the route in action in figure 9 below. successful Figure 9 The operation was successful, and the new course was added to the database! Figure 10 shows the course in the database! Figure 10 GET a specific course Getting a specific course is similar to getting all courses, but with one addition. Now, you use a clause to select a specific course. WHERE You pass the course "ID" in the URL, and then the SQL query matches that with a record from the database. But first of all, , you might ask! There are two options ( ): how do you get the ID? you need courses in the database first 1. You can visit the route and get the ID of any course you want. http://localhost:3000/courses 2. Go to the HarperDB studio, click on the course, and you'll see all the details. At the top of the page, you'll see that it specifies which one is the ID - see figure 10 above. getSpecificCourse = (req, res) => { course = client.query( );

    res.send({ course });
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
} const async const await `SELECT * FROM mydb.courses WHERE id=" "` ${req.params.id} module After that, it returns the course. Figure 11 below illustrates what happens when you make a GET request to get a specific course! Figure 11 DELETE a specific course You can delete a course by providing the course ID in the URL. For example, if you make a request to , you delete the course with the specified ID. DELETE http://localhost:3000/courses/0eb95456-255b-44da-b3df-b90671ef0908 deleteCourse = (req, res) => { course = client.query( );

    res.send({ : , : course
     });
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
    deleteCourse
} const async const await `DELETE FROM mydb.courses WHERE id=" "` ${req.params.id} message 'Course deleted!' deleteCourse module Once the course is deleted, it sends a message confirming that and some more extra information. Figure 12 illustrates a "DELETE" request. Figure 12 Build the PUT route Lastly, this route allows you to update an existing course. All the code before the block makes sure the users only use the allowed fields - , , and . If the user tries to update a field that does not exist, the server will respond with an error. try catch name description author link To edit the course, you're using a NoSQL operation again. You call the method on the database client, then pass the table name and the records you want to update. update (courses - in this case) Also, it's essential to specify the in the records array. If you do not, the server does not know which record to update. id editCourse = (req, res) => { updates = .keys(req.body); allowedUpdates = [ , , , ]; isValidOperation = updates.every( allowedUpdates.includes(update)); (!isValidOperation) {
        res.code( ).send({ : }); ;
    } { updatedCourse = client.update({ : , : [
                { : req.params.id, : req.body.name, : req.body.description, : req.body.author, : req.body.link
                }
            ]
        });

        res.send({ updatedCourse });
    } (error) {
        res.send({ error });
    }
}; .exports = {
    getCourses,
    addCourse,
    getSpecificCourse,
    deleteCourse,
    editCourse
} const async const Object const 'name' 'description' 'author' 'link' const ( ) => update if 400 error 'Not a valid operation! ' return try const await table 'courses' records id name description author link catch module If there is an error, the server sends back the error. Otherwise, it returns the updated course. The figure below shows how to make a PUT requests and what happens it's successful! Figure 13 That's all about routes and controllers! You can see the whole file in . I chose not to embed it because it is big and takes a significant space on the page! courseController.js this Gist Similarly, you can . To learn better, go through it, change stuff, break stuff, add things and so on! see the complete application on my GitHub How to start the application You can start the application by running node in the root folder. app.js However, if you clone the application from GitHub, you need to install the dependencies first. You can do so by running: npm install
node app.js Conclusion By this point, you should have a REST API that allows you to store and manipulate courses. The purpose of this tutorial was to get up-to-speed with the two technologies. As a result, other things could be added, such as: Authorization API rate limiting Adding a front-end and transform it into a full-stack application With that being said, if you like the tutorial and want part 2, where I build the front-end with Vue, let me know in the comments! Previously published at https://catalins.tech/build-a-rest-api-with-harperdb-and-fastifyjs

How to Create a RESTful API With HarperDB and FastifyJS

Many people learn or understand new things relative to things they already know. This makes sense; it’s probably a natural instinct. When it comes to products and technology, a lot of people ask “how are you different,” but different from what? You need some sort of baseline to start from, so you can say, “Similar to X, but different because of Y.” Because of this, comparisons, competitive analysis, and feature matrices are a great way to understand which technology solutions are right for you. So today, let’s do a comparison of three different database systems. The point of this article is not to determine which database is the best but to help uncover the factors to consider when selecting a database for your specific project. In most cases, it’s not that one database is better than the other, it’s that one is a better fit for a specific use case due to numerous factors. With MongoDB and PostgreSQL being two of the most popular tools out there, you may already know that there are tons of resources comparing the two. However, with HarperDB being a net new database, I thought it might be helpful to throw it in the mix to provide further clarity. Referring to my overview post: It’s important to understand things such as data type/structure, data volume, consistency, write & read frequency, hosting, cost, security, and integration constraints. That article provides a great high-level explanation across all different types of databases, but today we’ll get more specific. database architecture Similarities These technologies are all similar in that they are used to store data, but that simple concept is where the similarities end. However, while MongoDB and PostgreSQL are actually quite different from one another, HarperDB lies somewhere in the middle. As mentioned, there are numerous resources out there comparing MongoDB and PostgreSQL, which are both awesome databases. from Educative is one great place to start for understanding the differences between them. Therefore, to avoid redundancy, in this post I will focus a bit more on HarperDB compared to the two. This article Definitions is classified as a NoSQL database. It is document-oriented and uses JSON-like documents with optional schemas. MongoDB is a traditional RDBMS (relational database management system). Mainly used for relational data, it is object-oriented in nature. PostgreSQL is a geo-distributed database with a REST API and dynamic schema, that supports NoSQL and SQL including joins. (For example, you can ingest data via NoSQL JSON then immediately query it via SQL) HarperDB Differences Overview : PostgreSQL is a relational database handling more complex procedures, designs, and integrations. MongoDB is a NoSQL database often used for simpler, more unstructured data, great for app development. Ultimately, PostgreSQL enforces schema validations whereas Mongo does not. MongoDB vs. PostgreSQL : MongoDB is a document store that is great for unstructured data, whereas HarperDB offers full document store capability plus enterprise-grade ACID SQL. Benchmark tests found that HarperDB is 37 times faster than MongoDB at less than half the price. HarperDB also has a native REST API, supports SQL on JSON, and can be easier to use and manage. See the . HarperDB vs. MongoDB full benchmark here (Mongo is optimized for high scale writes, but not for reads. HarperDB's data storage algorithm written on top of LMDB enables both high scale reads and writes, resulting in high performance overall.) : HarperDB is more flexible than PostgreSQL, which is a great technology for complicated data or strict consistency, but HarperDB has simplified much of the work on the installation, configuration, and administration. HarperDB is allowing developers from relational backgrounds to use their existing knowledge with SQL with a database that also allows their team to use NoSQL from the same data model. HarperDB v. PostgreSQL Accessibility MongoDB, PostgreSQL, and HarperDB can each run anywhere; in the cloud, locally, data center, etc. (There is not a PostgreSQL cloud like there is with HarperDB Cloud and MongoDB Atlas, but some cloud providers offer PostgreSQL-as-a-service.) Under the hood MongoDB is mainly programmed in C, C++, and JavaScript language. PostgreSQL is mainly programmed in C. HarperDB is (LMDB is written in C, and Python is used in the install process). written in Node.js MongoDB uses BSON (binary JavaScript Object Notation) under the hood to make querying fast. This is then translated to JSON to be read when you open your collection. PostgreSQL ultimately employs SQL, a structured query language, to define, access, and manipulate the database. PostgreSQL also has a JSON datatype. HarperDB does not enforce data types; it currently stores all data attributes as strings, and data can be queried via SQL and/or NoSQL. (HarperDB is also working on enabling the ability for administrators to explicitly set attribute types (as numbers or strings) for performance tuning). Data Storage & Architecture MongoDB stores data as individual documents without regard to attributes, PostgreSQL stores data in traditional tables and rows, and HarperDB stores data in tables and rows/objects with all top-level attributes indexed by default. HarperDB has a unique data storage algorithm running on top of LMDB, this enables HarperDB to ingest JSON documents and relational data in a single product. As data comes in, HarperDB maps it to the data model; it's not a SQL engine or a NoSQL engine. Regardless of how data is ingested, data is stored according to the HarperDB data model and can be queried via SQL or NoSQL. MongoDB and HarperDB are more distributed architectures, whereas PostgreSQL might be considered a monolithic architecture. ACID Properties (atomic, consistent, isolated, & durable) HarperDB and PostgreSQL both have enterprise-grade ACID SQL transactions, meaning the validity of data is quite reliable. NoSQL databases like MongoDB usually adopt eventual consistency instead of ACID properties. (A study from May 2020 identified a bug that disputes claims that Mongo is ACID compliant, as MongoDB’s transactions are not fully isolated). HarperDB and PostgreSQL both follow the ANSI SQL standard. Schema & Tables With both MongoDB and HarperDB, using JSON allows you to change your schema flexibly without consequence. Documents can vary in terms of key/value pairs. Both MongoDB and HarperDB scale horizontally, which allows for speed. HarperDB has bidirectional table-level data replication. HarperDB uses a simple pub-sub model; data is replicated by publishing data to different “chat rooms” which different nodes subscribe to and are able to be distributed horizontally. PostgreSQL scales vertically (as it gets bigger, more space or more memory is needed), therefore it requires downtime to upgrade. With relational databases like PostgreSQL, altering your table is necessary to make any changes. The whole schema needs to be designed and configured at creation. You might be able to alter a table later on, but this may lead to database downtime and bugs in your application. PostgreSQL databases can use foreign keys which explicitly link data between tables and are used to keep the data normalized. Use Cases & Summary As stated in , because transactions in PostgreSQL follow ACID properties, it’s a good choice for industries such as fintech. When you absolutely need to control the state of your data, use a relational database like Postgres. Alternatively, if you only have unstructured data, or are working with big data, it might be a good idea to use the horizontal scaling approach with a tool like MongoDB. this article Use cases where HarperDB might be a better fit than existing systems include projects where you need SQL and NoSQL, rapid application development, hybrid cloud, integration, edge computing, and distributed computing. Our team will also be the first to tell you when HarperDB is not a good fit for your specific project. HarperDB is not recommended when you need full-text indexing, highly structured relational data, or strict consistency across systems. Both MongoDB and PostgreSQL are really great tools for certain use cases. They are loved by many and the team at HarperDB is thankful that they have paved the way for technologies like ours to exist. HarperDB was created to expand and blend the capabilities of awesome SQL, NoSQL, and products on the market to fill in the gaps and ensure that developers are continuously empowered to use the right tool for the job. NewSQL Sources https://www.educative.io/blog/mongodb-versus-postgresql-databases https://harperdb.io/ https://en.wikipedia.org/wiki/MongoDB https://en.wikipedia.org/wiki/PostgreSQL https://www.techshout.com/software/2020/19/mongodb-vs-postgresql-what-are-the-similarities-and-differences/ Also published at https://dev.to/harperdb/harperdb-vs-mongodb-vs-postgresql-49en

Which Database Is Right For You? HarperDB vs. MongoDB vs. PostgreSQL

Meet Stephen Goldberg Today, we hear from , the CEO of , Stephen began his career interning for Sequoia Software in the mid-’90s. He soon found success in the tech space, working as a CTO and CEO of multiple startups and in other roles at larger organizations like Red Hat. He is well-recognized as a leader in the industry. Stephen Goldberg HarperDB Denver’s Startup of the Year Winner. Q&A What does it mean for you to win this title? We are thrilled about winning Hackernoon's 2023 Startup of the Year! For us, community-driven awards are the most authentic and meaningful awards of all. It is further validation of what we have developed and continue to build upon as a company. We are grateful to all those who supported us in winning this award. What sets you apart from the competition? HarperDB was founded by developers who spent decades fighting the limitations of multi-system architectures. After exploring the logic behind systems like MongoDB, Kafka, Redis, and Node.js, it was apparent that combining the underlying systems into one distributed technology would have significant benefits from a performance, cost, and ease of development perspective.  From that realization, we created our unified system architecture that backs HarperDB. Aside from its performance advantage, I think HarperDB’s simplicity motivates companies to make the transition to our technology. We’ve seen distributed applications built in weeks with HarperDB that easily could have taken several quarters to plan and build with other technologies. Ultimately, HarperDB was a database built by developers for developers.  We built a product we would want to use. As our product continues to advance while we are focused on solving large enterprise problems, we still want to create a product that seamlessly integrates into a developer's workflow and is fun to build and deploy. What do you love about your team, and why are you the ones to solve this problem? Our team is made up of amazing people. One of our core values is authenticity, and what I love about our culture and our team is that people bring their authentic selves to HarperDB. We have a team of weirdos and misfit toys who are extremely passionate about innovation and solving complex problems.  We like working on hard things, and we love delighting developers and customers. Our team is made up of people who do not let existing paradigms and patterns stop them but will push through to create something new and incredible. If you weren’t building your startup, what would you be doing? Sleeping. We have been building HarperDB for almost seven years.  It has been an incredible journey, and we have had many amazing moments and a lot of hard ones. At the end of the day, everything that we have achieved has been due to the people who make up our team.  When we founded HarperDB, it was all about technology, and while that remains a huge part of what we do, I have learned that people matter more. If I weren’t running HarperDB, I would be doing something to help improve people's lives in meaningful ways. What that looks like exactly, I am not sure. At the moment, how do you measure success? What are your core metrics? From a business development perspective, proofs-of-concept (PoCs) are our number one metric. Although we also track npm and Docker downloads, PoCs are the key metric leading to business success. Since what we do is a first for the software development industry, people need to experience the benefits firsthand. We find that engineers tend to experience a light bulb moment with HarperDB once they fully internalize the breadth of value a unified system architecture can bring. After the light bulb moment, HarperDB tends to be part of nearly every deployment since building without it feels cumbersome and counterproductive. Tell us how your startup is changing the world. HarperDB changes the world on many levels. For , we simplify their workflow by giving them more capabilities with less effort. software engineers For , we empower innovation by providing them with systems that improve their agility and cost models. companies For , we make the technology they use substantially more performant, cutting down load times and delivering real-time experiences they did not have before. software users For the , we dramatically cut down on the infrastructure required to deliver software. Sometimes, by as much as 90%, making platforms that run on HarperDB more sustainable than systems that run on legacy alternatives. world How do you or your company intend to embrace the responsibility of this title in 2024? This title is a recognition that we are building a tool developers want to use. To embrace that, we will continue delivering features and performance improvements that help drive their software products forward. Today, nearly every feature we release is a direct response to requests from development teams worldwide. We will continue to serve them cutting-edge features that make their processes and products more performant. Furthermore, we want to continue to build a company where developers want to work.  We are successful due to the passion of the great technologists who have joined us on this journey, and we want to continue building a developer-led, techno-centric, authentic organization. What goals are you looking forward to accomplishing in 2024? 2023 was a breakthrough year for HarperDB in many ways. Our product gained significant traction with three releases that embedded data streaming capabilities deep into our technology. This was a welcome addition for many companies courting standalone data streaming solutions since they now had all the power they needed without the multi-year journey standalone streaming solutions required. 2024 so far has been customer and market-focused.  We have seen a strong reaction in the market to our product and tremendous validation, with many large organizations implementing mission-critical workloads on HarperDB.  We plan to spend 2024 growing our customer base while continuing to innovate. Which trend(s) are you most excited about in 2024? Share your reasons. For the last several years, people have been exploring what distributed computing really means. We have seen containers explode in their usage and other distributed deployment methodologies.  That said, that has created new paradigm shifts in how people think about application development, compute, deployment, security, and many other areas. We are seeing 2024 as the year where experimentation becomes a reality.  Organizations, large and small, are embracing new architectures that unlock the true power of distributed computing to improve customer experiences, do more with AI and ML, reduce cost, and build high-availability architectures. In a lot of ways, this leads to the democratization of technology.  It allows people to build bigger and better without being locked into a single cloud vendor.  I am most excited about the things people will build with this newfound freedom. 2023 had been another crazy year, especially in tech, with layoffs and the Generative AI takeover! Which trend are you most concerned about? Be as brief or as detailed as you like. We have had a pretty clear vision for where we wanted to take HarperDB from the beginning and what we wanted to build both as a product and as a company.  We try to stay focused on putting one foot in front of the other.  Ultimately, trends in the tech industry are cyclical, and there are booms and busts.  High valuations and low valuations.  Lots of hiring and lots of layoffs. Quite honestly, the trend that worries me the most is worrying about trends. People keep building great stuff.  Great people need jobs.  Good companies need funding.  Large companies need help solving big problems.  These things remain the same regardless of the current trends. I think the thing I worry about the most is people losing sight of the fact that this will all repeat itself. Share your biggest success so far and/or your biggest failure so far. Our biggest failure by far was that we were way too early for the market.  People thought we were CRAZY when we started HarperDB.  We were too early because we didn’t realize the problem we founded HarperDB to solve was not a common problem at that time, and only a handful of companies were experiencing it ten years ago. We should have done more research, talked to more companies, and paid closer attention to the market.  We were just so enamored with solving the problem that we didn’t do our homework.   We let passion get in the way of reason. Our biggest success has been our resilience as founders and as a team as a whole.  We are extremely proud of the team we have built and have had relatively little turnover in our time. We have also managed to stay alive and continue to build a tremendous product to reach the point where the market has caught up with our crazy ideas, which have become a commonplace everyday problem. We would love your feedback on HackerNoon as a tech publication! How has your experience been with us? We love HackerNoon! We have enjoyed using HackerNoon as a publication platform. In addition to our own articles, we see many independent creators writing HarperDB-related content for your site. It’s this robust community that has driven us to Hackernoon. Any words of wisdom you’d like to share with us? Be very careful who you take advice from.  We have gotten a lot more bad advice along the way than good advice.  It has cost us a lot of money and a lot of time.  We have gotten some really good advice as well.  Ultimately, if you're getting advice and following it feels inauthentic to who you are or the vision or values of your company, ignore it.  Trust your gut. This interview series celebrates all the phenomenal . Congratulations! Well-Deserved, Champion.The HackerNoon community is excited to learn more about you! Startups of the Year Winners Startups of The Year is HackerNoon’s Flagship community-driven event celebrating startups that survived and thrived in 2023. 30,000 startups across 4200+ cities and six continents participated this year to be crowned the best startup in their city. See our global winners announcement . here Companies that blog more get 67% more leads - Jumpstart your company’s blogging journey by purchasing our . Business Blogging Package

Meet HarperDB, Winner of the Startups of the Year in Denver

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, HarperDB has been nominated in HackerNoon's annual Startup of the Year awards in Denver, USA! Please vote for us here: https://hackernoon.com/startups/north-america/north-america-denver-co-usa Read more about us below to understand why we deserve your vote. Meet HarperDB HarperDB is a database and application development platform that is focused on performance and ease of use. With flexible user-defined APIs, a simple HTTP/S interface, and a high-performance single-model data store that accommodates both NoSQL and SQL workloads, HarperDB scales with your application from proof of concept to production. Install and manage on your hardware, or have us host it with . HarperDB Cloud Fun fact: HarperDB was named after our CEO's pup named Harper! We believe dogs symbolize loyalty, trust, and dependability, so we took these characteristics and built the brand around Harper and all of our other loyal employee pups. :) My Role I’ve worn a few different hats in my 4+ years at HarperDB, and currently am the Head of DevRel & Partnerships. This includes community engagement, content creation, and supporting our users to ensure that they have everything they need to be successful. I also manage our partnerships from a high level, coordinating training, events, timelines, etc., to make sure both parties are working towards their joint and individual strategic goals. I’m also the host of the and have written a . 😁 Select* podcast few articles How We're Disrupting/Improving the Software Dev/Data Management Industry HarperDB is disrupting the data management industry in a few ways. HarperDB reduces complexity, increases performance, and lowers costs by combining user-defined applications, a high-performance database, and an enterprise-grade message queue into a single package. The platform offers unlimited horizontal scale at the click of a button and synchronizes data across all the nodes in a cluster in milliseconds. HarperDB simplifies the process of delivering applications and the data that drives them to the edge, which dramatically improves both the user experience and total cost of ownership for large-scale applications. Deploying HarperDB on global infrastructure enables a CDN-like solution for enterprise data and applications. Standing Out From the Crowd - Streamlining the Journey to the Edge Even though edge computing is becoming mainstream, no one is talking about the complexity of deploying distributed applications and data to the edge. There are a lot of great articles about the edge being the future, and for the most part, edge companies are addressing this demand similar to how companies addressed the cloud. However, the edge is a fundamentally different problem. Once you consider the exponential complexity of building multiple data stores, deploying APIs that load balance across geo-distributed data stores, and the sneaky complexity of keeping everything in sync, there are very few companies that have actual solutions for the edge that won't stifle future innovation with mountains of system complexity. HarperDB is the only solution that actually decreases complexity while moving systems to the edge, and the only platform that gives innovation room to flourish while moving to the edge. Our Predictions/Thoughts on the Software Dev/Data Management Industry in 2023 2023 is poised to be a breakthrough year for Unified Backend technology in the Software Development Industry, heralding a monumental shift that will forever change the landscape of development. This transformative innovation blurs the line between edge and cloud, effectively commoditizing infrastructure. Unified Backend technology combines the finest elements of best-in-class database management, API server functionality, and data streaming capabilities, unifying their underlying systems at the core. Imagine a world where developers no longer grapple with the complexities of integrating disparate backend components, but instead embrace the power of a Unified Backend. In this new era, ease, scalability, and unprecedented performance take center stage. Applications seamlessly tap into a unified system processing data at the edge or sending real-time messages. The result is a paradigm shift, enabling faster development cycles, reduced latency, and lower infrastructure cost than ever before attainable. We predict that embracing the transformative power of Unified Backend technology is the key to forever changing the way software development is approached and executed, shaping the future of the industry. Why We Decided to Participate in HackerNoon’s Startup of the Year Awards We decided to participate in the awards because we love the HackerNoon community, and who doesn’t like a good ole’ friendly competition? Final Thoughts Thanks for reading! Please feel free to reach out with any questions/comments; we would love to hear from you. Vote for us today! https://hackernoon.com/startups/north-america/north-america-denver-co-usa

Startups of the Year 2023: Meet HarperDB - A Database and Application Development Platform

Suppose you are searching for information on a website. Let’s imagine a Twitter user writes about CRYPTOCURRENCY! What do you do? You can copy and paste the tweets about CRYPTOCURRENCY into your own file. But what if you want to retrieve massive volumes of information from Twitter? Such as vast quantities of information for your data science project? In this circumstance, copying and pasting won’t work! Then you will need to utilize Web Scraping. What is Web Scraping? The term “web scraping” refers to an automated process that can collect significant volumes of data from websites. The majority of this data is unstructured data that is stored in an HTML format. In order for this data to be utilized in a variety of applications, it must first be converted into structured data that is stored in a spreadsheet or a database. For many businesses, web scraping can be used to quickly and inexpensively gather data that can then be analyzed in a variety of ways such as news monitoring, sentiment analysis, email marketing, and others. Web scraping, the process of obtaining data from websites through automated means, can be carried out in a variety of different methods. Application Programming Interfaces (APIs) e.g from Twitter, StackOverflow & Google. Write code (e.g in the Python programming language). Online services from different providers e.g Octoparse. In this article, you will learn how to: How to execute web scraping on Twitter using the Python library. snsscrape How to store scraped data automatically in the database using . HarperDB How to share your data via API call by using Custom Function from . HarperDB So let’s get started. What is Snscrape? snscrape is a scraping tool for social networking services (SNS). It scrapes information like user profiles, hashtags, searches, and threads and returns the discovered items, e.g. the relevant posts. It was released on July 8, 2020, and it is capable of scraping data from a variety of platforms, including the following: Twitter Instagram Reddit Facebook Weibo Telegram Mastodon You can use snscrape by typing its command-line interface (CLI) commands into the command prompt/terminal. If you don’t feel comfortable using a terminal, you can use snscrape as a , but this is not yet documented. Python library On Twitter, it can scrape users, user profiles, hashtags, searches, tweets (single or surrounding thread), list posts, and trends. Note: What is HarperDB? is a lightning-fast and versatile platform for managing SQL and NoSQL data. You can put it to work for a wide variety of purposes, some of which include but are not limited to quick application development, distributed computing, edge computing, software as a service (SaaS), and many others. HarperDB HarperDB does not duplicate data, is fully indexed and can run on any device, from the edge to the cloud. Additionally, it may be used with any programming language, such as Javascript, Java, , and others. Python The following is a list of a few of the features that can be accessed with HarperDB: Allows JSON and CSV file insertions. Single endpoint API. Supports SQL queries for full CRUD operations. (Lambda-like application development platform with direct access to HarperDB’s core methods). Custom Functions Limited database configuration required. Math.js and GeoJSON are both supported. HarperDB has a built-in HTTP API, custom functions for user-defined endpoints, and a dynamic schema that can help you easily share your scraped data with your coworkers after storing them in a HarperDB cloud instance. HarperDB allows you to quickly download scraped data held in the HarperDB instance as a CSV file so that you can perform extra analysis before making a final choice. After being introduced to the tools (snscrape & harperDB) that you will use to automate the process of scraping data and saving it in the database. Then all you have to do is follow the steps that are described below Step 1: Create a HarperDB Account We will start by working on the HarperDB database first. You can visit and then click the navigation bar to see a link called “Start Free.” Click it in order to create your account. https://harperdb.io/ If you already have an account, use the following URL to sign in with your credentials. https://studio.harperdb.io/ Step 2: Create a HarperDB Cloud Instance After registration, you need to create a cloud instance to store and fetch your scraped data from Twitter. Click the Create New HarperDB Cloud Instance link to add a new instance to your account. You just need to follow all instructions provided by harperDB to create your cloud instance, such as: Note: Select Instance Types.Choose Cloud Provider. Add instance information. Select instance specification (RAM size, instance storage size, and instance region). Confirm and create a cloud instance. When the HarperDB Cloud Instance has been created successfully, you will see the status as OK for that particular instance, check the image below. Step 3: Configure the HarperDB Schema and Table To add the Twitter data that has been scraped into the database, you must first create a schema and a table. It only requires loading the HarperDB cloud instance you already created from the dashboard and creating the schema by giving it a name (like “data_scraping”). You then have to add a table (e.g ). Additionally, HarperDB will ask you to specify the hash attribute, which is equivalent to an ID number. “tweets” Step 4: Install the Required Packages You need to install the following package on your local machine. This is the we’ll use to implement different HarperDB API functions sucha as inserting data into to the cloud instance. It also provides wrappers for an object-oriented interface. (a) harper-sdk-python Python package pip install harperdb Snscrape requires Python 3.8 or higher. When you install snscrape, the dependencies for the Python package are automatically installed. (b) snscrape pin install snscrape Step 5:Import Important Packages The next step is to import Python packages to scrape data from Twitter and automatically store them on harperDB cloud instance. #import packages #snscrape import snscrape.modules.twitter as sntwitter # harperdb import harperdb import warnings # To ignore any warnings warnings.filterwarnings( "ignore" ) Step 6: Connect to HarperDB Cloud Instance You need to connect to the HarperDB cloud instance in order to insert scraped tweets into the table called . tweets Here you need to provide three parameters: Full URL of the HarperDB instance Your username Your password db = harperdb.HarperDB(url=URL, username=USERNAME, password=PASSWORD) db.describe_all() # connect to harperdb URL = "https://1-mlproject.harperdbcloud.com" USERNAME = "USERNAME" PASSWORD = "PASSWORD" # check if you are connected When you execute the above code, you will see output similar to that displayed below, indicating a successful connection to your HarperDB Cloud Instance. '__updatedtime__': 1660390877630,
   'hash_attribute': 'id',
   'id': 'd140645e-3af2-42d7-8594-2195826dabbc',
   'name': 'tweets',
   'residence': None,
   'schema': 'data_scraping',
   'attributes': [{'attribute': '__createdtime__'},
    {'attribute': '__updatedtime__'},
    {'attribute': 'id'}],
   'record_count': 0}}} {'data_scraping': {'tweets': {'__createdtime__': 1660390877630, Step 7:Create a Function to Record the Scrapped Tweets Using the from the harperdb-python package, the following function will insert the scraped tweets as data (in dictionary format) into the specified table.The insert function will receive three parameters: insert function SCHEMA name TABLE name data (scraped tweets) result = db.insert(SCHEMA, TABLE, [data]) # define a function to record scraped data into the table def record_tweets ( data ): #define the schema and table SCHEMA = "data_scraping" TABLE = "tweets" # insert data into the table return result Step 8:Scrape tweets by using snsscrape Now you can use T from snsscrape python package to scrap tweets with the particular search query. In this example, I will show you how to scrap 1,000 tweets about “ from 1st January 2022 to 13th August 2022. witterSearchScrapper method cryptocurrency” sntwitter.TwitterSearchScraper( data = { } result = record_tweets(data) #1 Using TwitterSearchScraper to scrape data and append tweets to list for i, tweet in enumerate ( 'crytocurrency since:2022-01-01 until:2022-08-13' ).get_items()): if i > 1000 : break #2 save data automatically to the HarperB cloud instance "user_name" : tweet.user.username, "content" : tweet.content, "lang" : tweet.lang, "url" : tweet.url, "source" : tweet.source # insert result into the HarperDB table As you can see from the code block above , harperDB will automatically store scraped data into the tweets table with the following attributes. (comment #2) Username Content Lang Url Source Step 10:View the Tweets Table If you open your HarperDB cloud instance, you will be able to see all records of your scraped data from Twitter. 🎉 You have successfully completed all required steps to automate the process of scraping data and saving it in the database. Congratulations What if you wish to share the scraped information with your colleagues? Custom Function provides a straightforward solution to this problem in HarperDB. What is a Custom Function? A Custom Function is a brand-new feature included in HarperDB’s 3.1+ release. You can use the feature to add your own to HarperDB. Custom functions are powered by Fastify, which is incredibly flexible and makes it simple to interact with your data by using HarperDB core methods. API endpoints You will learn how to use the HarperDB studio to create your very own custom function in this section. You can then use an API call to share the outcomes of your scraped data with your coworkers at the office. Here are the steps you need to follow:- The first step is to enable the Custom functions by clicking “ ” in your HarperDB Studio (it is not enabled by default). 1. Enable Custom Functions functions The next step is to create a project by specifying the name. For example It will also create setting files for the project including: 2. Create a Project tweets-api-v1. Routes folder File to add helper functions Static folder For this article, you will focus on the routes folder. Note: In this step, you will create the first route to fetch some data from the tweets table from the HarperDB Datastore. You also need to know that Route URLs are resolved in the following manner: 3. Define a Route [Instance URL]:[Custom Functions Port]/[Project Name]/[Route URL] It will include: Cloud Instance URL Custom Functions Port Project name you have created The route you have defined In the route file (example.js) from the function page, you will see some template code as an example. You need to replace that code with the following code: server.route({ request.body= { }; }
}); 'use strict' ; module .exports = async (server, { hdbCore, logger }) => { url : '/' , method : 'GET' , handler : ( request ) => { operation : 'sql' , sql : 'SELECT user_name,content,lang,url,source FROM data_scraping.tweets ORDER BY __createdtime__' return hdbCore.requestWithoutAuthentication(request); In the code above, the route is defined with the GET method and the handler function will send an SQL query to the database to get from the ordered by the /tweets-api-v1 user_name, content, lang, URL, and source tweets table __createdtime__ column. Finally, you can now use the route you have defined to get the data from the tweets table. Here you will send an API request by using the Python package. 4. Access data via API Endpoint requests r = requests.get(url = URL) data = r.json() #send an API request import requests # api-endpoint URL = "https://functions-1-mlproject.harperdbcloud.com/tweets-api-v1" # sending get request and saving the response as response object # extracting data in json format for experiment in data: print (experiment) Here is the sample output from the above code. {"user_name": "DailyCryptoTrad","content": "DXY forming a bullish bull flag on the daily - a break out of 106.6 will give crypto red days however if we fail below 105 will give crypto green days - Keep an eye on DXY #DXY #SPY #crypto #btc #eth #bitcoin #crytocurrency #cryptocurrencies https://t.co/AkF8Igf3Uc","lang": "en","url": "https://twitter.com/DailyCryptoTrad/status/1558211511461597188","source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>"},{"user_name": "Ariscrypto1970","content": "@scrypto_1977 @Epayme_uae #Saitama will go parabolic when it happens! This is the #WeAreSaitama and the world are waiting for. 🔥🔥🔥🚀🚀🚀🚀#crytocurrency #DeFi","lang": "en","url": "https://twitter.com/Ariscrypto1970/status/1558200674273345537","source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>"},{"user_name": "dan_nyeche","content": "Cryptocurrency market up 24Hrs. #Bitcoin #Dan_Trades #crytocurrency Emilokan Big Brother Modella FireBoy Giddyfia  GTBank President Obama #gayfish Gen Z Ethereum Chi Chi Obidatti2023 Sapa Lewandoski #HAPPYJAEMINDAY #Jalsa4K #GomoraMzanzi #SheggzOlu𓃵 #ViratKohli𓃵 https://t.co/QbU4ei3MGA","lang": "in","url": "https://twitter.com/dan_nyeche/status/1558188248362467329","source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>"}, With HarperDB, you can quickly and easily build API endpoints to share the scraped data with your team working on the same data science project. Note: Conclusion Congratulations 🎉, you have made it to the end of this article. You have learned: How to execute web scraping on Twitter using the Python library. snsscrape How to store scraped data automatically in the database using . HarperDB cloud instance How to create a from the HarperDB cloud instance to share your scraped data with your coworkers working on the project via an API endpoint. custom function If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter @Davis_McDavid.

How to Web Scrape Using Python, Snscrape & HarperDB

Adding authorization & authentication to an app is a common task for developers. When you're launching a new product, a startup, or a side project, it can feel like a mission. This tutorial will teach us how to make a music player app that uses SuperTokens for authentication and HarperDB for the backend. Some of the app's most essential features On Backend Side: Secure your API with almost significantly fewer and zero configurations. Secure your third-party API on the server side using SuperTokens. Session management is simple. On the Front-end side: For login, use SuperTokens' library component, which is available as an npm package. There's no need to develop any extra code for authentication. There's no need to maintain tokens on the UI side. You won't have to worry about managing token refresh calls on the UI side because SuperTokens will take care of it for you. TL;DR Here are the links to get to the source code or learn more about both products faster Source Code SuperTokens SuperTokens GitHub HarperDB What you'll discover How to add authentication using SuperTokens. How to secure 3rd party (HarperDB) endpoints. Let's talk about SuperTokens SuperTokens is an open-source authentication. They help you launch quickly so that you can focus on your core product offering. SuperTokens is 100% open source. Developers can own and manage their user's data. SuperTokens can be run on your premise for free and also has a generously hosted tier for those who don't want to manage it themselves. SuperTokens is a versatile authentication and authorization solution for your applications. Prerequisites This tutorial assumes the reader has the following: Node installed on their local development machine. You can verify if you do by running the command below in your terminal. node -v If otherwise, download from here A basic knowledge of HTML, CSS, JavaScript, and React Take a peek at how we developed our React app and used SuperTokens to authenticate it. A summary of the project's setup Yay! This section will create a React music player app and incorporate SuperTokens and HarperDB. Alternatively, you can utilize the start directory as your project root by cloning the . It includes the whole project setup that will get you started. GitHub repository How to Begin Using SuperTokens SuperTokens provides a variety of ready-to-use recipes from which to choose. You can see all the recipes. here We'll use the pre-built Recipe to access the , which will look something like this. ⬇ EmailPassword demo app The supertokens website has the documentation for this recipe. Click here Now, fork the GitHub repository Let's have a look at all of the project dependencies that were used to make this app Dependencies used:

"dependencies": {
        "axios": "^0.21.0",
        "body-parser": "^1.19.0",
        "cors": "^2.8.5",
        "express": "^4.17.1",
        "helmet": "^4.6.0",
        "morgan": "^1.10.0",
        "mysql": "^2.18.1",
        "npm-run-all": "^4.1.5",
        "react": "^17.0.1",
        "react-dom": "^17.0.1",
        "react-router-dom": "^5.2.0",
        "react-scripts": "4.0.0",
        "supertokens-auth-react": "^0.17.0",
        "supertokens-node": "^8.0.0",
        "web-vitals": "^0.2.4"
    },
    "devDependencies": {
        "@testing-library/jest-dom": "^5.12.0",
        "@testing-library/react": "^11.1.0",
        "@testing-library/user-event": "^12.6.2",
        "nodemon": "^2.0.12"
    }, Using the SuperTokens npm package in a React application The following use cases are demonstrated in this demo app. Login Sign up Logout Forgot password flow Session management & Calling APIs You can now install the project dependencies with npm: npm install Now it's time to put this demo app to work. npm run dev The example app will run on ( ), while the API server will run on ( ). http://localhost:3000 http://localhost:3001 On the frontend, authentication We use the supertokens-auth-react package on the front end to handle authentication (sending and receiving tokens). The Session is initialized in the app.js file: SuperTokens.init({
    appInfo: {
        appName: "Music Player", // TODO: Your app name
        apiDomain: getApiDomain(), // TODO: Change to your app's API domain
        websiteDomain: getWebsiteDomain(), // TODO: Change to your app's website domain
    },
    recipeList: [
        EmailPassword.init({
            emailVerificationFeature: {
                mode: "REQUIRED",
            },
        }),
        Session.init(),
    ],
}); To deliver the token to the server in an API request, we must include the following line. Session.addAxiosInterceptors(axios); The Supertokens-auth-react package will handle storing tokens on the client, transmitting tokens to the server, and updating tokens before they expire. Let's discuss the react component that we used to create the music player app In Visual Studio Code, the folder structure appears like this: 1- Music Container (MusicContainer.jsx) We manage all of the state-related stuff in this component and render all of the child components. Here, we call the songs endpoint exposed by API server.js to get the list of songs. import React, { useCallback, useState } from "react";
import SongList from "./SongList.jsx/SongList";
import Player from "./Player/Player";
import SongDetail from "./SongDetails/SongDetails";
import axios from "axios";
import { getApiDomain } from "../utils/utils";
import useAsync from "../Hooks/useAsync";

export default function MusicContainer() {
    const asyncCallback = useCallback(() => {
        return axios.get(`${getApiDomain()}/songs`);
    }, []);
    const { data } = useAsync(asyncCallback);
    const songs = data || [];
    const [selectedSong, setSelectedSong] = useState(0);

    return (
        <>
            <SongDetail selectedSongId={selectedSong} songs={songs} />
            <SongList selectedSong={selectedSong} songs={songs} selectedSongId={(id) => setSelectedSong(id)} />
            {songs.length > 0 && (
                <Player songs={songs} selectedSongId={selectedSong} selectSongById={(id) => setSelectedSong(id)} />
            )}
        </>
    );
} 2- Song List (SongList.jsx) All of the songs are rendered in this component. And the player component will play each of the songs in the panel when you click on them in the panel. import React from 'react'
import './SongList.css'
import logo from '../../playing.gif'

export default function SongList({songs, selectedSongId, selectedSong}) {
  return (
    <>
      <div className="header">
        <div className="track-number">#</div>
        <div className="track-title">Title</div>
        <div className="track-author">Author</div>
      </div>

      <div className="song-main">
        {' '}
        {songs.map((item, index) => (
          <div
            key={index}
            className={`song-list ${index === selectedSong ? 'active' : ''}`}
            onClick={() => selectedSongId(index)}
          >
            {index !== selectedSong ? (
              <div className="track-number">{index + 1}</div>
            ) : (
              <div className="index">
                <img alt="" src={logo} id="focused" className="small-icon" />
              </div>
            )}
            <div className="track-title">{item.name}</div>
            <div className="track-author">{item.author}</div>
          </div>
        ))}
      </div>
    </>
  )
} 3. Player (Player.jsx) The HTML5 audio element is used in this player component to play all of the songs in the app. import "./Player.css";
import { useCallback, useEffect, useRef, useState } from "react";
import { forwardsSvg, backwardsSvg, shuffleSvg } from "../svg";
import Progress from "../ProgressBar/ProgressBar";
import SongTime from "./SongTime";

export default function Player({ selectedSongId, songs, selectSongById }) {
    const [shuffled, setShuffled] = useState(false);
    const [currentTime, setCurrenTime] = useState(0);
    const [duration, setDuration] = useState(0);
    const [currentVolume, setCurrentVolume] = useState(100);
    const [playerState, setPlayerState] = useState(false);
    const audioRef = useRef();
    let intervalRef = useRef();
    let clicked = useRef(false);

    const spaceDownFunc = useCallback((event) => {
        if (event.keyCode === 32 && !clicked.current) {
            clicked.current = true;
            document.getElementsByClassName("main-control")[0].click();
        }
    }, []);
    const spaceUpFunc = useCallback((event) => {
        if (event.keyCode === 32 && clicked.current) {
            clicked.current = false;
        }
    }, []);

    useEffect(() => {
        document.addEventListener("keydown", spaceDownFunc);
        document.addEventListener("keyup", spaceUpFunc);
        return () => {
            clearInterval(intervalRef.current);
            document.removeEventListener("keydown", spaceDownFunc);
            document.removeEventListener("keyup", spaceUpFunc);
        };
    }, [spaceDownFunc, spaceUpFunc]);

    if (selectedSongId < 0 || selectedSongId > songs.length - 1) {
        selectSongById(0);
    }

    useEffect(() => {
        if (audioRef.current) {
            audioRef.current.volume = currentVolume / 500;
        }
    }, [currentVolume]);

    const onMusicPlay = (e) => {
        e.preventDefault();
        setPlayerState((prev) => !prev);
    };

    const onBackwardClick = () => {
        if (selectedSongId > 0) {
            selectSongById(selectedSongId - 1);
        }
    };
    const onForwardClick = () => {
        if (selectedSongId < songs.length - 1) {
            selectSongById(selectedSongId + 1);
        }
    };

    useEffect(() => {
        setPlayerState(true);
    }, [selectedSongId]);

    useEffect(() => {
        if (playerState) {
            audioRef.current.play();
        } else {
            audioRef.current.pause();
        }
    }, [playerState, selectedSongId]);

    return (
        <div id="player">
            <SongTime currentLocation={currentTime} duration={duration} />
            <div
                className="control"
                id={shuffled ? `active` : null}
                onClick={() => {
                    setShuffled(!shuffled);
                }}>
                {shuffleSvg}
            </div>
            <div className="control" onClick={onBackwardClick}>
                {backwardsSvg}
            </div>
            <div className="main-control control" onClick={onMusicPlay}>
                <i className={`fas fa-${playerState ? "pause" : "play"}-circle`}></i>
            </div>
            <div className="control" onClick={onForwardClick}>
                {forwardsSvg}
            </div>
            <Progress value={currentVolume} setVolume={(vol) => setCurrentVolume(vol)} />

            <audio
                id="main-track"
                controls
                src={songs[selectedSongId].url}
                preload="true"
                onEnded={() => {
                    selectSongById(shuffled ? Math.round(Math.random() * songs.length) : selectedSongId + 1);
                }}
                onLoadedMetadata={() => {
                    setDuration(audioRef.current.duration);
                    intervalRef.current = setInterval(() => {
                        if (audioRef.current) {
                            setCurrenTime(audioRef.current.currentTime);
                        } else {
                            clearInterval(intervalRef.current);
                        }
                    }, 1000);
                }}
                ref={audioRef}
                hidden>
                Your browser does not support the
                <code>audio</code> element.
            </audio>
        </div>
    );
} 4- Progress (Progress.jsx) The progress bar component is used to show the song's progress. import React from "react";
import "./ProgressBar.css";

export default class ProgressBar extends React.Component {
    constructor(props) {
        super(props);
        this.state = { showTooltip: false };
    }
    render() {
        return (
            <div className="progress">
                <input
                    type="range"
                    min="0"
                    max="100"
                    className="slider"
                    value={this.props.volume}
                    onChange={(e) => this.props.setVolume(e.target.value)}
                    onMouseEnter={() => this.setState({ showTooltip: true })}
                    onMouseLeave={() => this.setState({ showTooltip: false })}
                />
                {this.state.showTooltip && <span className="tooltip">{this.props.volume}</span>}
            </div>
        );
    }
} 5- Song Detail (SongDetail.jsx) The track title and album thumbnail image are displayed in this component. import React from 'react'
import './SongList.css'
import logo from '../../playing.gif'

export default function SongList({songs, selectedSongId, selectedSong}) {
  return (
    <>
      <div className="header">
        <div className="track-number">#</div>
        <div className="track-title">Title</div>
        <div className="track-author">Author</div>
      </div>

      <div className="song-main">
        {' '}
        {songs.map((item, index) => (
          <div
            key={index}
            className={`song-list ${index === selectedSong ? 'active' : ''}`}
            onClick={() => selectedSongId(index)}
          >
            {index !== selectedSong ? (
              <div className="track-number">{index + 1}</div>
            ) : (
              <div className="index">
                <img alt="" src={logo} id="focused" className="small-icon" />
              </div>
            )}
            <div className="track-title">{item.name}</div>
            <div className="track-author">{item.author}</div>
          </div>
        ))}
      </div>
    </>
  )
} Securing 3rd party APIs We're using an API server to use the Super tokens backend package for token management and session management. let supertokens = require("supertokens-node");
let Session = require("supertokens-node/recipe/session"); The supertokens node package must first be initialized: supertokens.init({
    framework: "express",
    supertokens: {
        // TODO: This is a core hosted for demo purposes. You can use this, but make sure to change it to your core instance URI eventually.
        connectionURI: "https://try.supertokens.io",
        apiKey: "<REQUIRED FOR MANAGED SERVICE, ELSE YOU CAN REMOVE THIS FIELD>",
    },
    appInfo: {
        appName: "SuperTokens Demo App", // TODO: Your app name
        apiDomain, // TODO: Change to your app's API domain
        websiteDomain, // TODO: Change to your app's website domain
    },
    recipeList: [EmailPassword.init(
        {
                override: {
                apis: (originalImplementation) => {
                    return {
                        ...originalImplementation,

                        signInPOST: async ({formFields, options}) => {
                                     let email = formFields.filter((f) => f.id === "email")[0].value;
                            let password = formFields.filter((f) => f.id === "password")[0].value;
                       
                           
                            // const res = await query(`select * from user where email='${email}'`)
                            if(userId[email]) {
                            let sessionHandles = await Session.getAllSessionHandlesForUser(userId[email]);
                            if(sessionHandles.length > 0) {
                                return {
                                    status: 'SESSION_ALREADY_EXISTS'
                                }
                            }
                        }
                            let response = await options.recipeImplementation.signIn({ email, password });
                            if (response.status === "WRONG_CREDENTIALS_ERROR") {
                                return response;
                            }
                            let user = response.user;
                            userId[email] = user.id;

                            await Session.createNewSession(options.res, user.id, {}, {});
                            // query(`insert into user (email, status) values ('${email}', 'ACTIVE')`)
                            return {
                                status: "OK",
                                user,
                            };
                        },
                       
                    }
                },
            }
    }
    ), Session.init(),
       
    ],
}); We exposed the Song endpoint to the react app to retrieve the music list. We're calling the HarperDB endpoint in this endpoint to receive a list of songs from DB. app.get("/songs", verifySession(), async (req, res) => {
    const resp = await axios.get('https://functions-custom-tyagi.harperdbcloud.com/ToDoApi/songs');
    res.send(resp.data);
}); As a result, the get method's second parameter, verifySession, does the validation(token, Session) for us. The super tokens make this method available. SuperTokens takes charge of token creation and session management when the user logs into the demo app. That was it for this blog. Today, I hope you learned something new, and if you did, please like and share it so that others can see it as well. Thank you for being a regular reader; you're a significant reason I've been able to share my life/work experiences with you. For the most up-to-date information, follow on Twitter. SuperTokens Follow me on Twitter at . TheAnkurTyagi If you want to understand more about SuperTokens, I recommend reading last year's blog article. ⬇ Everything about super tokens - an open-source alternative to Auth0, Firebase Auth, and AWS Cognito. First Published here

Implement Authentication & Security Using SuperTokens in Your React and HarperDB App

Introducing the newest innovation from HarperDB: . With the release of HarperDB 3.1, users are able to define their own API endpoints within HarperDB. What does that mean for you? . We’re collapsing the stack! HarperDB Custom Functions HarperDB grows from a distributed database to a distributed application development platform with integrated persistence - one that can serve as a single solution for all of your backend needs Alright, alright, what’s the big deal? Up until 3.1, in order to power an application, you would need to deploy and host your backend API code on additional servers, then have them call out to HarperDB for database needs. This is a pretty typical software stack, but at HarperDB, we’re far from typical. We’re constantly innovating and changing the game. Oh yeah, and it’s faster, significantly faster! Traditional architectures naturally introduce latency as data moves across multiple servers through a local network or potentially even the Internet. HarperDB is collapsing the stack onto a single server, which eliminates any and all network latency. Custom Functions enable developers to build their entire application backend in one place. This frees up headroom for achieving higher throughput from a single server. Capitalizing on HarperDB’s already powerful horizontal scalability means you can now distribute your APIs and your database to the edge. both For those familiar with modern cloud architectures, Custom Functions are just like AWS Lambda functions. For those familiar with relational databases, they’re like Stored Procedures. You define your logic and choose when to execute it. At a high level, it’s as simple as that! They’re low maintenance and easy to develop. You can develop HarperDB Custom Functions in the Studio or in your own IDE and Version Management System. HarperDB Custom Functions can be maintained like any other development project; in fact, the sample Custom Functions provided in the Studio are generated from our public GitHub repository. That means you can develop, maintain, and deploy your HarperDB Custom Functions code just like any other development project, so you don’t have to deviate from your existing development practices. That’s great news! HarperDB Studio Functions Editor What makes Custom Functions so powerful? They leverage the full power of and . HarperDB Custom Functions projects are effectively just Node.js projects, which means you can leverage the npm ecosystem, opening the doors to fast and efficient development. Node.js Fastify Fastify serves as the basis for the webserver, which means you can define and build a fully functional REST API with all the bells and whistles you’d expect. The key differentiator is that these Fastify routes have direct access to HarperDB core methods, bypassing the HarperDB API and, instead, interacting directly with HarperDB on the same machine. WebStorm IDE with HarperDB Functions Project By collapsing the stack, we deliver unparalleled performance and efficiency out of the box. Let’s take a look at some different ways Custom Functions can be used. I’m not going to cover everything here; in fact, I’m sure there are plenty of options that I haven’t even thought of. : Seamlessly connect third-party/external data with data stored in HarperDB within a single function. Integrate with third-party apps and APIs : Tightly integrate with third-party application providers to validate user requests within your API. Utilize third-party authentication : HarperDB is always adding features, but let’s say there’s a feature you need that’s missing. Build a HarperDB Custom Function to solve the problem. For example, if you need to enforce row-level security based on a user account, write a function! Define your own database functionality : Custom Functions can serve static content and serve as backend APIs, which means you can fully power a website or web app all with HarperDB. Serve a website These are just some of the ideas we’re kicking around with HarperDB Custom Functions. Since this will be the initial release of HarperDB Custom Functions, please let us know what else you’d like to see in future releases. You can submit your ideas to our feedback board here: . feedback.harperdb.io Also Published On: https://lzomedia.com/blog/harperdb-is-collapsing-the-stack-introducing-custom-functions/

HarperDB: Build Your Application Backend in One Place with Custom Functions

I recently had a fascinating conversation with Ron Lewis, the Director of Innovation and Engineering at Lumen Technologies. Ron brought up the notion that HarperDB is more than just a database, and for certain users or projects, HarperDB is not serving as a database at all. How can this be possible? on our podcast Database, Explained Well, what really is a database? states that “In computing, a database is an organized collection of data stored and accessed electronically from a computer system.” states that “​​A database is a systematic collection of data. They support electronic storage and manipulation of data. Databases make data management easy.” Wikipedia Another site So at its core, yes, is certainly a database and can fulfill this functionality (after all, that’s what the DB stands for). But it can do so much more. For example, there are many cases where organizations keep their existing database system(s) in place and use HarperDB to extend their current functionality or for a different capability altogether. HarperDB Especially when it comes to solving complex enterprise data management challenges, the answer rarely (if ever) comes down to this database vs. that database. There’s much more to it. There are many different moving parts related to capturing the right data, getting data to where it needs to be in a timely manner, analyzing and acting on that data, etc. This is really where HarperDB shines. HarperDB: A Runway for Launching Industry 4.0 Technology Ron mentioned in our podcast discussion: “The reason I see HarperDB as a disruptive technology is because you often call HarperDB a database, but it’s not really a database. It’s maybe what some people call a data mesh or data fabric... I see HarperDB more as a data surface, especially with Functions. The whole idea is to be able to converge and contextualize data to support decision making.” The new Ron refers to will enable users to define their own API endpoints within HarperDB, ultimately expanding HarperDB from a distributed database to a distributed application development platform with integrated persistence. Custom Functions So, now we’re thinking of HarperDB as a data mesh, fabric, or surface instead of a database. That’s a lot of buzzwords! Let’s take a step back. When I asked Ron what initially drew him and his team to HarperDB, he provided some great insight. Ron mentioned that they were working on a project for the Department of Defense (DoD) that required moving, contextualizing, and converging data and they needed something super fast and intuitive. They were essentially looking for something easy to use and easy to deploy that’s also flexible and scalable. Once HarperDB and Ron connected, he discovered that HarperDB could be deployed on devices as small as microprocessors like a Raspberry Pi or Tinker Board, all the way up to large-scale servers, cloud machines, or supercomputers. This piqued his interest, as he needed the ability to do large-scale analytics and move the data between devices in a simple manner. At a basic level, we quickly realized that Ron and the HarperDB team were asking the same questions: When we look at how much data has to move, how much is being created on an hourly basis from OT data onsite etc., how do we manage, transport, and take advantage of all of that data? How do we get the data to where it needs to be in the most efficient manner possible? Extended Functionality Ron said that with HarperDB, he and his team could “define the data movement and do all these crazy cool things.” As they were looking at different military adaptations, they were able to take data that is running integrated into a controller environment (OT) and expose that data without needing to have a human-machine interface (HMI). They could securely move that OT data into the cloud, into a highly scalable enterprise analytics domain powered by HarperDB in the cloud on compute nodes. There are many use cases similar to this, where HarperDB can provide a holistic solution that makes data sync and management easy. In the defense space, HarperDB’s bidirectional data movement enables the collection and movement of data and logic in real-time, shifting decision-making throughout the network as needed. Gaming and media industries benefit from HarperDB’s high performance and low latency, with clear implications for both the organization and the end-user. Retail and ticketing can recognize and block bad bots in real-time with HarperDB’s global replication and edge persistence. The list goes on! You can read more about . industries that benefit from a high-performing, low latency, geo-distributed database here Why HarperDB? Ron explained, “We started looking at all the different databases that are scalable like Couchbase and a bunch of others, but we ended up focusing on HarperDB because of the flexibility. Then, Stephen came up with the idea of Functions because a lot of what we did required us to put an API proxy in front of the data engine. He said, how about I make your life simpler? It’s just amazing how HarperDB checks all the boxes.” Ron continued, “If you think about how databases communicate and the different models, I love the way HarperDB does it through the native integration of all of these components. No matter what it’s running on or where, HarperDB is disruptive because I’m able to move the different types of data, and different types of assets like functionality, from place to place seamlessly without having to worry about the interoperability of different data engines, nor do I have to worry about the size and scale. Databases are not typically designed as persistent vs. non-persistent; they tend to be scaled vertically instead of horizontally. HarperDB scales beautifully; a containerized version of HarperDB tied to persistent storage allows me to scale HarperDB to meet my performance goals. The workload it can perform is amazing, and the ability to actually scale horizontally is amazing as well because it’s not typical for database engines.” Therefore, HarperDB is a unique solution for complex enterprise data challenges because the database engine is small and flexible enough to run on a microcontroller running on an onboard system, that can also be extended to edge bare metal or some edge computing environment for higher fidelity analysis, and can also be moved to the cloud -- all at the speed of the Internet. HarperDB can scale vertically horizontally while meeting performance needs. It really is more than just a database. and To sum it up, Ron stated: “From a data driven ecosystem, HarperDB is paving the path forward moving from mesh to fabric to actual data surface and providing that contextualization of data right out of the database engine, which will be key to a fundamental shift in application behavior.” Predictions for the Future To wrap up our conversation, I asked Ron about the future of technology. He mentioned a few key things: The move from cloud to edge is almost certain. The nature of applications will change to take on a more distributed nature, along the lines of distributed functionality with edge workloads managed and deployed from some cloud orchestrator. As we look at the nature of apps changing, data will be more contextualized from a database engine or persistence layer perspective rather than an application or business layer, and HarperDB is leading the charge on that. There you have it, folks. The future is all about data. There is a constant need for organizations to have their data where they need it, with the ability to orchestrate data where it’s both being created and consumed. If you’re not evaluating your data and how you’re handling your data assets, where will you be in 1, 5, or 10 years from now? Also published here

HarperDB is More Than Just a Database: Here's Why

Node is lightweight & scalable, allows us to develop quickly, and npm has incredible packages. The founding team at HarperDB built the first and only database written in Node.js. A little while back, our CEO Stephen Goldberg was invited to to share the story of this endeavor. Stephen discussed the architectural layers of the database, demonstrated how to build a highly scalable and distributed product in Node.js, and demoed the inner workings of HarperDB. You can watch his talk at the link above, and even read a , but since we all love Node.js and it’s an interesting topic, I’ll summarize here. speak at a Women Who Code meetup (what some called crazy) post from back in 2017 The main (and simplest) reason we chose to build a database in Node is because we knew it really well. We got flak for not choosing Go, but people now accept that Go and Node are essentially head to head ( ). Zach, one of our cofounders, recognized that with the time it would have taken to learn a new language, it would never be worth it. in popularity & community support Pros of building a Database in Node.js We already knew Node.js Lightweight Rapid development Highly scalable npm The HarperDB team has a background in large scale software development. The initial goal of our database was to create a tool that empowers developers to focus on coding, without having to devote time and effort to database maintenance, while still providing a powerful solution. We wanted people to feel comfortable and confident in the product they were using. Our team has extensive experience in languages other than Node, but we had great success programming in it. (Although coming from Java, Stephen thought Node was horrible at first, but after about 90 days he learned to love it). Node is lightweight, allows us to develop quickly, and npm has incredible packages. Cons of building a Database in Node.js At the time was not accepted as an “enterprise grade language” Does not have direct control of Operating System/File System Not as performant as C/C++ (sometimes) Did not have native threading (now it does) We did have some troubles... being the first database written in Node.js we didn’t have the option to follow in anyone’s footsteps. We’re probably one of the first enterprise products ever built in Node, at least the most data-centric one. People questioned this. One guy told Stephen that he would rather cut his heart out with a spoon than program a database in Node.js. We did run into challenges around not having direct control of OS in the file system. Also, C/C++ are faster, but can be more complicated and not necessarily as scalable horizontally. It really depends if you're looking for vertical or horizontal computing. Now people have realized this was a great idea because we have all these incredible features in our product that we didn’t have to build and are inherent in what we do. Tech Stack This is what our tech stack looks like. We consider our to be part of the HarperDB stack, and that is built in React with a Node back end. The green box signifies any application built on top of HarperDB, for example our can be used to build custom workflows. The HarperDB technology is built entirely in Node.js, which encompasses our interfaces and HarperDB core. Management Studio Node-RED node Our product presents itself as a REST API which, under the hood, is essentially just an Express application, that’s the primary interface for how you interact with HarperDB. Our NoSQL parser is a custom solution we built internally. We use AlaSQL for our SQL parsing functionality which you can , we extend their functionality with custom code on top of that, it’s an amazing npm package for parsing SQL. We offer drivers, like ODBC and JDBC, built by a partner of ours. Finally, we use SocketCluster for distributed computing and clustering which . read more about here our CTO presented on in August The HarperDB core technology encompasses the “secret sauce.” This is what makes it possible for us to be fully indexed with no data duplication and offer various interface options to a single data model. Within the core there are numerous npm packages implemented to extend our functionality. Finally we have various options for storage media. We bundle LMDB by default as it provides significant performance gains over the other options. HarperDB core contains extensible code that allows us to add additional storage media options in the future. REST API HarperDB is a set of microservices A single endpoint All operations are post Stateless/RESTful (Sample code found at https://docs.harperdb.io/ ) At a former company our team dealt with the headache of hundreds of API’s with different endpoints, which was simply insane. People might think it’s weird that HarperDB is just one endpoint, but if you look in the body of the code, for every operation you do- all you ever have to change is the body, those first few lines. This is super simple, and when writing a REST-based application you can make it really straightforward. This is something you can take from us and use in any application! Basically you post a single message to the API, we see what operation you’re performing, and handle it with a standard set of methods. We’ve rewritten a lot of our application over the last couple years but this part has stayed mostly the same. Management Studio Built on the HarperDB REST API Written in React Native Allows for control of your HarperDB instances via GUI The HarperDB is a React front end built on top of our microservices (so we eat our own dog food). React is amazing, it’s changed the quality of front end development and allows us to make our application more accessible. By building on top of this, we’re also testing our own API’s at the same time - which makes it really powerful. Jaxon our VP of Product chose React for the Studio, while Stephen wrote our back end reporting in Express. Management Studio One awesome thing about JavaScript is how lightweight it is, regardless of what framework you're using (Node, React, etc), and you can easily couple together these different layers. AlaSQL SQL search is built on AlaSQL https://github.com/agershun/alasql Allows for enhanced SQL Our devs contribute to the project , it has some great things in it that we don’t, and allows us to wire in things like Math.js and GeoJSON so it’s an incredible package. If we had to build our own SQL parser we’d probably still be building HarperDB. It took one of our competitors, FaunaDB, about 4 years just to get to market, but we launched the beta of our product in 6 months, the original version in 12 months, and we just released our cloud product a few months ago (about 3 years later). We chose AlaSQL for HarperDB's back end functionality One amazing benefit of using Node for a language like this is as technology is advancing, most of the cool stuff that you want and need is on npm. We’re not saying we’re geniuses, but by developing in Node we got to stand on the shoulders of people like AlaSQL developers which is what we find amazing about the npm community. Maths.js HarperDB uses math.js functions inside our SQL Allows for enhanced math capability while leveraging the capabilities of npm community Maths.js is another incredible package for things like averages, data science, etc., that we wired into our SQL capability. It’s not hard to use and very powerful in combination with AlaSQL. Clustering/Replication Built on SocketCluster.io Fault tolerant Peer-to-Peer Table level replication Globally shared schema Distributed Computing Another very cool feature of building something in Node.js is that it’s stateless by nature, meaning it does not require holding data in memory that is critical to serving clients across sessions, which is very resource efficient. Most enterprise grade applications have background processes and stateful variables that can become highly unstable. Node is stateless, designed for the web, designed to scale horizontally and to be peer-to-peer. An amazing benefit from using a Node framework is that we were able to wire in SocketCluster to power our clustering and replication. HarperDB uses a simple pub-sub model, so we replicate data by publishing data to different chat rooms which different nodes subscribe to and are able to be distributed horizontally. Node can be horizontally scalable and less resource intensive than other languages, and its stateless nature makes it incredibly stable. By putting Node on lots of computers (horizontally scaling) you can make the framework significantly more powerful while driving down costs, having easier development, and being part of an awesome community. LMDB & File System Originally built our exploded data model on the file system Problematic due to the generation of many files taking up inodes and excess disk space, and other issues Rebuilt data model on LMDB Massive performance gain Originally we were using the file system directly with the above HarperDB data model, this is what makes the product unique. As data comes in, we map it to our data model, it's not a SQL engine or NoSQL engine. We exploded that data into individual attributes and stored them in a folder structure on the file system. We store each thing atomically, and you can query via SQL and NoSQL. We did run into some challenges at scale, so , a key value store that we operate on top of. We were able to implement our exact data model on top of that and it has provided incredible performance gains. , largely thanks to LMDB. more recently we wired in a package called LMDB In a recent benchmark we were about 37 times faster than MongoDB Once again, by leveraging the amazing Node community we are able to focus on what we’re good at. Also published at https://dev.to/harperdb/building-a-database-written-in-node-js-from-the-ground-up-2hgm

HarperDB - How and Why We Built It From The Ground Up on NodeJS

New tech terms and lingo are constantly popping up in the development world. It’s hard to keep up! People also love to share and debate about their favorite tech stacks and tools / frameworks. That said, there is no ultimate right or wrong answer when it comes to choosing a tech stack because it really depends on the specific use case it’s meant to solve for. Today, let’s take a look at the HERN stack. Ever heard of it? It’s a new “buzz word” as they say, and I thought it might be helpful to provide some clarity. The HERN stack stands for HarperDB, Express, React, and Node, the four technologies that make up this particular stack, and they’re all written using JavaScript. This is a full-stack solution that follows a common 3-tier architecture (front end, application, and database). As mentioned, there are numerous stacks out there and different factors play a role in the decision around which is right for your specific end goal. Remember that you have the freedom to choose and try different stacks! To make sure we’re all on the same page, Stackshare defines a tech stack as “the set of technologies an organization uses to build a web or mobile application. It is a combination of programming languages, frameworks, libraries, patterns, servers, UI/UX solutions, software, and tools used by its developers.” So as you can see, the combinations and possibilities are endless! What Makes up the HERN Stack? Distributed SQL / NoSQL hybrid database accessed via a REST API. HarperDB : Open source, back end web application framework for Node.js, designed for building web applications and APIs. Express.js : Open source, front end, JavaScript library for building user interfaces or UI components. React.js : Open source, cross-platform, back-end JavaScript runtime environment that executes JavaScript code outside a web browser. Node.js : There are several reasons that these technologies work well together and serve as a great full-stack solution for projects like app development and web development. There are also variants of this stack that would work just as well, such as swapping out and incorporating frameworks like Vue, Angular, or . HEAN, HEVN, HFRN… don’t be afraid to get creative and choose what technologies make the most sense for you! Fastify How Does the HERN Stack Work? Like most tech stacks, this one has a frontend, backend, and database tier. The HERN stack allows you to write everything in Javascript, ultimately simplifying and streamlining the development process. Each component of the HERN stack provides a unique benefit, and data flows naturally through each of the three tiers, enabling rapid development and simple upkeep. ( !) Fun fact - HarperDB is actually the only database that was built from the ground up in Node.js, which you can read more about here Frontend React serves as the frontend or web tier of the HERN stack. Among the many reasons React is , it enables faster development, simplifies coding and integration, and comes with a strong community. The HarperDB was written in React. React has improved the quality of front end development and allowed us to quickly make changes and improvements to our application. taking over frontend development Management Studio HarperDB also provides a great , if that’s your style, or you could make calls directly to HarperDB via our built-in, easy to use API. Either way, you can query data directly from within your client-side application if you choose. are a great place to manage your API calls and how responses are handled within your app. React SDK React Hooks Backend Node.js and Express.js serve as the backend, server, or application tier of the HERN stack. Express essentially operates inside a Node server, as the server-side framework. is a minimal and flexible framework, with numerous HTTP utility methods and middleware at your disposal. Express The idea here is that you build your custom API functions (GET, POST, etc.) with Express to be called by the React front end. These functions then make calls from the server to HarperDB to access and modify the database. You can utilize the HarperDB , which natively supports promises and callbacks. The library makes HarperDB interactions feel more native to JavaScript by exposing each HarperDB operation as a function on the client object. Node.js client Database If you need to store or transport any type of data, which you probably do, you will also need a database. Why not choose the quickest, easiest, and most intuitive database out there? HarperDB provides full document store capability plus enterprise grade ACID-compliant SQL, effectively allowing you to run SQL on JSON. The database is accessed via a single endpoint REST API using SQL and NoSQL operations. HarperDB has simplified much of the work when it comes to database installation, configuration, and administration. It’s really a great option for developers of any skill level. Check out if you’re building in the cloud, or you can also do a local . HarperDB Cloud install on your server Is HERN Your Next Full-Stack Solution? You can essentially build anything with the HERN stack, and it’s especially great for JavaScript developers and projects with lots of JSON data. With a dynamic schema, HarperDB was designed to ingest and handle any type of data at scale. You no longer need to pick a database based on the structure of your data. HarperDB provides the flexibility to work with the frameworks of your choice, and ability to connect to your favorite reporting or analysis tools. Scaling is easy with HarperDB’s clustering and replication features, and our data model, built on top of LMDB, is enabling us to break records on both read and write speeds. HarperDB works extremely well with Express, React, and Node, and with this tech stack you can maximize your time writing code, and work on your project instead of debugging and other administration tasks. Don’t get stuck in your old ways! Try the new HERN, HEAN, or HEVN stack and let us know what you think. We would love to hear what you’re working on! Also published at https://dev.to/harperdb/hern-stack-comin-in-hot-pa6

Along Came HERN Stack

Following our ’ ranking, just got a major upgrade, all in service of representing your company’s internet footprint as accurately as possible on our news database of 11,000+ top tech companies in the world! Tech Companies HackerNoon’s Tech Companies News Pages Let’s dive into it. The New(er) Evergreen Pages If you visit , you’ll see four main pages: home, news, about, and blog: https://hackernoon.com/company/hackernoon features a logo, name, slogan, email, employee numbers, funding date, ranking, and description. It also includes a graphic of the company’s ranking and stock price (for public companies). The HomePage Scroll down for a list of similarly ranked companies ( ), brand mentions in HackerNoon stories, and a company wiki. with option to hide list reserved for Premium Brands only The company ranking’s top variable is interest from HackerNoon readers. To improve your ranking, direct more traffic to your evergreen page or write more stories on HackerNoon and boost both your content and your page. exhibits all stories that mention the company and presents a related companies’ listicle. Here’s what it looks like: The News page will lead you to the company’s wiki, which has all the essential info about the foundation and story of the brand, and the blog button will direct you to the brand’s profile page. Check out to fully grasp our updates. The About page Microsoft's page currently is reserved for our , who hosts their blogs on HackerNoon; and for Premium brands, who can link any external blog post to this section. (See more about Unlocking Evergreen Premium below) The Blog page business blogging customers In the future, primary brand profile pages, such as , will be replaced with Tech Company News pages as the primary business page (such as ). The collection of stories by the business will still be available as [company name] blog. this one this one Get Full Control Over Your Evergreen Pages How To Find Your Evergreen Page Congrats - if you are one of the 11,000 companies (thanks to ) that currently have an evergreen page built for you! Startups of the Year You can simply search for your company’s name or url in our search bar at the top left corner of every page. Or type your company’s name in the wide search bar of this page which ranks the top companies: hackernoon.com/companies Don’t see an Evergreen Page of your own? Apply via this form . here How To Claim Your Existing Evergreen Page To Claim an existing Evergreen Page, you need to have a company email that matches exactly with the domain of that company. For example, awesomehacker@hackernoon.com will be able to claim our page, HackerNoon. Then, click the yellow button on your page ⬇️ A pop-up will show on your screen asking you to: Select automatic claim: login/sign up to HackerNoon with an email matching the company’s URL Submit for review: login with any email and go through an editorial review of your claim. This is reserved for PR agencies that represent companies, or companies with different subsidiaries/parent companies. Here’s a preview of the overlay: You’ll need to be logged in to claim the page and only after your page has been claimed can you make changes to its content. ! Get your Tech Company News page today How to Update Your Evergreen Pages Once you claim your Tech Company News page, here’s what you’ll be able to edit (click on the “edit” button on the top right) edit your company name, description, wiki, website, employee number, and expertise here. About: add all your links - HackerNoon, LinkedIn, Facebook, GitHub, Twitter, YouTube and Instagram. Socials: add your company logo and ( ) . Images: for Premium Brands only a video header pick the top 6 stories that accurately represent your brand. HackerNoon Stories: curate which news you’d like to be shown on your profile. Web Articles (reserved for Premium Brands Only): See some examples of companies with very comprehensive internet presence here: https://hackernoon.com/company/minio?embedable=true https://hackernoon.com/company/microsoft?embedable=true https://hackernoon.com/company/harperdb?embedable=true How to Unlock Premium Features Loving your (already awesome) Evergreen Pages? Sweet! Why not unlock for even more premium features 😉 To preview Premium features as are on your Evergreen Pages, Click the “Show Upgrades” button on top right, and you’ll see ability to: Add A wide video banner Hide Competitor List Add non-HackerNoon links via the Web Articles section Link either your HackerNoon Blog or your corporate Blog via the Blog Section Premium features are all about control, so you can capture your company details and comprehensive presence on the interwebs. There are several ways they could make your company better, but we want to focus specifically on your ability to curate your 6 best articles around the web, as demonstrated below: As seen above, you can: Add an article with header image, title, published date, author, and URL Arranged your article in order by deleting/adding new ones Have total control over the top 6 most important links about your company The left column, in the Evergreen Premium example, is full of your company stories on HackerNoon, when/if your company name is mentioned. The best way to take advantage of this automatic feature, is of course, our . Or, if you are not interested in telling your own story and would rather the community talk about you instead, you might wanna check our . Business Blogging our Writing Contests Until next time, please enjoy your brand new curated Evergreen pages, if you have one, or apply for one , if you don’t! here

How to Find, Claim, Edit, and Upgrade Your Evergreen Tech Company News Page

আপনার চিরসবুজ টেক কোম্পানির সংবাদ পৃষ্ঠা কীভাবে সন্ধান করবেন, দাবি করবেন, সম্পাদনা করবেন এবং আপগ্রেড করবেন

So finden, beanspruchen, bearbeiten und aktualisieren Sie Ihre Evergreen Tech Company-Newsseite

Cómo encontrar, reclamar, editar y actualizar la página de noticias de su empresa Evergreen Tech

Comment trouver, réclamer, modifier et mettre à niveau la page d'actualités de votre entreprise Evergreen Tech

अपने सदाबहार टेक कंपनी समाचार पृष्ठ को कैसे खोजें, दावा करें, संपादित करें और अपग्रेड करें

Evergreen Tech Company ニュース ページを検索、請求、編集、アップグレードする方法

Evergreen Tech Company 뉴스 페이지를 찾고, 청구하고, 편집하고, 업그레이드하는 방법

Como encontrar, reivindicar, editar e atualizar a página de notícias da sua Evergreen Tech Company

Как найти, заявить права, отредактировать и обновить страницу новостей вашей компании Evergreen Tech

Evergreen Teknoloji Şirketi Haber Sayfanızı Nasıl Bulunur, Talep Edilir, Düzenlenir ve Yükseltilir

Cách tìm, xác nhận quyền sở hữu, chỉnh sửa và nâng cấp Trang tin tức về công ty công nghệ Evergreen của bạn

如何查找、认领、编辑和升级您的常青科技公司新闻页面

232 Stories To Learn About Product

278 Stories To Learn About Machine Learning

461 Stories To Learn About Slogging

143 Stories To Learn About Data Visualization

121 Stories To Learn About Databases

Let's learn about via these 53 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Data Scraping /Learn Repo 1. How I Successfully "Reverse-Engineered" ChatGPT to Create an Unofficial API Wrapper Scraping ChatGPT with Python 2. How POST Requests with Python Make Web Scraping Easier To scrape a website, it’s common to send GET requests, but it's useful to know how to send data. In this article, we'll see how to start with POST requests. 3. AutoScraper Introduction: Fast and Light Automatic Web Scraper for Python In the last few years, web scraping has been one of my day to day and frequently needed tasks. I was wondering if I can make it smart and automatic to save lots of time. So I made AutoScraper! 4. How To Scrape Google With Python Ever since Google Web Search API deprecation in 2011, I've been searching for an alternative. I need a way to get links from Google search into my Python script. So I made my own, and here is a quick guide on scraping Google searches with requests and Beautiful Soup. 5. My Journey Building a Scraper with Ruby Last week I finished my Ruby curriculum at Microverse. So I was ready to build my Capstone Project. Which is a solo project at the end of each of the Microverse technical curriculum sections. 6. How to Scrape NLP Datasets From Youtube Too lazy to scrape nlp data yourself? In this post, I’ll show you a quick way to scrape NLP datasets using Youtube and Python. 7. How to Scrape Data From Any Website With JavaScript Learn how to scrape the web using scripts written in node.js to automate scraping data off of the website and using it for whatever purpose. 8. Playwright Vs Selenium: Comparing the Two A brief comparison between Selenium and Playwright from a web scraping perspective. Which one is the most convenient to use? 9. How Web Scraping Helps Businesses Outperform Their Competition It’s safe to say that the amount of data available on the internet nowadays is practically limitless, with much of it no more than a few clicks away. However, gaining access to the information you need sometimes involves a lot of time, money, and effort. 10. Web Scraping Using Node.js While there are a few different libraries for scraping the web with Node.js, in this tutorial, i'll be using the puppeteer library. 11. Web Scraping con Python: Guía Paso a Paso La necesidad de extraer datos de sitios web está aumentando. Cuando realizamos proyectos relacionados con datos, como el monitoreo de precios, análisis de negocios o agregador de noticias, siempre tendremos que registrar los datos de los sitios web. Sin embargo, copiar y pegar datos línea por línea ha quedado desactualizado. En este artículo, le enseñaremos cómo convertirse en un "experto" en la extracción de datos de sitios web, que consiste en hacer web scraping con python. 12. America's Secret Pager Giant Early January 2022, I spontaneously bought a pager. I looked into the US pager market, and to my surprise... 13. 8 Browser Extensions for Scraping Google Maps like a Pro These extensions for scraping Google maps can be used for a number of purposes in various situations that can be either data collection or market research. 14. How Do I Build a LinkedIn Scraper For Free? Check out this step-by-step guide on how to build your own LinkedIn scraper for free! 15. What is Web Data Collection? Everything you need to know to automate, optimize and streamline the data collection process in your organization! 16. How To Scrape Amazon, Yelp and GitHub Profiles in 30 Seconds The most talented developers in the world can be found on GitHub. What if there was an easy, fast and free way to find, rank and recruit them? I'll show you exactly how to to this in less than a minute using free tools and a process that I've hacked together to vet top tech talent at BizPayO. 17. Web Scraping con Python: Guía Paso a Paso La necesidad de extraer datos de sitios web está aumentando. Cuando realizamos proyectos relacionados con datos, como el monitoreo de precios, análisis de negocios o agregador de noticias, siempre tendremos que registrar los datos de los sitios web. Sin embargo, copiar y pegar datos línea por línea ha quedado desactualizado. En este artículo, le enseñaremos cómo convertirse en un "experto" en la extracción de datos de sitios web, que consiste en hacer web scraping con python. 18. Web Crawling vs Scraping: What's the Difference Between Crawlers and Scrapers? Learn the fundamental distinctions between web crawling and web scraping, and determine which one is right for you. 19. Where Do I Find the Right Social Media Marketing Data? As a marketer, you probably know that social media marketing is part art, part science. 20. Scraping Google Search Console Backlinks Learn how to emulate a normal user request and scrape Google Search Console data using Python and Beautiful Soup. 21. How to Use Web Scraping to Empower Marketing Decisions Learn how to leverage web scraping in marketing. In this article, we unpack use cases and tips for getting started. 22. Scraping Glassdoor Job Data Glassdoor is one of the biggest job markets in the world but can be hard to scrape. In this article, we'll legally extract job data with Python & Beautiful Soup 23. Scraping Tweet Replies with Python and Tweepy Twitter API [A Step-by-Step Guide] A Quick Method To Extract Tweets and Replies For Free 24. Las 15 preguntas más frecuentes sobre Web Scraping Previously published at https://www.octoparse.es/blog/15-preguntas-frecuentes-sobre-web-scraping 25. A Step-by-Step Guide to Building a Football Data Scraper Scraping football data (soccer in the US) is a great way to build comprehensive datasets to help create stats dashboards.  Check out our football data scraper! 26. An Intro to No-Code Web Scraping Web scraping has broken the barriers of programming and can now be done in a much simpler and easier manner without using a single line of code. 27. How is Web Crawling Used in Data Science No-Code tools for collecting data for your Data Science project 28. How to Build a Web Crawler from Scratch How often have you wanted a piece of information and have turned to Google for a quick answer? Every piece of information that we need in our daily lives can be obtained from the internet. You can extract data from the web and use it to make the most effective business decisions. This makes web scraping and crawling a powerful tool. If you want to programmatically capture specific information from a website for further processing, you need to either build or use a web scraper or a web crawler. We aim to help you build a web crawler for your own customized use. 29. How to Develop a Price Comparison Tool in Python Online Shopping for various commodities is no more a luxury but has rather become a necessity now. Getting your desired product on your doorstep has made it easier for consumers to shop effortlessly. As a result, several niche e-commerce or generic shopping sites pop up every year. This trend is not limited to some specific region rather it’s a global phenomenon now, as more and more people are preferring online shopping over visiting outlets due to traffic congestions and ease of purchasing. This is why it’s predicted that by 2021, overall 15.5% of sales will be generated via online websites. 30. PHP Web Scraping Using Goutte When you talk about web scraping, PHP is the last thing most people think about. 31. How To Monitor a Forum for Keywords Using Python and AWS Lambda While building ScrapingBee I'm always checking different forums everyday to help people about web scraping related questions and engage with the community. 32. The Evolution of Big Data And Web Scraping As the CEO of a proxy service and data scraping solutions provider, I understand completely why global data breaches that appear on news headlines at times have given web scraping a terrible reputation and why so many people feel cynical about Big Data these days. 33. How To Create A Slick iOS Widget In JavaScript With a Scriptable app, it’s possible to create a native iOS widget even with basic JavaScript knowledge. 34. Scraping Amazon using Puppeteer and Browserless An easy tutorial showcasing the power of puppeteer and browserless. Scrape Amazon.com to gather prices of specific items automatically! 35. 3 Mejores Formas de Crawl Datos desde Website La necesidad de crawling datos web ha aumentado en los últimos años. Los datos crawled se pueden usar para evaluación o predicción en diferentes campos. Aquí, me gustaría hablar sobre 3 métodos que podemos adoptar para scrape datos desde un sitio web. 36. An Intro to Web Scraping: What it is and How to Start A quick introduction to web scraping, what it is, how it works, some pros and cons, and a few tools you can use to approach it 37. Data Scraping in Node.js 101 How to gather data without those pesky databases. 38. How to Scrape Bestbuy Products with Scrapezone SDK Welcome to the new way of scraping the web. In the following guide, we will scrape BestBuy product pages, without writing any parsers, using one simple library: Scrapezone SDK. 39. A Quick Primer on Data Scraping Suppose you want to get large amounts of information from a website as quickly as possible. How can this be done? 40. A Guide to Web Scraping With JavaScript and Node.js With the massive increase in the volume of data on the Internet, this technique is becoming increasingly beneficial in retrieving information from websites and applying them for various use cases. Typically, web data extraction involves making a request to the given web page, accessing its HTML code, and parsing that code to harvest some information. Since JavaScript is excellent at manipulating the DOM (Document Object Model) inside a web browser, creating data extraction scripts in Node.js can be extremely versatile. Hence, this tutorial focuses on javascript web scraping. 41. How to Extract Knowledge from Wikipedia, Data Science Style As Data Scientists, people tend to think what they do is developing and experimenting with sophisticated and complicated algorithms, and produce state of the art results. This is largely true. It is what a data scientist is mostly proud of and the most innovative and rewarding part. But what people usually don’t see is the sweat they go through to gather, process, and massage the data that leads to the great results. That’s why you can see SQL appears on most of the data scientist position requirements. 42. How to Scrape a Medium Publication: A Python Tutorial for Beginners A while ago I was trying to perform an analysis of a Medium publication for a personal project. But getting the data was a problem – scraping only the publication’s home page does not guarantee that you get all the data you want. 43. Scraping Data With Selenium: Upwork Series #2 Hi Devs! 44. How To Scrape Amazon Using Python Scrapy Library [Tutorial] Scrapy is an application framework for crawling web sites and extracting structured/unstructured data which can be used for a wide range of applications such as data mining, information processing or historical archival.As we all know, this is the age of “Data”. Data is everywhere, and every organisation wants to work with Data and take its business to a higher level. In this scenario Scrapy plays a vital role to provide Data to these organisations so that they can use it in wide range of applications. Scrapy is not only able to scrap data from websites, but it is able to scrap data from web services. 45. Scraping Amazon Reviews using Scrapy in Python [Tutorial] Are you looking for a method of scraping Amazon reviews and do not know where to begin with? In that case, you may find this blog very useful in scraping Amazon reviews. In this blog, we will discuss scraping amazon reviews using Scrapy in python. Web scraping is a simple means of collecting data from different websites, and Scrapy is a web crawling framework in python. 46. 5 Técnicas Anti-Scraping que Puedes Encontrar Con el advenimiento de los grandes datos, las personas comienzan a obtener datos de Internet para el análisis de datos con la ayuda de rastreadores web. Hay varias formas de hacer su propio rastreador: extensiones en los navegadores, codificación de python con Beautiful Soup o Scrapy, y también herramientas de extracción de datos como Octoparse. 47. How Can The Travel Industry Benefit From Data Scraping The travel industry is a major service sector in most countries these days. It is also a major employment and revenue provider. This demands a lot of constant innovation and maintenance. The travel industry is a dynamic industry where the needs and preferences of a customer change every moment. The market players in this field need to keep up with the trends in the industry, the choices of the customers and even on the details of their own historical performance to perform better as time progresses. Thus, as you would presume, the companies working in the travel sector need a lot of data from multiple sources and a pipeline to assess and use that data for insights and recommendations. 48. The A-Z of Web Scraping in 2020 [A How-To Guide] Web data extraction or web scraping in 2020 is the only way to get desired data if owners of a web site don't grant access to their users through API. 49. How to Web Scrape Using Python, Snscrape & HarperDB Learn how  to execute web scraping on Twitter using the snsscrape Python library and store scraped data automatically in database by using HarperDB. 50. Big Data: 70 Increíbles Fuentes de Datos Gratuitas que Debes Conocer para 2020 Por favor clic el artículo original：http://www.octoparse.es/blog/70-fuentes-de-datos-gratuitas-en-2020 51. Scraping with Selenium 101: The Big Hole on Data Scientists Toolset [Part 1] Usually forgotten in all Data Science masters and courses, Web Scraping is, in my honest opinion a basic tool in the Data Scientist toolset, as is the tool for getting and therefore using external data from your organization when public databases are not available. 52. Python for Data Science: How to Scrape Website Data via the Internet's Top 300 APIs In this post we are going to scrape websites to gather data via the  API World's top 300 APIs of year. The major reason of doing web scraping is it saves time and avoid manual data gathering and also allows you to have all the data in a structured form. 53. How To Build a First Strike OTM Call Options Watchlist from Cashtags wHAOR Today, We're going to build a script that scrapes Twitter to gather stock ticker symbols. We'll use those symbols to scrape yahoo finance for stock Options data. To ensure we can download all the Options data, we’ll make each web request with High Availability Onion Routing. In the end, we’ll do some Pandas magic to pull the first out of the money call contract for each symbol into the final watchlist. Thank you for checking out the 53 most read stories about Data Scraping on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

Stories, News & Mentions about HARPERDB

Harperdb’s Stories on HackerNoon

Harperdb's News & Mentions