Wepipeis in the Sales Force Automation Industry

Wepipe

wepipe

Sometimes, you can't simply open your in Chrome to test your website. You need the features of a real web server like NGINX. index.html A common case where directly loading falls short is trying to serve local files using the Fetch API. You run into Cross-Origin Resource Sharing (CORS) errors. I hit this little bump when creating the example for this . In addition to preventing CORS errors, it's more realistic to use a production web server in your development environment. index.html blog post about the ClipBoard API To get started, you only need to and run one command. install Docker $ docker run -d --name nginx-development -v $(pwd):/usr/share/nginx/html -p 80:80 nginx The above command does a few things. The flag ensures that you don't get stuck attached to the container. Using the keyword argument gives this container instance a name, so we can reference it later. The maps our local machine's port to the container port that NGINX uses to serve HTTP. This port-mapping allows us to go to to see our development site. -d --name nginx-development -p 80:80 80 http://localhost The keyword argument needs a little more explaining. It links our current working directory to a directory the running container instance. This directory in the container is the default location where NGINX looks for files to serve. If you want to serve a different directory, then you need to change the argument. For example, I like to put my at in my project directory. To adjust for that, my argument would look like . -v $(pwd):/usr/share/nginx/html in --volume index.html ./src/index.html -v -v "$(pwd)/src:/usr/share/nginx/html" If you use your browser to navigate to , you should be able to see your site. But if you make changes to your or any other files on your site, refreshing your browser won't show them. To reload the site after you make changes, use the following command. http://localhost index.html $ docker restart nginx-development It's good that we can reload our changes, but it requires the extra step: running the above command. With the previous workflow – opening up directly – we only needed one step: refreshing our browser. Let's fix that 😤 docker restart index.html Writing an Auto-Reload Script To create this script, we need to either Poll the directory every so often and keep track of changes ourselves Have some way to be notified when there is a change To avoid managing our own state and to keep our bash script simple, let's go with option 2. How can we get these notifications? On Linux, there is a command-line tool called that relies on the underlying . We could use this, but then we would be locked-in to Linux. I develop on Mac OS and Linux, so I'd like to support both – even better if we can support Windows too. inotifywait inotify C API There is another tool called that works on Mac OS, Linux, and Windows. It may not be installed by default on your operating system. On Mac OS, I needed to install using the following command. fswatch fswatch $ brew install fswatch Now let's see how this tool works. Let's try running the tool and pointing it at our source directory. $ fswatch src My console just hangs with a blinking cursor. Let's see what happens if we modify one of the files in the directory. I'll try editing one of my css files... src $ fswatch src
/Users/curtis/git/licorice-works/src/styles/4913
/Users/curtis/git/licorice-works/src/styles/projects.css
/Users/curtis/git/licorice-works/src/styles/projects.css
/Users/curtis/git/licorice-works/src/styles/projects.css~ Wow 😲 Almost immediately, gives us the above output with name of the file that I modified. fswatch This is a lot of change notifications for modifying one file though. It looks like we don't want notifications for the file or for the file. These seem to be oddities of how saving files works on Mac OS. Luckily, the tool has some arguments that we can use to filter the change notifications. By default, it includes everything. This behavior makes including exactly the file types we want a little weird. First, we need to use to exclude everything. Then we can use to re-include only files that end in . With our new command, we only get notifications for the CSS file that we changed – not those other weird files. 4913 .css~ fswatch -e '.*' -i 'css$' css $ fswatch -e '.*' -i 'css$' src                     
/Users/curtis/git/licorice-works/src/styles/projects.css
/Users/curtis/git/licorice-works/src/styles/projects.css This is still two notifications though. We only want to reload NGINX once per change. Thankfully, has a flag that we can use to batch the changes together. fswatch --one-per-batch $ fswatch -o -e '.*' -i 'css$' src
2 After adding , we only get one line of output when we modify a file. Let's use this output to trigger a restart of our Docker container. -o $ fswatch -o -e '.*' -i 'css$' -i 'html$' -i 'js$' src | \
while read -r line; do docker restart nginx-development; done That command looks more complicated than the commands we've used so far. Let's dig in. First, I added two more include filters. One to include any updates we make to files and another to include files. Then we pipe the output of this command into a while loop. If you're not familiar with pipe, it just sends the output of the first command as input to the second command. The second command that we're piping into is a while loop. The general syntax of a while loop looks like the following. -i .html .js | while <condition>;
do
  <command to execute>
done It's a little easier to read when it's not squished onto one line. For the part, we used a command . Whenever there is a line of input to read, this command reads it, stores it in the variable , and returns true. In our script, this means whenever prints out a batch of changes, our loop will execute once. So each time there is a batch of changes, we will execute our restart command which is the same as above . <condition> read -r line line fswatch docker restart nginx-development You could get started by running the two commands. $ docker run -d --name nginx-development -v $(pwd):/usr/share/nginx/html -p 80:80 nginx
$ fswatch -o -e '.*' -i 'css$' -i 'html$' -i 'js$' $(pwd) | while read -r line; do docker restart nginx-development; done But this is a lot to remember or to copy-paste each time, so I wrote a script that takes only one argument: your source directory. You can copy this script and easily reuse it for each of your projects. nginx-autoreload.sh #!/bin/bash

CONTAINER_NAME="nginx-development"

# Use the first argument as our source dir.
# Default to current working directory if no argument is supplied.
SOURCE_DIRECTORY=$1
if [ ! $1 ]; then
    SOURCE_DIRECTORY="$(pwd)";
    echo "No source directory provided. Defaulting to current working directory: ${SOURCE_DIRECTORY}";
else
    # Get the full path
    SOURCE_DIRECTORY=$(realpath "${SOURCE_DIRECTORY}")
fi

docker rm --force "${CONTAINER_NAME}" &> /dev/null;

docker run -d --name "${CONTAINER_NAME}" -v "${SOURCE_DIRECTORY}":/usr/share/nginx/html -p 80:80 nginx > /dev/null;
if [ $? -ne 0 ]; then
    echo "Failed to initialize container";
    exit 1;
else
    echo "Successfully started container: ${CONTAINER_NAME}";
fi

echo "Waiting for changes..."
fswatch -o -e '.*' -i 'css$' -i 'html$' -i 'js$' "${SOURCE_DIRECTORY}" |
while read -r line; do
    docker restart "${CONTAINER_NAME}" > /dev/null;
    if [ $? -eq 0 ]; then
        echo "Successfully reloaded ${CONTAINER_NAME}";
    else
        echo "Failed realoading ${CONTAINER_NAME}";
    fi
done You can use the script like , and it will take care of setting up the container and watching for changes. $ ./nginx-autoreload.sh <your-source-directory Note: remember to install Docker and fswatch first! We did it! We wrote a single script to host and auto-reload your development site with NGINX 🥳 Also published . here

How to Use NGINX to Host a Local Site Using Docker

You can’t really invest in a data team too early. That’s something we learned as a small but growing team. We invested in a data team at an early stage, so we could establish product usage trends, create business insights, and identify areas to improve our product. If you’re an early-stage startup, start building your data strategy team and architecture as soon as you can. You don’t have to go big—a one-person data team can make a massive difference to your success as you grow. Hire a Generalist First When it comes to launching your data team, you might be tempted to hire an expert from the largest brand name you possibly can. Our advice: find someone with early-stage startup experience and has worked with data at scale too. There are wonderful people with brilliant minds working on large enterprise data teams, but these large teams often have their employees specialize in specific parts of their data platform. That makes sense when you have 10 data engineers on a team or when you have a large team of machine learning data scientists working on pricing algorithms. But when you’ve only got one data role for the entire organization, you need someone who can do a little bit of everything. It’s important to first hire an all-rounder who can create data insights, develop business hypotheses, and create a scalable data architecture from scratch. You need someone ready to start building the data team from the ground up, preferably someone who can hire out the rest of your team once you’re ready to do so. This first hire should be someone who has experience building (or at least working on) a data team at an early-stage startup. They need to be able to generate value quickly and have the business acumen to ask the right questions. They should be able to approach data in an 80/20 manner to deliver immediate results instead of diving into the hardest problems that you have. For the data engineering side, it’s common to hire a consultant or contractor to help set up your architecture. We decided to complement our first hire with a consultant. This way, we have the best of both worlds – an expert in data engineering part time, and a generalist who can maintain data architecture on a day to day basis. We set up strong foundations and also kept our knowledge in house. A Small Data Team Can Answer Big Questions Even if you can’t hire right away for all the roles you want on your data team long term, you can do a lot with only one or two of the right people. The key is thinking in terms of scalability and efficiency. What helps your data team do more with less? And how will this work when your team is bigger? Scalable analyses are extremely important with a small data team. By thinking ahead about how your processes and workflows will function as your team grows, you’ll save time and create valuable solutions for some of your employees’ most time-consuming issues. For example, we immediately realized that our sales team had no visibility into prospective customer product usage, and it was going to come up often. So we built our sales reps a workspace dash so they can pull the data themselves. Our data team’s job is to enable data-driven decision-making. When we saw an opportunity for a one-time data investment that could lead to hundreds of saved hours on the sales side, we acted on it. A small data team should also be focused on preventing issues in the future. If you only have the headcount for a one-person data team at the moment, keep them focused on your biggest problems but also budget time to maintain and scale your data architecture to reduce tech debt. The Tools Our Data Team Uses From the beginning, our goal was to be forward-thinking with every decision. Like all startups, we wanted to generate immediate ROI. We use a centralized data warehouse that is our single source of truth – nothing hurts data teams more than having fragmented or untrustworthy data. We’re able to consolidate our data thanks to automations and integrations. This helps us provide valuable insights to the entire company with low lift. We pipe in all of our data using ELT tools like Segment (which is also our data collection tool), Fivetran, and Snowflake external stages. We use to clean all of our raw data into a more usable format. For example, our dim_user table has ~20 tables powering it under the hood, that have a lot of critical info on our customers, such as how many notifications they send and what plan type they are on. By making this so readily available, our non-technical users can quickly draw business insights. We can quickly look at non-paying or self-serve customers that are sending a lot of notifications to identify business contract prospects. Our customer success team also built reports to look at the health of our business contract customers. We also use dbt to clean our data of sensitive information. dbt We use Census to send our warehouse data back to places such as HubSpot and Courier. This allows our sales and marketing teams to access this information on the platform that they are using on a daily basis. Metabase is our BI tool. It allows us to create dashboards, write SQL queries, and provide a self-serve format for non-technical users. We’ve found that creating these kinds of self-serve data dashboards allows users to answer a lot of their own questions. They are quick to spin up without large investments upfront like teams that use Looker or Tableau. For example, we have a KPI dash that shows the top metrics across the company. We can also track how individual workspaces are performing and identify workspaces that we can target for our sales team. We have also brought data-driven decision-making to our product development by deep-diving into our customers’ product usage. One example is that we found that a lot of our users were creating their own Courier instance instead of joining their colleague’s existing Courier instance. Using this insight, our product team went back to the drawing board to create a page that surfaces Courier workspaces that are created by the same business email domain so you can easily request access. We also use Propel to power our in-app analytics to provide visibility for our end users. One way that we use Propel is to deliver template analytics for our Business-tier customers. Propel allows our engineers to easily put GraphQL visualizations on top of our metrics from our warehouse. Our Data Team Is Evolving Again Now, we’re entering a period of growth, so naturally, we’re evolving our data approach again. Our current goal is to enable our data scientists to build subject matter expertise while still maintaining centralized data standards. Our current structure is centralized, with a senior analytics engineer reporting to the head of data. It works now since we’re still in the earlier stages of building out a data team. But when the team expands, we will be transitioning to a hybrid model. Data scientists will cover certain business areas (like the product team), but they’ll still report to the Head of Data. This will allow the data scientists to build subject matter expertise while still maintaining centralized data standards as we continue to grow. Because the data team will always be changing and growing along with the rest of the organization, you need standards that will scale with the business. If you’d like to learn more about how we approach data standardization, check out our blog post on . how Courier became HIPAA compliant

Establish Your Data Team Early Using These Tools and Techniques 

Establezca su equipo de datos con anticipación usando estas herramientas y técnicas

Établissez tôt votre équipe de données à l'aide de ces outils et techniques

इन उपकरणों और तकनीकों का उपयोग करके अपनी डेटा टीम को जल्दी स्थापित करें

これらのツールと手法を使用してデータ チームを早期に確立する

Estabeleça sua equipe de dados com antecedência usando essas ferramentas e técnicas

Thành lập sớm nhóm dữ liệu của bạn bằng cách sử dụng các công cụ và kỹ thuật này

使用这些工具和技术尽早建立您的数据团队

Over time most Linux aficionados accrue a sparkling war chest full of hard-won tricks that can come in tremendously handy when a situation demands quick thinking at your terminal. I stashed away many of these bits of knowledge over the years and whenever I had the opportunity to watch over the shoulder of somebody exceptionally well-versed with Linux. Today, you're pairing with me at the terminal. We're exploring the depths of Linux filesystem and shell tools and tricks. Take a with me /proc One of the most useful directories in a Linux system is . From the page for : /proc man proc The proc filesystem is a pseudo-filesystem which provides an interface to kernel data structures. When the page says "pseudo-filesystem", it means that if you were to peek underneath your disk where you might expect to find bits representing a file like you would for a text file at , there's nothing at . It's present and alive on a running Linux system, but entirely absent if you were to pull the disk out and inspect it. a control panel for your running kernel! man /tmp/launch-codes.txt /proc /proc If you were to take a look in your own right now, you might find a lot of directories like the following: /proc ls /proc 1
10
10021
10059
10144
...hundreds more files... Each of those numbers represents a process ID, or – yes, the same that identifies the process for your browser or terminal program. In fact, you can interrogate lots of information about the process itself. For example, you might recall that process on a Linux system is traditionally the top-level process, which in most modern systems is systemd-based. Let's see the command that kicked off on my system: PID PID 1 init PID 1 cat /proc/1/cmdline /run/current-system/systemd/lib/systemd/systemd is a file that tells us the command that kicked off process - in this case, itself. cmdline 1 systemd There's a file in that is particularly useful - , which actually shows you the arguments passed to your kernel itself at boot time. Mine is very wordy, but tells me the that my system booted with, along with any other flags, which in my case is and : cmdline /proc /proc/cmdline initrd init loglevel cat /proc/cmdline initrd=\efi\nixos\hx5g5rmvq748m64r32yjmpjk3pmgqmr1-initrd-linux-5.17.11-initrd.efi init=/nix/store/9zvklk45yx41pak2hdxsxmmnq12n712k-nixos-system-diesel-22.05.20220604.d9794b0/init loglevel=4 My NixOS hostname is . Please note that I do not put petroleum in my laptop. diesel isn't just read-only, either. Like its man page says, is an to the kernel, which includes interacting with the kernel itself. The directory holds a variety of knobs and dials, but I want to show you , which lets us peek at the kernel's virtual memory. Want to get more adventurous? /proc /proc interface /proc/sys /proc/sys/vm Consider my machine's current memory usage. free -h total        used        free      shared  buff/cache   available
Mem:           31Gi        22Gi       3.0Gi       4.4Gi       5.6Gi       3.6Gi
Swap:          31Gi       130Mi        31Gi Nothing too unusual here – but what if I wanted to free up my memory aggressively? Most of the time, the kernel knows best when it comes to using memory for caching, but there are some situations in which you may want to clear any memory that is safe to clear – we don't want to break any running processes, just reclaim memory if possible. It turns out that there's a file for that. We pipe an command into because is usually write-protected and only can write to the file we're interested in. echo sudo tee /proc/sys/vm root echo 1 | sudo tee -a /proc/sys/vm/drop_caches What this command does is effectively signal to the kernel, "please drop any caches in memory that you can afford to lose without breaking any running processes on my system." On my machine this opens up about 500M of memory: total        used        free      shared  buff/cache   available
Mem:           31Gi        22Gi       3.5Gi       4.4Gi       5.1Gi       3.6Gi
Swap:          31Gi       130Mi        31Gi Cool! There are all sorts of useful file-like objects in that can do interesting things like this. Feel free to open up if you'd like to learn more. /proc man proc as a prehistoric /dev curl A character device like represents an attached disk, but there's another use for the path: a little-known way to send network requests. /dev/sda /dev The path isn't actually a file-like device exposed by the Linux kernel, but actually a feature of your chosen shell like . Shells can intercept operations on this path in order to open up low-level socket connections to remote endpoints like web servers listening on port . /dev/tcp bash 80 To begin, open a new file descriptor connected to a file path in that indicates the desired endpoint and port. Much like how file descriptor numbers 0, 1, and 2 represent , , and , respectively, you can think of this new file descriptor 3 as representing a pipe to a remote network endpoint. We'll assume the use of from here on out. /dev/tcp stdin stdout stderr bash exec 3<>/dev/tcp/httpbin.org/80 Next, send the plaintext form of a simple HTTP request to the open file descriptor. This request to also requires the header to be set in order to be properly handled by most reverse proxies. Two newlines signal termination of the request: GET /status/200 Host echo -e "GET /status/200 HTTP/1.1\r\nHost: httpbin.org\r\n\r\n" >&3 Finally, a simple read operation retrieves the HTTP response: cat <&3 You should see a response similar to the one below: HTTP/1.1 200 OK
Date: Fri, 10 Jun 2022 21:39:43 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true Congratulations! You've just sent an HTTP request using nothing but your shell. Swimming in /sys There's one more root-level directory to explore after diving into and : the enigmatic directory. /proc /dev /sys Like and , is another file-like interface to low-level mechanisms that sit very close to the operating system. Unlike - which is relatively process-focused - and - which models block devices and more - is a useful interface to many abstractions that the kernel models. /proc /dev /sys /proc /dev /sys For example, take the directory . Inside of this directory, you'll find a list of links that represent the network interfaces on your host. Here's what mine looks like: /sys/class/net ls /sys/class/net enp0s20f0u6u4u1
lo
tailscale0
wlan0 As you can see, the active network connections that my system is managing include a wired interface (the interface that begins with ), the loopback interface, a interface, and my wireless interface . Listing the contents of one of these directories reveals a long list of files, but let's look more closely at two files in particular for my wired network interface: en lo Tailscale wlan0 cat /sys/class/net/enp0s20f0u6u4u1/statistics/rx_bytes
cat /sys/class/net/enp0s20f0u6u4u1/statistics/tx_bytes 11281235262
274308842 Each of these files represents the number of received bytes and transmitted bytes, respectively. Check out how the numbers change if I use the same command a few seconds later: cat /sys/class/net/enp0s20f0u6u4u1/statistics/rx_bytes
cat /sys/class/net/enp0s20f0u6u4u1/statistics/tx_bytes 11289633209
274760138 Bigger numbers! Apparently I'm making the most of my bandwidth. How is this useful? Have you ever wondered how network usage widgets are written? Well, how about making your own? Check out this small script that uses the aforementioned files in the directory to derive a network activity rate. bash statistics interval=1
interface=$1

rx_bytes=$(cat /sys/class/net/$interface/statistics/rx_bytes)
tx_bytes=$(cat /sys/class/net/$interface/statistics/tx_bytes)

rx_bytes_rate=0
tx_bytes_rate=0

function fmt() {
  numfmt --to=iec-i --suffix=B $1
}

while true
do
        echo -en " $(fmt $tx_bytes_rate)/s ⬆ $(fmt $rx_bytes_rate)/s ⬇\t\r"
        sleep $interval
        old_rx_bytes=$rx_bytes
        old_tx_bytes=$tx_bytes
        rx_bytes=$(cat /sys/class/net/$interface/statistics/rx_bytes)
        tx_bytes=$(cat /sys/class/net/$interface/statistics/tx_bytes)
        tx_bytes_rate=$(( ($tx_bytes - $old_tx_bytes) / $interval ))
        rx_bytes_rate=$(( ($rx_bytes - $old_rx_bytes) / $interval ))
done You can place this script somewhere in your , make it executable with , and try it out with . Here's what the output looks like on my machine: $PATH chmod +x <script> script.sh <interface name> 13KiB/s ⬆  379KiB/s ⬇ That's pretty cool! You can imagine some uses for this: for example, as a widget for a tool that can render command output or as a quick way to look at network activity for a particular network interface. In either case, the file-based interface to this data makes accessing and using it exceptionally easy. Further Exploration This was just a small dive into the types of information available to you as you look further into the capabilities of a modern Linux system. You can search for additional guides like this one or go straight to the source by reading pages for entries like in order to read the function and purpose of various directories in . man man hier / Have fun exploring!

20,000 Leagues Under Your Shell

20.000 leguas bajo tu caparazón

20 000 lieues sous votre coquille

20,000 लीग अंडर योर शेल

シェルの下で 20,000 マイル

20.000 léguas sob sua concha

20.000 giải đấu dưới vỏ bọc của bạn

20,000 联赛在您的壳下

is one of the most well-known p2p libraries. Libp2p It is the base of the and it will be the base of , so it is a very good choice if you want to develop decentralized applications. Interplanetary Filesystem Ethereum 2.0 In this article, I will show the basic conceptions of libp2p and we will develop a minimalistic browser-based (no own server needed) chat application using the JavaScript implementation of libp2p and vue.js. PeerId is one of the main elements of libp2p. PeerId It is something like a BitCoin or Ethereum address that is generated from the public key part of a keypair. Like a BitCoin address, it is unique and can be used to sign a message by the private key that can be validated by the public key that is associated with the peerId. You can generate a peerId by the utility. peer-id You can simply install peer-id by npm: sudo npm i peer-id -g After the installation, you can simply generate your very own peer-id by the peer-id CLI tool: peer-id { "id": "QmPMKmmRyepsh4Wq24msEdeGJQWSszy7eWNSjL53bxz3sS", "privKey": "CAASqAkwggSkAgEAAoIBAQC6/4DQXtW9HYbx5wPNbcsNbwMITH8Mg70O+zhQ9t9w/9Z7BuOzwQqu/ABaRclkl1FIuGfIayKSzqkczYkdY3I6bMUtn8dyHuJi1JzJj5PtaGfK81ra6hNvkVRqttYKJdgpPdGWCa6ouRbFPf3l0w2sK0ejf1Im/h4bAL1ltgZK7G+fXZTlUdAoEvQJja3bl9b47zqWiq+oVaez7uBhemrJQ6Ao09Tlr/RpVfbsc48NAczuDqh9nZzUq27LQEu9OXOf6xk/hPaDcof0ubQs9REXDCcDq01Cg6dQ5odKKHcT9hTeytZlvzfZZgleU8pME9r1EVpG5aEMn5qC0zfoe3PTAgMBAAECggEAWBoa+ZFEyHYJ5xy9WOMaoLilyBoqXZ4Py+gmj1bQzS9sQMhtLXqM6waFsAJjMUZtoIJpOy7muh4t5QkdScBZyBcJC0bVM/pDFOcw+3Hu8xKWnDLtomhYQd9J04FS9LMB1eRvQ25KYOnbRZDAd7BpJ624cdqBvSdKzdQaZ7pL2q4d/P1DUVLVfNDHzHlHyiLoxL1Ilm5YQtGGA3PwTLmSMqWGPvrzK4n0qL2rXUMbio0wf7w3lWDnBNQpncwdcPqUSTPDItwa2GNImH5yZRIF/vP2Qdky7LvNoxIzr+UhyQ6s2YSfRLwgnSPNL4IRuRaUWCtFprdaMd/x65xkBDVP2QKBgQDqXV922hbpMWgGPUco0uv7C8UApWiA/TP95pDzJuoD/SL0pn15QQrPMYTbOf32DrwC/sVMuuIbStqJd8RSApTsQO48KiPgmJM6WgD35umlYpmPQh8djvhPTe3nCadoxy+40i7NuuDde9AtvDlqwti4t1E/6lnYNZecbrXnT3ELZwKBgQDMQryboWcrUS+pfrJADtNiBOBtAAIQz7aHZRmk9ZFY7L8ekSB1Cuee4/FXswkrie+jE1rGbV97OQ1f6qW10SkoN64OIs8AVx408MxLdEejLg0vLwjEqEVXE2Wc1/TkOuEOpsrniKgisIZThrdC2RhmOqtaKRnAWz761ypCluP8tQKBgQCzqzV+Zh9eUpQPBHdDIr/qS9GRdz0wdeyf31yMK+8Hc86Sg/h5NpXU1X+mmUTKl+0m1q3m7vZcOfxjmr+Up4oHvJdm5F9w1Uc5WrqXUh0Yvwg+PVChVnOiSHnzvwDqYJmDNQ7QhU3SPhMQnNXftNR0d4UAXObXy+4Y7P7i/5IITQKBgQCtIxh6Frq7lep/kjwHXknA+P8+hVY658YBSCoPkHOuW6a4gy1u6FpibTZCLyjjtdzhbuNv9H+NlFOI7P2fevaW93Na2hh6Yl3hZAbXIm4inENirkyRXUzBPVjRNKCI4HuqDqlIzqYuGVES9crbJ+etp6ddGh+Q1AczWjShEwOXTQKBgF2cuW8w7ghAXapb2BKQU9bZTMAevohw6cxEmajNngVGSGMQJIJdhEoaGUXdtBOpOxAfYdycegf9VkPraVERXAbCMDk5mAh4Aor1SsHUDnQ7VKMLQfGzH3Jfk2TpXqy/11rJlLAl05LCyAeHGLVM4CQrvpLOIeM5n6j/3QWO3Jdh", "pubKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC6/4DQXtW9HYbx5wPNbcsNbwMITH8Mg70O+zhQ9t9w/9Z7BuOzwQqu/ABaRclkl1FIuGfIayKSzqkczYkdY3I6bMUtn8dyHuJi1JzJj5PtaGfK81ra6hNvkVRqttYKJdgpPdGWCa6ouRbFPf3l0w2sK0ejf1Im/h4bAL1ltgZK7G+fXZTlUdAoEvQJja3bl9b47zqWiq+oVaez7uBhemrJQ6Ao09Tlr/RpVfbsc48NAczuDqh9nZzUq27LQEu9OXOf6xk/hPaDcof0ubQs9REXDCcDq01Cg6dQ5odKKHcT9hTeytZlvzfZZgleU8pME9r1EVpG5aEMn5qC0zfoe3PTAgMBAAE=" } Calling peer-id will generate a JSON with the private and the public key, and the id which is a multihashed public key. This id is your unique peer address. Multihash is a future-proof hashing format that is used by libp2p and IPFS. It is built from 2 parts. Multihash The first 2 characters are the hash header that defines the hashing algorithm, and the second part is the hash itself. In the above example (in the id part of the peerId JSON) Qm means it is an SHA-256 hash and the second part is the base58 encoded 256 bits hash. Multiaddress is something like an HTTP URL, but it’s more general. A multiaddr is a list of protocol/value pairs that can point to a host, a service, a peer, etc. Multiaddress Let’s see some examples: - this multiaddr points to a host. /ip4/1.2.3.4 - this points to TCP port 80 of the host. /ip4/1.2.3.4/tcp/80 - this points to the UDP port 123 of . - this points to a peerId. - this multiaddr exactly points to a host and a TCP port where we can communicate to the given peer. /dns4/example.com/udp/123 example.com /p2p/QmbEKwwsuzArDenmWJfcFgyQ8uATQrmkjAsaT8VaGocV1x /ip4/7.7.7.7/tcp/4242/p2p/QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N Now we know the basics, let’s go forward to the implementation. Libp2p is a modular network stack with many implementations. In this article, I will use the JavaScript implementation that can run on the server-side in node.js or in your browser. Let’s see some code to understand how js-libp2p works: 'use strict'

const Libp2p = require('libp2p')
const TCP = require('libp2p-tcp')
const { NOISE } = require('@chainsafe/libp2p-noise')

const createNode = async () => {
  const node = await Libp2p.create({
    addresses: {
      listen: ['/ip4/0.0.0.0/tcp/4321']
    },
    modules: {
      transport: [ TCP ],
      connEncryption: [ NOISE ]
    }
  })

  await node.start()
  return node
} The above example shows how can we start a libp2p node. The Libp2p.create method has two options here. The first is the address where the node will listen. In this case, the node will listen on every IP address and on 4321 port. The second option is the definitions of the modules. Libp2p is a strongly modular stack. Every module is defined by an interface, and you can freely choose which one would you use, or you can create your own modules. Every part of the stack is modular. You can define the transport layer, the encryptions, you can define your own peer discovery mechanism, etc. In this case, we will use TCP transport and NOISE encryption. The above code works fine in node.js, but in a web browser, it’s not possible to open server ports. To solve this problem, libp2p can use relay servers that can open the required server ports and forward the traffic to the browser. this.libp2p = await Libp2p.create({
  addresses: {
    listen: [
     "/dns4/wrtc-star1.par.dwebops.pub/tcp/443/wss/p2p-webrtc-star",
     "/dns4/wrtc-star2.sjc.dwebops.pub/tcp/443/wss/p2p-webrtc-star",
    ],
  },
  modules: {
    transport: [Websockets, WebRTCStar],
    connEncryption: [NOISE],
    streamMuxer: [Mplex],
    peerDiscovery: [Bootstrap],
    dht: KadDHT,
  },
  config: {
    peerDiscovery: {
      [Bootstrap.tag]: {
      enabled: true,
      list: [
"/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
      ],
      },
    },
    dht: {
      enabled: true,
    },
  },
}); This more complex example uses the wrtc-star1.par.dwebops.pub and wrtc-star2.sjc.dwebops.pub servers as relays through the webrtc protocol. These servers are provided by the libp2p community, only for testing. If you building a real-world application, you should . install your own webrtc star servers Another new thing is the bootstrap nodes. These nodes are the starting point of peer discovery. When your node starts, it will connect to these nodes first. After the connection, our node will get other nodes addresses that are also connected to the bootstrap nodes, and it will connect some of them, so after the bootstrap, the bootstrap nodes are not needed anymore, the p2p network builds automatically. The bootstrap nodes in the example are also provided by the libp2p community. By defining different bootstrap nodes, you can define your own, separated p2p networks. The last thing that I want to talk about is the DHT. is a distributed key/value store that is stored on the nodes. This key/value store is used to store the current internet address of the peerId. Distributed Hash Table It is something like the DNS system, where the DNS servers store the IP addresses for the domain names. DNS is also distributed like the DHT, but in this case, we use the peerId instead of the domain and the endpoint’s multiaddress instead of the IP. let peerId = PeerId.parse(this.otherPeerId);
let result = await this.libp2p.peerRouting.findPeer(peerId);
this.otherPeerMultiaddrs = result.multiaddrs; The above example shows, how you can find a peer by its peerId. The peerRouting module’s findPeer method tries to discover the peer by the peerId and if it finds it the multiaddress will be available in the result. Now we can start our own nodes in the browser, and find other peers. The next step is communicating with them. First of all, we have to define a protocol and handle the incoming connections. It is something like opening a server port in a traditional client-server application. this.libp2p.handle('/chat/1.0.0', 
  ({ connection, stream, protocol }) => { 
    this.remotePeerId = connection.remoteAddr.getPeerId(); 
    pipe( 
      stream, 
      (source) => { 
        return (async function* () { 
         for await (const buf of source) 
           yield array2str(buf.slice()); 
        })(); 
      }, 
      async (source) => { 
       for await (const msg of source) { 
        this.messages.push(“> “ + msg); 
       } 
      } 
    ); 
}); The above code defines the ‘chat/1.0.0’ (this is the standard format of a protocol) protocol and handles the incoming connections to it. The handler function has 3 parameters where the 2. is the most important, the incoming stream. JavaScript streams are very tricky things, but hopefully, there are some nice tools to handle them. In this example, we are using the pipe function. The first parameter of the pipe is a stream and the last is a consumer. The middle parameters can be stream transformers. In our case, we transform the stream elements by the array2str function because the stream is binary and we need simple UTF8 strings. The last function (the consumer) simply read the messages from the stream and write them out. Now we have nodes in the browser that can find each other, and we have a chat protocol handler. The last step is connecting from one peer to another, and send some messages on the chat protocol. const { stream, protocol } = await this.libp2p.dialProtocol( 
  peerId, chatProtocol 
); 
this.chatQueue = pushable(); 
pipe( 
  this.chatQueue, 
  (source) => { 
    return (async function* () { 
      for await (const msg of source) yield str2array(msg); 
    })(); 
  }, 
  stream 
); Libp2p has a dialProtocol method for connecting other peers. The method has 2 parameters. The target peerId and the protocol identifier. The result of the method is a stream, so we use the pipe function again. The first parameter of the pipe is a pushable. Pushable is a queue that can be read as a stream. We will push the chat messages to this queue. The second stage of the pipe is a transformer that calls str2array to convert the string messages to binary form. The last stage of the pipe is the stream itself. You can find the full code here: https://github.com/TheBojda/js-libp2p-browser-chat You can test it online on . Open this URL on 2 browser tabs. Copy the one tab’s peerId to the other tab and push the ‘Find’ button. If everything goes well, the browser node will find the other node in the other tab. Then push ‘Dial protocol’ button and you can start chatting between the two tabs. https://thebojda.github.io/js-libp2p-browser-chat/index.html Libp2p is the basis of many p2p applications. I hope, this short introduction will help you to start developing your own p2p applications (maybe a new sharing economy application, or your very own new blockchain). So, let’s code! Initially published here.

Writing Decentralized Applications in JavaScript - Libp2p Basics

Since the seminal paper “ ” of Vaswani et al, Transformer models have become by far the state of the art in NLP technology. With applications ranging from NER, Text Classification, Question Answering, or text generation, the applications of this amazing technology are limitless. Attention is all you need More specifically, BERT — which stands for Bidirectional Encoder Representations from Transformers— leverages the transformer architecture in a novel way. For example, BERT analyses both sides of the sentence with a randomly masked word to make a prediction. In addition to predicting the masked token, BERT predicts the sequence of the sentences by adding a classification token [CLS] at the beginning of the first sentence and tries to predict if the second sentence follows the first one by adding a separation token[SEP] between the two sentences. In this tutorial, I will show you how to fine-tune a BERT model to predict entities such as skills, diploma, diploma major, and experience in software job descriptions. Fine-tuning transformers requires a powerful GPU with parallel processing. For this,we use Google Colab since it provides freely available servers with GPUs. For this tutorial, we will use the newly released to fine tune our transformer. Below is a step-by-step guide on how to fine-tune the BERT model on spaCy 3. spaCy 3 library Data Labeling: To fine-tune BERT using spaCy 3, we need to provide training and dev data in the spaCy 3 JSON format ( ) which will be then converted to a .spacy binary file. We will provide the data in IOB format contained in a TSV file then convert to spaCy JSON format. see here I have only labeled 120 job descriptions with entities such as , , and for the training dataset and about 70 job descriptions for the dev dataset. skills diploma diploma major, experience In this tutorial, I used the annotation tool because it comes with extensive features such as: UBIAI ML auto-annotation Dictionary, regex, and rule-based auto-annotation Team collaboration to share annotation tasks Direct annotation export to IOB format Using the regular expression feature in UBIAI, I have pre-annotated all the experience mentions that follows the pattern “\d.*\+.*” such as “5 + years of experience in C++”. I then uploaded a csv dictionary containing all the software languages and assigned the entity skills. The pre-annotation saves a lot of time and will help you minimize manual annotation. For more information about annotation tool, please visit the page and my previous post “ ”. UBIAI documentation Introducing UBIAI: Easy-to-Use Text Annotation for NLP Applications The exported annotation will look like this: MS B-DIPLOMA O
electrical B-DIPLOMA_MAJOR
engineering I-DIPLOMA_MAJOR
or O
computer B-DIPLOMA_MAJOR
engineering I-DIPLOMA_MAJOR
. O + B-EXPERIENCE
years I-EXPERIENCE I-EXPERIENCE
industry I-EXPERIENCE
experience I-EXPERIENCE
. I-EXPERIENCE
Familiar O O
storage B-SKILLS
server I-SKILLS
architectures I-SKILLS O
HDD B-SKILLS in 5 of with with In order to convert from IOB to JSON (see documentation ), we use spaCy 3 command: here !python -m spacy convert drive/MyDrive/train_set_bert.tsv ./ -t json -n -c iob
!python -m spacy convert drive/MyDrive/dev_set_bert.tsv ./ -t json -n -c iob 1 1 After conversion to spaCy 3 JSON, we need to convert both the training and dev JSON files to .spacy binary file using this command (update the file path with your own): !python -m spacy convert drive/MyDrive/train_set_bert.json ./ -t spacy
!python -m spacy convert drive/MyDrive/dev_set_bert.json ./ -t spacy Model Training: Open a new Google Colab project and make sure to select GPU as hardware accelerator in the notebook settings. In order to accelerate the training process, we need to run parallel processing on our GPU. To this end we install the NVIDIA 9.2 cuda library: !wget https: !dpkg -i cuda-repo-ubuntu1604– – -local_9 – _amd64.deb
!apt-key add / /cuda-repo – -local/ fa2af80.pub
!apt-get update
!apt-get install cuda //developer.nvidia.com/compute/cuda/9.2/Prod/local_installers/cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64 -O cuda-repo-ubuntu1604–9–2-local_9.2.88–1_amd64.deb 9 2 .2 .88 1 var -9 2 7 -9.2 To check the correct cuda compiler is installed, run: !nvcc --version Install the spacy library and spacy transformer pipeline: pip install -U spacy
!python -m spacy download en_core_web_trf Next, we install the pytorch machine learning library that is configured for cuda 9.2: pip install torch== +cu92 torchvision== +cu92 torchaudio== -f https: 1.7 .1 0.8 .2 0.7 .2 //download.pytorch.org/whl/torch_stable.html After pytorch install, we need to install spacy transformers tuned for cuda 9.2 and change the CUDA_PATH and LD_LIBRARY_PATH as below. Finally, install the cupy library which is the equivalent of numpy library but for GPU: !pip install -U spacy[cuda92,transformers]
! CUDA_PATH=”/usr/local/cuda export -9.2 "
!export LD_LIBRARY_PATH=$CUDA_PATH/lib64:$LD_LIBRARY_PATH
!pip install cupy SpaCy 3 uses a config file config.cfg that contains all the model training components to train the model. In , you can select the language of the model (English in this tutorial), the component (NER) and hardware (GPU) to use and download the config file template: spaCy training page The only thing we need to do is to fill out the path for the train and dev .spacy files. Once done, we upload the file to Google Colab. Now we need to auto-fill the config file with the rest of the parameters that the BERT model will need; all you have to do is run this command: !python -m spacy init fill-config drive/MyDrive/config.cfg drive/MyDrive/config_spacy.cfg I suggest to debug your config file in case there is an error: !python -m spacy debug data drive/MyDrive/config.cfg We are finally ready to train the BERT model! Just run this command and the training should start: !python -m spacy train -g drive/MyDrive/config.cfg — output ./ 0 P.S: if you get the error cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed, just uninstall cupy and install it again and it should fix the issue. If everything went correctly, you should start seeing the model scores and losses being updated: At the end of the training, the model will be saved under folder model-best. The model scores are located in meta.json file inside the model-best folder: “performance”:{
“ents_per_type”:{
“DIPLOMA”:{
“p”: ,
“r”: ,
“f”: },
“SKILLS”:{
“p”: ,
“r”: ,
“f”: },
“DIPLOMA_MAJOR”:{
“p”: ,
“r”: ,
“f”: },
“EXPERIENCE”:{
“p”: ,
“r”: ,
“f”: }
},
“ents_f”: ,
“ents_p”: ,
“ents_r”: ,
“transformer_loss”: ,
“ner_loss”: } 0.5584415584 0.6417910448 0.5972222222 0.6796805679 0.6742957746 0.6769774635 0.8666666667 0.7844827586 0.8235294118 0.4831460674 0.3233082707 0.3873873874 0.661754386 0.6745350501 0.6494490358 1408.9692438675 1269.1254348834 The scores are certainly well below a production model level because of the limited training dataset, but it’s worth checking its performance on a sample job description. Entity Extraction with Transformers To test the model on a sample text, we need to load the model and run it on our text: nlp = spacy.load(“./model-best”)
text = [ s degree Computer Science or related field
(Equivalent experience can substitute earned educational qualifications)
- Minimum years experience C# and .NET
- Minimum years overall experience developing commercial software '' 'Qualifications
- A thorough understanding of C# and .NET Core
- Knowledge of good database design and usage
- An understanding of NoSQL principles
- Excellent problem solving and critical thinking skills
- Curious about new technologies
- Experience building cloud hosted, scalable web services
- Azure experience is a plus
Requirements
- Bachelor' in for 4 with 4 in '' '
]
for doc in nlp.pipe(text, disable=["tagger", "parser"]):
    print([(ent.text, ent.label_) for ent in doc.ents]) Below are the entities extracted from our sample job description: [
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , ),
( , )
] "C" "SKILLS" "#" "SKILLS" ".NET Core" "SKILLS" "database design" "SKILLS" "usage" "SKILLS" "NoSQL" "SKILLS" "problem solving" "SKILLS" "critical thinking" "SKILLS" "Azure" "SKILLS" "Bachelor" "DIPLOMA" "'s" "DIPLOMA" "Computer Science" "DIPLOMA_MAJOR" "4 years experience with C# and .NET\n-" "EXPERIENCE" "4 years overall experience in developing commercial software\n\n" "EXPERIENCE" Pretty impressive for only using 120 training documents! We were able to extract most of the skills, diploma, diploma major, and experience correctly. With more training data, the model would certainly improve further and yield higher scores. Conclusion: With only a few lines of code, we have successfully trained a functional NER transformer model thanks to the amazing spaCy 3 library. Go ahead and try it out on your use case and please share your results. Note, you can use annotation tool to label your data, we offer free 14 days trial. UBIAI As always, if you have any comments or questions, please leave a note below or email us at admin@ubiai.tools! Also published at https://walidamamou.medium.com/how-to-fine-tune-bert-transformer-with-spacy-3-6a90bfe57647

Train a NER Transformer Model with Just a Few Lines of Code via spaCy 3 

Intro Organizations are increasingly looking to containers and distributed applications to provide the agility and scalability needed to satisfy their clients. While doing so, modern enterprises also need the ability to benchmark their application and be aware of certain metrics in relation to their infrastructure. In this post, I am introducing you to a cloud-native bench-marking tool known as . This tool is meant to assist your development teams with getting performance metrics from your Kubernetes clusters. Kubestone How does Kubestone work? At it's core, Kubestone is implemented as a in with the help of . You can find more info on the Operator Framework via . Kubernetes Operator Go language Kubebuilder this blog post Kubestone leverages Open Source benchmarks to measure Core Kubernetes and Application performance. As benchmarks are executed in Kubernetes, they must be containerized to work on the cluster. A certified set of benchmark containers is provided via . Here is a list of currently supported benchmarks: xridge's DockerHub space Intro Organizations are increasingly looking to containers and distributed applications to provide the agility and scalability needed to satisfy their clients. While doing so, modern enterprises also need the ability to benchmark their application and be aware of certain metrics in relation to their infrastructure. In this post, I am introducing you to a cloud-native bench-marking tool known as . This tool is meant to assist your development teams with getting performance metrics from your Kubernetes clusters. Kubestone How does Kubestone work? At it's core, Kubestone is implemented as a in with the help of . You can find more info on the Operator Framework via . Kubestone leverages Open Source benchmarks to measure Core Kubernetes and Application performance. As benchmarks are executed in Kubernetes, they must be containerized to work on the cluster. A certified set of benchmark containers is provided via . Here is a list of currently supported benchmarks: Kubernetes Operator Go language Kubebuilder this blog post xridge's DockerHub space Let's try installing Kubestone and running a benchmark ourselves and see how it works. Installing Kubestone Requirements v1.13 (or newer) Kubernetes Kustomize v3.1.0 Cluster admin privileges Deploy Kubestone to namespace with the following command: kubestone-system $ kustomize build github.com/xridge/kubestone/config/default | kubectl create -f - Once deployed, Kubestone will listen for Custom Resources created with the group. kubestone.xridge.io Benchmarking Benchmarks can be executed via Kubestone by creating Custom Resources in your cluster. Namespace It is recommended to create a dedicated namespace for benchmarking. $ kubectl create namespace kubestone After the namespace is created, you can use it to post a benchmark request to the cluster. The resulting benchmark executions will reside in this namespace. Custom Resource rendering We will be using to render the Custom Resource from the . kustomize github repository Kustomize takes a , and patches with an to render the final yaml file, which describes the benchmark. base yaml overlay file $ kustomize build github.com/xridge/kubestone/config/samples/fio/overlays/pvc The Custom Resource (rendered yaml) looks as follows: apiVersion: perf.kubestone.xridge.io/v1alpha1
kind: Fio
metadata:
  name: fio-sample
spec:
  cmdLineArgs: --name=randwrite --iodepth=1 --rw=randwrite --bs=4m --size=256M
  image:
    name: xridge/fio:3.13
  volume:
    persistentVolumeClaimSpec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
    volumeSource:
      persistentVolumeClaim:
        claimName: GENERATED When we create this resource in Kubernetes, the operator interprets it and creates the associated benchmark. The fields of the Custom Resource controls how the benchmark will be executed: : Identifies the Custom Resource. Later, this can be used to query or delete the benchmark in the cluster. metadata.name : Arguments passed to the benchmark. In this case we are providing the arguments to (a filesystem benchmark). It instructs the benchmark to execute a random write test with 4Mb of block size with an overall transfer size of 256 MB. cmdLineArgs Fio : Describes the Docker Image of the benchmark. In case of , we are using , which is built from . image.name Fio xridge's fio Docker Image this repository : Given that Fio is a disk benchmark, we can set a for the benchmark to be executed. The above setup instructs Kubernetes to take 1GB of space from the default StorageClass and use it for the benchmark. volume.persistentVolumeClaimSpec PersistentVolumeClaim Running the benchmark Now, as we understand the definition of the benchmark, we can try to execute it. Note: Make sure you installed the kubestone operator and have it running before executing this step. $ kustomize build github.com/xridge/kubestone/config/samples/fio/overlays/pvc | kubectl create --namespace kubestone -f - Since we pipe the output of the command into , it will create the object in our Kubernetes cluster. kustomize build kubectl create The resulting object can be queried using the object's type ( ) and it's name ( ): fio fio-sample $ kubectl describe --namespace kubestone fio fio-sample
Name:         fio-sample
Namespace:    kubestone
Labels:       <none>
Annotations:  <none>
API Version:  perf.kubestone.xridge.io/v1alpha1
Kind:         Fio
Metadata:
  Creation Timestamp:  2019-09-14T11:31:02Z
  Generation:          1
  Resource Version:    31488293
  Self Link:           /apis/perf.kubestone.xridge.io/v1alpha1/namespaces/kubestone/fios/fio-sample
  UID:                 21cdbe92-d6e3-11e9-ba70-4439c4920abc
Spec:
  Cmd Line Args:  --name=randwrite --iodepth=1 --rw=randwrite --bs=4m --size=256M
  Image:
    Name:  xridge/fio:3.13
  Volume:
    Persistent Volume Claim Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  1Gi
    Volume Source:
      Persistent Volume Claim:
        Claim Name:  GENERATED
Status:
  Completed: Running: Events:
  Type    Reason           Age   From       Message
  ----    ------           ----  ----       -------
  Normal  Created  11s   kubestone  Created /api/v1/namespaces/kubestone/configmaps/fio-sample
  Normal  Created  11s   kubestone  Created /api/v1/namespaces/kubestone/persistentvolumeclaims/fio-sample
  Normal  Created  11s   kubestone  Created /apis/batch/v1/namespaces/kubestone/ /fio-sample true false jobs As the section shows, Kubestone has created a , a and a for the provided Custom Resource. The field tells us that the benchmark has completed. Events ConfigMap PersistentVolumeClaim Job Status Inspecting the benchmark The created objects related to the benchmark can be listed using command: kubectl $ kubectl get pods, ,configmaps,pvc --namespace kubestone
NAME                   READY   STATUS      RESTARTS   AGE
pod/fio-sample-bqqmm   0/1     Completed   0          54s

NAME                   COMPLETIONS   DURATION   AGE
job.batch/fio-sample   1/1           15s        54s

NAME                   DATA   AGE
configmap/fio-sample   0      54s

NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/fio-sample   Bound    pvc-b3898236-c698-11e9-8071-4439c4920abc   1Gi        RWO            rook-ceph-block   54s jobs As shown above, Fio controller has created a PersistentVolumeClaim and a ConfigMap which is used by the Fio Job during benchmark execution. The Fio Job has an associated Pod which contains our test execution. The results of the run can be shown with the command: kubectl logs $ kubectl logs --namespace kubestone fio-sample-bqqmm
randwrite: (g=0): rw=randwrite, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.13
Starting 1 process
randwrite: Laying out IO file (1 file / 256MiB)

randwrite: (groupid=0, =1): err= 0: pid=47: Sat Aug 24 17:58:10 2019
  write: IOPS=470, BW=1882MiB/s (1974MB/s)(256MiB/136msec); 0 zone resets
    clat (usec): min=1887, max=2595, avg=2042.76, stdev=136.56
     lat (usec): min=1953, max=2688, avg=2107.35, stdev=142.94
    clat percentiles (usec):
     |  1.00th=[ 1893],  5.00th=[ 1926], 10.00th=[ 1926], 20.00th=[ 1958],
     | 30.00th=[ 1991], 40.00th=[ 2008], 50.00th=[ 2024], 60.00th=[ 2040],
     | 70.00th=[ 2057], 80.00th=[ 2073], 90.00th=[ 2114], 95.00th=[ 2409],
     | 99.00th=[ 2606], 99.50th=[ 2606], 99.90th=[ 2606], 99.95th=[ 2606],
     | 99.99th=[ 2606]
  lat (msec)   : 2=34.38%, 4=65.62%
  cpu          : usr=2.22%, sys=97.78%, ctx=1, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,64,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all ):
  WRITE: bw=1882MiB/s (1974MB/s), 1882MiB/s-1882MiB/s (1974MB/s-1974MB/s), io=256MiB (268MB), run=136-136msec

Disk stats ( /write):
  rbd7: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% jobs jobs read Listing benchmarks We have learned that Kubestone uses Custom Resources to define benchmarks. We can list the installed custom resources using the command: kubectl get crds $ kubectl get crds | grep kubestone
drills.perf.kubestone.xridge.io         2019-09-08T05:51:26Z
fios.perf.kubestone.xridge.io           2019-09-08T05:51:26Z
iopings.perf.kubestone.xridge.io        2019-09-08T05:51:26Z
iperf3s.perf.kubestone.xridge.io        2019-09-08T05:51:26Z
pgbenches.perf.kubestone.xridge.io      2019-09-08T05:51:26Z
sysbenches.perf.kubestone.xridge.io     2019-09-08T05:51:26Z Using the CRD names above, we can list the executed benchmarks in the system. Kubernetes provides a convenience feature regarding CRDs: one can use the shortened name of the CRD, which is the singular part of the fully qualified CRD name. In our case, can be shortened to . Hence, we can list the executed benchmark using the following command: fios.perf.kubestone.xridge.io fio fio $ kubectl get --namespace kubestone fios.perf.kubestone.xridge.io
NAME         RUNNING   COMPLETED
fio-sample false true Cleaning up After a successful benchmark run the resulting objects are stored in the Kubernetes cluster. Given that Kubernetes can hold a limited number of pods in the system, it is advised that the user cleans up the benchmark runs time to time. This can be achieved by deleting the Custom Resource, which initiated the benchmark: $ kubectl delete --namespace kubestone fio fio-sample Since the Custom Resource has ownership on the created resources, the underlying pods, jobs, configmaps, pvcs, etc. are also removed by this operation. Next steps Now you are familiar with the key concepts of Kubestone, it is time to explore and benchmark. You can play around with Fio Benchmark via it's cmdLineArgs, Persistent Volume and Scheduling related settings. You can find more information about that in Fio's benchmark page. Hopefully you gained some valuable knowledge from this post! About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of . He likes sharing his knowledge through writing for , , and many more, and while he is not doing that, he must be fishing or playing chess. Javelynn Hackernoon DZone Appfleet Previously posted at https://appfleet.com/ .

How To Run Cloud-Native Performance Benchmarks with Kubestone

Here is the scenario. You work for a startup that develops a platform that allows photographers (both professionals and amateurs) to upload and share their lovely pictures with other photography lovers. The platform has been online for more than 2 years now and has been very successful. Users have uploaded more than 5 million pictures already. Your product owner comes to you requesting a killer feature that allows users to search for photos, even those that haven’t been tagged nor received any description yet. You are requested to come up with a cheap and fast solution to the problem. Now what? Google Cloud to the rescue! In this series of articles, we are going to see how to build a solution that leverages Android, Firebase and Google Cloud Vision API for streamlining image tagging in the cloud with a . serverless structure In this first part, we are going to focus on configuring a Google Cloud project with Google Cloud Vision API enabled. In the parts that follow, we’ll see how to configure Firebase services and how to create a basic Android app for uploading the photos. Let’s get it started! Google Cloud Vision API Google Cloud Vision API showcase Google describes this service as follows: Google Cloud Vision API enables developers to by encapsulating in an easy to use REST API. It quickly into thousands of categories (e.g., “sailboat”, “lion”, “Eiffel Tower”), , and finds and reads printed words contained within images. You can build metadata on your image catalog, moderate offensive content, or enable new marketing scenarios through image sentiment analysis. or integrate with your image storage on Google Cloud Storage. understand the content of an image powerful machine learning models classifies images detects individual objects and faces within images Analyze images uploaded in the request For our use case, we are going to focus on . We are going to use it to tag our images, so they can be searched by our platform users. image classification The first thing we need to do in order to use the API is to create a Google Cloud Project. To do so, follow these steps: Access and sign in using your Google account; Google Cloud console Click in the menu right to the console logo: Project selection menu In the dialog that opens, click the button for creating a new project: + Project selection dialog Give your new project a name and wait for its creation. After the project is created, click the menu right to the console logo again and select your new created project as the active one; Cool! You have just created your Google Cloud project, which is going to be used for managing the APIs used in this series. Time to enable the for the project. To do so, follow the steps below: Google Cloud Vision API Open the top left menu and select > : API Manager Library Opening API Manager Library Next, search for the Google Cloud Vision API and select it: Searching for the Google Cloud Vision API Google Cloud Vision API page (for this tutorial, we are going to use Server-to-server interaction) After clicking , the page changes to show a dashboard for the API: ENABLE Google Cloud Vision API dashboard Next and final step for using the API is creating an API Key for authenticating our calls to the service. Let’s get it done: Click the top left menu again and select > : API Manager Credentials Opening API Manager Credentials Click , then select : Create credentials API key Creating our API Key A new dialog displays your key. You can later access and copy this key by accessing > at any time. API Manager Credentials Testing what we have done Time to test our precious work! In order to do this, we are going to perform a HTTP request to the API. For this, we are going to use the following JSON template: Notice we use as the value for field. This placeholder will be replaced with when we call the API: __IMAGE_URI__ imageUri sed What’s happening here? First, we use to replace with the URL for the image we are analyzing; sed __IMAGE_URI__ Then, we pipe the to our command. This is going to POST our JSON changed by sed to (make sure you use the API KEY you generated above here). sed output curl https://vision.googleapis.com/v1/images:annotate?key=__API_KEY__ The result is the following: From the results, we can see the API is more than and more than . Below is the image we analyzed. As you can see, the API yielded pretty good results: It works! 97% sure it’s a mug 94% sure it’s orange Our not so beauty analyzed orange mug Finally, if you check the Google Cloud Vision API dashboard again, you should see it now displays updated data: Google Cloud Vision API dashboard now displays some data Conclusion In this first part, we have seen what is Google Cloud Vision API about, how easy it is to create a Google Cloud project and enable the Vision API for it, how easy it is to use the API and, of course, how impressively accurate the API is. Stay tuned for the next parts, as we are going to see how to configure Firebase and integrate it with the API, as well as how to create a basic Android. Hope you liked it! Thanks for reading! If you found this helpful or interesting please ❤ it. Also, feel free to share your thoughts in the comments below.

Leveraging Google Cloud for building a serverless image tagging service — part 1

I wanted to get a quick picture of what the on my were doing in the background, so I turned to (shout out to Julia Evans for ), which can tell me handy things like “who’s opening up which files?” and “what system calls are being made?” processes computer DTrace getting me excited about it Just to start, I decided to run this to see which programs were making the most system calls: one-liner sudo dtrace -n 'syscall:::entry { @num[execname] = count(); }' After running the above for a couple of minutes (and browsing idly in the background), I got this output (truncated for brevity): Now, I want to reboot into a relatively clean state, and run the command for exactly ten minutes, without having to manually stop it. This is what I came up with: The ampersand makes DTrace run as a background job. Running gives you back a list of the process IDs of jobs: jobs -p bash-3.2$ jobs -p10038 I pipe that through to take only the last one (in case you had other jobs running or suspended in the same shell instance). And then because doesn’t take input from stdin, we pipe the PID to which will pass its stdin as arguments to . And we make sure that we run with , because that’s how the DTrace process was started. tail -1 kill xargs, kill kill sudo By the way, because is a bash builtin, if you want help with it you need to use , not . jobs help jobs man jobs Let me know if there’s a better way of doing this! I’m by no means a bash expert. is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising & sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Running a process for exactly ten minutes

What I learned about logging when using cron as a “quick and dirty” scheduler in my app. I’m sure I logged the error somewhere I usually use a scheduling framework to run scheduled (duh) tasks. But this really isn’t a heavyweight app, and I didn’t want to add extra tooling and configuration. Opting for using to run my app tasks from outside look. cron Cron is the built in scheduler. And a user “MyAppUser” has an optional crontab (read “cron table”) the crond (the cron daemon) reads configuration from. Mission accomplished, the tasks are running and everything is ok. Open source rocks and a quick win for me. linux And then IT HAPPENED. Something critical was broken, really broken. And I really needed the logs from the command I was running through cron. But where are they? Following the recommendations My app logs to stdout and stderr streams, letting the streams be taken care of from outside. For the regular app logs, it’s supervisor that is handling the stdout and stderr streams, writing to log files, rotating, etc.. 12 factor app But supervisord does not run the scheduled tasks, but crond. So who takes care of my logs? I remembered seeing cron entries in syslog so I took a look there and found entries like: Sep 23 10:00:01 ip-172–30–0–129 CRON[2021]: (myuser) CMD (/bin/bash /home/myuser/run_task.sh) Sep 23 10:00:05 ip-172–30–0–129 CRON[2018]: (CRON) info (No MTA installed, discarding output) OH. so cron did try to run my_task.sh but “discarding output”. But why? And what is an MTA and what does it have to do with cron logs? Googling supplied an . for “Message Transfer Agent” which usually translates an email sending service (POSTFIX is quite common as one). So Cron jobs output is logged through email. And if you don’t have the correct configuration it goes into an internal /var/mail/myuser file (In email format). explanation MTA is an acronym While I can see why having cron job reporting by email might look like a good idea in certain use cases, I have never actually encountered such a use case :). Actually configuring the MTA to send the emails would flood me with endless reporting emails with no value (I look at the logs when I want to search for something and not all the time). Not configuring it would flood /var/mail/myuser with this kind of output. We’ll get the text but without line by line order and context, date stamp (unless I take care for that in my app logging logic). But a worse problem is that I DONT WANT TO INSTALL AN EMAIL SERVER in my app server. Opening another possible attack vector, another thing to worry about and configure, this doesn’t look like an attractive option. After ruling out installing an email service what can we do otherwise? I found lots of problematic answers to this problem, as least in my view. Most of them either suggesting log file handling in the applicative code (Would force me to handle double logging logic (the regular stdout and file logging , log to somewhere my user has access too and handle log rotation etc). Another common solution is to redirect output from the scheduled task to a manual log file: 15 15 * * 0–5 /bin/bash /home/myuser/run_task.sh >> /var/log/mylog.log 2>&1 This is a better solution but might not be good enough, logs are without a date, unless I add in the app. And again, this is a file that I need to take care of adding a logrotate configuration (which doesn’t look like a fun thing to do and adds another thing to configure and worry about) The solution I opted for (adapted from a stackoverflow question I can’t find right now) was using the following: 15 15 * * 0–5 /bin/bash /home/myuser/run_task.sh 2>&1 | /usr/bin/logger -t tag_name We redirect stderr to stdout We pipe the results to the system “logger” utility which is in charge of writing the system logs (duh), written to /vat/log/syslog in recent ubuntu distribution (some linux distribution would log to /var/log/messages) We add a tag for an easier searching and grepping Try: echo “hey there” 2>&1 | logger -t my_tag And check /var/log/syslog for the result Sep 24 09:37:47 ip-172–30–0–129 my_tag: hey there So the result would be logging the output from cron tasks into a reliable log file with all the needed context of other things happening on the server in the same time. Mission accomplished. Changing this logging configuration uncovered the bug I was hunting for soon enough.

Cron job — a tale of a missing log

THE POLE.

Various Tragedies

The Tansy Patch

A Special Providence

I faced a problem when I started learning programming. That problem was: I thought I needed to have a deep understanding of every field to do projects and make cool stuff. So, I started learning to code seriously, but after a few weeks, I lost my motivation, as it was not clear to me how coding was going to help me make something that I'd like. This GitHub repository features 50+ Python applications with only 10 lines of code! In this repository, you'll find a wide range of topics such as Machine Learning, Deep Learning, GUI, Computer Vision, and API development. Author: @xiaowuc2 This is only the project version of this course: repository Computational Thinking: Python I am a Machine Learning Researcher currently working at IBM with expertise in solving real-world problems through coding. I taught 20,000+ students & working professionals to crack coding interviews and land their dream job at TCS, Walmart, Accenture, and Cognizant. My research has been recognized through publications in esteemed journals such as Springer and O'Reilly . 1 2 : Python course to crack coding interviews (Recorded / Live) Computational Thinking: Python : rohitmandal814566@gmail.com Let's Have an intellectual conversation : 50+ Python Applications (10 Lines of Code) Repository Projects: Python 📼 : Simple voice recorder with a custom time limit. Voice Recorder 🔑 : Protect a pdf with a custom password. Password Protech PDF 🗏 s: Merge multiple pdfs with Python scripting. Merge Multiple PDF 🔔 : Custom Windows notification maker. Windows Notification 🎬 : Awesome audio visualization tool! Audio Visualization Tool 📟 : Random secured password generator app. Random Password Generator 🎶 : Extract audio from video with parsing. Extract mp3 from mp4 🔗 : URL shortener from the terminal. Link Shortner 🔋 : Cool terminal tricks #scripting. Terminal Tricks 🎂 : Birthday reminder for lazy coders. Birthday Reminder 📻 : Audiobook creator from a text file. Audiobook ⏰ : Friendly alarm for programmers to take a break. Alarm ⏱️ : Python script will play a YouTube video at a scheduled time. Schedule YouTube Video 📆 : A tkinter(GUI toolkit) based calendar app. Calendar ✏️ : A tkinter(GUI toolkit) based interactive paint clone. Paint 💻 : A tkinter-based screenshot app with a clickable button. Screenshot taker 📖 : Wekipedia API integrated tkinter-based search engine. Wikipedia Search Engine Projects: Machine Learning ✒️ : Tool to automate news briefing and blogging from custom senders (mail). email-automation ⭐ Custom-chatbot: ask the chatbot to do custom work on the bases of the task (eg., scriptwriter) 📟 : OpenAI's other API to convert text from audio. whisper-speech-text ⚙️ : Train chatGPT on your custom data, & ask queries from that data. finetuned-gpt 💠 : Voice assistant based on ChatGPT and WhisperAPI (Audio input & output). voice-assistant 🐻 : This tool scrapes a given website and summarizes the main context. web-scraping-summarizer ⌚ : You can train ChatGPT to perceive things the way you do, and it will imitate you! your-perspective 📖 : A religious book that contains all the answers to find our purpose and to live it fully. bhagavad-gita-gpt 🏜 : This is how you can send big text files to ChatGPT and avoid the token limits. vector-database Keynote Video: https://youtu.be/B0_0gK_CUpM?embedable=true GitHub org: https://github.com/qxresearch/qxresearch-event-1 Join #qxresearch: Mozilla Group: https://community.mozilla.org/en/groups/qx-research Telegram Group: https://t.me/qxresearch

50 + Python Projects (10 Lines of Code)

HackPedia: 16 Solidity Hacks/Vulnerabilities, their Fixes and Real World Examples

Let's learn about via these 53 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Dogecoin /Learn Repo If you google 'anonymous crypto jokes,' Dogecoin is the third result. 1. An Intro to Shitcoins for Beginner Crypto Traders Shitcoins are the subject of most crypto discussions and for a good reason. Find out more about them in this introductory article. 2. My Picks: Five Top-Performing Cryptos in 2021 My Five top-performing cryptos in 2021: Bitcoin, Axie Infinity, The Sandbox, Dogecoin and Solana. What's yours? 3. Lets Graph Simple Moving Averages Using Rust Simple Moving Averages are calculated by getting the mean closing price over a period of time. Lets see on how we can program that idea into Rust. 4. Is Tesla Adding Dogecoin as a Payment Option? Twitter user @BitDeepu tweeted an image of Tesla’s source code that shows that Dogecoin holders can buy Tesla Model Y with their DOGE tokens. 5. Mining Crypto for Profit: It is Time Full-scale mining requires a large number of computers with powerful GPUs. 6. Who Showed Up to Dogepalooza and Who Cares? (Much Surprise) What happened at the first-ever Dogecoin-themed charity music festival Dogepalooza? Hear it from the perspective of one of the performers there, Rito Rhymes. 7. Elon Musk and Crypto Markets: "Seriously, playing?" or "Playing seriously"? Elon Musk and crypto market manipulation - is it a joke or a serious crime? 8. Doge Wars: Attack Of The Clones Dogecoin's popularity has inspired more clones with very similar names to emerge, creating a situation where the name itself is in contention between coins. 9. From Dogecoin to Twitter: The Making of the Everything App Depending on whom you ask, Elon is either an egotistical villain or a savior. He is currently the world’s richest man, with a net worth of about 200 Billion. Well, with that ind of money it becomes easy to create products that have a direct impact on people’s daily lives- with huge repercussions. Now, Elon is not your average billionaire. Most of the time, he has swayed the masses to his favor, much to the chagrin of big tech. This has turned him onto somewhat of a cult, with a huge following and an enviable portfolio of companies that seek to upend conventional beliefs and corporations. From Neuralink to Space X , Tesla to the Boring Company, he is an entrepreneur who seeks to “disrupt” global order as he strives to achieve what may somewhat be thought to be impossible, until its done. 10. Should You Invest In Meme Coins? Find out everything you need to know about meme-based cryptocurrencies 11. How to Pronounce Doggie Coin The doggie is in heat! One of the inventors confirms Dogecoin is "DOhj"coin. The name is made up but "sounds" Japanese. 12. An Interview with a Dogecoin Core Developer Interview with Max Keller, a Dogecoin Core developer, on cryptocurrency today, goals of the Dogecoin team, Elon Musk, all the hype around DOGE and much more! 13. The Summer of Doge A revolution just like the Summer of Love is nigh. 14. Elon Musk Buys Twitter: Now Dogecoin to the Moon Elon Musk, the founder of Tesla and SpaceX, has just announced his latest venture: buying Twitter. This move comes no surprise to many Musk is known game change 15. Is Dogecoin Headed to Twitter, The Moon, or Throughout The Universe? The direction of dogecoin in 2022. Rito Rhymes explains his experiences with tipping on Twitter and shares knowledge of various parts of its growing eco-system. 16. Greed Threatens to Destroy Dogecoin, But the Motto May Be Redemption #DoOnlyGoodEveryday is just an awesome motto. One I can embrace. Greed threatens to destroy #dogecoin but I cling to the motto. The community has been around for years. It has shown resilience. They believe the motto Do Only Good Everyday. They were doing it when dogecoin was going for less than a penny. 17. The Elon-Twitter Love Affair: Is it Steamy Enough for Crypto and Tech? Elon Musk Twitter association is expected to pave the way for the next-gen of social networking, web 3.0 adoption, and other tech-powered developments. 18. My 5 Favorite Proof of Work Cryptocurrencies Proof of Work is an important aspect of the cryptocurrency solution. Here are my favorite Proof of Work cryptocurrencies. 19. The Power of Memes: Tendies, Dogecoin, and the Potential for Social Impact Ask yourself this question: what is the most powerful, yet relatively untapped force on earth at this moment? You might think of the ability of sustainable energy to replace fossil fuels, the strength of youth voices as they attempt to take power back from the boomer powers that be, or as it's been hammered through the heads of technologists over the past 5 years, decentralization. The promise of decentralization is as follows: instead of the corporations owning all the services you pay for, the services run themselves and act in a truly fair way towards all users. This means no more price gouging of Epipens, no more ridiculously high medical bills, no more violations of net neutrality, and so on. To get to this idealized form of world decentralization, however, we need to think about the how. Think about the one omnipresent force that is on every single person's mind right now: virality. 20. Why You Should Be Careful Investing in Dogecoin Even though there has been plenty of DOGE price momentum lately, it will come to an end sooner rather than later. 21. Shiba Inu vs. Akita Inu: The Great Memecoin War Meme-based coins have taken the world by storm lately. Two new cryptocurrency entrants are Shiba Inu and Akita Inu, inspired by Dogecoin and based on Ethereum. 22. Anndy’s Opinion: Meme Tokens With Real Utility The cryptocurrency market can be extremely unpredictable. Defi, NFT and now MEME. Meme Tokens With Real Utility is the way forward. Check it out. 23. Crypto research of the July 2019 cryptocurrency activity Corindex July 2019 crypto report is comprised of parsed data on key cryptocurrency exchanges. Indicators, specifically designed by this crypto analytic platform, are calculated past the analysis of the blockchain network of various crypto assets. To best approximate the overall state of the cryptocurrency market, the report shows data only on the top cryptocurrencies with the largest capitalizations, and, occasionally, currencies that showed biggest relative changes by one indicator or another over the analyzed period. The report contains nine chapters, each of which encompasses the changes to its designated indicator over the analyzed period. 24. What Elon Musk's Twitter Purchase Means For DOGE Dogecoin's price jumped as high as $0.1428, marking a 12 percent rise in its price in 24 hours. 25. Comparing Bitcoin ATMs Which Bitcoin ATM offers the best exchange rate, dollars for crypto? 26. Dogecoin (DOGE) Up 20% Despite Bitcoin (BTC) Dip Over the past week, Dogecoin (DOGE) was up 20%, compared to a 3% jump in Bitcoin (BTC) price. 27. Dogecoin: The Value Proposition That's Worth More Than a Penny Dogecoin started its initial coin production schedule with 100 billion coins in circulation. By mid-2015 the 100 billionth Dogecoin had been mined though. 28. An Explainer on Why Stablecoins are a Better System for Money Stablecoins have become a key part of the cryptocurrency industry. 29. Will Meme Coins Keep Going to the Moon? While meme coin may generate spectacular short-term gains as people are willing to speculate on the possibility of making money in a short amount of time. 30. The Multiverse of Madness: Warren Buffett, Elon Musk, Inflation & Bitcoin Warren Buffett vs. Elon Musk - Inflation vs. Bitcoin 31. What's Driving The DogeCoin Mania? You've got that friend that came to you and asked about Dogecoin? That's how crazy Doge is all about. Why did it skyrocketed in 2021? Read to find out more now! 32. Is Dogecoin Worth a Dime? - All Things Doggie Are Good Dogecoin is holding at a nickel so a dime seems possible. Crypto is SO volatile, there is no actual set value in your head. 33. Why Bitcoin Boomed and Dipped, and What's Coming Next Bitcoin and cryptocurrencies, although they are using high amounts of energy, create greater efficiency because they are taking out the middle man. 34. Who's the Top Crypto Dog? Shiba Inu vs. Dogecoin This article compares the various value propositions presented by the Shiba Inu coin (SHIB) and Dogecoin (DOGE). 35. It's Time to Give Meme Tokens a Purpose Each meme token is moving to offer a unique value proposition in a space more known for inside jokes than serious efforts toward innovation. 36. Meme Coins That Still Have Market Traction in 2022 Coins based on memes and online pranks, known as meme coins, are a form of digital currency. 37. Is There Such a Thing as Too Many Cryptocurrencies? An excessive number of cryptocurrencies and an endless rollout of ICOs can be problematic for the image of the crypto world. 38. Bitcoin vs Dogecoin: An Honest Comparison Bitcoin and Dogecoin are two popular cryptocurrencies. Here is how they compare regarding market supply, concentration, mining, and transactions. 39. In The BTC Vs. USD Battle, My Money's on DOGE: Here's Why Could it really be true Dogecoin is better than Bitcoin for widespread adoption and usage by a national or pseudo-national entity? 40. THE HELLADOGE MANIFESTO: Why Web3 Ads are Like Instagram Ads on Steroids Ok, this might sound like shameless self-promo, but I’m hoping you will indulge me. I hope Hackernoon lets me run this article - I think they’ll let it slide cause Utsav is the homie, and also because if you can look past the self-promotion you will see I’m making an important philosophical point. 41. The Rollerball Cryptocurrency Hypothesis The Rollerball Cryptocurrency Hypothesis: I was putting real money into cryptocurrency based upon the belief there would be a Rollerball League. 42. Bitcoin Dodging Doge as Doge Dodges Bit by Bit An argument for a marketplace of currencies on the bitcoin standard 43. Is Elon Musk WRONG About Dogecoin? The CEO of Koinos Group explains Dogecoin and what Elon Musk might be getting wrong. 44. How a Single Elon Musk Tweet can Change Cryptocurrency Prices Elon Musk tweets about Dogecoin and everyone starts buying: find out how he and other crypto influencers affect the prices of cryptocurrency in this piece. 45. Understanding Meme Economy While traditional investing moves along the lines of traditional media, the meme economy is developed as a result of user-generated investment strategies. 46. You HODL Dogecoin? What seems like millennia ago, but was only about a year ago, I wrote on HN about Dogecoin's value proposition being worth a penny. I just wrote my response to my adult sons when they had asked why I had started accumulating Dogecoin. This essay is similar. This is essentially my response to their queries when they learned I was still HODL'ing my Dogecoin. 47. Cats, Computers, & Working From Home Fails Top writers and editors at Hackernoon share their working from home fails with fluffy individuals jealous for attention otherwise known as cats. 48. Always Bet Musk's Oxen! Fresh off the Virgin Martian Express, Jackson had a layover at the Blue Origin moon base. He was trying to get to the Starbase Autonomous Zone 49. Security.org Knows 15 Million Americans Will Be Buying Cryptocurrency in the Next Year Twelve years after the Bitcoin network came into existence in 2009, we asked nearly 1,300 U.S adults about their familiarity, ownership, and confidence in cryptocurrencies, from Dogecoin to Ethereum, and everything in between. One highlight?  Nearly 15 million Americans plan to purchase crypto for the first time in the next year. 50. How I Built a Bot to Help Buy Doge or BTC When Elon Musk’s Tweets About Them this post contains a lot of affiliate link because all tools that I recommend are necessary to run the bot and provide a simple indirect way to funding the project without charging end-user 51. Dogecoin: Everything You Wanted To Know About This Memecoin Making A Mockery of 'Serious' Crypto Can a cryptocurrency that started as a joke become a serious asset? Here is a quick guide on the Dogecoin ecosystem, how to start mining some, and its future. 52. Analyzing Dogecoin Tweet Sentiment in Real Time How to analyze Dogecoin tweet sentiment in real-time with a new managed Kafka platform. 53. The Dogecoin Salut on The HackerNoon Podcast Welcome to this beautiful corner of the internet - Dogecoin meets partying & decentralization. Amy Tom talks to Gary Lachance, a Vancouver Dogecoin millionaire. Thank you for checking out the 53 most read stories about Dogecoin on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

53 Stories To Learn About Dogecoin

Exploring a Product’s Path to Success Ahhh 😌. Wouldn't it be nice if there was a world with short instant-success business tips? Yes, that one, the same world with free housing and magic carpet rides. Unfortunately, there is no such world, and unfortunately, there is no TLDR for successful product management 🤗. This is a long read. Hopefully though, like the Harry Potter novels it'll be so good you wouldn't want it to end 😁. Picture this. You have a product. You're most likely either a founder, if that's you, by the way, congratulations on beginning the adventure that is entrepreneurship, or, you're someone who likes a challenge, working in a company where your task is bringing growth. Brilliant. Either way, you have a product. The next step is crucial. As you may imagine from the contrasting fortunes of Google's Android going from launch to success and Microsoft's Windows Phone going from launch to failure, the path to either is not always obvious. In this article, I’ll list some theories that would help you understand what’s best. Symbiosis One avenue worth exploring is finding a symbiotic relationship with another business. An example could be Company A offering a bonus item or discount code with the purchase of an item from Company B. Such as Spotify offering premium membership codes inside cans of Pringles. Ideally, look for companies of similar size or companies that are keenly interested in efforts to boost their sales. This would increase the likelihood that such a deal could be done without requiring a cash payment. Companies are also likely to accept deals that require minimal to no effort on their part. This also has the added benefit of effectively hedging a bet on marketing methods, as your product would benefit from all the marketing methods used by your partner companies too. It's essential that partners are chosen carefully so as to avoid detrimental brand association. Once You Get to Know Me It is very helpful to know how your target customers feel about your product. After all, customer sentiment is usually primarily responsible for customers picking products, or not. Surveys are your friend here. Directly speaking to customers is even better. During surveys of your target market, learning from those who have heard of your products and chose not to use it could also be as helpful as those who use your product. For example, knowing what to remove could be as important as knowing what to add. Be Careful Being the first in a new industry that proves to be successful in the long run could be a great growth strategy. For example, being the first website or YouTube channel to host tutorials on a new programming tool could lead to prominence in that field. An alternate example is a programmer who discovers a new platform, e.g Windows Phone at its time of launch. As a developer, putting a lot of resources into a young platform may mean great exposure at a time of little competition. It could also mean wasted efforts on a platform that ends up in failure. There are some growth opportunities that aren't particularly clear as to whether they are a good idea or not. There isn't a particular magic formula for business success and it is in instances like this that careful analysis and planning are required. A case is the game streaming platform Mixer by Microsoft. It launched as a competitor to the more popular platform Twitch. In the early days, two successful categories on Mixer were new Mixer creators who rose to prominence through the less crowded platform and prominent creators who were paid a lot of money to leave Twitch and use Mixer exclusively. This meant that even though Mixer eventually shut down a few years later, for some, their time on Mixer was well worth it. The larger creators even though returning to reduced fan bases on Twitch, had the benefit of the Mixer signup bonus to make the exercise a possible net benefit. Smaller creators who could not carry over their followers made on Mixer to other platforms and had to start over with much smaller fan bases may not feel so. However, the argument could be made that those smaller creators would not have gotten their fame if they had started on Twitch. The lesson here is that it's not always a simple decision of committing or not if you're chasing maximum benefit. Sometimes the decision needs to be a careful consideration of the entry point and exit strategy. Care should be taken on growth strategies that alter the product significantly. An example could be taken from the book Growth Hacking, where author Shawn Ellis described how raising prices at his startup, Qualaroo, led to a 400% increase in revenue. Apparently, the higher-priced product gave the product an appearance of quality. Though this may have led to a loss of smaller customers, this was more than offset by the gain in large corporate customers. In other words, they essentially switched out their customers. It would be up to the individuals to determine whether they want to alter their product focus as such. Whenever you receive information or advice, it's a good idea to examine its basis. For example, the book, Hacking Growth (published in 2017), cited a 2005 study on software to share the belief that companies should not release too many features in a product at once. Herein lies a problem. 2005 was a long time ago. No social media, no smartphones, The PlayStation 2 was a current-gen console. It also followed up by talking about a presentation that showed Microsoft Word with every toolbar enabled. This could be seen as an over-generalization and misrepresentation of a product leading to a faulty conclusion. The idea was never to utilize MS Word with every toolbar enabled. How features are presented and enabled is what matters, not how many they are. The focus should be on giving users the things they need. Don't Be Evil Be careful of growth strategies that come at a non-obvious cost to the company. An example would be the case of the pharmaceutical company Mylan (now Viatris after a merger). They manufacture the product EpiPen, a first aid drug for allergic reactions, which they acquired in 2007. At that time the price was 100 USD for a two-pack. By 2009 the price was about $103.50. A series of steep price increases raised the price to about $609 in May 2016. All this occurred during a period that saw less than 4% annual inflation in the US. Through a combination of marketing,  lobbying, and price hikes, the $200 million a year income of EpiPen rose to $1.5 billion by May 2015. Now from a mathematical point of view, this represents fantastic growth. However, revenue was not the only thing Mylan had grown. It had also grown resentment. Eventually, push-back came, in the form of a class-action lawsuit settled for $264 million. Now some of you may have noticed that $0.264 billion once is a lot less than $1.5 billion (though revenue not profit). Viatris is not the first in history to use this 'cheaper to do the crime' model of business, but it does come at a reputational cost. per year Another example is the company FaceBook (now Meta). For many, its unethical data and business practices are legendary. These helped it grow its advertising revenue as well as its product addictiveness. After breaking a previous FTC settlement on data handling that helped achieve high growth. Facebook was fined $5 Billion by the FTC. The prior quarter saw revenues of over three times that amount. Once again a fine, but not exactly a calamity. Like Mylan, Facebook had its users over a barrel with an effective monopoly. They had essentially utilized the 'drug dealer model' and now users were so dependent on them. It was hard to regulate them effectively. However, if you push them hard enough, just like The North, consumers will remember. While Viatris and Meta are still profitable companies today, their reputations have taken a solid dent. Facebook will have realized this when they launched their ill-fated cryptocurrency project . Almost immediately the pushback was significant. were aware of how low the stock of their reputation was, they prominently promoted it as the effort of a consortium. They were not successful in swaying public opinion enough. Partners began to drop from the project and even a rebranding exercise was not enough to save it. Diem Libra Even they Another large tech company with a similar reputation problem is Google. In November 2019 Google launched Stadia, a cloud gaming service. It was immediately hit with a wave of pessimism. With it's reputation of shuttering products, see , people didn't trust paying for games that would be locked to Google's control, but most of all they didn't trust Google to keep the service running for longer than a few years. Google sought to combat those fears with campaigns but pessimism remained. Well, if it walks like a duck, looks like a duck and quacks like a duck, it is very likely, a duck. In late September 2022 Google announced it was shutting Stadia down. A rather simple prophecy had been fulfilled. Part of the reason Google might have shut down Stadia was because they did not see a strong enough user adoption. Ironically enough, a problem that came about because people did not trust Google to not shut Stadia down prematurely. Coupled with its privacy concerns, Android monopoly ( )) and app store fees, Google was now company with a growing reputational problem. Google had a phrase, "Don’t Be Evil" during their early days. They have since dropped it. killedbygoogle.com see story It's Not Thaaat Hard Some growth hacks are 'common sense'. I put that in quotes because perspective and background really do matter. An example was FaceBook discovering that a reduced data version of the FaceBook app, FaceBook Lite, was able to spur growth in developing countries. An example: India, Facebook's largest market by user count, has a median monthly wage of about $200. The median monthly wage in the US is $54,000. People with low-cap data plans would therefore be more likely to prefer low-data usage apps. As data would be relatively more valuable to them due to its cost. Another example is the Microsoft Xbox One launch 'drama'. Microsoft initially planned to require the Xbox One to connect to Microsoft every hour to retain the ability to play games. The reaction to this, along with the removal of the ability to play used games, was brutal. It was so bad that Microsoft backtracked on the idea. One theory was that people in Redmond (Microsoft HQ location) were so used to their constant and easy access to the internet that they subconsciously assumed their customers' lives were that connected as well. Having a good product design overview helps to avoid scenarios such as these. Pay attention to your customers before you design the product. Data, Data, Data Do data analysis of your product usage. How are people using it? Could a small feature be in fact the most used part of your product? If it's a shopping app, on what screen do most of your users quit using the app? Is there a problem there? Information gathered here could help you decide if changes are needed. However, be careful when analyzing data. Causation is not always equal to correlation. Also, know your users and their types. For example, there are many people who spend hours a day scrolling through social media but do not post anything. There are others who post but may spend less than an hour a day online. Both categories are useful. Ideally do not try to solve the 'problem' of people spending less time using your app if that is not a problem for that category of users. One of the most important but also difficult tools to get is direct feedback from your users. Feedback from your users could help you plan marketing campaigns by knowing what to highlight and who to target. Feedback could also help you know what new features to implement. When you are testing new features in your product, it is advisable to introduce them gradually and test them as you do. As opposed to infrequent large changes. This would help avoid building on a mistake. Quality x Product Here is a question many entrepreneurs need to ask themselves before starting work on a project: Should your product exist? Struggling for a product to be used is not a great idea. If your product falls under the category of frivolous. It might be best to seriously consider if putting time and money into it is a good idea. Your initial market may not be your whole target market. Facebook for example started via targeting university students in the US. Consider what to change in your strategy and product if you’re trying to drive growth to a larger market. If a pivot is likely, try to do so as early as possible. This would help avoid spending a lot of resources marketing a product only to change it significantly not long after. There are cases, of course, like Netflix that pivoted while they were well into being a mature company. But new companies may not always be able to financially afford a rebranding. On quality. Lots of mobile app developers are now aware that notification reminders to use an app can help drive engagement. However there is a risk of notification fatigue among users. Ideally only send notifications to users when it is of clear use to them e.g. an item they were seeking is back in stock. There is a risk that your app would be deleted entirely if it's seen to pester. A strong first impression is key. There are lots of mobile apps that are opened only once and then forgotten. Similarly there are lots of websites that users leave after only checking the page they arrived on. Impressing your audience at first attempt is key, because even if not all become your customers, this impression can lead to your product being spread via word of mouth. Organics Ahhh, the dream of creating a product with viral growth. There's nothing quite like watching a product launch and do the marketing job for you. I can only imagine it's the same feeling tired new parents might get if their toddler suddenly started taking care of him/herself. And as it happens, unlike toddlers, products do sometimes launch to marketing success of magical proportions. Whether intentionally or not. However viral growth can work both ways, and once something goes viral, it's hard to keep a handle on. Take the launch of the video game Fallout 76 for example. A much-anticipated game from a franchise with millions of fans. However, the game was launched with a litany of problems, from mistakes in design to bad choices in management. The fans were not pleased. The ensuing protest that spread over the Internet like wildfire was, . The result was that even people, like me, who had never paid any attention to the game's launch heard about its troubles. Ladies and gentlemen, that, is not how we would like to launch a product. strong The moral of that story is that in creating and launching a product, care should be taken to create something worth spreading organically. It is worth remembering in this day and age of marketing = digital marketing, that a real world exists. Yes, large parts of that real-world might involve people's heads glued to their phone screens, but word of mouth is still a powerful organic growth tool (organic growth in this case refers to growth obtained without paid marketing efforts). In-person conversations are likely a more convincing medium than online messages. So whether it's by friends or sales representatives, remember the offline channel. When you are online and seek to spread via word of mouth, consider the point of view that the recommendation is framed. For example, Friend A sending a message to Friend B about a product is arguably better than the company taking the email address of Friend B, and sending a generalized advertisement message. Companies could give message design tools to customers, via a web form, for example, to help them send bright, well-designed, but still personalized messages. Obtaining new customers via your existing ones is an effective method. Sometimes incentivizing your customers via offering referral programs where customers get some benefit is worth considering. Careful Now Realistically, no one is a perfect business expert. So assess all the advice you receive. Yes, even this one. Cover Photo by on Unsplash Kristopher Roller

How to Find X When Your Product + X = Success

Cómo encontrar X cuando su producto + X = Éxito

Comment trouver X lorsque votre produit + X = succès

एक्स कैसे खोजें जब आपका उत्पाद + एक्स = सफलता

製品 + X = 成功の場合に X を見つける方法

Como encontrar X quando seu produto + X = sucesso

Cách tìm X khi Sản phẩm của bạn + X = Thành công

当您的产品 + X = 成功时如何找到 X

PROBLEM STATEMENT Marine plastic pollution has been at the forefront of climate issues for the past decade. Not only does plastic in the ocean capable of killing marine life by strangulation or starvation but it is also a major factor in warming the ocean by trapping CO2. In recent years, there have been numerous attempts at cleaning the plastic that has been circling our oceans such as by the non-profit group, The Ocean Cleanup. The problem with a lot of cleanup processes is that it requires human labor and is not cost-efficient. There has been a lot of research done to automate this process by using Computer Vision and Deep Learning to detect marine debris to utilize ROVs and AUVs for cleanup. Note: This is theoretical and we are working on publishing our paper. Once it gets peer-reviewed and gets published in a journal I will update this article. So, watch this space for the release of our academic paper! : , , and Reading Resources Our Paper on Using CV and DL to detect Epipelagic plastic The University Of Minnesota's paper on a general-purpose marine debris detecting algorithm The Ocean Cleanup’s paper on floating plastic detection on rivers. The major issue for this approach is regarding the availability of datasets to train the computer vision models. The JAMSTEC-JEDI dataset is a good collection of Marine Debris off the coast of Japan on the seafloor. But, apart from this dataset, there is a huge discrepancy in the availability of datasets. For this reason, I’ve utilized the help of Generative Adversarial Networks; The DCGAN in particular to curate synthetic datasets that could theoretically be a close replica to real datasets over time. (I wrote theoretically because there are a lot of variables such as quality of real-images available, progress with GANs, training, etc). GAN and DCGAN GANs or Generative Adversarial Networks were proposed by Ian Goodfellow et al in 2014. GANs consist of a simple 2 components called Generators and Discriminators. An oversimplification of the process is as follows: The Generators role is to generate new data and the Discriminators role is to distinguish between the generated data and the actual data. In the ideal scenario, the discriminator is unable to distinguish between the generated data and real data which results in ideal synthetic data points. DCGAN is a direct extension of the GAN architecture mentioned above except it uses Deep Convolutional Layers in the discriminator and generator respectively. It was first described by Radford et. al in the paper . The discriminator is made up of strided convolutional layers while the generator is made up of convolutional transpose layers. Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks GAN architecture (source: ) Here PyTorch Implementation In this method, we are going to be applying the DCGAN architecture on the dataset. If you’re not familiar with the DeepTrash dataset, consider reading my paper . DeepTrash is a collection of plastic images in the epipelagic layer and abyssopelagic layer of the ocean curated for marine plastic detection using computer vision. DeepTrash A Robotic Approach towards Quantifying Epipelagic Bound Plastic Using Deep Visual Models Let's begin coding! Installing Requirements We start by installing all the basic requirements to building our GAN model such as Matplotlib and Numpy. We will also be utilizing all the tools (such as neural networks, transformations) from Pytorch. If you’re not familiar with PyTorch I recommend giving this article a read: Initializing our Hyperparameters This step is fairly straightforward. We’re going to set up the hyperparameters that we want to train the Neural Network with. These hyperparameters are borrowed straight from the paper and PyTorch’s tutorial on training GANs. Generator and Discriminator Architectures Now, we define the architectures for the generator and discriminator. Defining the Training Function After defining the Generator and Discriminator classes, we move on to defining the training function. The training function takes in the generator, discriminator, optimization functions, and the number of epochs as the arguments. We train the generator and discriminator by recursively calling the train function until the required number of epochs. We do this by iterating through the dataloader and updating the discriminator with new images from the generator and calculating and updating the loss function. Monitoring and Training the DCGAN After we’ve established the Generator, Discriminator, and Train functions the final step is to simply call the train function for the number of epochs we defined. I’ve also used which allows us to monitor our training. Wandb Results We plot the loss incurred by the generator and discrimination during training. IMG - Generator and Discriminator Loss during Training We can also pull up an animation of how the generator was generating images to see a difference between the real and the fake images. Which looks something like this: And the final resulting image after training (the same image as above). CONCLUSION In this article, we discussed the use of Deep Convolution Generative Adversarial Networks to generate synthetic images of marine plastic that can be used by researchers to extend their current dataset of marine plastic. This can help with giving researchers the ability to extend their dataset with a mix of real and synthetic images. As you can see with the results, the GAN still needs a lot of work. The ocean is a complex environment with varying illumination, turbidity, blur, etc. But this is a theoretical starting point that other researchers can build off of. If you’re interested in improving the results and building a better network architecture, please reach out to me at gautamtata.blog@gmail.com.

Stories, News & Mentions about WEPIPE

Wepipe’s Stories on HackerNoon