Get the most recent info and news about Work Inspire on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Work Inspire

workinspire

Whether you’re a nonprofit, business, or community leader or you are just looking for inspiration, the upcoming “Transformational Leadership” workshop is for you. Hosted by the Telluride Foundation, t

A get-together to inspire change

Indiana Owned is on a mission to help you identify and support local businesses faster and easier! Today, were exploring Burke Metal Work!

Indiana Owned Burke Metal Work: Crafting unique Christmas ornaments and keychains

Should a retailer lack a specific component necessary for integration, Chicory will work in partnership with the retailer ... www.chicory.co. About Chicory Chicory believes in the power of recipes to inspire consumers and drive commerce.

Chicory Builds Direct Integration With Albertsons to Enhance Online Shopping Experience for the Retailer

Anyone wanting to find out how Virtual Reality (VR) can improve their health, help train staff, or inspire students in education, should get along to Northampton Central Library on Thursday, 2 November.

Free virtual reality experience day on Thursday

For startup founders, the ability to inspire has everything to do with the ability to get others to see the world the way you do. Once you can clearly and concisely articulate both that problem and your solution, you need to work on the ability to get ...

How to be an inspirational startup founder

Jordan Strauss/Invision/AP NEW YORK — Now that she’s stepped back from the sport she dominated like few others, Serena Williams is ready to reflect. The tennis great has a two-book deal with ...

Serena Williams has two-book deal for ‘intimate’ memoir and an ‘inspirational’ work

The exhibit features work by over 50 artists depicting generations ... “The hardest question is, what will inspire change?” Mr. Goodell said in a tour of the show with Simone Wicha, director ...

What Would Inspire Climate Action? Perhaps an Orange Sky.

It's called Made in China 2.0. Chong asks audiences not to discuss the content of his show, so instead he joins us to discuss the work of others and how those works have inspired him. And we delve ...

Bridal laments and vanishing vultures inspire new work at OzAsia

The Awards celebrate extraordinary achievements of people of all ages, from children of courage to lifesaving emergency workers, inspirational fundraisers and people who make the world a better ...

Daily Mirror Pride of Britain Awards 2023 - full list of amazing winners

Garcia Olmos speaks to KVNF's Laura Palmisano about being a first generation Latiné in the United States, challenges she faced when learning to navigate life in a new country, and her work with La Plaza.

Firsthand experiences inspire Mesa County resident to work with fellow migrants, immigrants

Good Morning Britain Young Fundraiser of the Year - To reward a young person aged 17 and under for inspirational charity work. This Morning Emergency Services Award - For police, fire ...

The Pride of Britain Awards 2023

Dozens of Native American students from Crandon High School as well as the middle and high schools in Wabeno and Laona will gather at the Forest County Potawatomi Community Center Friday. The event will be focused on civic engagement, democracy, and ...

Forest County Potawatomi Community and Wisconsin Native Vote work to inspire next generation to be civically engaged

But what should you do? Figure out how to get involved in their work without stepping on their toes. Ask ChatGPT for suggestions. “I want to inspire my team to work harder by demonstrating that ...

5 ChatGPT Prompts To Get Your Team To Work Harder (and Achieve More)

Introducing the InfoSec colour wheel — blending developers with red and blue security teams.

Presentamos la rueda de colores de InfoSec — que combina desarrolladores con equipos de seguridad rojos y azules.

Here is a quick guide on how to Install Gitea with on Debian 11 Postgresql Start with a quick system update: sudo apt-get update -y
sudo apt-get upgrade -y Setup postgresql sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

# Import repo signing key with:
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

sudo apt-get update
sudo apt-get -y install postgresql System setup sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo systemctl status postgresql Next, You need to switch to ’SCRAM-SHA-256’ scheme from md5 encryption scheme for better security. If you want to connect to PostgreSQL remotely, then you need to allow your IP address in the PostgreSQL configuration file. Open the configuration file to make changes using the following command: sudo vim /etc/postgresql/15/main/postgresql.conf

# Inside file uncomment:

listen_addresses = 'localhost, 45.32.225.46'
password_encryption = scram-sha-256 Then, restart the systemd service: sudo systemctl restart postgresql Login into postgresql sudo -u postgres psql Inside psql: postgres=# CREATE ROLE gitea WITH LOGIN PASSWORD 'secure@123';

postgres=# CREATE DATABASE giteadb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE giteadb TO gitea;

postgres=# exit if you receive the error: The database settings are invalid: migrate: sync: pq: permission denied for schema public you might have to do the following in psql after the install is complete, I did: ALTER DATABASE gitea OWNER TO gitea Within /etc/postgresql/15/main/pghba.conf file: host    giteadb        gitea            134.122.38.0/32           scram-sha-256 Install and configure git: sudo apt install git
git --version

git config --global user.name "Your Name"
git config --global user.email "youremail@domain.com”
git config --list Create git user for gitea sudo adduser \
   --system \
   --shell /bin/bash \
   --gecos 'Git Version Control' \
   --group \
   --disabled-password \
   --home /home/git \
   git Install gitea: wget https://dl.gitea.com/gitea/1.20.3/gitea-1.20.3-linux-amd64 -O /usr/local/bin/gitea
sudo chmod +x /usr/local/bin/gitea Make directory structure for gitea: sudo mkdir -p /etc/gitea

sudo mkdir -p /var/lib/gitea/{custom,data,indexers,public,log}

sudo chown -R git:git /var/lib/gitea/

sudo chown root:git /etc/gitea

sudo chmod -R 750 /var/lib/gitea/

sudo chmod 770 /etc/gitea Create systemd service file: sudo vim /etc/systemd/system/gitea.service Inside: [Unit]
Description=Gitea
After=syslog.target
After=network.target
After=postgresql.service

[Service]
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/gitea/
ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini
Restart=always
Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start gitea
sudo systemctl enable gitea
sudo systemctl status gitea

# Verify running on port 3000:
netstat -tulpan | grep 3000 Navigate to port 3000 on your server and you will see it running! Congrats, you have now dropped GitHub as your Git server. Also published here.

Step-by-Step Guide for Installing Gitea on Debian 11

MinIO is one of the most widely implemented object stores because it offers high-performance, massive scalability, high-availability and adheres to the industry-standard S3 protocol. MinIO is capable of tremendous performance - a achieved 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. No matter where you run MinIO – bare metal, virtual instances or Kubernetes – MinIO’s performance and scalability provide an excellent foundation for such cloud-native applications as data lakes, analytics, AI/ML and more. recent benchmark In addition to Kubernetes, customers run MinIO on virtual instances and bare metal, frequently relying on in the datacenter and clouds like , and . Linux’s small footprint coupled with efficient resource utilization make it a versatile and flexible choice for running MinIO. Yet, the greater numbers of Linux machines and instances require automation to reduce administrative burden. There is a need for managing MinIO servers as an init service using SystemD because it helps to automate the service lifecycle, particularly during startup, shutdown and restart. Supermicro hardware AWS GCP Azure is a service manager that maintains services in Linux systems during boot up, shutdown, initialization and more. Think of it as the successor to init scripts where you had to write the whole thing from scratch, but with SystemD you just fill in a few parameters and the commands and logging are standardized. SystemD One of the several challenges of using the older init.d service management was that you had to write the entire file from scratch. Each init.d file was unique and had a different way of managing the lifecycle of the service. SystemD takes the lessons learned from init.d and standardizes them. For instance, the SystemD service files will work any service as long as it's written to run – only the paths, names, and semantics are different – but the basic structure, how to restart services, reading logs, gracefully un-mounting filesystems, waiting for network connections to come up, among other things, are now common among all services, so no matter which service you run you know exactly where to look for its logs. Not only is SystemD versatile among different services, but you can take the same SystemD service file and use it across multiple operating systems, as long as the operating system is also using SystemD to manage the lifecycle of its services. This simplifies automation as you can be working on Ubuntu while creating the initial file, but the same file can then be deployed to CentOS/RedHat operating system as long as the paths and other things remain the same, which in most cases they do. There are several main components to SystemD but a few that you will come across more often are: : This is what is used to control the processes, whether to stop, start and restart mostly. The most common commands used are: systemctl systemctl start <service> systemctl stop <service> systemctl restart <service> journalctl: The logs for the services when they do their startup, shutdown and restart operations. It’s a good way to get insight if something is not working properly. The most common command I use, which shows logs enough to fill up your terminal window and scrolls to the very end: journalctl -e -u <service> To run most of the commands in this guide you need to have `sudo` or root access because SystemD config files need to be permissioned as root. You can also find in our repository. additional instructions for MinIO and SystemD MinIO Binary Fetch the MinIO binary from upstream # wget https://dl.min.io/server/minio/release/linux-amd64/minio


--2022-07-24 11:31:33--  https://dl.min.io/server/minio/release/linux-amd64/minio
Resolving dl.min.io (dl.min.io)... 132.68.11.115, 138.118.66.212
Connecting to dl.min.io (dl.min.io)|132.68.11.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96649216 (92M) [application/octet-stream]
Saving to: 'minio'

minio                       100%[==========================================>]  92.17M  50.2MB/s    in 1.8s

2022-07-24 11:31:35 (50.2 MB/s) - 'minio' saved [96649216/96649216] Make it executable so SystemD can run MinIO as a service # chmod +x minio Move it to a location that is executable under $PATH # mv minio /usr/local/bin/ Service File MinIO can be installed in a few different ways on Linux based on the operating system. RedHat/CentOS and its derivatives rely on the.rpm package and Debian/Ubuntu  use the .deb package. In both cases, the following SystemD service file is included in the installation package. However, we manually fetched the MinIO binary from upstream, so we’ll manually create the SystemD service file. Use your favorite text editor to create a SystemD file at with the following contents. /etc/systemd/system/minio.service [Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name}) : The Linux system group the Minio daemon will run as. Create it using the following command: Group=minio-user groupadd -r minio-user : The Linux system user the MinIO daemon will run as. Create the user using the following command: User=minio-user useradd -M -r -g minio-user minio-user : This prevents creating a home directory for the user since this is a service. -M : Systems users have a separate UID/GID range for tracking purposes, this flag will create the user from the predetermined range. -r : Group to add the user under. -g <group_name> MinIO Distributed Setup There are a couple of prerequisites that are required to set up on the bare metal node before MinIO service can be started. Create a new disk and ensure it is not on the same partition as root volume, to avoid the following message: Error: Disk /mnt/disk1/minio is part of root disk, will not be used Create 4 directories on your local host where the new disk was just mounted, in this case : /mnt/data mkdir -p /mnt/data/disk1 \
mkdir -p /mnt/data/disk2 \
mkdir -p /mnt/data/disk3 \
mkdir -p /mnt/data/disk4 the directories with the MinIO user and group chown chown minio-user:minio-user /mnt/data/disk1 /mnt/data/disk2 /mnt/data/disk3 /mnt/data/disk4 Environment Service Files Update this file with the following content /etc/default/minio # Set the hosts and volumes MinIO uses at startup

# The command uses MinIO expansion notation {x...y} to denote a

# sequential series.

#

# The following example covers four MinIO hosts

# with 4 drives each at the specified hostname and drive locations.

# The command includes the port that each MinIO server listens on

# (default 9000)


MINIO_VOLUMES="https://minio1.example.com:9000/mnt/data/disk{1...4}/minio"


# Set all MinIO server options

#

# The following explicitly sets the MinIO Console listen address to

# port 9001 on all network interfaces. The default behavior is dynamic

# port selection.


MINIO_OPTS="--console-address :9001"


# Set the root username. This user has unrestricted permissions to

# perform S3 and administrative API operations on any resource in the

# deployment.

#

# Defer to your organizations requirements for superadmin user name.


MINIO_ROOT_USER=minioadmin


# Set the root password

#

# Use a long, random, unique string that meets your organizations

# requirements for passwords.


MINIO_ROOT_PASSWORD=minioadmin


# Set to the URL of the load balancer for the MinIO deployment

# This value *must* match across all MinIO servers. If you do

# not have a load balancer, set this value to to any *one* of the

# MinIO hosts in the deployment as a temporary measure.

MINIO_SERVER_URL="https://minio.example.net:9000" Start the MinIO Process We have all the pieces we need to enable and start the service # systemctl enable minio.service
# systemctl start minio.service Check the status of the service and logs # systemctl status minio.service
# journalctl -e -u minio.service Verify that the MinIO service has come up. In the logs you should see something similar to this: Aug 01 13:27:06 aj-test-3 systemd[1]: Starting MinIO...
Aug 01 13:27:06 aj-test-3 systemd[1]: Started MinIO.
Aug 01 13:27:07 aj-test-3 minio[3241]: Formatting 1st pool, 1 set(s), 4 drives per set.
Aug 01 13:27:07 aj-test-3 minio[3241]: WARNING: Host minio1.example.com:9000 has more than 2 drives of set. A host fai>
Aug 01 13:27:07 aj-test-3 minio[3241]:  You are running an older version of MinIO released 4 days ago
Aug 01 13:27:07 aj-test-3 minio[3241]:  Update: Run `mc admin update`
Aug 01 13:27:07 aj-test-3 minio[3241]: MinIO Object Storage Server
Aug 01 13:27:07 aj-test-3 minio[3241]: Copyright: 2015-2022 MinIO, Inc.
Aug 01 13:27:07 aj-test-3 minio[3241]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Aug 01 13:27:07 aj-test-3 minio[3241]: Version: RELEASE.2022-07-26T00-53-03Z (go1.18.4 linux/amd64)
Aug 01 13:27:07 aj-test-3 minio[3241]: Status:         4 Online, 0 Offline.
Aug 01 13:27:07 aj-test-3 minio[3241]: API: https://minio.example.net:9000
Aug 01 13:27:07 aj-test-3 minio[3241]: Console: http://10.128.0.4:9001 http://127.0.0.1:9001
Aug 01 13:27:07 aj-test-3 minio[3241]: Documentation: https://docs.min.io Log in to Console Using a browser, log in to the MinIO console using the and from the configuration we made earlier. MINIO_ROOT_USER MINIO_ROOT_PASSWORD http://<server_host_ip>:9001/ Note the above setup is for you to quickly get up and running with MinIO. You can expand from a single node to a multi-node distributed configuration for additional testing. If you would like to deploy and configure MinIO  in a production environment, please refer to the . documentation SystemD Simplifies MinIO Virtual and Bare Metal Deployments Integration with SystemD is very versatile. The syntax of the service file is the same across all services The same service file will work across any OS that supports SystemD You can use SystemD with bare metal or on VMs in any cloud such as AWS, GCP and Azure. This can help with automation for DevOps by standardizing the format and making it simple to automate deployments. Automation goes a long way towards maintaining availability when running a mission critical service like object storage. Automation is a requirement for operating at scale across multiple clouds and environments. With the help of SystemD and MinIO you can automate your cloud object storage deployments and ensure the service lifecycle is managed smoothly and successfully. Got questions? Want to get started? Reach out to us on . Slack Also published . here

Automating Cloud Storage Deployments With MinIO and SystemD

Automatización de implementaciones de almacenamiento en la nube con MinIO y SystemD

Automatisation des déploiements de stockage cloud avec MinIO et SystemD

मिनिआईओ और सिस्टमडी के साथ क्लाउड स्टोरेज परिनियोजन को स्वचालित करना

MinIO と SystemD によるクラウド ストレージ デプロイメントの自動化

Automatizando implantações de armazenamento em nuvem com MinIO e SystemD

Tự động triển khai lưu trữ đám mây với MinIO và SystemD

使用 MinIO 和 SystemD 自动化云存储部署

In this article, I want to tell you how to connect and further generate Allure reports for a test project if your test project is made on the Citrus Framework. I hope you know about Allure framework, which is designed to generate reports after passing tests. The report opens on a browser page, where you can see which tests passed successfully and which did not, you can also see at which test step the expected result did not match. Since the Citrus Framework can use the TestNG or Junit engines, you need to include the Allure library according to the engine you are using. I will show the example of TestNG and the project builder we use Maven. 1. First of all, the Allure distribution must be installed locally and the path to the distribution must be written in PATH, similar to installing Maven. The distribution can be taken from here by choosing the latest version: https: //repo.maven.apache.org/maven2/io/qameta /allure/allure-commandline You can check if you have Allure locally with this command in cmd: allure --version If everything is ok, you will see the current version installed locally: 2. In the pom.xml file, you need to add the dependency “allure-testng” and two plugins “maven-surefire-plugin” and “allure-maven”, probably “maven-surefire-plugin” will already be in your list, then add the configuration, as shown below: </properties> <properties> < aspectj.version > 1.9.4 </ aspectj.version > < allure.version > 2.22.0 </ allure.version > < dependencies > < dependency > < groupId > io.qameta.allure </ groupId > < artifactId > allure-testng </ artifactId > < version > ${allure.version} </ version > </ dependency > </ dependencies > < build > < plugins > < plugin > < groupId > org.apache.maven.plugins </ groupId > < artifactId > maven-surefire-plugin </ artifactId > < version > 2.22.2 </ version > < configuration > < failIfNoTests > false </ failIfNoTests > < workingDirectory > ${project.build.directory} </ workingDirectory > < argLine > -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </ argLine > < systemPropertyVariables > < allure.results.directory > ${project.build.directory}/allure-results </ allure.results.directory > </ systemPropertyVariables > </ configuration > < dependencies > < dependency > < groupId > org.aspectj </ groupId > < artifactId > aspectjweaver </ artifactId > < version > ${aspectj.version} </ version > </ dependency > </ dependencies > </ plugin > < plugin > < groupId > io.qameta.allure </ groupId > < artifactId > allure-maven </ artifactId > < version > 2.10.0 </ version > < configuration > < reportVersion > ${allure.version} </ reportVersion > < allureDownloadUrl > https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/${allure.version}/allure-commandline-${allure.version}.zip </ allureDownloadUrl > </ configuration > </ plugin > </ plugins > </ build > After Reload Maven, the connected Allure plugin will appear in the list of plugins: 3. Now we can add Allure annotations to tests as shown in the screenshot: private TestContext context; @CitrusTest sendRequest(); getResponse();
    } $(http() .send() } .receive()
                        .response(HttpStatus.OK)
                        .message()
                        .type(MessageType.JSON) );
    } user.setData(data); user.setSupport(support); }
} @Story( "GET REQUEST" ) public class FirstTestGetUser extends TestNGCitrusSupport { @Test(description = "Getting information about user" ) public void getTestActions ( ) { this .context = citrus.getCitrusContext().createTestContext(); //        "Send GET request" //        "Get response" @Step( "Send GET request" ) public void sendRequest ( ) { .client( "restClientReqres" ) .get( "users/" + context.getVariable( "userId" ))); @Step( "Get response" ) public void getResponse ( ) { $(http().client( "restClientReqres" ) .body( new ObjectMappingPayloadBuilder(getJsonData(), "objectMapper" )) public User getJsonData ( ) { User user = new User(); Data data = new Data(); data.setId(Integer.valueOf(context.getVariable( "userId" ))); data.setEmail( "janet.weaver@reqres.in" ); data.setFirstName( "Janete" ); data.setLastName( "Weaver" ); data.setAvatar( "https://reqres.in/img/faces/2-image.jpg" ); Support support = new Support(); support.setUrl( "https://reqres.in/#support-heading" ); support.setText( "To keep ReqRes free, contributions towards server costs are appreciated!" ); return user; We added Allure annotations @Story for class and @Step for methods in the test﻿ above. 4. After passing the test, we can generate a report by calling the "allure:serve" command: 5. Browser will automatically launch with the report page: Thus, we connected Allure reports to our test project. If you are using the Junit engine in your test project, then you will need to follow the same steps, only you will need the “allure-junit4” or “allure-junit5” dependency. This is all. I wish you happy testing).

Let's Connect Allure Reports to a Test Project Made on Citrus Framework

As part of my personal development, I've created a personal health platform that uses various different microservices (Built using Azure Functions) that extract data from my Fitbit account and store them in an Azure Cosmos DB database. I have other microservices that pass messages between different services via Azure Service Bus. For this project, I use Azure DevOps to build my artifacts, run my unit tests and deploy my microservices to Azure. The great thing about DevOps is that we can do all of this within the YAML pipeline. Yes I said YAML. Honestly, I don't know what the fuss is all about 😂 In a previous post, I talked about how we can deploy NuGet packages to a private feed in Azure Artifacts using YAML pipelines. If you haven't read that post yet, you can check it out below! https://dev.to/willvelida/publishing-nuget-packages-to-a-private-azure-artifacts-feed-with-yaml-build-files-3bnb In this article, we will turn our attention to building and deploying C# Azure Functions using a single build file. What we'll cover We've got quite a bit to cover, so I'll break down my YAML file and talk about each stage in the following order: Triggering a Build 👷‍♂️👷‍♀️ Using User-Defined Variables in our pipelines 👨‍🔬👩‍🔬 Defining Stages 💻 Building our project 🔨 Running our tests 🧪 Getting code coverage 🧾 Producing a Build Artifact 🏠 Using Secrets from Key Vault 🔑 Deploying our Function to Azure ⚡ Running our build pipeline 🚀 Triggering a Build 👷‍♂️👷‍♀️ In our build pipelines, we can use triggers to run our pipelines. We can use different types of triggers, ranging from triggering a build when we push our code to a defined branch, to . scheduled triggers based on a CRON schedule For this project, I'm just triggering my build every time I push to the main branch. To do so, I define this in my YAML file like so: trigger:
  - main Now if I wanted to do this properly, we could define multiple branch names to trigger builds from like so: trigger:
  - feature/*
  - bugfix/* This would trigger the builds when we push code to either our feature or bugfix branch. When we raise Pull Requests in DevOps, we can create policies on our PR to only be eligible to merge back into our main branch provided that the build was successful. This helps us increase our confidence that the code we are merging (and deploying) is of high quality, particularly if we are running tests as part of our build pipeline. To enable build validation on your branch policies, check out this documentation Using User-Defined Variables in our pipelines 👨‍🔬👩‍🔬 We can use user-defined variables in our YAML files to use throughout our pipeline. Doing this gives us the benefit of re-using common values across various stages and tasks in our pipeline, and it allows us to track changes to them over time using version control. We can define variables within out file with the following scope: , making it available to all jobs in our pipeline. At the root level , making it available to a specific stage in our pipeline. At the stage level , making it available to a specific job in our pipeline. At the job level In my case, I'm going to be re-using these variables throughout my pipeline, so I'm defining them at the root level. variables:
  buildConfiguration: 'Release'
  vmImageName: 'vs2017-win2016'
  functionAppName: 'famyhealthfitbitbody'
  azureSubscription: '<azureSubscription-id-or-name>'
  workingDirectory: '$(System.DefaultWorkingDirectory)/MyHealth.Fitbit.Body'
  projectName: 'MyHealth.Fitbit.Body' We're not just limited to user-defined variables in YAML pipelines! We can use System and Environment variables as well. To learn more about defining variables in YAML pipelines, check out this article Defining Stages 💻 Within our YAML pipelines, we can organize our jobs into Stages. These divide our pipeline into logical....well, stages that perform certain tasks, such as 'build the app', 'deploy to production'. In this example, I just have a stage and deploy stage. Every YAML pipeline has at least one stage even if we don't define it. As I have done in my file, we can also run stages depending on whether the previous stage has succeeded or not, like so: stages:
  - stage: Build
    displayName: Build Stage

    jobs:
      - job: Build
        displayName: Build MyHealth.Fitbit.Body
        pool:
          vmImage: $(vmImageName) Using YAML pipelines, we can define multiple stages of our pipeline and store it as code. Check out the if you want to learn more about how Stages work in YAML pipelines. documentation Building our project 🔨 Let's turn our attention to our Function. First up, I want to restore and build my .csproj files. In our YAML pipelines, we can se .NET Core tasks to run dotnet commands against our artifacts. To build my project, I'll and . restore build I'm using a custom NuGet feed hosted on Azure Artifacts, so I will need to point to this feed when restoring. For the build task, I want to build a release package, so I'll pass in the variable as an argument when running the command. buildConfiguration build For both commands, I'm going to point to all .csproj files in my repository. steps:
- task: DotNetCoreCLI@2
  displayName: Restore
  inputs:
   command: 'restore'
   feedsToUse: 'select'
   vstsFeed: '<feed-id>'
   projects: '**/*.csproj'
    
- task: DotNetCoreCLI@2
  displayName: Build
  inputs:
   command: 'build'
   projects: '**/*.csproj'
   arguments: --configuration $(buildConfiguration) Running our tests 🧪 Alongside my function code, I have a project for my Unit Tests. As part of my build, I want to run my unit tests to make sure that my changes haven't broken any of our tests and that they still pass. If any of my tests fail, I want the build to fail. To run our tests, we can use a .NET Core task that runs the command and point the task at our Unit Test .csproj file. test I also want to collect code coverage metrics with Coverlet. Coverlet is .NET tool that allows us to get unit test coverage for line, branch and method coverage. In our arguments parameters, I'm telling my task to run Coverlet to get the code coverage and then publish the test results to the */TestResults/Coverage folder on my build agent: - task: DotNetCoreCLI@2
  displayName: Run Unit Tests
  inputs:
   command: 'test'
   projects: '**/*UnitTests/*.csproj'
   arguments: '--configuration $(buildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/'
   publishTestResults: true Getting code coverage 🧾 Once we've generated our coverage report, we can publish it to see the results. I've got two tasks within my pipeline to achieve this. First, I'm using a bash script task to install the report generator onto the Build agent. Once that's installed, I'm creating the xml report into my TestResults/Coverage folder on the agent. We can also specify the report type that we want to generate, but for this tutorial I'm just generating an inline HTML report. If you want to use a task for this instead of running an inline script, you can try installing the following onto your DevOps organisation. task Once we have generated a report, I've got a second task that publishes the code coverage report. - script: |
    dotnet tool install -g dotnet-reportgenerator-globaltool
    reportgenerator -reports:$(Build.SourcesDirectory)/TestResults/Coverage/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura
    displayName: Create Code coverage report
  
- task: PublishCodeCoverageResults@1
  displayName: 'Publish Code Coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/TestResults/Coverage/' When our build finishes, we can review our code coverage report within DevOps! It should look similar to this: Producing a Build Artifact 🏠 We've built our project and our tests pass! We now need to produce a build artifact that we will deploy as our Function. Starting with publish, all we need here is a .NET Core task that runs the command, publishes our project as a 'Release' package, zip it and then publish it to our Artifact Staging Directory on our build agent. publish We then use a to publish an artifact called 'drop' into our staging directory. Publish Build Artifact task - task: DotNetCoreCLI@2
  displayName: Publish
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '**/*.csproj'
    arguments: '--configuration $(buildConfiguration) --output $(build.artifactstagingdirectory)'
    zipAfterPublish: True
            
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)' Just a mini-recap before we move onto deployment. In our Build Stage we have: Built our .NET Projects Run our Unit Tests to ensure our build is a quality build Generate and publish a code test coverage report. Publish a build artifact to deploy to Azure. With that achieved, we are now ready to move onto our deployment stage! Deploying our Function Stage For our deployment stage, we'll need to create a new stage in our YAML file. In this case, I've added a and property to our Stage YAML block. In the YAML file, I'm saying that the 'Deploy' stage our 'Build' stage and that the 'Deploy' stage will only run on the condition that the 'Build' Stage has succeeded. condition dependsOn depends on - stage: Deploy
    displayName: 'Deploy MyHealth.Fitbit.Body'
    dependsOn: Build
    condition: succeeded()

    jobs:
      - deployment: Deploy
        displayName: Deploy MyHealth.Fitbit.Body
        environment: Production
        pool:
          vmImage: $(vmImageName) When we use multiple stages in a pipeline, they will run in the order that we define them in the YAML file. We can add dependencies to our stages using the parameter. dependsOn We can control which stages run by specifying the condition of that Stage. In my case, I'm stating that the 'Deploy' stage will only run provided that the 'Build' stage has run successfully. To learn more about how conditions work in YAML pipelines, check out the following . documentation Using Secrets from Key Vault 🔑 In my function, I'm doing performing a variety of tasks such as sending messages to Service Bus Queues and Topics, calling APIs using Access tokens etc. etc. In order to make these operations, I need to use connection strings and secrets that I'm storing in a Azure Key Vault. Because these are secrets and I don't want to expose them, I need to use a Azure Key Vault task within my pipeline. For this purpose, I can use the to download the secrets I need. Azure Key Vault task As a pre-requisite for this task, I need to create a that's linked to my Azure Subscription and I need a Key Vault that contains my secrets. Azure Resource Manager service connection Once I have this setup, I can use the following snippet in my pipeline: steps:
  - task: AzureKeyVault@1
    inputs:
      azureSubscription: '$(azureSubscription)'
      KeyVaultName: '<key-vault-name>'
      SecretsFilter: '*'
      RunAsPreJob: false Here, we are defining the subscription that contains our Key Vault with our secrets, and we are retrieving all of the secrets in our Key Vault. We could filter these out to use a comma-separated list of the secret names we want to download, but for now I'm just downloading everything. Deploying our Function to Azure ⚡ We're finally ready to deploy our Function! ⚡ For this I'm going to use a Azure App Service Deploy Task We can use this task to deploy a variety of different app service types, but in my YAML snippet, I'm specifying as our type. functionApp As a pre-req for this task, you'll need an App Service instance to deploy your code to. I'm also passing the package that we built earlier as our , using the variable for the name of our Function and then passing in key-values for our App Settings. I can pass in the secrets I need for my Function since we downloaded them in our Key Vault task. package $functionAppName - task: AzureRmWebAppDeployment@4
  displayName: 'Deploy Azure Function'
  inputs:
    azureSubscription: '$(azureSubscription)'
    appType: functionApp
    WebAppName: $(functionAppName)                   
    package: '$(Pipeline.Workspace)/drop/MyHealth.Fitbit.Body.zip'
    appSettings: '-FUNCTIONS_WORKER_RUNTIME "dotnet" -FUNCTIONS_EXTENSION_VERSION "~3" -KeyVaultName "<key-vault-name>" -AccessTokenName "<secret-name>" -ServiceBusConnectionString "$(<secret-value>)" -APPINSIGHTS_INSTRUMENTATIONKEY "<some-key>" -ExceptionQueue "myhealthexceptionqueue" -BodyTopic "myhealthbodytopic" -WEBSITE_TIME_ZONE "New Zealand Standard Time"'
    enableCustomDeployment: true
    RemoveAdditionalFilesFlag: true Our full YAML file Before we finish, here's my full YAML build pipeline: trigger:
  - main

variables:
  buildConfiguration: 'Release'
  vmImageName: 'vs2017-win2016'
  functionAppName: 'famyhealthfitbitbody'
  azureSubscription: '<azureSubscription-id-or-name>'
  workingDirectory: '$(System.DefaultWorkingDirectory)/MyHealth.Fitbit.Body'
  projectName: 'MyHealth.Fitbit.Body'

stages:
  - stage: Build
    displayName: Build Stage

    jobs:
      - job: Build
        displayName: Build MyHealth.Fitbit.Body
        pool:
          vmImage: $(vmImageName)
        
        steps:
          - task: DotNetCoreCLI@2
            displayName: Restore
            inputs:
              command: 'restore'
              feedsToUse: 'select'
              vstsFeed: '<artifact-feed-id>'
              projects: '**/*.csproj'
    
          - task: DotNetCoreCLI@2
            displayName: Build
            inputs:
              command: 'build'
              projects: '**/*.csproj'
              arguments: --configuration $(buildConfiguration)

          - task: DotNetCoreCLI@2
            displayName: Run Unit Tests
            inputs:
              command: 'test'
              projects: '**/*UnitTests/*.csproj'
              arguments: '--configuration $(buildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/'
              publishTestResults: true
  
          - script: |
              dotnet tool install -g dotnet-reportgenerator-globaltool
              reportgenerator -reports:$(Build.SourcesDirectory)/TestResults/Coverage/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura
            displayName: Create Code coverage report
  
          - task: PublishCodeCoverageResults@1
            displayName: 'Publish Code Coverage'
            inputs:
              codeCoverageTool: Cobertura
              summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml'
              reportDirectory: '$(Build.SourcesDirectory)/TestResults/Coverage/'
    
          - task: DotNetCoreCLI@2
            displayName: Publish
            inputs:
              command: 'publish'
              publishWebProjects: false
              projects: '**/*.csproj'
              arguments: '--configuration $(buildConfiguration) --output $(build.artifactstagingdirectory)'
              zipAfterPublish: True
            
          - task: PublishBuildArtifacts@1
            displayName: 'Publish Artifact'
            inputs:
              PathtoPublish: '$(build.artifactstagingdirectory)'
  
  - stage: Deploy
    displayName: 'Deploy MyHealth.Fitbit.Body'
    dependsOn: Build
    condition: succeeded()

    jobs:
      - deployment: Deploy
        displayName: Deploy MyHealth.Fitbit.Body
        environment: Production
        pool:
          vmImage: $(vmImageName)

        strategy:
          runOnce:
            deploy:
              
              steps:
                - task: AzureKeyVault@1
                  inputs:
                    azureSubscription: '$(azureSubscription)'
                    KeyVaultName: '<key-vault-name>'
                    SecretsFilter: '*'
                    RunAsPreJob: false

                - task: AzureRmWebAppDeployment@4
                  displayName: 'Deploy Azure Function'
                  inputs:
                    azureSubscription: '$(azureSubscription)'
                    appType: functionApp
                    WebAppName: $(functionAppName)                   
                    package: '$(Pipeline.Workspace)/drop/MyHealth.Fitbit.Body.zip'
                    appSettings: '-FUNCTIONS_WORKER_RUNTIME "dotnet" -FUNCTIONS_EXTENSION_VERSION "~3" -KeyVaultName "<key-vault-name>" -AccessTokenName "<secret-name>" -ServiceBusConnectionString "$(<secret-value>)" -APPINSIGHTS_INSTRUMENTATIONKEY "<some-key>" -ExceptionQueue "myhealthexceptionqueue" -BodyTopic "myhealthbodytopic" -WEBSITE_TIME_ZONE "New Zealand Standard Time"'
                    enableCustomDeployment: true
                    RemoveAdditionalFilesFlag: true Running our build pipeline 🚀 We've successfully created a build pipeline in YAML that will be triggered every time we push to our branch. We can view the status of our build in Azure DevOps, which looks like this: main As you can see at the bottom of the image, we can see the result of each stage within our pipeline. If a particular stage had failed, we could use this to re-run a particular stage instead of having to push our code again to . main Wrapping Up Multi-Stage Build pipelines are awesome for defining our build and release process all within the same file. By using source control, we can track changes to our build process over time. We can also do useful things like generate and publish code coverage reports, download secrets from key vault and deploy apps all within one file! Hopefully, you found this article useful! As always, if you have any questions, feel free to comment below or ask me on ! Twitter Happy Coding! 💻👨‍💻👩‍💻 Also published on: https://dev.to/willvelida/building-and-deploying-c-azure-functions-using-multi-stage-pipelines-in-azure-devops-3ne9

How to Build and Deploy C# Azure Functions using Multi-Stage Pipelines in Azure DevOps

It is becoming a common pattern to see websites and web apps written as a front end single page application (SPA) connected to a backend API. For this reason, the Visual Studio provides a several project templates for getting up and going with a Web API + SPA project. However, at time of writing, these project templates only support Angular and React. They completely miss out on supporting Vue projects, despite it being one of the big 3 frontend frameworks. Of course, it is totally possible to just run your Vue project and ASP.NET Core project separately, but this is less than ideal for a number of reasons. Firstly, it means having multiple servers running, one for your each application, but you'll also run into issues with , , and doing things with cookies such as protecting against becomes trickier. CORS SSL CRSF In my opinion, it is a much more ideal situation to have you SPA being served by the same application that provides your API. Thankfully someone has on how to serve a Vue SPA from an ASP.NET Core web application. However, I was recently working on a project using Nuxt (which is based on Vue), and I had to make some adjustments to the article to get it to work with Nuxt. If you're looking at how to integrate your ASP.NET Core web application with Nuxt, please keep reading. written an in depth article How to run Nuxt from a ASP.Net Core Web Application Start by creating a new ASP.NET Core Web Application and select the API project template. From a terminal window navigate to the main folder of your project (this will be the same folder that has your Program.cs and Startup.cs files). You can install Nuxt using either NPM or Yarn, depending on your preference. Please note that the rest of the tutorial relies on the Nuxt app being called client-app, so please follow that instruction. npx create-nuxt-app client-app yarn create nuxt-app client-app # install using NPM # OR # install using Yarn Follow through the installation instructions, selecting the various addons that you desire. For the purpose of this tutorial, I just selected all of the default options. Once that is installed, we need to modify the projects *.csproj file. In Visual Studio, you can do this by double clicking on the project name. The following markup will allow the Nuxt files to be built and published properly when the main ASP.NET Core project is built. Your *.csproj file should look like this: netcoreapp3.1 true Latest false client-app\ $(DefaultItemExcludes);$(SpaRoot)node_modules\** %(DistFiles.Identity) PreserveNewest < = > Project Sdk "Microsoft.NET.Sdk.Web" < > PropertyGroup < > TargetFramework </ > TargetFramework < > TypeScriptCompileBlocked </ > TypeScriptCompileBlocked < > TypeScriptToolsVersion </ > TypeScriptToolsVersion < > IsPackable </ > IsPackable < > SpaRoot </ > SpaRoot < > DefaultItemExcludes </ > DefaultItemExcludes </ > PropertyGroup < > ItemGroup < = = /> PackageReference Include "Microsoft.AspNetCore.SpaServices.Extensions" Version "3.0.0-preview6.19307.2" </ > ItemGroup < > ItemGroup  < = /> Content Remove "$(SpaRoot)**" < = /> None Remove "$(SpaRoot)**" < = = /> None Include "$(SpaRoot)**" Exclude "$(SpaRoot)node_modules\**" </ > ItemGroup < = = = > Target Name "DebugEnsureNodeEnv" BeforeTargets "Build" Condition " '$(Configuration)' == 'Debug' And !Exists('$(SpaRoot)node_modules') "  < = = > Exec Command "node --version" ContinueOnError "true" < = = /> Output TaskParameter "ExitCode" PropertyName "ErrorCode" </ > Exec < = = /> Error Condition "'$(ErrorCode)' != '0'" Text "Node.js is required to build and run this project. To continue, please install Node.js from https://nodejs.org/, and then restart your command prompt or IDE." < = = /> Message Importance "high" Text "Restoring dependencies using 'npm'. This may take several minutes..." < = = /> Exec WorkingDirectory "$(SpaRoot)" Command "npm install" </ > Target < = = > Target Name "PublishRunWebpack" AfterTargets "ComputeFilesToPublish"  < = = /> Exec WorkingDirectory "$(SpaRoot)" Command "npm install" < = = /> Exec WorkingDirectory "$(SpaRoot)" Command "npm run generate"  < > ItemGroup < = /> DistFiles Include "$(SpaRoot)dist\**" < = = > ResolvedFileToPublish Include "@(DistFiles->'%(FullPath)')" Exclude "@(ResolvedFileToPublish)" < > RelativePath </ > RelativePath < > CopyToPublishDirectory </ > CopyToPublishDirectory </ > ResolvedFileToPublish </ > ItemGroup </ > Target </ > Project After that, we need to create a helper class that will allow the Nuxt development server to be proxied by the ASP.NET Core web application. This helper is only useful in development, since in production the Nuxt project will be being served by the server as static files. In your main project folder, create a folder called Helpers and inside create a file called NuxtHelper.cs: Microsoft.AspNetCore.Builder; Microsoft.AspNetCore.SpaServices; Microsoft.Extensions.DependencyInjection; Microsoft.Extensions.Logging; System; System.Diagnostics; System.IO; System.Linq; System.Net.NetworkInformation; System.Runtime.InteropServices; System.Threading.Tasks; { { Port { ; } = ; Uri DevelopmentServerEndpoint { ; } = Uri( ); TimeSpan Timeout { ; } = TimeSpan.FromSeconds( ); DoneMessage { ; } = ; {
            spa.UseProxyToSpaDevelopmentServer( () =>
            { loggerFactory = spa.ApplicationBuilder.ApplicationServices.GetService<ILoggerFactory>(); logger = loggerFactory.CreateLogger( ); (IsRunning())
                { DevelopmentServerEndpoint;
                } isWindows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows); processInfo = ProcessStartInfo
                {
                    FileName = isWindows ? : ,
                    Arguments = ,
                    WorkingDirectory = ,
                    RedirectStandardError = ,
                    RedirectStandardInput = ,
                    RedirectStandardOutput = ,
                    UseShellExecute = ,
                }; process = Process.Start(processInfo); tcs = TaskCompletionSource< >();
                _ = Task.Run(() =>
                { { line; ((line = process.StandardOutput.ReadLine()) != )
                        {
                            logger.LogInformation(line); (!tcs.Task.IsCompleted && line.Contains(DoneMessage))
                            {
                                tcs.SetResult( );
                            }
                        }
                    } (EndOfStreamException ex)
                    {
                        logger.LogError(ex.ToString());
                        tcs.SetException( InvalidOperationException( , ex));
                    }
                });
                _ = Task.Run(() =>
                { { line; ((line = process.StandardError.ReadLine()) != )
                        {
                            logger.LogError(line);
                        }
                    } (EndOfStreamException ex)
                    {
                        logger.LogError(ex.ToString());
                        tcs.SetException( InvalidOperationException( , ex));
                    }
                }); timeout = Task.Delay(Timeout); ( Task.WhenAny(timeout, tcs.Task) == timeout)
                { TimeoutException();
                } DevelopmentServerEndpoint;
            });

        } => IPGlobalProperties.GetIPGlobalProperties()
                .GetActiveTcpListeners()
                .Select(x => x.Port)
                .Contains(Port);
    }
} using using using using using using using using using using using namespace NuxtIntegration.Helpers public static class NuxtHelper // default port number of 'npm run dev' private static int get 3000 private static get new $"http://localhost: " {Port} private static get 30 // done message of 'npm run dev' command. private static string get "DONE  Compiled successfully in" ( ) public static void UseNuxtDevelopmentServer ISpaBuilder spa this async var var "Nuxt" // if 'npm dev' command was executed yourself, then just return the endpoint. if return // launch Nuxt development server var var new "cmd" "npm" $" run dev" {(isWindows ? : )} "/c npm " "" "client-app" true true true false var var new int try string while null if 1 catch new "'npm run dev' failed." try string while null catch new "'npm run dev' failed." var if await throw new return ( ) private static bool IsRunning Please note that this presumes you are running Nuxt on the default port of 3000. If you are using a different port, you can update the Port property in the NuxtHelper class. Finally, we need to configure the Startup.cs file to use this helper during development and to serve the Nuxt files during production. Go to the ConfigureServices method and add the following line: services.AddSpaStaticFiles(options => options.RootPath = ); "client-app/dist" Then go to the Configure method and add the following lines after the app.UseEndpoints statement: app.UseSpaStaticFiles();
            app.UseSpa(spa =>
            {
                spa.Options.SourcePath = ; (env.IsDevelopment())
                { spa.UseNuxtDevelopmentServer();
                }
            }); "client-app" if // Launch development server for Nuxt Everything is now set up! Run your application and go to the root page in your browser (i.e. https://localhost:{port}). You should be able to see your Nuxt app home page there. Occasionally (especially on the first build), you might get a TimeoutException: The operation has timed out message. This is usually because the Nuxt project is taking longer to build than expected. Just wait a few moments then refresh the page. That should fix it. Conclusion In this article, I have shown you how to run a Nuxt project from a ASP.NET Core Web Application. This is based on the information given for integrating Vue projects. here I have made a . Please feel free to fork it to help you get started with your own Nuxt/ASP.NET Core project. public GitHub repo available of this project I post mostly about full-stack .NET and Vue web development. To make sure that you don't miss out on any posts, please follow this blog and . If you found this post helpful, please like it and share it. You can also find me on . subscribe to my newsletter Twitter

Essential Guide to Running Nuxt from an ASP.NET Core Web Application

Home automation comes with plenty of potential to make our lives easier. But in order to succeed in its task, it often requires you to fill your house with bridges that can connect your smart devices to your Wi-Fi network. Unless you buy a smart device that communicates directly over Wi-Fi (like a or ), odds are that many of your favourite smart devices use either Bluetooth, Zigbee or Z-Wave to communicate. These protocols solve some of the issues of Wi-Fi when it comes to smart devices — like latency, centralised topology and relatively high power requirements — but they do require some physical hardware in between to do the smart protocol <-> Wi-Fi translation and make the devices actually controllable from a Wi-Fi-connected client. TP-Link Belkin smart plug The bad thing is that you’ll probably need a different bridge or hub for each class of devices you want to use. Philips Hue lights come with their own bridge, same for Lutron, same for HomeKit, same for Belkin, same for Switchbot, and the list goes on. What’s ironic is that most of these devices actually speak the same protocol (either Zigbee or Z-Wave) but, in most of the cases, they can only control their own devices. Try to imagine an alternative reality where all the ethernet cards can send and receive TCP/IP packets, but you’ll need an adapter just for HTTP traffic, one for FTP, one for SMTP, and so on: that’s more or less the reality today when it comes to smart bridges and hubs. In order to solve the problem, many companies are throwing on the market even hubs and bridges (from Samsung SmartThings, to the plethora of Amazon Echo, Google and Apple devices etc.), often compatible only with a subset of the devices out there and incompatible with one another, often incompatible by design with devices produced by competitors: that only brings even more fragmentation to the current smart home hell. more Moreover, many hubs can usually talk only to their smartphone app, adding an app hell to the bridge hell. One of the most sought goals nowadays when it comes to home automation is (or it should be) to find consistent ways to communicate with as many devices as possible using the same tool, the same box and the same interface. I’ve recently worked on achieving this goal in platypush. Other platforms have tackled the issue too (from HASS.io to Openhab) but, in my opinion, they all still require a certain degree of user configuration (like specifying what each device is and what it’s supposed to do) that in my opinion shouldn’t be required. In this article, we’ll go through what Zigbee and Z-Wave actually are, what you need to set up a universal bridge for them using just a Raspberry Pi and a USB dongle, and how to permanently move your existing smart bridges to the storage room. Zigbee vs. Z-Wave Zigbee and Z-Wave have arguably risen to become the dominant communication protocols in the smart home and IoT industries. They share some common features, such as: They are better suited for low-powered devices. Wi-Fi is a relatively power-hungry protocol, needs to deal with packet losses, relies on a centralised topology, and isn’t ideal for communication with devices that can be asleep for a long time because of its high overhead on reconnection. Zigbee and Z-Wave have instead been designed to work well with low-powered devices, be low-latency and optimized for sending less bytes on the network. Some Z-Wave or Zigbee devices can theoretically run on the same battery for a couple of years; a similar Wi-Fi device won’t usually last more than a couple of days.While Wi-Fi networks are usually configured using a star-based topology (one access point/router in the middle and all the devices connect to it), Zigbee and Z-Wave have been designed to support more flexible topologies. Each device can be both a client and a repeater on the network, each can directly connect to other devices and expand the network with no need to introduce further routers or access points. This allows the creation of scalable networks that can include devices that would otherwise be out of range. mesh When it comes to the differences: : Z-Wave networks are limited to 232 devices, while Zigbee supports up to 65k devices. Number of devices : Z-Wave supports up to four hops between a device and the network controller, while Zigbee has theoretically no such limitations. Maximum hops : Most Zigwave devices operate around the 2.4 GHz spectrum — although some specific devices also use 784 MHz in China, 868 MHz in Europe and 915 MHz in the US and Australia. Z-Wave devices instead operate around the 850–900 MHz range (at 868.42 MHz in Europe, at 908.42 MHz in North America and other frequencies in other countries depending on their regulations). Operating range : A thumb rule when it comes to electromagnetic waves is that, at fixed transmission power, the higher the frequency the lower the range. It means that Z-Wave devices, which operate at lower frequencies, have usually a longer range compared to Zigbee. An unobstructed Z-Wave signal can travel up to 100 meters outdoor, although a common guideline is 30 meters indoor for unobstructed signals and 15 meters if there are walls in between. The indoor range for Zigbee is instead usually limited around 12 meters. However, Zigbee networks theoretically support an unlimited number of hops between the nodes and the controller, so the range can be easily extended by adding more nodes to the network. Signal range : Z-Wave has a longer range than Zigbee. It means that, at a fixed distance, its signal is usually more reliable. Moreover, Zigbee operates in the relatively “quieter” 800–900 MHz spectrum, therefore it doesn’t have to share the crowded 2.4 GHz spectrum with Wi-Fi and Bluetooth devices. That overall leads to more reliable communication. Signal reliability : Z-Wave is a proprietary technology developed and maintained by Sigma Designs. Sigma (acquired by Silicon Labs in 2018) owns the protocol, licenses the compatible devices and runs the Z-Wave Alliance, and grants certifications to the devices that comply with the standard. Its selling point is, in my opinion, the strong enforcement of a shared protocol both on hardware and software side. Each sensor, configuration value or switch is defined by a variable type, a range, a read-only vs. read-write policy and a structured representation that applies to all the compliant devices. That makes it very easy to develop consistent interfaces that can comprehensively represent and control any device as long as it speaks Z-Wave, even if the implementation doesn’t know exactly which device it is. Zigbee, on the other hand, is an open standard maintained by the Zigbee Alliance. It also has a certification process in place, but that comes in two parts — one part certifies the hardware, the other certifies the software. It is possible to produce Zigbee certified hardware even if the software isn’t certified nor compatible with other clients. While this makes the protocol much more open than Z-Wave, it also makes the task of writing an all-purpose interfacing software much harder, as different devices may name their properties following different conventions. Ownership and protocol That should cover most of the knowledge you need when it comes to the theory of Zigbee and Z-Wave. You’ll find many Zigbee and Z-Wave compatible smart devices around. Some examples of Zigbee devices are the Philips Hue and Ikea smart bulbs, Honeywell thermostats, Belkin smart plugs and bulbs, Bosch sensors and Osram products. Z-Wave includes around 65,000 compatible devices out there, including many garage doors, presence and temperature sensors, thermostats, dimmers, remote controls, smoke detectors and so on. Hardware and software We’ll use a RaspberryPi in the following examples as a DIY bridge (any model and any distribution should work fine), and as a home automation platform that also runs the Zigbee and Z-Wave integrations. platypush Making your own Zigbee bridge On the hardware side you’ll need: One or more Zigbee compatible devices, like Philips Hue, Ikea or Osram lights or Belkin switches. A Zigbee-to-USB adapter/sniffer. The is one of the most popular options out there.A Zigbee , that you’ll need in order to flash the firmware on the dongle. CC2531 debugger+adapter cable A RaspberryPi or similar device (any model and distribution should work fine), or any computer that you want to use as a bridge. On the software side: platypush uses as a backend to interact with the Zigbee dongle. Install the zigbee2mqtt firmware on the dongle by following the instructions (for Windows, macOS and Linux) on . You can check a list of the compatible devices .Install, start and enable an MQTT instance on the local machine, if you don’t have a server already running in your network. If you’re running Debian/Ubuntu/Raspbian and want to use Mosquitto, for example: zigbee2mqtt their website here [sudo] apt-get install mosquitto [sudo] systemctl start mosquitto .service [sudo] systemctl enable mosquitto .service Install zigbee2mqtt: [sudo] apt- install nodejs git g++ gcc
git clone http //github. /Koenkk/zigbee2mqtt zigbee2mqtt
npm install data/configuration
# Change mqtt.server serial.port respectively
# your MQTT server USB dongle device.

npm start get make s: com cd vi and to match and You can also make a systemd service out of it: =zigbee2mqtt =network.target =/usr/bin/npm start =/path/to/zigbee2mqtt =inherit =inherit =always =multi-user.target [Unit] Description After [Service] ExecStart WorkingDirectory StandardOutput StandardError Restart [Install] WantedBy Note that the zigbee2mqtt configuration file also includes a permit_join option. Set this to true while you’re pairing your Zigbee devices for the first time, and set it to false afterwards to prevent other devices from accidentally or malignantly join the network — you can always temporarily allow joins later. Once started and in permit_join mode, you can start pairing Zigbee devices to your new network. That is usually done by doing a factory reset of the device. The procedure varies with the device: Philips Hue bulbs, for example, can be reset either from the app (if they are paired to a bridge) or by pressing the ON/OFF buttons of a Hue dimmer simultaneously for 10 seconds while keeping the dimmer within 10 cm from the lightbulb. Other Zigbee devices may include a reset button instead.Once a Zigbee device joins the network, the zigbee2mqtt logs should show a trace like the following: Successfully interviewed '0x 8d dc126a', device has successfully been paired 0015 0001 Install Redis and platypush with the Zigbee, HTTP and MQTT extensions: [sudo] apt-get redis- [sudo] systemctl redis.service
[sudo] systemctl redis.service
pip install server start enable install 'platypush[zigbee,http,mqtt]' Edit your file to enable the Zigbee and HTTP services: ~/.config/platypush/config.yaml backend.http: port: 8008 zigbee.mqtt: host: localhost backend.zigbee.mqtt: enabled: true Start platypush (either by running platypush or starting the platypush.service systemd service).Open in a browser. The web panel should include the Zigbee icon in the navigation bar, you’ll be able to control your network from there. http://host-or-ip:8008/ You can send requests through the over HTTP, Python code or through whichever platypush backend you have configured: supported API curl -XPOST -H 'Content- : application/json' -d '
{ : , : , : { : , : , : }
}' http: # HTTP request Type "type" "request" "action" "zigbee.mqtt.device_set" "args" "device" "White Bulb" "property" "state" "value" "ON" //localhost:8008/execute platypush.context get_plugin

get_plugin( ).device_set(device= , property= , value= ) # Python usage from import 'zigbee.mqtt' 'White Bulb' 'state' 'ON' Or hook any custom logic to the : supported events event.hook.OnLightBulbOn: :
        type: platypush.message.event.zigbee.mqtt.ZigbeeMqttDevicePropertySetEvent
        device: White Bulb
    then:
        - ${properties.get( ) == }:
            - action: tts.say
              args:
                  text: if if 'state' 'ON' "The light went on" Congratulations, you’re now ready to use your Zigbee devices and build automation without bridges! Making your own Z-Wave bridge Making a DIY Z-Wave bridge is even simpler than making a Zigbee bridge, as you won’t need a debugger to flash a custom firmware nor an MQTT service in between. You’ll need a Z-Wave USB adapter dongle. I use , but any compatible dongle should work. Take note of where the adapter is mapped on your system — e.g. /dev/ttyUSB0. this one Install Redis and platypush with the Z-Wave and HTTP extensions: [sudo] apt-get redis- [sudo] systemctl redis.service
[sudo] systemctl redis.service
pip install server start enable install 'platypush[zwave,http]' Configure your integrations in : ~/.config/platypush/config.yaml backend.http: port: 8008 zwave: device: /dev/ttyUSB0 backend.zwave: enabled: true Start platypush (by running platypush or through systemd service) and point your browser to . You’ll see a new tab for the Z-Wave integration.Some screenshots from the Z-Wave web panel. http://host-or-ip:8008/ Each Z-Wave compatible device has its own way of pairing to a network. All you need to do is to press on the + button to put the network in pairing mode and then pair your devices within one minute through the procedure referred in the user manual. Since Z-Wave has a stricter protocol and all the compliant devices publish their values using the same format, the Z-Wave interface is much more granular and detailed compared to Zigbee. You can, of course, send commands to the new network through the and subscribe custom hooks on : available API Z-Wave events curl -XPOST -H 'Content- : application/json' -d '
{ : , : , : { : , : }
}' http: # HTTP request Type "type" "request" "action" "zwave.get_value" "args" "value_label" "Temperature" "node_name" "Kitchen Sensor" //localhost:8008/execute platypush.context get_plugin
get_plugin( ).get_value(value_label= , node_name= ) # Python usage from import 'zwave' 'Temperature' 'Kitchen Sensor' { : : , : { : [], : { : , : , : , : , : , : , : , : , : , : , : , : }
    },
} # Example output "type" "response" "target" "http" "response" "errors" "output" "command_class" 49 "data" 26.799999237060547 "data_items" "Read only" "genre" "User" "index" 1 "is_read_only" true "is_write_only" false "label" "Temperature" "node_id" 3 "type" "Decimal" "units" "C" "value_id" 72057594093256722 event.hook.OnZWaveValueEvent: :
        type: platypush.message.event.zwave.ZwaveValueChangedEvent
    then:
        - ${node[ ] == value[ ] == value[ ] }:
            - action: tts.say
              args:
                  text: Motion has been detected # Trigger a custom action when motion is detected if if 'name' 'Motion Sensor' and 'label' 'Sensor' and 'data' is True You should now have all the ingredients to build your custom IoT networks and ditch those bridges for good! Previously published at https://medium.com/the-monolith/transform-a-raspberrypi-into-a-universal-zigbee-and-z-wave-bridge-b686cdd1849e

How to Transform a RaspberryPi Into a Universal Zigbee and Z-Wave Bridge

Cómo transformar una RaspberryPi en un puente universal Zigbee y Z-Wave

A little backstory on how this started. My team needed to save some on money on the infrastructure we were requesting and since most of the load for the application we wanted to build would be on the client-side rather than the service side. We decided to see if we could combine a Spring Application with a React app and serve up a single war file. Find out how to combine Create React App with Spring Boot to give you a single war file The basic idea of what Spring Boot and Create React App do. Create React App helps you start a React project very quickly. It gives you all the basic things you need to get up and running asap. Spring boot helps you start and maintain a spring application quickly and easily. Goals: Frontend and backend in a single war file with optimized productions build Keeping the benefits Create React App gives such as hot reloading Setup: Must have java installed. Head over to download a version here Must have maven installed. For Mac, I used Homebrew (brew install maven )but otherwise, head here Must have node installed. For Mac, I used Homebrew (brew install node )but otherwise, head over here * Side note: my IDE of choice is IntelliJ. When working on react code I usually switch over to VS Code. Feel Free to use what makes you feel comfortable Create an empty repo on Github and add a Read Me, gitignore, license, etc. Head over to to create your spring application and download locally. Group and Artifact can also be anything you want it to be. https://start.spring.io : e.the.awesome GroupId : spring-react-combo-app Artifact 3. Unzip the downloaded project into your git directory. Commit, Commit, Commit. Your SpringReactComboAppApplication should look like this. package e.the.awesome.springreactcomboapp; org.springframework.boot.SpringApplication; org.springframework.boot.autoconfigure.SpringBootApplication; org.springframework.boot.web.servlet.support.SpringBootServletInitializer;

@SpringBootApplication
public {

	public main( [] args) {
		SpringApplication.run(SpringReactComboAppApplication.class, args);
	}

} import import import class SpringReactComboAppApplication extends SpringBootServletInitializer static void String 4. Let’s create a basic service now. We’ll call it the DadJokesController. This should be created in the same folder as SpringReactComboAppApplication file. I’m aware this is not a proper Rest API format but let’s ignore that for now. package e.the.awesome.springreactcomboapp; org.springframework.web.bind.annotation.GetMapping; org.springframework.web.bind.annotation.RestController;
@RestController
public {
    @GetMapping( )
    public dadJokes() { ;
    }
} import import class DadJokesController "/api/dadjokes" String return "Justice is a dish best served cold, if it were served warm it would be just water." 5. In your terminal run mvn spring-boot: run Then in your browser check . You should see the dad joke we added to our controller. http://localhost:8080/api/dadjokes 6. To create your React app, just run in the root directory npx create-react-app basic-frontend- app You can call it whatever you want, I just called mine basic-frontend-app 7. To run the front end application: basic-frontend- < >npm start cd app br After starting it should look like: 8. Since we want to integrate our Dad Jokes service into the frontend, first we will address proxy issues. If you’ve noticed, your service starts on localhost:8080 while the frontend starts on localhost:3000. If we try calling our service from the frontend, depending on your browser, you’ll get a CORS error. The simplest way to address the issue is to have your front end proxy any requests from port 3000 to 8080. This change will be made in your package.json file { : , : , : , : { : , : , : }, : { : , : , : , : }, : { : { : , : }
  }
} "name" "basic-frontend-app" "version" "0.1.0" "private" true "dependencies" "react" "^16.3.1" "react-dom" "^16.3.1" "react-scripts" "1.1.4" "scripts" "start" "react-scripts start" "build" "react-scripts build" "test" "react-scripts test --env=jsdom" "eject" "react-scripts eject" "proxy" "/api" "target" "http://localhost:8080" "ws" true Add the following to your frontend App.js file React, {Component} ; logo ; ; {

    state = {};

        componentDidMount() { .dadJokes()
        }

    dadJokes = {
        fetch( )
            .then( response.text())
            .then( { .setState({ : message});
            });
    };

    render() { ( <header className="App-header">
            <img src={logo} className="App-logo" alt="logo"/>
            <h3 className="App-title">{this.state.message}</h3>
            </header>
            <p className="App-intro">
            To get started, edit <code>src/App.js</code> and save to reload.
        </p>
        </div> import from 'react' import from './logo.svg' import './App.css' class App extends Component this => () '/api/dadjokes' => response => message this message return < = > div className "App" );
    }
}

export default App; Restart the front end and you should be good. if you happen to get this error: I deleted my package-lock.json file and node_modules folder reinstalled npm packages and ran it again 9. Your application should now look like. You can see the results of the dad jokes API call. 10. Now that our basic front end and backend is complete, it’s time to create a production build and single war file. Under the add this in <dependencies>  com.github.eirslett < > groupId </ > groupId /artifactId>
<version>1.6</ </ > dependency Under the section of the pom file, we will add the following commands which will do the following things when mvn clean install is run. <plugins> npm install with the specified version of node run a production build of our frontend deposit the production build <plugin> <artifactId>frontend-maven-plugin< version> <workingDirectory>basic-frontend-app</workingDirectory> <installDirectory>target</installDirectory> <executions> <id>install node and npm</id> <goals>
            <goal>install-node-and-npm</goal>
         </goals> <configuration>
            <nodeVersion>v8.9.4</nodeVersion>
            <npmVersion>5.6.0</npmVersion>
         </configuration> <execution> <goals> < > < d> <goal>npm</goal> <configuration> < execution> < artifactId> <execution>
         <phase>generate-resources</phase>
         <configuration>
            <target>
               <copy todir="${project.build.directory}/classes/public">
                  <fileset dir="${project.basedir}/basic-frontend-app/build"/>
               </copy>
            </target>
         </configuration>
         <goals>
            <goal>run</goal>
         </goals>
      </execution>
   </executions> com.github.eirslett < > groupId </ > groupId /artifactId>
   <version>1.6</ < > configuration </ > configuration < > execution </ > execution npm install < > id </ > id npm < > goal </ > goal /goals>
         <configuration>
            <arguments>install</ arguments </ > configuration /execution>
      <execution>
         <id>npm run build</i < > goals </ > goals run build < > arguments </ > arguments /configuration>
      </ </ > executions /plugin>
<plugin>
   <artifactId>maven-antrun-plugin</ < > executions </ > plugin Side note: order is important for your plugins so make sure your node/npm install execution is before copying the build file execution 11. After adding this, run mvn clean install  and verify that the target/classes directory contains both frontend files and java files. And you should be good to go. A final look at my . pom file So that’s all I got. If you would like to take a look at the repo or use it. It can be found here on my Github . Next up is an article on how to deploy your war file on Heroku. Look forward to it!

Package Your React App with Spring Boot [A How-To Guide]

The workflow for building machine learning models often ends at the evaluation stage: you have achieved an acceptable accuracy, and “ ” ta-da! Mission Accomplished. Beyond that, it might just be sufficient to get those nice-looking graphs for your paper or for your internal documentation. In fact, going the extra mile to put your model into production is not always needed. And even when it is, this task is delegated to a system administrator. However, nowadays, many researchers/engineers find themselves responsible for handling the complete flow from conceiving the models to serving them to the outside world. Whether it is a university project or a personal experiment, demonstrating our work is generally a great way to get a wider audience interested. Few people would make the extra effort to use a system for which the value is not instantly perceived. In this article, we’ll go together through this workflow; a process that I had to repeatedly do myself. The assumption is that you have already built a machine learning or deep learning model, using your favorite framework (scikit-learn, Keras, Tensorflow, PyTorch, etc.). Now you want to serve it to the world at scale via an API. sitting idly at your institution or in the cloud. This entails serving multiple requests at a time, spawning new processes as the load increases, and reducing the number of processes as load decreases. You also want the additional guarantee that your server would be restarted after unexpected system failures. By “at scale” , we’re not talking about an industrial scale of a huge company. The goal is to make the best out of that server with lots of CPUs and a large RAM, If that is what you have in mind, let’s go through it together. We’ll be considering the context of Python-based frameworks on Linux servers. Our setup will involve: : for managing package installation and creating an isolated Python 3 environment. Anaconda a high-level neural networks API, that is capable of running on top of , , or . Keras : TensorFlow CNTK Theano : a minimalistic python framework for building RESTful APIs. Despite being easy to use, Flask’s built-in server serves only one request at a time by default; hence on its own for deployment in production. Flask it is not suitable : the highly stable , which provides benefits such as load-balancing, SSL configuration, etc. nginx web server a highly configurable (Web Server Gateway Interface) that allows forking multiple workers to serve multiple requests at a time. uWSGI : WSGI server an system used in multiple Linux distributions to manage system processes after booting. systemd : init Nginx will be our interface to the internet, and it will be the one handling clients’ requests. Nginx has native support for the binary uWSGI protocol, and they communicate via Unix sockets. In turn, the uWSGI server will be invoking a callable object within our Flask application directly. That is the way that requests will be served. A few notes at the beginning of this tutorial: . For example, Keras can be easily replaced with PyTorch, Flask can be easily replaced with , and so on. - Most components above can be easily replaced by equivalent components with little to no change in the rest of the steps Bottle The typical case is having access to a server with a lot of CPU cores and trying to make use of these cores to serve the models. GPUs, on the other hand, are more expensive to get in large numbers. Moreover, depending on your application, the speed gain you get from using a GPU at prediction time might not be that significant (especially in NLP applications). - We will only consider the case of serving models over CPU. Setting up the Environment To begin with, we need to install the systemd and nginx packages: sudo apt-get install systemd nginx Next, we have to install Anaconda by following the instructions , which consist of downloading the executable, running it, and adding Anaconda to your system’s PATH. Below, we will assume that Anaconda is installed under the home directory. on the official site All the code and configuration files in this article are available from the accompanying . But make sure you follow the steps below to get the full workflow. Github repository github.com harkous/production_ml production_ml — Scaling Machine Learning Models in Production Next, let’s create the isolated Anaconda environment from the environment.yml file. Here is how this file looks like (it already contains several of the frameworks we’ll be using): name: production_ml_env channels: - conda-forge dependencies: - python=3.6 - keras - flask - uwsgi - numpy - pip - pip: - uwsgitop We run the following to create the environment: < = >conda env create --file environment.yml</ > code class "markup--code markup--pre-code" code When we want to activate this environment, we run: source production_ml_env activate By now, we have Keras installed, alongside flask, uwsgi, uwsgitop, etc. So we are ready to get started. Building the Flask Web App For the purposes of this tutorial, we will not dive deeply into how to build your ML model. Instead, we will adapt the example of topic classification using the bundled within Keras. This is the code for building the classifier: Reuters newswire dataset __future__ print_function os numpy np keras keras.datasets reuters keras.models Sequential keras.layers Dense, Dropout, Activation keras.preprocessing.text Tokenizer keras.callbacks ModelCheckpoint

MODEL_DIR = max_words = batch_size = epochs = print( )
(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words,
                                                         test_split= )
print(len(x_train), )
print(len(x_test), )

num_classes = np.max(y_train) + print(num_classes, )

print( )
tokenizer = Tokenizer(num_words=max_words)
x_train = tokenizer.sequences_to_matrix(x_train, mode= )
x_test = tokenizer.sequences_to_matrix(x_test, mode= )
print( , x_train.shape)
print( , x_test.shape)

print( )
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
print( , y_train.shape)
print( , y_test.shape)

print( )
model = Sequential()
model.add(Dense( , input_shape=(max_words,)))
model.add(Activation( ))
model.add(Dropout( ))
model.add(Dense(num_classes))
model.add(Activation( ))

model.compile(loss= ,
              optimizer= ,
              metrics=[ ]) os.path.exists( ):
    os.makedirs(MODEL_DIR)

mcp = ModelCheckpoint(os.path.join(MODEL_DIR, ), monitor= ,
                      save_best_only= )

history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose= ,
                    validation_split= ,
                    callbacks=[mcp])

score = model.evaluate(x_test, y_test,
                       batch_size=batch_size, verbose= )
print( , score[ ])
print( , score[ ]) '''Trains and evaluate a simple MLP
on the Reuters newswire topic classification task.
''' from import import import as import from import from import from import from import from import './models' 1000 32 5 'Loading data...' 0.2 'train sequences' 'test sequences' 1 'classes' 'Vectorizing sequence data...' 'binary' 'binary' 'x_train shape:' 'x_test shape:' 'Convert class vector to binary class matrix ' '(for use with categorical_crossentropy)' 'y_train shape:' 'y_test shape:' 'Building model...' 512 'relu' 0.5 'softmax' 'categorical_crossentropy' 'adam' 'accuracy' if not '' 'reuters_model.hdf5' "val_acc" True 1 0.1 1 'Test score:' 0 'Test accuracy:' 1 To replicate the setup we use here, simply run the following to train a model without a GPU: =-1<br>KERAS_BACKEND=theano python build_classifier.py export CUDA_VISIBLE_DEVICES This will create a model file reuters_model.hdf5 in the folder models. Now, we are ready to serve the model via Flask on port 4444. In the code below, we provide a single REST endpoint /predict that supports GET requests, where the text to classify is provided as a parameter. The returned JSON is of the form{"prediction": "N"}, where N is an integer representing the predicted class. flask Flask flask request keras.models load_model keras.datasets reuters keras.preprocessing.text Tokenizer, text_to_word_sequence flask jsonify os

MODEL_DIR = max_words = app = Flask(__name__)

print( )
model = load_model(os.path.join(MODEL_DIR, )) word_index = reuters.get_word_index()
tokenizer = Tokenizer(num_words=max_words) word_sequence = text_to_word_sequence(text)
    indices_sequence = [[word_index[word] word word_index word word_sequence]]
    x = tokenizer.sequences_to_matrix(indices_sequence, mode= ) x :
        text = request.args.get( )
        x = preprocess_text(text)
        y = model.predict(x)
        predicted_class = y[ ].argmax(axis= )
        print(predicted_class) jsonify({ : str(predicted_class)}) :
        response = jsonify({ : })
        response.status_code = response __name__ == :
    app.run(host= , port= ) from import from import from import from import from import from import import './models' 1000 "Loading model" 'reuters_model.hdf5' # we need the word index to map words to indices : def preprocess_text (text) if in else 0 for in 'binary' return @app.route('/predict', methods=['GET']) : def predict () try 'text' 0 -1 return 'prediction' except 'error' 'problem predicting' 400 return if "__main__" '0.0.0.0' 4444 To start the Flask application server, we run: app. python py Voila! Now we have the simple, lightweight server running. You can test the server with your favorite REST client (e.g., ) or by simply going to this URL in your web browser (replace by your server’s URL): :4444/predict?text=this is a news sample text about sports and football in specific Postman your_server_url http:// your_server_url You should get back a response as { : } <br> "class" "11" <br> Configuring the uWSGI Server Now, we are off to scaling our simple application server. uWSGI will be the key here. It communicates with our Flask application by invoking the callable object app in the file app.py. uWSGI includes most of the parallelization features we are after. Its configuration file looks as follows: [uwsgi] # placeholders that you have to change my_app_folder = /home/harkous/Development/production_ml my_user = harkous socket = %(my_app_folder)/production_ml.sock chdir = %(my_app_folder) file = app.py callable = app # environment variables env = CUDA_VISIBLE_DEVICES=-1 env = KERAS_BACKEND=theano env = PYTHONPATH=%(my_app_folder):$PYTHONPATH master = true processes = 5 # allows nginx (and all users) to read and write on this socket chmod-socket = 666 # remove the socket when the process stops vacuum = true # loads your application one time per worker # will very probably consume more memory, # but will run in a more consistent and clean environment. lazy-apps = true uid = %(my_user) gid = %(my_user) # uWSGI will kill the process instead of reloading it die-on-term = true # socket file for getting stats about the workers stats = %(my_app_folder)/stats.production_ml.sock # Scaling the server with the Cheaper subsystem # set cheaper algorithm to use, if not set default will be used cheaper-algo = spare # minimum number of workers to keep at all times cheaper = 5 # number of workers to spawn at startup cheaper-initial = 5 # maximum number of workers that can be spawned workers = 50 # how many workers should be spawned at a time cheaper-step = 3 Depending on your needs and file locations, you might need to modify/add too. On your side, you have to modify the option my_app_folder to be the folder of your own app directory and the option my_user to be your own username. other options One important section in the uwsgi.ini is the part where we use the in uWSGI, which allows us to run multiple workers in parallel to serve multiple concurrent requests. This is one of the cool features of uWSGI, where dynamically scaling up and down is attainable with a few parameters. With the above configuration, we will have at least 5 workers at all times. Upon load increase, Cheaper will allocate 3 additional workers at a time until all the requests find available workers. The maximum number of workers above is set to 50. subsystem Cheaper In your case, the best configuration options depend on the number of cores in the server, the total memory available, and the memory consumption of your application. Take a look at the for advanced deployment options. official docs Connecting uWSGI with nginx We’re almost there. If we start uWSGI now (which we’ll do in a while), it will take care of invoking the app from the file app.py, and we will benefit from all the scaling features it provides. However, in order to get REST requests from the internet and to pass them to the Flask app via uWSGI, we will be configuring nginx. Here is a barebone configuration file for nginx, with just the part we depend on for this application. Of course, nginx can be additionally used for configuring SSL or to serve static files, but that is out of the scope of this article. { ; YOUR_SERVER_NAME_OR_IP; / { uwsgi_params; unix:/home/harkous/Development/production_ml/production_ml.sock;
    }
} server listen 4444 # change this to your server name or IP server_name location include # change this to the location of the uWSGI socket file (set in uwsgi.ini) uwsgi_pass We place this file in /etc/nginx/sites-available/nginx_production_ml (you will need sudo access for that). Then, to enable this nginx configuration, we link it to the sites-enabled directory: <code = >sudo ln -s nginx nginx_production_ml nginx code> class "markup--code markup--pre-code" /etc/ /sites-available/ /etc/ /sites-enabled</ We restart nginx: sudo nginx restart service Configuring the systemd Service Finally, we will launch the uWSGI server we configured earlier. However, in order to ensure that our server does not die forever after system restarts or unexpected failures, we will launch it as a systemd service. Here is our service configuration file, which we place in the /etc/systemd/system directory using: sudo vi /etc/systemd/system/production_ml.service [Unit] Description=uWSGI instance to serve production_ml service [Service] User=harkous Group=harkous WorkingDirectory=/home/harkous/Development/production_ml/ ExecStart=/home/harkous/anaconda3/envs/production_ml_env/bin/uwsgi --ini /home/harkous/Development/production_ml/uwsgi.ini Restart=on-failure [Install] WantedBy=multi-user.target We start the service with: sudo systemctl start < = >production_ml</ >.service code class "markup--code markup--pre-code" code To allow this service to start when the machine is rebooted: sudo systemctl <code = >production_ml</code>.service enable class "markup--code markup--pre-code" At this stage, our service should start successfully. If we update the service later, we simply have to restart it: sudo systemctl restart < = >production_ml</ >.service code class "markup--code markup--pre-code" code Monitoring the Service To keep an eye on the service and see the load per worker, we can use . In uwsgi.ini, we have already configured a stats socket within our application folder. To see the stats, execute the following command in that folder: uwsgitop < class= >uwsgitop stats </code> code "markup--code markup--pre-code" .production_ml .sock Here is an example of the workers in action, with additional workers that have already been spawned. To simulate such a heavy load on your side, even with simple tasks, you can artificially add a time.sleep(3) in the prediction code. One way to send concurrent requests to your server is using curl (remember to replace YOUR_SERVER_NAME_OR_IP by your server’s URL or IP address. url= i {0..10} curl -s & #!/usr/bin/env bash "http://YOUR_SERVER_NAME_OR_IP:4444/predict?text=this%20is%20a%20news%20sample%20text%20about%20sports,%20and%20football%20in%20specific" # add more URLs here for in do # run the curl job in the background so we can start another job # and disable the progress bar (-s) echo "fetching " $url $url done wait #wait for all background jobs to terminate In order to monitor the log of the application itself, we can make use of journalctl: < class= >sudo journalctl -u production_ml -f</code> code "markup--code markup--pre-code" .service Your output should look like this: Final Notes If you have reached this stage, and your application has run successfully, then this article has served its purpose. Some additional notes are worth mentioning though at this stage: To keep this article general enough, we have used the lazy-apps mode in uwsgi, which loads the application one time per worker. , this will require O(n) time to load (where n is the number of workers). It also likely requires more memory but results in a clean environment per worker. By default, uWSGI loads the whole application differently. It starts with one process; then it forks itself multiple times for additional workers. This results in more memory savings. However, this does not play well with all of the ML frameworks. For example, the TensorFlow backend in Keras fails without the lazy-apps mode (e.g., check , , and ). The best could be to try first without lazy-apps = true, and shifting to it if you encounter similar issues. lazy-apps mode in uwsgi: According to the docs this this this Because uWSGI invokes app as a callable, the parameters of the app itself should not be passed via the command line. You’re better off using a configuration file with the likes of to read such parameters. - Parameters of the Flask App: configparser - The guide above does not discuss the case of multiple servers. Luckily enough, this can be achieved without a significant change in our setup. Benefiting from the , you can setup multiple machines, each with the uWSGI setup we described above. Then, you can configure nginx to route the requests to the different servers. nginx comes with multiple methods to distribute the load, ranging from a simple round-robin to accounting for the number of connections or the average latency. Scaling across multiple servers: load balancing feature in nginx The above guide uses port 4444 for illustration purposes. Feel free to adapt to your own ports. And make sure that you open these ports in the firewall or ask your institution’s administrators to do so. - Port selection: We have been permissive in the socket permissions by giving write access to all users. Feel free to also adjust these permissions to your purposes and to run the service with a different user/group. Make sure that your nginx and uWSGI can still talk to each other successfully after your changes. - Socket permissions: So that’s it! I hope this guide and will be helpful for all those trying to deploy their models into production as part of a web application or as an API. If you have any feedback, feel free to drop a comment below. the associated repository And thanks for reading! You might also be interested in checking my other articles on my Medium page: medium.com Hamza Harkous — Medium Read writing from Hamza Harkous on Medium. Postdoc at EPFL, Switzerland; working at the intersection of Privacy, NLP… … or in seeing what I tweet about: twitter.com Hamza Harkous (@hamzaharkous) | Twitter The latest Tweets from Hamza Harkous (@hamzaharkous). Postdoc @EPFL; working at the intersection of Privacy, NLP, and…

A Guide to Scaling Machine Learning Models in Production

Coding is not an easy process, especially if you have to be super focused to do it relatively fast and without bugs. Keeping the focus on coding only is quite a difficult challenge, but this story is more about how to make the ultimate setup to code from anywhere with the internet and browser! We at traveling a lot, and sometimes switching computers to check our tools for a few days intensively, that’s why I needed an environment that will support the same coding flow on any kind of computer with internet and browser. Hexometer.com Probably this is not for nerd low-cost travelers, because they usually getting cheap places with a bad internet connection, and as you can imagine, the only way of making a laptop independent coding setup is to make it over the internet using a basic browser because guess what Browsers are everywhere! What are the advantages? I’ve been always chasing to get the most powerful ultrabook around to make my coding process faster and code compile time seamlessly. As you can guess that could be expensive, and the power you get is affecting the laptop’s battery time. Usually, MacBooks are fine to code about 4–5 hours without changing them, which is not bad, however, you are blocked to that specific laptop and sometimes moving to a new laptop getting harder and harder over time. I will be honest, for the past 4–5 years I’ve been using only MacBooks primarily for my coding projects, and I love the way how they built and how they are running OS specifically optimized for them, BUT when I started to work with remote servers and DevOps stuff, I realized that moving entire coding flow to server will give more advantages than disadvantages this is a HUGE benefit, specifically if you are installing remote packages or pulling/pushing Docker containers quite often. - 1GB/s internet network! You can get powerful Xeon or i7 CPU cores for your coding server cheaper than you can buy them as a laptop bundle, but the most significant difference is that laptop’s OS’s (all) using 3–5% CPU just to keep Desktop UI operating, which drains batteries faster if you combine powerful IDE with it. Powerful CPU Cores without any UI overheads. As a web developer, I used to run a tool called ngrock to make some remote tunneling, for showing up some experimental projects to clients. Now it is extremely useful to have even dedicated domain attached to my dev environment, code and share stuff online directly from one place! Having your localhost accessibly remotely! I’ve been using laptops with 256GB SSD almost all the time, but having the ability to scale it dynamically is just giving the ability to run some good data collection tests directly inside your coding environment. Very flexible disk space. All of these points coming as a bonus to the main benefit of Coding inside the browser literally on any device with a browser. VS Code inside your browser There is several tools/services which are giving the ability to run VS Code like editor inside your browser. However, there is one open-source project, which is doing the exact VS Code copy inside the browser, by replacing Electron’s functionality with WebSocket based server communication. It works fine even with the 3G network over a mobile hotspot, however, sometimes when the internet connection is bad, you will be very disappointed, but that’s the tradeoff for being laptop independent. Anyway, you are getting more benefits than you can even imagine at the beginning. is based on VSCode BUT they changed all connection parts with the Electron desktop app to work over Browser + Websocket connection, to run services on server-side and deliver results to the browser. Code-Server works on all platforms including MAC, Windows, and Linux, but for most cases, I’m sure you will use Linux as a server OS to run code-server. Code-Server # Running code-server CLI
~# code-server --auth none --port from 8888 This will start code-server, and now you can access to VSCode functionality from browser by navigating to http://localhost:8888 If you are going to run this for a long term and would like to manage all your projects over it, you are going to make it run as a background service, let’s say using with SystemD (that’s what I’m using). Basic SystemD configuration for my case looks like this [Unit]
Description=Code Server
[Service]
PIDFile= coder pid
User=coder
Group=coder
Restart=always
KillSignal=SIGQUIT
WorkingDirectory= coder
ExecStart= local/bin/code-server --auth none --port [Install]
WantedBy=multi-user.target /tmp/ -99. /home/ /usr/ 8080 As you can see I’m not using public port directly from code-server, this is because it actually can expose all my projects without any access control (their password thing is not that secure). To make it more secure I’m using Nginx in front of it, and making specific configurations just to proxy stuff and set an HTTP Authentication + some SSL configurations to secure it only accessible over VPN. Is it really working on ANY device? I’ll say Yes! and No! Some devices have Unicode input problems, where they are using some character converting service, like most of the Android devices, BUT for Apple devices, it works great! Even on iPad Pro you will feel the complete joy of coding and accessing server CLI over VSCode terminal. I worked for a while on Acer Chromebook 14, which is a very low power device, BUT it worked for me pretty fine because most of the autocomplete and service processing work on the Server-side not blowing up your local laptop’s CPU. This is defiantly not for everyone, if you are mainly coding on applications not requiring desktop view or specific OS, then you can do it. I even did React Native coding with it using Expo, where they are covering all OS independent flows. I hope you will enjoy coding inside the browser and be laptop independent as I’m now!

Stories, News & Mentions about WORKINSPIRE

Workinspire’s Stories on HackerNoon

Workinspire's News & Mentions