Get the most recent info and news about shotstack on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Edit, automate and personalise videos, at scale, in the cloud, using the Shotstack video editing API

shotstack

With Shotstack, we remove all that hassle. You can take any video you have and add localised voiceovers to it in a matter of minutes. That’s right, removing weeks or even months from your video ...

Why text-to-speech is every marketer's secret weapon

What Are the Top Startups in Oceania?

Thank you to the problem solvers who campaigned extra hard in the first-ever competition! HackerNoon Startups of the Year focuses on recognizing the passion and stamina of those willing to find and create effective tech solutions for their cities. HackerNoon’s Startups of the Year To see who is a winner in each respective city, visit and pick a city. The name at the top of the list is the winner. (Awards, where cheaters were caught, are under review.) startups.hackernoon.com From all of us at , thank YOU for your excellent campaign perseverance🥺! We look forward to seeing you again in the 2022 campaign (stay tuned 😉)! HackerNoon Without further adieu, let us kick off by sharing the cities with top votes per region for 2021 🎉 . our series of Best Startups of the Year Stay tuned for the rest of the winners per region to be over the course of this week (Feb 14, 2022). announced North and Central America America is the concrete jungle where dreams are made of, the world’s capital of fashion, showbiz, business, major industry, etc., and surely, of Startups! Representing , we have: North and Central America (14,563 votes): Winner: Boulder Bury.com (10,521 votes) - Winner: New York City Hum Capital (7,987 votes) - Winner: San Francisco Zentist (7,713 votes) - Winner: Redwood City TigerGraph (469 votes) - Winner: Los Altos Stobox To see more awards in this region please click here. Europe Charming, sophisticated, cosmopolitan. There are many beautiful words used to describe this region. With its eccentric vibe, is also a place where new Startups thrive. Europe Representing Europe we have: (10, 010 votes) - Winner: Belfast Bob & Berts (5, 066 votes) - Winner: Ghent WeGroup (4, 712 votes) - Winner: Porto TAIKAI (3, 779 votes) - Winner: City of London Crypto Quantique To see more awards in this region please click here. Oceania The heat, the beaches, roaming kangaroos, or tarantulas? No. Startups! Representing the region of are: Oceania (1, 011 votes) - Winner: Sydney shotstack (430 votes) - Winner: Melbourne Doohly (118 votes) - Winner: Gold Coast Easil (76 votes) - Winner: Tauranga LawVu (40 votes) - Winner: Western Junction Today We Learned To see more awards in this region please click here. Asia Diverse in both ethnicity and culture, known for exotic sceneries as well as friendly people. Many Asian countries are on the verge of mass globalization and modernization. These fast-developing economies give way for many potential Startups. We present, for : Asia (12, 686 votes) - Winner: Singapore Cake DeFi (3,355 votes) - Winners: and (It’s a tie, we’re working on it!) Tel Aviv Applitools hyro (2, 402 votes) - Winner: Beirut Hovi Digital Lab (1.006 votes) - Winner: Hyderabad Quixy (756 votes) - Winner: Hanoi DeFi For You To see more city awards in this region please click here. Africa The second biggest continent in the world, home to many big animals, the longest river in the world AND the largest desert in the world. is known for many big things and sure enough, big Startups. Africa Representing Africa, we have: (884 votes) - Winner: Johannesburg MiBRAND (804 votes) - Winner: Lagos Goodtalent (526 votes) - Winner: Cape Town Revix (420 votes) - Winner: Lusaka ComGrow (224 votes) - Winner: Nairobi UTU To see more city awards in this region please click here. South America “South America is the land of the extreme.” - National Geographic Extremely beautiful people? Yes. Extremely worth-noticeable Startups? Hell yeah! We proudly present, Startups in : South America (217 votes ) - Winner: São Carlos Predify (193 votes) - Winner: Belo Horizonte CM Technologia (106 votes) - Winner: Porto Alegre Privacy Tools (82 votes) - Winner: Quito Agrodemanda (55 votes) -Winner: Rio De Janeiro Bybrand To see more city awards in this region please click here. Once again, and to all of the top Startups of each region this year! HUGE CONGRATULATIONS BIG PROPS The whole team at would like to thank you for your excellent campaign perseverance and we look forward to seeing you in the 2022 campaign. HackerNoon As part of honoring your participation, we will continue to host your company in this directory until further notice ( for legal reasons we hesitate to say ‘forever’..but pretty much forever).  As traffic is directed to the site, it will act as a promotional platform for your company. Have a great Valentine’s Day, HackerNoon loves y’all <3 More Information: If you’re a winner, please answer ! If you’re a reader, participant, just a curious individual, consider answering one of our interviews . this interview here Read the FAQs about or . Startups here here To see all of the winners you may visit , pick any city and view the name that appears at the top. this site If you’re interested in partnering with us, we invite you to look through some of these or reach out to our business department. unique sponsorship options Perhaps creating your very own tickles your fancy or maybe you have a whole new idea for how we can partner! writing contest Thank you to all of our sponsors! We truly appreciate your contribution to the tech world and thus the future! The Startups Awards are sponsored by: , , , , , , , , , , Blockster Brex FreeCodeCamp LaunchDarkly Coil Udacity Udemy Vercel Asia Pacific Transgender Network Bybit Everscale Network This post was written by Tuan Anh Vu. If you have any questions or concerns or would like to report a discrepancy please email ellen@hackernoon.com.

To the Winners of #Startups2021, With Love:

Startups of the Year, 2021 FIND OUT HERE Hello there, Hacker, Vision, strength, determination, resilience… these are all key building blocks to moving the world forward. People with these skills create solutions, not only muse about problems. The Startups campaign celebrates the baby companies created by the thinkers and doers that – regardless of what life tossed at them this year— have taken the situation by the horns and came out of it with a business plan and the balls (etc.) to launch it. I decided to highlight a few of the startups doing well so far. Below is a mini list of some companies in the running with concepts that I personally find interesting! Head on over to to cast your vote, nominate a startup (maybe even yours), or simply browse the cool things that people are doing in your city. https://startups.hackernoon.com/ Startups voting and nominations are still open, so go vote and nominate now! You can vote on as many cities as you like, but you only get one vote per city. Key Details: 4,553 cities. 37,689 startups. 152,672 votes. North and Central America Representing North and Central America in , currently trending in we have: New York City, New York second place Inpher Inpher has pioneered cryptographic Secret Computing® technology for secure, privacy-preserving analytics and machine learning. The total amount of votes cast for the city is 4343. Thank you to for sponsoring this category. Brex Oceania Representing Oceania as a currently trending Startup in we have: Sydney, Australia Shotstack Amazing video and image editing API infrastructure as a service startup that allows you to generate video and images at scale. The total amount of votes cast for the city is 960. . Thank you to for sponsoring this category Bybit Read more about Sydney below: Sydney remains the capital of fintech in Australia with over 50% of fintech domiciled in New South Wales. The city is ranked 30th in the Global Ecosystem Talent with $768 million total yearly stage funding, followed by $11 bn startup ecosystem value. Europe Representing Europe as the current top trending Startup in we have: Budapest, Hungary Bitrise Bitrise is a platform for mobile app development, enabling tens of thousands of developers to automate integration, testing, and deployment. There have been 30,046 votes cast for this city! Thank you to for sponsoring this category. Free Ton Asia Representing Asia as the current top trending we have: Startup in Tel Aviv, Israel Applitools Applitools delivers a Next-Generation Test Automation Platform through Visual AI and Ultrafast Test Cloud. Applitools' Visual AI uses computer vision algorithms, enabling engineering teams to rapidly write, run, analyze, and maintain tests to ultimately release high-quality applications at incredible speed and reduced cost. There have been 1701 votes cast for this city. Thank you to for sponsoring this category. Udemy Read more about Tel Aviv below: - Seventy three percent of Israeli startups are based in Tel Aviv, and 79% of all industry jobs in Israel. A startup employee in Tel Aviv makes on average NIS 22,100 (approximately $6,390) a month, twice as much as the country’s general average wage. Africa Representing Africa as a current trending Startup in in fourth place, we have: Johannesburg, South Africa Wilderness Safaris Wilderness Safaris is a company that protects wilderness areas with the flaura and fauna in their areas. There have been 882 votes cast for this city! Thank you to for sponsoring this category. Bybit South America Representing South America as the top trending startup in we have: Porto Alegre, Brazil Privacy Tools Privacy Tools is a privacy compliance platform, based on AI/Blockchain, which helps global companies to keep compliant with the data protection requirements. There have been 88 votes cast for this city. Thank you to for sponsoring this category. Bybit For More Information See our CEO and team have a beautiful discussion about this campaign here. Read the FAQs about Startups here . Interested in more details? Read about them . here VOTE/NOMINATE HERE. This post was written/compiled by Ellen Stevens.

Find out Which Startup is Winning in Your City

In this article I want to show you how to build 1,000 personalised birthday videos for 1,000 different people. Currently this is close to impossible to do using traditional video editors such as Adobe and , with specialised personalisation agencies for this type of service. Premiere After Effects charging tens of thousands of dollars In this article I’ll aim to show you a way. One that doesn’t cost you hours of time, and would cost you only a fraction of what agencies would charge you. better To make this all happen I’ll be using the . This API allows you to render tens of thousands of videos in the cloud, and personalise each individual video through small changes to a JSON file. Shotstack API ( : ) Disclosure The author is a Co-founder at Shotstack.io. With this API rendering up to 200 videos concurrently in the cloud, . the ability to work at enormous scale pushes the total time to render all 1,000 of our videos to just under 100 seconds Getting started The speed behind this example was made possible through a Shotstack Enterprise account, but you can that provides you with all the same functionalities and allows you to test the service out. sign up for a free account The example in this article has some throttling built in, but you’ll have to be aware of the usage limits of 10 requests per second, and 2,000 requests per month for free accounts if you plan to try this out yourself. I’ll use to build the application, but feel free to use what works for you. The minimum you’ll need is a way to loop over a CSV file and send POST requests to the API. Nothing fancy. Node.js Choosing our footage I built a promotional video which we’ll use to offer our fictitious customers a discount on their birthday. We’ll remove some of the video's content, and use those empty spaces for our personalised data. This data will be placed into a JSON file, and will place back all of those missing items with new values and animations. For this particular template we’ll personalise the name, age, their younger “fake” age, the discount code, and the discount amount. Shotstack uses a JSON object that acts like an editable timeline. This works in a similar fashion to how any desktop editing solution would work, with a timeline representing the video duration, and individual clips allowing you to manipulate its content. For this article I won’t go into how this video was built, but if you’re interested in understanding how this particular video was edited you can find the JSON template . And if you’re really keen goes a little deeper into how you can design these videos yourself. here this tutorial For this example we’ll use a simplified template where the animations have been pre-rendered, making it easy for us to add in specific HTML assets where our personalised content should go: Our customers For our customers we’ll use a dataset with information on about 1,000 concocted users. This will include their name, age, fake age, their discount code, and the discount amount. We will then use the data in this CSV to fill in the JSON template and send it to the API. You can find the complete list . here The application The only thing you need to do now is iterate over the CSV file, add those personalised datapoints to the JSON template, and send each video edit to the API for rendering. The below script works using an .env environment file with the Shotstack API key details in it. You can find an example of what that looks like . here The result Once you run the script you’ll see all thousand personalised templates be sent through to the API for rendering, and all output data will end up in a CSV file called video-list.csv, which will include the urls for each individual video. The below list contains the first 20 videos, but if you don’t believe me you can find the whole list . here So what next? Personalised marketing has to lead to higher email open rates, higher click-through rates, better engagement, and provides you with a new way to build relationships with your audience. shown The above only shows you a small example of what can be done by personalising videos at scale. And this is all done at lightning speeds, with this particular  campaign not costing more than $30 to render. We’re always talking to artists, marketers and other creative people on how video and code can be combined to create new and exciting ways to communicate with the world. If you want to follow along with that conversation , and if you have an idea or suggestion let us know in the comments below. sign up to our newsletter

How I Built 1,000 Personalised Videos in Under 100 seconds

Video editing has never been something I truly enjoyed. I often found video editors cumbersome, and wasn’t willing to spend the time learning to use professional video editing applications. Let alone pay hundreds of dollars per year on an . Adobe subscription I did always enjoy programming, primarily because of the flexibility it afforded me to create whatever I wanted — bound only by my imagination and my skills as a developer. No editing software has been used to design or render this video, with only a 40 line piece of JSON being needed to put together all individual elements to create a fully composited video using the . The video shown above has been created with nothing other than code. Shotstack API enables video editing at scale. The API allows you to describe your video edit in JSON, and then use your favourite programming language to render hundreds to thousands of data-driven videos concurrently in the cloud. Shotstack I wrote this guide to show you how you can use code instead of software to build cool and artistic videos, and I hope this small guide provides you with some inspiration to start creating your own videos. We’ll start off by creating something simple (such as compositing a title on a video clip), and then overlaying a soundtrack to give it some emotion. At the end, I’ll show you some more advanced features to finish off the video with some professional-looking effects. Let’s get started. Sign up for an API key In order to build your first video you’ll need to . This is completely free and will allow you to access all of Shotstack’s video editing functionalities to build videos and develop applications. After registering just log in via the Shotstack website to receive your API key. sign up for a Shotstack developer account Curl vs Postman This tutorial presumes a basic knowledge of interacting with API’s. For this guide we’ll be using but you can use any other application such as as well. Curl Postman The basics Before diving in and writing any code it is a good idea to acquaint yourself with some basic video editing principles and how these relate to Shotstacks JSON video editing format. — An in Shotstack is essentially a JSON file describing how assets such as titles, video clips and images are arranged to create a video. Edit Edit The represents the entire duration of your video in seconds. For example, a 30 second video edit would have a 30 second timeline. Clips are placed along the timeline via a . Timeline — timeline track — A runs for the entire duration of the timeline and is used to layer clips on top of each other. For example, you can layer text on top of an image or a video clip. You can place multiple tracks on top of each other to create complex compositions. Tracks track — A is a placeholder for an asset (titles, images, videos, audio, html, and ) and are positioned on the timeline (via a track) at specific time intervals. For example, you could add a clip that starts playing at the 5th second until the 10th second. Here you can also apply transitions, filters, and motion effects. Clips clip Luma assets — When creating a , you specify the type of asset to be displayed. This can be a , , , , or a . Each asset has options specific to its type. For example a title asset will have a type style, and a video asset might have a trim point to cut the start and end of the clip. Assets clip text title image video clip HTML audio Luma asset — The specifies the final render properties, with the API supporting mp4 video files and gifs in a range of resolutions such as HD, SD and mobile. Output output If this does not all make sense yet, it will all become clearer once we create a few examples. Your first Edit Let’s create a very simple “Hello World” title video. Create a text file called with the following contents: hello.json { : { : , : [
      { : [
          { : { : , : , : }, : , : }
        ]
      }
    ]
  }, : { : , : }
} "timeline" "background" "#000000" "tracks" "clips" "asset" "type" "title" "text" "Hello World" "style" "future" "start" 0 "length" 5 "output" "format" "mp4" "resolution" "sd" In this JSON snippet we specify a timeline with a black background, a track with a single clip which plays for 5 seconds, and a title asset with the text ‘Hello World’. We specify the output as an mp4 video with SD resolution. Now we post the Edit to the API for rendering by running the command below. Remember to replace the x-api-key parameter value above with your own staging key. If you don’t have a key yet you can get one by . signing up here curl -X POST \
  -H \
  -H \
  -d @hello.json \
  https://api.shotstack.io/stage/render "Content-Type: application/json" "x-api-key: YOUR_API_KEY" You should see a response similar to: { : , : , : { : , : }
} "success" true "message" "Created" "response" "message" "Render Successfully Queued" "id" "d2b46ed6-998a-4d6b-9d91-b8cf0193a655" Copy the response id returned by the API as we will need it in the next step. To check the status of the render run the following command, making sure to replace with the id just copied. d2b46ed6–998a-4d6b-9d91-b8cf0193a655 curl -X GET \
  -H \
  -H \
  https://api.shotstack.io/stage/render/d2b46ed6-998a-4d6b-9d91-b8cf0193a655 "Content-Type: application/json" "x-api-key: YOUR_API_KEY" The video should only take a few seconds to render. Once it is ready the response will show a status of done and include a URL to the final video file: { : , : , : { : , : , : , : , : , : { : {...}, : {...}
    }, : , : }
} "success" true "message" "OK" "response" "id" "d2b46ed6-998a-4d6b-9d91-b8cf0193a655" "owner" "hckwccw3q3" "plan" "sandbox" "status" "done" "url" "https://s3-ap-southeast-2.amazonaws.com/shotstack-api-stage-output/hckwccw3q3/d2b46ed6-998a-4d6b-9d91-b8cf0193a655.mp4" "data" "output" "timeline" "created" "2019-04-16T12:02:42.148Z" "updated" "2019-04-16T12:02:51.867Z" If the status is not yet done (it might show queued, rendering, saving, fetching, or failed) wait a few seconds and run the command again. Once your video is completed you can copy the URL and open it in your browser. You should see a simple 5 second video with the words ‘Hello World’ on a black background similar to the below. Congratulations! You have created your first video using the Shotstack API. It is obviously still a little bit boring so let’s spice it up by adding a soundtrack and a background video. Adding a video clip and soundtrack Replace the contents of with the following snippet: hello.json { : { : { : }, : , : [
      { : [
          { : { : , : , : }, : , : }
        ]
      },
      { : [
          { : { : , : }, : , : }
        ]
      }
    ]
  }, : { : , : }
} "timeline" "soundtrack" "src" "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/music/freepd/motions.mp3" "background" "#000000" "tracks" "clips" "asset" "type" "title" "text" "Hello World" "style" "future" "start" 0 "length" 15 "clips" "asset" "type" "video" "src" "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/earth.mp4" "start" 0 "length" 15 "output" "format" "mp4" "resolution" "1080" In this example we have added a new soundtrack file as a second track ( ), in addition to the video asset . Both the title and video clip have a length of 15 seconds and start at 0 seconds, meaning we have composited these tracks on top of each other. motions.mp3 earth.mp4 This makes sure that the text is rendered on top of the video file and not hidden underneath. Note that the video asset is positioned below the title asset. Go ahead and use the previous Curl commands to post the video and check its status. You should end up with a video like the one below: This video is already a lot more interesting, but it could still do with a bit of refinement to give it a more professional touch. Adding the final touches to our edit To add some final polish we will reposition and resize the title and fade in and out both the video clip and soundtrack, in addition to a few other styling improvements. Replace the file with the following and post it to the API using the Curl command and check its status. hello.json { : { : { : , : }, : , : [
      { : [
          { : { : , : , : , : , : }, : , : , : { : , : }, : }
        ]
      },
      { : [
          { : { : , : , : }, : , : , : { : , : }
          }
        ]
      }
    ]
  }, : { : , : }
} "timeline" "soundtrack" "src" "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/music/freepd/motions.mp3" "effect" "fadeOut" "background" "#000000" "tracks" "clips" "asset" "type" "title" "text" "Hello World" "style" "future" "position" "left" "size" "x-small" "start" 4 "length" 11 "transition" "in" "fade" "out" "fade" "effect" "zoomIn" "clips" "asset" "type" "video" "src" "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/earth.mp4" "trim" 5 "start" 0 "length" 15 "transition" "in" "fade" "out" "fade" "output" "format" "mp4" "resolution" "sd" In the Edit above we’ve added the following: A effect is added on the soundtrack; fadeOut We use a fade in and fade out on the title asset and on the video asset; transition We the title asset to the left and adjust its size; position We displaying the title asset after 4 seconds in time to the music; start We add a to the title asset; zoomIn motion effect We the first 5 seconds off the start of the video asset. trim The result is below: A lot more to explore Hopefully, this article has inspired you to use code to create beautiful videos. We’re always really interested in seeing what people are able to come up with so if you’ve managed to create something cool please share! If you are keen to learn more please check out the . We are always working on building more guides and resources on and have examples available using our and SDKs. documentation shotstack.io PHP NodeJS There’s a lot to explore, so if you’re interested in learning more. I’ll write another article soon that covers how you can use HTML and CSS to build your videos. Here a glimpse of what that looks like: subscribe to our newsletter Also published here.

Create Videos With Code [Free Tutorial]

Hey Hackers! On April 13th, Canon, the camera and photography giant, announced that they will be launching an NFT Marketplace, called , offering tokenized photographs. Cadabra Cadabra is described as a curated photography marketplace that offers NFTs from a variety of innovative creators. Canon states that the goal of the Cadabra marketplace is “to empower creators by connecting them to enthusiastic collectors who celebrate the art of photography through a shared appreciation for these digital artworks”. Decrypt has that the NFT marketplace will be launched on the Ethereum network. reported “The digital art will range from landscape, wildlife, lifestyle, sports, abstract, and other categories to spotlight the stunning work of the invited creators.” - Canon USA Canon also took to to announce the exciting news to it’s customers and followers. Twitter On Cadabra, the NFT photos will be minted by Creator drops, and there will reportedly be a secondary marketplace for NFT collectors to trade their minted NFTs. Canon has stated that Cadabra creators will have the choice of offering “purchasers prints” with their initial drops, which entitles NFT purchasers to also receive a physical print of the NFT image. Based off the small amount of information we have about Cadabra, it does sound like there will be intuitive options for creators to present their work and possibly even earn a good amount. There is no official word on when Cadbara will be launched, but Canon did allude that it will be later this year. Canon has also stated that the Cadbara marketplace will only be available in the United States at launch, but they will be expanding the project internationally in the future. Last but not least, Canon has confirmed that both cryptocurrency, as well as debit and credit cards, will be accepted as payment options on Cadabra. Here is the direct link to sign up for Cadabra news and updates: https://shortstack.usa.canon.com/cadabra What are your thoughts on this news? Do you think this NFT Marketplace could be a big player? Are you interested in Photography? Will you be signing up for Cadabra when it launches? Check out my Socials: https://linktr.ee/zerorequiem0x See ya! :) Also published . here

Canon Throws Hat in the NFT Ring With Cadabra: a Curated Marketplace for Tokenized Photography

Where Do We Go From Here? This weekend a groundbreaking story in the New York Times, , which was covered by media internationally, was the first acknowledgement by the Pentagon since the 1960’s that they were taking UFOs seriously. Glowing Auras and Black Money: The Pentagon’s Mysterious UFO Program It is being hailed by many in the world of UFOs as the beginning of a drip drip disclosure not only about the existence of so-called black government projects to study UFO sightings, but also what is perhaps a more interesting revelation for scientists and engineers: that in our airspace there exists technology that defies our known laws of physics, and we are to reverse engineer it. actively trying Until now, if you asked any serious scientist about UFOs publicly, they usually gave a CYA answer. It reminds me of the old saying in the business world, “no one ever got fired for going with IBM.” No academic or researcher ever got fired for saying that “there is no evidence that UFOs exist, therefore we shouldn’t study them”. On the other hand, if they said that the matter should be taken seriously and given funding, they would subject themselves to all kinds of ridicule and possibly lose their job. This brings to mind an exchange in the movie (based on the book written by very respectable astronomer Carl Sagan), between Ellie Arroway (played by Jodie Foster) and her mentor, the very respectable scientist David Drumlin (played by Tom Skerritt). He basically told her: “You’re a promising scientist. Don’t throw away your career … looking for little green men.” Contact With this bombshell report and the DoD’s admission that it takes UFO seriously, is that attitude about to change? A Little History Before we answer that, let’s look at a little history. Almost 50 years ago today, on December 17, 1969, Project Blue Book, a government funded effort to study UFO sightings was shut down (on a strange personal note, this was also the day I was born, though it happened half way around the world!). The popular narrative for the shutdown of “Project Blue Book” (and related projects) was that most UFOs were mis-identifications or natural occurrences, and since the military didn’t consider them a threat, the government was officially . no longer in the UFO business There were 3 problems with this narrative: A number of sightings were left “unexplained” — these were different from the number of sightings that were “incomplete” (i.e. there wasn’t enough information to figure out what they were). Many suspected that the government continued to show interest in and pursue UFOs in top secret programs in the DoD even after claiming they were “no longer interested”. In fact, there are many claims that the most interesting UFO cases were investigated by the DoD directly and never turned over to BB. Dr. J. Alan Hynek, who was the chief scientific consultant on the project, later became an advocate of serious UFO research, often debating with none other than Carl Sagan, who (publicly at least) believed that money should be spent on “serious” science like radio astronomy. Hynek not only invented the terminology “Close Encounters of the Third Kind” but also had a cameo in Spielberg’s 1977 film. The Big Announcement This year, on December 16, 2017, along with the NYTimes story, the Department of Defense released its first “official footage” of UFO encounters with the military. They admitted for the first time since 1969 that there was another secret organization that was formed and funded by Senator Harry Reid and others to study the phenomenon. If you read the information contained in that report and others, the real story wasn’t just that the DoD admitted they were studying UFOs — but that this was the first time they having real evidence — including video, radar, eyewitness, photographs and even physical artifacts — of encounters between US military aircraft and unknown craft which were defying the known laws of physics. admitted publicly of The aspect of the story that’s most interesting to me, and perhaps to you if you are an engineer or scientist is right here: Under Mr. Bigelow’s direction, the company modified buildings in Las Vegas for the storage of metal alloys and other materials that Mr. Elizondo and program contractors said had been recovered from unidentified aerial phenomena. In fact, this may be the first time since the 1947 Roswell incident (where the military did a press release saying that it had captured a flying saucer, only to retract the story over the next few days) that anyone in the government is actually admitting to having flyer saucer/UFO/UAP technology in their possession. This is an astonishing admission. How Will Scientists, Engineers and Silicon Valley React? Will this cause mainstream scientists to be a little more careful when asked about UFOs? A few years ago when I was at SETI-CON (sponsored by SETI, the Search for Extra Terrestrial Intelligence, whose executive director, Jill Tartar, was a respectable radio astronomer and who was partly the inspiration behind heroine), I asked Seth Shostack, the chief radio astronomer there why SETI wasn’t studying UFOs. Contact’s Seth’s response to me was something along the lines of: “Well the UFO people are crazy. They think they have evidence that UFOs are here when there is zero evidence. I repeat zero evidence.” I always suspected this was a standard CYA answer, since there was actually tons of evidence that he didn’t want to take the time to sort through. Now that the Pentagon has pretty much confirmed that there are UFOs flying around in our skies, and they have not only been studying them, but that we have from them, the real question is: artifacts Shouldn’t civilian scientists in academia and in industry be demanding that we try to figure out how this stuff works? There should be a big cry from Silicon Valley — which is supposed to be the heart of innovation ( I live here and am just down the road from both Google and SETI in Mountain View). Instead of Elon Musk spending his time trying to make rockets (a 50+ year old technology) work better, if he truly wants us to become an inter-planetary (and perhaps inter-stellar species) shouldn’t he be investing some of his billions in figuring out this not yet understood form of transportation? While Google has the Lunar X prize for us to reconstruct technology that we had working 40 years ago to go to the moon, shouldn’t they be investing more $$ in an X prize that would mark a real advancement in physics and human exploration — demonstration of a non-rocket based system of flight and propulsion to the stars? Re-engineering/Hacking UFO Technology? The truth is we don’t really know how this UFO tech works, and in the realm of basic research, Silicon Valley may be more of a laggard than an innovator. And by “we”, I’m saying that it’s our mainstream science that really doesn’t know how it works. Arthur C. Clarke once said that sufficiently advanced technology is indistinguishable from magic -and there’s nothing that our “respectable”mainstream scientists hate more than something that appears to be “magic” because we don’t understand it. From the same New York Times Story: “We’re sort of in the position of what would happen if you gave Leonardo da Vinci a garage-door opener,” said Harold E. Puthoff, an engineer who has conducted research on extrasensory perception for the C.I.A. and later worked as a contractor for the program. “First of all, he’d try to figure out what is this plastic stuff. He wouldn’t know anything about the electromagnetic signals involved or its function.” I can say that I have personally spoken to researchers from top universities (Stanford, MIT, Harvard) who have seen the “artifacts” that the article references, and other similar ones that are even more secretive (and perhaps more functional). Although I’m usually the one who brings up science fiction scenarios, one researcher brought up an apt one to me: In the movie , scientists and engineers were given limited access to “it” — the pieces of the microchip and the skeleton of the terminator which had come from the future. They were reverse engineering it trying to figure out how it worked. It was far ahead of anything that had been developed to date. Terminator 2 Sounds familiar? I’ve been told that the reverse engineering effort is just like that. This is a very big challenge. Think of the advances we made in science and tech in the past 100 years, and imagine the advances we’ll make in the next 100 years. Now imagine that we have tech from a civilization that is ! a million years ahead of us The Way Forward We will need the best minds on this — not only from the military or a few covert researchers who are sworn to secrecy if we are to figure out how this stuff works. This is a much bigger project than anything else we’ve endeavored to do — because once the basic materials questions are answered, we may need new kinds of electromagnetic signals and a new type of computer science than what we’ve been taught. I’m told that some of the tech that’s been discovered and hasn’t been talked about involves interfacing consciousness with physical artifacts, something that also sends most of our mainstream scientists and engineers scurrying to their CYA corners, muttering about “pseudo-science” and “anecdotal evidence.” Those of us who have researched this phenomenon know that this weekend’s article is just the tip of the Iceberg. I’ve personally seen old photos from the 1950s and 1960s that were much clearer than anything released thus far. Tom DeLong, the former blink182 musician who formed and who was instrumental in getting this announcement going himself tweeted: “There’s a lot more s — t coming!” To The Stars Academy The big question in my mind is, what will the “respectable” scientists do now that the cat is out of the bag and as more information comes out? I see one of three possible ways forward: 1. Scientists and engineers from the mainstream continue to deny, deny, deny and give a CYA answer: There is no evidence of UFOs or UFO technology and it doesn’t deserve to be studied. 2. As more information leaks out, at some point, the happens. What is the ? In the same movie, , after an actual signal from another world was discovered, the same scientist that ridiculed Ellie Arroway suddenly became an “expert” on the new signal and aliens and tried to co-opt the work, never acknowledging their prior attitude of closed-mindedness. David Drumlin effect David Drumlin effect Contact 3. Scientists and engineers work together with UFO researchers and the DoD, all the while staying humble and keeping an open mind. A new open-minded approach may be needed where scientists and engineers may have to start with an attitude that “we were wrong”, and that “we have not discovered all there is to discover, not by a long shot” as a starting point. This may result in a re-discovery of some fundamental ideas about aerodynamics, quantum physics, and even, I’m told consciousness and its ability to affect the material world. As more comes to light, we may need to reconsider our arrogance of not just being alone in the universe, but also in thinking our science/tech is technologically advanced that we can make definitive statements about other civilizations and their ability to “visit” us. Those of us who are “pro-science” may also have to reconsider some fundamental aspects of our own science and technology, and even more difficult, our prejudices and close-mindedness to things we don’t understand. PS — a big thanks to the efforts of people like Tom DeLong, former DoD officer Luis elizondo, journalists Helene Cooper, Ralph Blumenthal, and the tireless Leslie Keane, and others who have stuck with this subject and were instrumental in getting this announcement out via the NYTimes, etc.

Hacking UFO Tech, SETI, and the Boundaries of “Respectable” Science

A DevOps transformation without implementing Infrastructure as Code will remain incomplete: Infrastructure Automation is a pillar of the modern Data Center. With tools like SaltStack and Docker things are becoming easier. This tutorial is part of a . Painless Docker Course If you are interested in discovering our new courses about AWS, and we will be sharing interesting stuff with you ! register using this form Update: After sharing this story to Reddit, I read some comments saying that I am reinventing AWS Lambda or a serverless system in general. I’d like to say yeah ! why not !? After all, the purpose of this course is educational and you can consider it as a reverse hack. Happy Hacking ! Introduction We are going to use a set of tools like SaltStack, Boto3 (Python3), AWS CLI, EC2 in order to create a master and some minions running a Docker container each. This diagram explains a lot of what we will do. Creating the First EC2 Machine Using AWS CLI We will need the ID of the VPC, the ID of the Security Group, the ID of the Subnet, the name of the key pair and the AMI of the OS we are going to use (Ubuntu 16.04). The latter depends on the zone. Describing VPCs Let’s see what we have as VPCs: aws ec2 describe-vpcs You should get a detailed list of your active VPCs: {"Vpcs": [{"DhcpOptionsId": "dopt-xxxxxx","CidrBlock": "172.31.0.0/16","InstanceTenancy": "default","State": "available","IsDefault": true,"VpcId": "vpc-xxxxx"},{"DhcpOptionsId": "dopt-xxxxx","CidrBlock": "172.20.0.0/16","InstanceTenancy": "default","State": "available","Tags": [{"Key": "xxxxx","Value": "xxxxx"},{"Key": "xxxxxx","Value": "xxxxx"},{"Key": "xxxxxx","Value": "xxxxxx"}],"IsDefault": false,"VpcId": "vpc-xxxxxx"}]} Listing the Security Groups aws ec2 describe-security-groups You should get a list: {"IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"UserIdGroupPairs": [ \],  
      "PrefixListIds": \[  
          
      \]  
    }  
  \],  
  "Tags": \[  
    {  
      "Key": "Name",  
      "Value": "xxxxxx"  
    }  
  \],  
  "OwnerId": "xxxxx",  
  "GroupName": "xxxxxxxx",  
  "VpcId": "vpc-xxxxxxx",  
  "Description": "xxxxxx",  
  "IpPermissions": \[  
    {  
      "IpProtocol": "-1",  
      "IpRanges": \[  
        {  
          "CidrIp": "0.0.0.0/0"  
        }  
      \],  
      "UserIdGroupPairs": \[  
          
      \],  
      "PrefixListIds": \[  
          
      \]  
    },  
    {  
      "IpRanges": \[  
        {  
          "CidrIp": "0.0.0.0/0"  
        }  
      \],  
      "ToPort": 22,  
      "UserIdGroupPairs": \[  
          
      \],  
      "PrefixListIds": \[  
          
      \],  
      "IpProtocol": "tcp",  
      "FromPort": 22  
    },  
    {  
      "IpRanges": \[  
        {  
          "CidrIp": "0.0.0.0/0"  
        }  
      \],  
      "ToPort": 2376,  
      "UserIdGroupPairs": \[  
          
      \],  
      "PrefixListIds": \[  
          
      \],  
      "IpProtocol": "tcp",  
      "FromPort": 2376  
    }  
  \],  
  "GroupId": "sg-xxxxxx"  
}      
... Describing Availability Zones & Choosing the AMI: aws ec2 describe-availability-zones Even if we’re not going to use this directly but to find the AMI, we should get this information first: {"AvailabilityZones": [{"Messages": [],"RegionName": "eu-west-1","ZoneName": "eu-west-1a","State": "available"},{"Messages": [],"RegionName": "eu-west-1","ZoneName": "eu-west-1b","State": "available"},{"Messages": [],"RegionName": "eu-west-1","ZoneName": "eu-west-1c","State": "available"}]} Since I am going to use Ubuntu, I used this website: https://cloud-images.ubuntu.com/locator/ec2/ Describing the Subnets aws ec2 describe-subnets And you get a similar list to this: {"AvailableIpAddressCount": 4091,"MapPublicIpOnLaunch": true,"AvailabilityZone": "eu-west-1b","VpcId": "vpc-xxxxxx","State": "available","DefaultForAz": true,"CidrBlock": "172.31.0.0/20","Tags": [{"Value": "xxxxxxxxxx","Key": "Name"}],"SubnetId": "subnet-xxxxxxxxxx"} Getting the Key Pairs aws ec2 describe-key-pairs And it will show the list of existent key pairs: {"KeyPairs": [{"KeyFingerprint": "xxxxxxxxxxx","KeyName": "xxxxxxxx"},{"KeyFingerprint": "xxxxxxxxx","KeyName": "xxxxxxxx"}]} Let’s Create the Machine aws ec2 run-instances --image-id ami-785db401 --count 1 --instance-type t2.micro --key-name .xxxxx --security-group-ids sg-xxxxx --subnet-id subnet-xxxxx --associate-public-ip-address --query 'Instances[0].InstanceId' --output text This should show us the id of the instance: i-0ed688fc95b1feeb2 Get the Public DNS In order to connect using SSH, we should get this ! aws ec2 describe-instances --instance-ids i-0ed688fc95b1feeb2 --query 'Reservations[0].Instances[0].PublicDnsName' --output text And the output was: ec2-34-253-201-12.eu-west-1.compute.amazonaws.com Let’s Try the Public IP ! Same as DNS, type: aws ec2 describe-instances --instance-ids i-0ed688fc95b1feeb2 --query 'Reservations[0].Instances[0].PublicIpAddress' --output text Output: 34.253.201.12 I Need the Private IP In my case I am going to connect the Salt Minion to the Salt Master using the private IP, you can do the same if you will create both machines in the same VPC. If you want to use the private IP : aws ec2 describe-instances --instance-ids i-0ed688fc95b1feeb2 --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text Output: 172.31.4.106 Installing the Master Using the public DNS and your Key Pair connect to the created machine: apt install salt-master Creating & Provisioning Multiple Minion Machines Using Boto3 & Cloudinit To use Boto3 (Python), you can create a Python virtual environment if you don’t want to install it to your machine. This is what I do in all cases. virtualenv -p python3 learning cd learning/. bin/activatepip install boto3mkdir app && cd app Now let’s create the Python script: import boto3import sys AWS_ACCESS_ID = "xxxxxxxxxx"AWS_SECRET_KEY = "x/xxxxxx"ImageId = "ami-785db401"KeyName = ".xx"InstanceType = "t2.micro" MinCount = 1MaxCount = 1SubnetId = "subnet-xxxxx"SecurityGroupIds = ["sg-xxxxxx"]UserData = open('init.sh') conn = boto3.client('ec2',aws_access_key_id=AWS_ACCESS_ID,aws_secret_access_key=AWS_SECRET_KEY) for i in range( int(sys.argv[1]) ):reservation = conn.run_instances(ImageId=ImageId,KeyName=KeyName,InstanceType=InstanceType,MinCount=MinCount,MaxCount=MaxCount,SecurityGroupIds=SecurityGroupIds,SubnetId = SubnetId,UserData=UserData.read()) This is the user data file: #!/bin/bash # exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1echo BEGIN https://alestic.com/2010/12/ec2-user-data-output/ sudo apt-get update -ysudo apt-get upgrade -y # Install saltstacksudo apt-get install salt-minion -y # Set salt master location and start minionsudo sed -i 's/#master: salt/master: 34.253.201.12/g' /etc/salt/minionsudo salt-minion -d # Install Dockercurl -fsSL | sudo apt-key add - https://download.docker.com/linux/ubuntu/gpg sudo add-apt-repository \"deb [arch=amd64] \$(lsb_release -cs) \stable" -y https://download.docker.com/linux/ubuntu sudo apt-get update -y || :sudo apt-get install docker-ce -y sudo apt install python-pip -y && sudo pip install docker-py -y echo END It will provision the machine on its startup by installing Salt Minion and Docker. Note in the script above that we already set the server of the minion to the private IP address of the master. sudo sed -i 's/#master: salt/master: 34.253.201.12/g' /etc/salt/minion To create 5 machines, just type: python <script_name> 5 To run 1000 machines, just type: python <script_name> NO WAY :) Now we have 5 machines where Salt Minion and Docker are installed and a a Salt Master. Connecting the Minions to the Master On the Master type: salt-key -L and you’ll get the Minions waiting to be accepted: Accepted Keys:Denied Keys:Unaccepted Keys:ip-172-31-5-169.eu-west-1.compute.internalip-172-31-5-168.eu-west-1.compute.internalip-172-31-5-167.eu-west-1.compute.internalip-172-31-5-166.eu-west-1.compute.internalip-172-31-5-165.eu-west-1.compute.internalip-172-31-5-164.eu-west-1.compute.internal..etc Rejected Keys: Accept all using: salt-key -A then uncomment the following part from /etc/salt/master: file_roots:base:- /srv/salt We will have a file server for Salt states and Salt high states in the location mentioned above. Now restart the Master: pkill salt-mastersalt-master -d In order to install Docker containers on the Minions, you should not forget this: salt * pip.install docker-py>=1.4.0 This command will install a superior version to 1.4.0 of Python docker package. Now create two files, the Top file In Salt, the file which contains a mapping between groups of machines on a network and the configuration roles that should be applied to them is called a . top file And the SLS file: The core of the Salt State system is the SLS, or a t tate file. The SLS is a representation of the state in which a system should be in, and is set up to contain this data in a simple format. This is often called configuration management. S L S (both quotes are taken from the official Salt documentations) Here is the structure of file tree: tree /srv/salt/ /srv/salt/├── top.sls└── webserver.sls Starting Docker Containers We are going to run multiple Nginx containers. Nginx is just an example here, you can run whatever image you want. Let’s modify the content of the webserver.sls and tell the Master to run Nginx: my_service:dockerng.running:- name: nginx- image: nginx Now let’s modify the content of the Top file and tell the Master to apply the state to the servers having Ubuntu as an OS. This is a targeting technique, you can choose any other targeting pattern. (read this for more information: ) https://docs.saltstack.com/en/latest/topics/targeting/ #top.sls file base:'os:Ubuntu':- match: grain- webserver In order to start the containers, type: salt '*' state.apply and you will get the execution logs: [...]ip-172-31-5-169.eu-west-1.compute.internal:----------ID: my_serviceFunction: dockerng.runningName: nginxResult: TrueComment: Container 'nginx' is already configured as specifiedStarted: 01:03:40.062033Duration: 28.66 msChanges: Summary for ip-172-31-5-169.eu-west-1.compute.internal------------Succeeded: 5Failed:    0------------Total states run:     5 This Combination of Salt, Docker, Boto3 and AWS could be used in different automation scenarios and even for self-healing and auto-scalable infrastructures. You can find similar tutorials in my . Painless Docker Course That’s all folks ! Connect Deeper If you resonated with this article, you can find more interesting contents in . Painless Docker Course If you liked this course, subscribe to one or more of our newsletters: : The DevOps Ecosystem is Growing Fast. Each Week, We Hand-curate Must-read News, Tutorials, Reviews and More ! DevOpsLinks : An Independent Newsletter Focused On Serverless, Containers, FaaS & Other Interesting Stuff Shipped : A #Kubernetes Community Hub, Hand Curated Newsletter, Team Chat, Training & More Kaptain You can find me on , or my and you can also check my books: . Twitter Clarity website SaltStack For DevOps Don’t forget to join our jobboard ! Jobs For DevOps If you liked this post, please recommend it and share it with your followers.

Creating an Army of Docker Containers using SaltStack, Boto3 & CloudInit on AWS

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
Authors:
(1) Ehud Shapiro,  Department of Computer Science and Applied Math, Weizmann Institute of Science, Israel and ehud.shapiro@weizmann.ac.il. Table of Links Abstract and Introduction
Grassroots Protocols
Grassroots Implementation of Grassroots Dissemination
Applications of Grassroots Dissemination Supporting Digital Sovereignty
Future Work & Discussion
References
A. Preliminaries: Asynchronous Distributed Multiagent Transition Systems
B. All-to-All Dissemination is not Grassroots
C. Proofs References 1 Martin R Albrecht, Jorge Blasco, Rikke Bjerg Jensen, and Lenka Mareková. Mesh messaging in large-scale protests: Breaking bridgefy. In Cryptographers’ Track at the RSA Conference, pages 375–398. Springer, 2021. 2 Juan Benet. Ipfs-content addressed, versioned, p2p file system. arXiv preprint arXiv:1407.3561, 2014. 3 Sonja Buchegger, Doris Schiöberg, Le-Hung Vu, and Anwitaman Datta. Peerson: P2p social networking: early experiences and insights. In Proceedings of the Second ACM EuroSys Workshop on Social Network Systems, pages 46–52, 2009. 4 Vitalik Buterin et al. A next-generation smart contract and decentralized application platform. white paper, 3(37), 2014. 5 Manuel Castells. The city and the grassroots: a cross-cultural theory of urban social movements. Number 7. Univ of California Press, 1983. 6 Miguel Castro, Barbara Liskov, et al. Practical byzantine fault tolerance. In OsDI, volume 99, pages 173–186, 1999. 7 Gregory Chockler, Roie Melamed, Yoav Tock, and Roman Vitenberg. Constructing scalable overlays for pub-sub with many topics. In Proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing, pages 109–118, 2007. 8 Gregory Chockler, Roie Melamed, Yoav Tock, and Roman Vitenberg. Spidercast: a scalable interest-aware overlay for topic-based pub/sub communication. In Proceedings of the 2007 inaugural international conference on Distributed event-based systems, pages 14–25, 2007. 9 Bram Cohen. Incentives build robustness in bittorrent. In Workshop on Economics of Peer-to-Peer systems, volume 6, pages 68–72. Berkeley, CA, USA, 2003. 10 Alan Demers, Karin Petersen, Mike Spreitzer, Doug Terry, Marvin Theimer, and Brent Welch. The bayou architecture: Support for data sharing among mobile users. In 1994 First Workshop on Mobile Computing Systems and Applications, pages 2–7. IEEE, 1994. 11 Bitcoin Developer. P2p network, Retrieved 20222. URL: https://developer.bitcoin.org/ devguide/p2p_network.html. 12 Ethereum Developer. go-ethereum/params/bootnodes.go, Retrieved 2023. URL: https://github.com/ethereum/go-ethereum/blob/master/params/bootnodes.go. 13 Adem Efe Gencer, Soumya Basu, Ittay Eyal, Robbert van Renesse, and Emin Gün Sirer. Decentralization in bitcoin and ethereum networks. In International Conference on Financial Cryptography and Data Security, pages 439–457. Springer, 2018. 14 Gabriele Goldacker. Digitale souveränität [white paper of kompetenzzentrum öffentliche it (öfit)]. retrieved april 20, 2021, 2017. 15 Idit Keidar, Eleftherios Kokoris-Kogias, Oded Naor, and Alexander Spiegelman. All you need is dag, 2021. arXiv:2102.08325. 16 Idit Keidar, Oded Naor, and Ehud Shapiro. Cordial miners: A family of simple and efficient consensus protocols for every eventuality. arXiv preprint arXiv:2205.09174, 2022. 17 Antony Lewis. A gentle introduction to self-sovereign identity. 2017. URL: https://bitsonblocks.net/2017/05/17/a-gentle-introduction-to-self-sovereign-identity/. 18 Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system, 2019. 19 Romualdo Pastor-Satorras, Claudio Castellano, Piet Van Mieghem, and Alessandro Vespignani. Epidemic processes in complex networks. Reviews of modern physics, 87(3):925, 2015. 20 Julia Pohle and Thorsten Thiel. Digital sovereignty. Internet Policy Review: Journal on Internet Regulation, 9(4):1–19, 2020. 21 Aravindh Raman, Sagar Joglekar, Emiliano De Cristofaro, Nishanth Sastry, and Gareth Tyson. Challenges in the decentralised web: The mastodon case. In Proceedings of the Internet Measurement Conference, pages 217–229, 2019. 22 Vinay Setty, Maarten van Steen, Roman Vitenberg, and Spyros Voulgaris. Poldercast: Fast, robust, and scalable architecture for p2p topic-based pub/sub. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, pages 271–291. Springer, 2012. 23 Devavrat Shah et al. Gossip algorithms. Foundations and Trends® in Networking, 3(1):1–125, 2009. 24 Ehud Shapiro. Multiagent transition systems: Protocol-stack mathematics for distributed computing. arXiv preprint arXiv:2112.13650, 2021. 25 Ehud Shapiro. Sovereign personal cryptocurrencies: A grassroots foundation for a digital economy. arXiv preprint arXiv:2202.05619, 2022. 26 Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. Conflict-free replicated data types. In Symposium on Self-Stabilizing Systems, pages 386–400. Springer, 2011. 27 Robert Shostak, Marshall Pease, and Leslie Lamport. The byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3):382–401, 1982. 28 Ion Stoica, Robert Morris, David Karger, M Frans Kaashoek, and Hari Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. ACM SIGCOMM computer communication review, 31(4):149–160, 2001. 29 Maofan Yin, Dahlia Malkhi, Michael K Reiter, Guy Golan Gueta, and Ittai Abraham. Hotstuff: Bft consensus with linearity and responsiveness. In Proceedings of the 2019 ACM Symposium on Principles of Distributed Computing, pages 347–356, 2019. 30 Shoshana Zuboff. The age of surveillance capitalism: The fight for a human future at the new frontier of power: Barack Obama’s books of 2019. Profile books, 2019. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Ehud Shapiro,  Department of Computer Science and Applied Math, Weizmann Institute of Science, Israel and ehud.shapiro@weizmann.ac.il. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: Authors: (1) Ehud Shapiro,  Department of Computer Science and Applied Math, Weizmann Institute of Science, Israel and ehud.shapiro@weizmann.ac.il. Table of Links Abstract and Introduction Grassroots Protocols Grassroots Implementation of Grassroots Dissemination Applications of Grassroots Dissemination Supporting Digital Sovereignty Future Work & Discussion References A. Preliminaries: Asynchronous Distributed Multiagent Transition Systems B. All-to-All Dissemination is not Grassroots C. Proofs Abstract and Introduction Abstract and Introduction Grassroots Protocols Grassroots Protocols Grassroots Implementation of Grassroots Dissemination Grassroots Implementation of Grassroots Dissemination Applications of Grassroots Dissemination Supporting Digital Sovereignty Applications of Grassroots Dissemination Supporting Digital Sovereignty Future Work & Discussion Future Work & Discussion References References A. Preliminaries: Asynchronous Distributed Multiagent Transition Systems A. Preliminaries: Asynchronous Distributed Multiagent Transition Systems B. All-to-All Dissemination is not Grassroots B. All-to-All Dissemination is not Grassroots C. Proofs C. Proofs References 1 Martin R Albrecht, Jorge Blasco, Rikke Bjerg Jensen, and Lenka Mareková. Mesh messaging in large-scale protests: Breaking bridgefy. In Cryptographers’ Track at the RSA Conference, pages 375–398. Springer, 2021. 2 Juan Benet. Ipfs-content addressed, versioned, p2p file system. arXiv preprint arXiv:1407.3561, 2014. 3 Sonja Buchegger, Doris Schiöberg, Le-Hung Vu, and Anwitaman Datta. Peerson: P2p social networking: early experiences and insights. In Proceedings of the Second ACM EuroSys Workshop on Social Network Systems, pages 46–52, 2009. 4 Vitalik Buterin et al. A next-generation smart contract and decentralized application platform. white paper, 3(37), 2014. 5 Manuel Castells. The city and the grassroots: a cross-cultural theory of urban social movements. Number 7. Univ of California Press, 1983. 6 Miguel Castro, Barbara Liskov, et al. Practical byzantine fault tolerance. In OsDI, volume 99, pages 173–186, 1999. 7 Gregory Chockler, Roie Melamed, Yoav Tock, and Roman Vitenberg. Constructing scalable overlays for pub-sub with many topics. In Proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing, pages 109–118, 2007. 8 Gregory Chockler, Roie Melamed, Yoav Tock, and Roman Vitenberg. Spidercast: a scalable interest-aware overlay for topic-based pub/sub communication. In Proceedings of the 2007 inaugural international conference on Distributed event-based systems, pages 14–25, 2007. 9 Bram Cohen. Incentives build robustness in bittorrent. In Workshop on Economics of Peer-to-Peer systems, volume 6, pages 68–72. Berkeley, CA, USA, 2003. 10 Alan Demers, Karin Petersen, Mike Spreitzer, Doug Terry, Marvin Theimer, and Brent Welch. The bayou architecture: Support for data sharing among mobile users. In 1994 First Workshop on Mobile Computing Systems and Applications, pages 2–7. IEEE, 1994. 11 Bitcoin Developer. P2p network, Retrieved 20222. URL: https://developer.bitcoin.org/ devguide/p2p_network.html. 12 Ethereum Developer. go-ethereum/params/bootnodes.go, Retrieved 2023. URL: https://github.com/ethereum/go-ethereum/blob/master/params/bootnodes.go. 13 Adem Efe Gencer, Soumya Basu, Ittay Eyal, Robbert van Renesse, and Emin Gün Sirer. Decentralization in bitcoin and ethereum networks. In International Conference on Financial Cryptography and Data Security, pages 439–457. Springer, 2018. 14 Gabriele Goldacker. Digitale souveränität [white paper of kompetenzzentrum öffentliche it (öfit)]. retrieved april 20, 2021, 2017. 15 Idit Keidar, Eleftherios Kokoris-Kogias, Oded Naor, and Alexander Spiegelman. All you need is dag, 2021. arXiv:2102.08325. 16 Idit Keidar, Oded Naor, and Ehud Shapiro. Cordial miners: A family of simple and efficient consensus protocols for every eventuality. arXiv preprint arXiv:2205.09174, 2022. 17 Antony Lewis. A gentle introduction to self-sovereign identity. 2017. URL: https://bitsonblocks.net/2017/05/17/a-gentle-introduction-to-self-sovereign-identity/. 18 Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system, 2019. 19 Romualdo Pastor-Satorras, Claudio Castellano, Piet Van Mieghem, and Alessandro Vespignani. Epidemic processes in complex networks. Reviews of modern physics, 87(3):925, 2015. 20 Julia Pohle and Thorsten Thiel. Digital sovereignty. Internet Policy Review: Journal on Internet Regulation, 9(4):1–19, 2020. 21 Aravindh Raman, Sagar Joglekar, Emiliano De Cristofaro, Nishanth Sastry, and Gareth Tyson. Challenges in the decentralised web: The mastodon case. In Proceedings of the Internet Measurement Conference, pages 217–229, 2019. 22 Vinay Setty, Maarten van Steen, Roman Vitenberg, and Spyros Voulgaris. Poldercast: Fast, robust, and scalable architecture for p2p topic-based pub/sub. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, pages 271–291. Springer, 2012. 23 Devavrat Shah et al. Gossip algorithms. Foundations and Trends® in Networking, 3(1):1–125, 2009. 24 Ehud Shapiro. Multiagent transition systems: Protocol-stack mathematics for distributed computing. arXiv preprint arXiv:2112.13650, 2021. 25 Ehud Shapiro. Sovereign personal cryptocurrencies: A grassroots foundation for a digital economy. arXiv preprint arXiv:2202.05619, 2022. 26 Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. Conflict-free replicated data types. In Symposium on Self-Stabilizing Systems, pages 386–400. Springer, 2011. 27 Robert Shostak, Marshall Pease, and Leslie Lamport. The byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3):382–401, 1982. 28 Ion Stoica, Robert Morris, David Karger, M Frans Kaashoek, and Hari Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. ACM SIGCOMM computer communication review, 31(4):149–160, 2001. 29 Maofan Yin, Dahlia Malkhi, Michael K Reiter, Guy Golan Gueta, and Ittai Abraham. Hotstuff: Bft consensus with linearity and responsiveness. In Proceedings of the 2019 ACM Symposium on Principles of Distributed Computing, pages 347–356, 2019. 30 Shoshana Zuboff. The age of surveillance capitalism: The fight for a human future at the new frontier of power: Barack Obama’s books of 2019. Profile books, 2019.

Grassroots Distributed Systems for Digital Sovereignty: References

Laying the ground for post-capitalist blockchain protocols Blockchain Protocols and the Byzantine Generals Problem Section 1.1 Background Blockchain technology has emerged as a transformative force, disrupting traditional systems of trust and enabling decentralized, tamper-resistant data storage. At its core, blockchain is a distributed ledger technology that relies on consensus mechanisms to ensure the integrity and security of transactions. Before delving into the various blockchain protocols, it is essential to understand the fundamental coordination challenges that these protocols aim to address. One such challenge is the Byzantine Generals Problem, a classical problem in computer science and distributed systems. 1.2 The Byzantine Generals Problem The Byzantine Generals Problem was first introduced by Lamport, Shostak, and Pease in their landmark paper titled "The Byzantine Generals Problem" published in 1982. The problem provides a theoretical framework for understanding the difficulties associated with achieving consensus in a distributed network when some participants (nodes) may behave maliciously or fail to operate correctly. In the scenario described by the Byzantine Generals Problem, a group of Byzantine army generals surrounds an enemy city, and they must coordinate their attack or retreat strategy. The challenge is that some generals may be traitors, providing conflicting orders to different subgroups of the army. The loyal generals need to reach a consensus despite the potential misinformation from traitorous generals, ensuring a unified and coordinated action. This problem mirrors the challenges faced by distributed systems where nodes must agree on a shared state or transaction without a central authority. The Byzantine Generals Problem highlights the need for consensus protocols that can withstand the presence of faulty or malicious nodes. 1.3 Coordination in Distributed Systems The broader context of the Byzantine Generals Problem extends to the field of distributed systems, where achieving consensus among nodes is a critical challenge. As articulated by Leslie Lamport in his seminal paper "Time, Clocks, and the Ordering of Events in a Distributed System" (1978), the lack of a global clock and the asynchronous nature of distributed systems make achieving consensus a non-trivial task. Distributed systems must contend with issues such as message delays, network partitions, and the potential for malicious nodes. Various consensus protocols have been developed to address these challenges, and blockchain technology represents one application of these protocols to create decentralized and secure systems. 1.4 Objectives of This Section This section aims to lay the groundwork for understanding the Byzantine Generals Problem as a coordination challenge in distributed systems. Subsequent sections will explore different consensus protocols employed by blockchain networks to tackle this problem. By examining the strengths and weaknesses of these protocols, we can gain insights into the evolution of blockchain technology and its potential impact on various industries. In the following sections, we will delve into specific blockchain protocols, such as Proof of Work (PoW), Proof of Stake (PoS), and other novel protocols. These protocols represent diverse approaches to addressing the Byzantine Generals Problem, each with its unique advantages and trade-offs. Section 2.1 Introduction: A Post-Capitalist Perspective As blockchain technology advances, decentralized consensus protocols play a pivotal role in shaping the nature of distributed networks. In the context of a post-capitalist society, where the focus shifts from profit-driven motives to collective well-being and directly democratic decision-making, it becomes essential to critically assess existing consensus protocols and explore their implications for fostering a decentralized and equitable socio-economic system. 2.2 Proof-of-Work (PoW) Proof-of-Work, the pioneering consensus algorithm introduced by Bitcoin, relies on computational power to secure the network and validate transactions. While effective in achieving security, PoW has faced criticism for its environmental impact and resource-intensive nature. In a post-capitalist framework, the ecological concerns associated with PoW raise questions about sustainability and the alignment of incentives with broader social and environmental goals. Research suggests that PoW consumes significant energy resources, contributing to carbon footprints and potentially exacerbating environmental issues. In a post-capitalist society, where sustainability and ecological balance are paramount, PoW's resource-intensive model may be incompatible with the overarching objectives of the system. 2.3 Proof-of-Stake (PoS) Proof-of-Stake addresses some of the environmental concerns associated with PoW by replacing computational power with ownership stakes in the network's native cryptocurrency. PoS has been praised for its energy efficiency and reduced environmental impact. However, critics argue that PoS may lead to centralization, as participants with larger stakes have more influence over the consensus process. In a post-capitalist framework, the potential concentration of power within a small group of stakeholders challenges the ideals of decentralization. A shift in influence from computational power to economic power introduces new dynamics that may not align with the principles of equity and inclusivity. 2.4 Proof-of-Authority (PoA) Proof-of-Authority relies on identity and reputation, where consensus nodes are known and trusted entities. This model aims to enhance security by reducing the risk of malicious actors. However, it introduces a level of centralization and relies on pre-established trust. In a post-capitalist society, the reliance on trusted authorities may be perceived as a potential impediment to achieving true decentralization. The concentration of authority could result in power imbalances and undermine the ethos of an inclusive and distributed system. 2.5 Proof-of-Space-Time and Novel Protocols Proof-of-Space-Time, often associated with storage-based consensus, leverages participants' unused storage space to secure the network. While it addresses some environmental concerns, its effectiveness in fostering decentralization remains an area of exploration. Several novel consensus protocols, such as Delegated Proof-of-Stake (DPoS) and Directed Acyclic Graphs (DAGs), offer unique approaches to consensus. DPoS introduces a voting mechanism for selecting block producers, while DAGs eliminate the traditional block structure, allowing for concurrent block creation. From a post-capitalist perspective, the assessment of these protocols involves evaluating their ability to provide security, scalability, and decentralization while aligning with broader societal objectives. 2.6 Conclusion In a post-capitalist society, the evaluation of decentralized consensus protocols extends beyond technical efficiency to consider environmental sustainability, equity, and the distribution of influence. Each protocol presents a unique set of trade-offs, and their appropriateness depends on the specific goals and values of the decentralized system. As we explore the landscape of consensus protocols, it is crucial to critically assess their impact on society and their alignment with the principles of a post-capitalist ethos. delegat0x is a libertarian anti-capitalist R&D Engineer in the crypto space. They write about the intersections between philosophy, politics, media, alternatives to capitalism, social movements, and collective autonomy. References: Lamport, L., Shostak, R., & Pease, M. (1982). The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems (TOPLAS), 4(3), 382-401. Lamport, L. (1978). Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM, 21(7), 558-565. Buterin, V., & Griffith, V. (2017). Casper the Friendly Finality Gadget. Ethereum Foundation. Zamfir, V. (2018). On Stake. Medium. Atzei, N., Bartoletti, M., & Cimoli, T. (2017). A Survey of Attacks on Ethereum Smart Contracts (SoK). In International Conference on Principles of Security and Trust (POST) (pp. 164-186). Springer. Sompolinsky, Y., & Zohar, A. (2013). Accelerating Bitcoin’s Transaction Processing. Fast Money Grows on Trees, Not Chains. IACR Cryptology ePrint Archive, 2013, 881. Bächle, T., & Ylitalo, J. (2018). Proof-of-Authority: Reputation-Based Consensus in Permissioned Networks. Medium. Hyperledger. (n.d.). Cactus: A Modular Framework for Enterprise and Permissioned Ledgers. Retrieved from Hyperledger Cactus Also published here Feature Image depicts the concept of an anti-capitalist blockchain, showing a network of interconnected nodes with a focus on sustainable and cooperative community activities. Image produced by delegate0x DALL-E

Rethink Cryptoeconomics - Part 2: A Post-Capitalist Critique of Decentralized Consensus Protocols

Automated deployment strategies have emerged as a game-changing method to simplify , empowering organizations to achieve release cycles and improve their development flexibility. This article delves into the details of automated deployment strategies examining their importance, advantages, and how they effectively tackle challenges. software delivery What Are Automated Deployment Strategies? Automated deployment strategies consist of various tools and techniques that streamline the process of deploying software applications to production environments. These strategies remove the requirements for intervention greatly reducing the potential for mistakes and minimizing downtime during deployment. By automating the deployment process organizations can guarantee consistency, reliability, and repetition, in their software delivery cycles. Significance of Automated Deployment Strategies The significance of automated deployment strategies lies in their ability to address several critical challenges faced by software development teams: Manual deployment processes are often time-consuming, error-prone, and require significant effort from developers and operations personnel. Automated deployment strategies eliminate these inefficiencies, allowing teams to focus on higher-value tasks. Manual Deployment Inefficiencies: Manual deployments often introduce bottlenecks in the release cycle, delaying the delivery of new features and updates to users. Automated deployment strategies accelerate release cycles, enabling organizations to respond rapidly to market demands and user feedback. Release Cycle Bottlenecks: Manual deployments often involve downtime, disrupting user access to applications. Automated deployment strategies minimize downtime, ensuring seamless and uninterrupted service delivery. Reduced Deployment Downtime: Automated deployment strategies ensure consistent and repeatable deployment processes, eliminating the risk of variations and inconsistencies that can lead to errors and failures. Improved Deployment Consistency: Challenges of Manual Deployment Processes Manual deployment processes pose several challenges that hinder software development agility and reliability: Manual deployments are susceptible to human error, which can lead to configuration mistakes, compatibility issues, and application downtime. Human Error: Manual deployments lack repeatability, as the process is often dependent on individual knowledge and experience. This can lead to inconsistencies and variations in deployment outcomes. Lack of Repeatability: Manual deployments require significant time and effort from developers and operations personnel, diverting resources away from more strategic initiatives. Time-Consuming Efforts: Manual deployments become increasingly challenging as the scale and complexity of software applications increase. Limited Scalability: Automated deployment strategies effectively address these challenges, paving the way for streamlined, reliable, and scalable software delivery. Types Of Automated Deployment Strategies Automated deployment strategies encompass a spectrum of approaches tailored to specific deployment scenarios and application requirements. Each strategy offers unique advantages and considerations, making the choice of deployment method crucial for ensuring optimal performance and user experience. Rolling deployment , also known as incremental deployment, involves releasing a new version of the application in a gradual manner, rolling out the update to a small subset of users first. This approach minimizes disruption and allows for continuous monitoring of the deployment's impact. If any issues arise, the deployment can be rolled back or halted, preventing widespread disruption. Rolling deployment Blue-green deployment involves maintaining two identical production environments, designated as "blue" and "green." When a new version is ready, it is deployed to the "green" environment, while the "blue" environment continues to serve live traffic. Once the new version is thoroughly tested and validated, the traffic is switched to the "green" environment, while the "blue" environment is decommissioned. This approach ensures seamless transitions and minimal downtime. Blue-green deployment Canary deployment Canary deployment is a technique that involves releasing a new version of the application to a small, controlled group of users, often referred to as "canaries." This group is selected to represent a diverse range of user demographics and usage patterns. By observing the canaries, organizations can detect potential issues early on before rolling out the update to a broader audience. A/B testing deployment deployment involves releasing two different versions of an application to separate groups of users. This approach enables organizations to compare the performance and user acceptance of the two versions. By analyzing user behavior and feedback, organizations can determine which version is more effective and make informed decisions about the wider rollout. A/B testing The choice of automated deployment strategy depends on various factors, including application architecture, deployment frequency, and risk tolerance. Rolling deployment is particularly suitable for applications that can tolerate minor disruptions, while blue-green deployment is ideal for mission-critical applications where downtime is unacceptable. Canary deployment is valuable for identifying potential issues early on, while A/B testing deployment is crucial for data-driven decision-making. Advantages and Disadvantages of Automated Deployment Strategies Each automated deployment strategy offers unique benefits and considerations, making the choice of deployment method crucial for ensuring optimal performance and user experience. Rolling Deployment Advantages: Minimal downtime and disruption Continuous monitoring and rollback capability Suitable for applications with high availability requirements Disadvantages: Requires careful configuration and monitoring May introduce performance degradation during deployment Not suitable for applications with strict consistency requirements Real-world example: Netflix utilizes rolling deployment to release new features and updates to its streaming platform. By gradually rolling out updates to a small subset of users, Netflix can monitor the impact of the update and quickly roll back if necessary, minimizing disruption to its global user base. Blue-Green Deployment Advantages: Zero downtime during deployments Seamless traffic switching Suitable for mission-critical applications Disadvantages: Requires maintaining two identical production environments May increase infrastructure costs Complexity of setup and maintenance Real-world example: 's e-commerce platform, Amazon.com, employs blue-green deployment to ensure seamless transitions during major software releases. By maintaining two identical production environments, Amazon can thoroughly test new versions without impacting live traffic, ensuring a smooth and reliable user experience. Amazon Canary Deployment Advantages: Early detection of potential issues Controlled rollout to a limited audience Suitable for identifying performance bottlenecks Disadvantages: May introduce performance variations Requires careful selection of canary users Not suitable for applications with strict consistency requirements Real-world example: Google utilizes canary deployment to release new features to its search engine. By deploying new features to a small group of canary users, Google can gather feedback and identify potential issues before rolling out the update to a broader audience, ensuring a high-quality user experience. A/B Testing Deployment Advantages: Data-driven decision-making Comparison of different feature implementations Suitable for optimizing user engagement and conversion Disadvantages: Requires careful experiment design and analysis May introduce performance variations Not suitable for applications with strict consistency requirements Real-world example: E-commerce platforms like Shopify and Etsy use A/B testing deployment to optimize their checkout processes and landing pages. By comparing different versions of these pages, e-commerce platforms can identify the most effective design and maximize conversion rates. Implementing Automated Deployment Pipelines Automated deployment pipelines are the backbone of modern software delivery, enabling organizations to streamline the process of releasing new software to production environments. These pipelines encompass a series of automated steps that transform code changes into deployable artifacts, test the artifacts, and deploy them to the target environment. Key Components of Automated Deployment Pipelines Effective automated deployment pipelines typically consist of the following key components: CI involves automatically building and testing code changes as they are committed to a version control system. This ensures that code is always in a buildable and testable state. Continuous Integration (CI): CD extends CI by automatically deploying tested code changes to production environments. This minimizes the time between code changes and deployment, enabling rapid delivery of new features and updates. Continuous Delivery (CD): IaC treats infrastructure as software, enabling automated provisioning and management of infrastructure resources. This ensures that infrastructure is consistently configured and aligned with the application's requirements. Infrastructure as Code (IaC): Continuous monitoring and logging are crucial for tracking the health and performance of deployments. This enables early detection of issues and prompt corrective actions. Monitoring and Logging: Tools and Technologies for Automated Deployment A rich ecosystem of tools and technologies supports automated deployment pipelines. Some popular examples include: Jenkins, GitLab CI/CD, Azure DevOps Pipelines, CircleCI CI/CD tools: Terraform, Ansible, Chef, Puppet Infrastructure as Code tools: Ansible, Chef, Puppet, SaltStack Configuration management tools: Prometheus, Grafana, Splunk, ELK Stack Monitoring and logging tools: Building and Implementing Automated Deployment Pipelines The process of building and implementing automated deployment pipelines typically involves the following steps: Determine the stages of the deployment pipeline, such as development, testing, staging, and production. Define deployment stages: Integrate CI tools to automatically build and test code changes as they are committed. Automate code building and testing: Configure CD tools to automatically deploy tested code changes to a staging environment for final validation. Automate deployment to staging: Utilize IaC tools to automate the provisioning and configuration of infrastructure resources. Automate infrastructure provisioning: Integrate monitoring and logging tools to provide continuous visibility into deployment health and performance. Implement continuous monitoring: Establish automated rollback and recovery procedures to handle deployment failures. Automate rollback and recovery: Continuously refine and improve the deployment pipeline based on experience and feedback. Continuous improvement: By following these steps and leveraging appropriate tools and technologies, organizations can effectively implement automated deployment pipelines, enabling rapid, reliable, and consistent software delivery. Conclusion Automated deployment strategies have completely transformed the landscape of software development empowering organizations to achieve progress and bring value to users at an unprecedented pace and effectiveness. By automating the process of deploying software organizations can eliminate the potential for mistakes, reduce downtime and guarantee an experience, across different environments. The key takeaways from this article are as follows: Automated deployment strategies encompass a range of techniques that automate the process of releasing new software versions to production environments. Different deployment strategies, such as rolling deployment, blue-green deployment, canary deployment, and A/B testing deployment, offer unique advantages and considerations tailored to specific deployment scenarios and application requirements. Automated deployment pipelines are the backbone of modern software delivery, enabling organizations to streamline the process of releasing new features and updates. Implementing automated deployment pipelines requires careful planning, selection of appropriate tools and technologies, and continuous improvement based on experience and feedback. Automated deployment strategies are not just a trend; they are an essential component of modern software development practices. By embracing automation, organizations can unlock the full potential of agile methodologies and deliver software that meets the ever-changing demands of today's digital world. For further learning and exploration, consider the following resources: Continuous Delivery: Reliable Software Releases in the Age of DevOps: Here The Phoenix Project: A Novel About IT Operations: Here

Efficient Strategies for Automated Deployment

I sometimes surprise fellow technologists when I tell them that a successful digital payments company we built runs on . FreeBSD It’s not that they haven’t heard of FreeBSD—the open source operating system turns 30 this year, after all. Rather, it’s that they hadn’t previously thought of FreeBSD as an enterprise OS. But that’s exactly what it is and how we use it: FreeBSD is a critical enterprise-grade OS that is on par with—and in some ways surpasses—other operating systems when working in enterprise environments. I’ve been a FreeBSD user for 16 years now. Like many others, I got started as a hobbyist. I’d been working with various OSes in enterprise environments, mostly in the payments industry, and getting frustrated with some of the drawbacks of other well-known options: random error codes, endless support tickets, black boxes. A colleague tipped me off to FreeBSD, and I immediately loved it for its simplicity and capabilities. We rebuilt that company on the FreeBSD stack and the project was highly successful. When we launched our digital payments company, we built it on FreeBSD and never looked back; it is still our OS of choice today. We have nearly 60 servers running across two different U.S. data centers. We run on an active-active architecture, with our own IPV4 blocks that we Anycast. We operate several highly segmented VLANs for isolation and security purposes. And we do all of this with FreeBSD. One reason this surprises some people is that, as a payments company, we handle millions of credit card and bank account numbers. To put it in simple terms: a data breach would be extraordinarily bad for us. We do intensive pen testing and vulnerability scanning on a daily basis to make sure we’re as strong as possible. We regularly undergo security and compliance assessments, which we pass with flying colors. And in the process, I’ve spent many hours talking with auditors, InfoSec pros, CIOs, CISSPs, and others in the industry to make the business case for FreeBSD and teach with them what we’ve learned first-hand: FreeBSD is a remarkably powerful, reliable enterprise OS that can stand up to the most stringent security standards and regulatory scrutiny. Below, I’ll unpack five fundamentals behind why FreeBSD is our enterprise OS of choice—which can double as five fundamentals for making the business case for FreeBSD in your own organization. Using FreeBSD as an enterprise OS: 5 things to know Security & Stability One of the best features of FreeBSD—along with all of the products in its ecosystem—is that it’s a completely free and open source project. Historically, open source sometimes got a bad rap in enterprise environments, but that is changing as IT pros recognize that, when properly configured and managed, open source software can be just as secure (or even more so) than proprietary code. A key reason for this is that open source eliminates all of the black boxes. You can look under the hood and see everything that is going on; there are no mysteries. Even more importantly, there are lots of eyes on the code, which means issues tend to be discovered and mitigated much faster, especially where there is a longstanding, engaged community around the project (like there is with FreeBSD). Moreover, I’ve never seen another OS that has such a consistent behavior from release-and-patch cycle to release-and-patch cycle. With FreeBSD, we never encounter the common scenario where you install an update or new version and, suddenly, a bunch of tools stop working because an underlying library changed. FreeBSD’s base install also runs a very minimal set of services that are exposed to the outside world, which supports the many security frameworks that emphasize zero-trust or least-privilege principles by only running what is actually necessary. It’s easy to add what you need, while not needing to worry about uninstalling or disabling a bunch of things that you don’t. Similarly, it’s very easy to tune and configure everything from how your network stack runs to how your intrusion detection runs and more. The documentation is great; again, no black boxes or endless support tickets. The open source nature also helps deliver excellent performance, minimizing calls and other potential drags on your environment. Finally, from a security and stability standpoint, there’s separation between that base OS and everything else that gets installed on that OS. High Availability High availability (HA) is a hot topic and an increasingly common requirement in many sectors. In the past, I’d never been able to actually achieve it with other OSes without purchasing expensive firewalls and routing equipment. We built our active-active, multi-homed, Anycast-ed environment more or less using out-of-the-box FreeBSD. Even our firewalls are FreeBSD. The only third-party application we’ve had to add is BIRD for BGP routing. After two years, we have had zero downtime. We’ve load-tested the current configuration to 20,000 concurrent credit card transactions – and that didn’t even stretch the platform to its outer limits. And we’ve done this essentially with simple hardware and out-of-the-box FreeBSD. Patching This is probably one of the most compelling reasons to consider using FreeBSD in an enterprise environment—and one of my favorite aspects to talk about as a result. I am regularly asked some version of the question: “Wait, you’re using FreeBSD as the OS for running a firewall?” And my answer is always: “Yes, absolutely.” The reality is that a lot of auditing and assessment firms and CISSPs are used to seeing the brand-name OSes. There’s nothing wrong with that, per se, but I like to share my belief that many of those platforms are using a lot of open source—like FreeBSD and its PF firewall—under the hood. They’re wrapping their own UI and code around it, of course, but essentially it’s the same thing we do on our own. And what happens is that when we run things like pkg audit nightly to check our systems against CVEs and other known issues, we can patch those vulnerabilities much faster than would be possible if we had to wait on a commercial OS vendor. When a vulnerability becomes known, we have a fresh, patched build up and running soon after. Configuration Management Nobody loves the mysterious checkbox UI style of configuration management. It’s another form of black box, in that you don’t really know what’s going on in the background. For this reason, I love the simplicity of configuring FreeBSD. It’s all flat files. You don’t have to log into a UI to make changes. FreeBSD’s file-based configuration arms you with a persuasive case in terms of security and auditing because it creates a simple but powerful change management system. Just about any security and assessment framework requires some type of change management process in place. File-based configuration makes it simple to GIF those file configurations, or use version control on them, inherently adding change management to your configuration management. If you want to take it to the next level, we use SaltStack for centralized automation and orchestration of our configuration management. There is not a single manually created or updated configuration file on any server in our environment; everything is done via SaltStack and Git, which makes it easy for us to track configuration management and change management throughout the environment. That gives us a centralized audit trail of any change made on any system. Compliance Management Regulatory compliance is a major requirement for us as a payments company, as it is for other firms in the banking and financial services sector (as well as most industries now). We regularly work with PCI DSS and the NIST Cybersecurity Framework. In our experience, frameworks like these are heavily tuned toward one of the major commercial operating systems[, and, as a result, many of the auditors and assessors are more experienced with that OS as well. That’s created a learning curve when doing something like a PCI level 1 assessment or a NIST audit, where we’ve basically had to teach the assessors all of the above, because they weren’t familiar with how FreeBSD gets configured and secured. (As a result, there are now some vendors who know very well how it works!) If you do work with a vendor that doesn’t know FreeBSD, the first assessment or audit will likely be more painful than future ones, because you’ll have to help them navigate that learning curve. But here’s the great news: Every time we’ve completed a course with a security assessor who wasn’t previously familiar with FreeBSD, by the end of it they always become convinced FreeBSD is the best OS from the security, compliance, patching, and configuration perspectives. Seriously. And successfully completing those audits and assessments is ultimately what qualifies an OS as “enterprise-ready,” and I can say emphatically that FreeBSD exceeds that bar. That’s what I love sharing our experience and what we’ve learned—because inevitably that might help be a catalyst for helping our technologists and teams reap the benefits of such a great open source OS.

5 Reasons We Use Open Source FreeBSD as Our Enterprise OS

In this article, we’ll dive into bytecode, and examine the bytecode generation and constant pool in detail. 1. Bytecode Generation The GenerateUnoptimizedCodeForToplevel is responsible for transforming an AST tree into bytecodes. 1.  MaybeHandle<SharedFunctionInfo> GenerateUnoptimizedCodeForToplevel(
2.      Isolate* isolate, ParseInfo* parse_info, AccountingAllocator* allocator,
3.      IsCompiledScope* is_compiled_scope) {
4.  //omit............
5.     while (!functions_to_compile.empty()) {
6.  //omit............
7.       if (job->ExecuteJob() == CompilationJob::FAILED ||
8.           FinalizeUnoptimizedCompilationJob(job.get(), shared_info, isolate) ==
9.               CompilationJob::FAILED) {
10.           return MaybeHandle<SharedFunctionInfo>();
11.         }
12.         if (FLAG_stress_lazy_source_positions) {
13.           SharedFunctionInfo::EnsureSourcePositionsAvailable(isolate, shared_info);
14.         }
15.         if (shared_info.is_identical_to(top_level)) {
16.           *is_compiled_scope = shared_info->is_compiled_scope();
17.           DCHECK(is_compiled_scope->is_compiled());
18.         }
19.       }
20.       parse_info->ResetCharacterStream();
21.       return top_level;
22.     } Line 7 reads an AST tree from parse_info and generates bytecode. By debugging it, we step into the following function which analyses every AST node and generates a corresponding bytecode. job->ExecuteJob() 1.  void BytecodeGenerator::GenerateBytecodeBody() {
2.    VisitArgumentsObject(closure_scope()->arguments());
3.    VisitNewTargetVariable(closure_scope()->new_target_var());
4.    FunctionLiteral* literal = info()->literal();
5.    if (IsResumableFunction(literal->kind())) {
6.      BuildGeneratorObjectVariableInitialization();
7.    }
8.    if (FLAG_trace) builder()->CallRuntime(Runtime::kTraceEnter);
9.  //省略.........................
10.    BuildIncrementBlockCoverageCounterIfEnabled(literal, SourceRangeKind::kBody);
11.    VisitDeclarations(closure_scope()->declarations());
12.    VisitModuleNamespaceImports();
13.    builder()->StackCheck(literal->start_position());
14.    if (IsBaseConstructor(function_kind())) {
15.      if (literal->requires_brand_initialization()) {
16.        BuildPrivateBrandInitialization(builder()->Receiver());
17.      }
18.      if (literal->requires_instance_members_initializer()) {
19.        BuildInstanceMemberInitialization(Register::function_closure(),
20.                                          builder()->Receiver());
21.      }
22.    }
23.    VisitStatements(literal->body());
24.    if (!builder()->RemainderOfBlockIsDead()) {
25.      builder()->LoadUndefined();
26.      BuildReturn();
27.    }
28.  } In the above function, lines 11, 12, and 23 first analyze the declaration, import modules, and statements respectively, and then generate corresponding bytecodes. In our case, we will examine line 11 VisitDeclarations() in more depth. In VisitDeclarations(), the following function is called. 1.  BytecodeArrayBuilder& BytecodeArrayBuilder::LoadConstantPoolEntry(
2.      size_t entry) {
3.    OutputLdaConstant(entry);
4.    return *this;
5.  }
6.  //============================
7.  #define DEFINE_BYTECODE_OUTPUT(name, ...) In line 3, OutputLdaConstant stores one global constant into the constant-pool and generates bytecode LdaGlobal for loading this constant, so that we can get out the constant through the LdaGlobal during execution. About the constant pool, I will talk later. By debugging OutputLdaConstant(), we step into the EmitBytecode() which generates one bytecode for a corresponding statement and also is the function that generates the LdaGlobal for our global constant. 1.  void BytecodeArrayWriter::EmitBytecode(const BytecodeNode* const node) {
2.  //omit..........
3.     for (int i = 0; i < operand_count; ++i) {
4.       switch (operand_sizes[i]) {
5.         case OperandSize::kNone:
6.           UNREACHABLE();
7.           break;
8.         case OperandSize::kByte:
9.           bytecodes()->push_back(static_cast<uint8_t>(operands[i]));
10.           break;
11.         case OperandSize::kShort: {
12.           uint16_t operand = static_cast<uint16_t>(operands[i]);
13.           const uint8_t* raw_operand = reinterpret_cast<const uint8_t*>(&operand);
14.           bytecodes()->push_back(raw_operand[0]);
15.           bytecodes()->push_back(raw_operand[1]);
16.           break;
17.         }
18.         case OperandSize::kQuad: {
19.           const uint8_t* raw_operand =
20.               reinterpret_cast<const uint8_t*>(&operands[i]);
21.           bytecodes()->push_back(raw_operand[0]);
22.           bytecodes()->push_back(raw_operand[1]);
23.           bytecodes()->push_back(raw_operand[2]);
24.           bytecodes()->push_back(raw_operand[3]);
25.           break;
26.         }
27.       }
28.     }
29.   } From lines 4 to 25, calculate the operand size for our LdaGlobal, because in execution, a bytecode needs the operand size to operate the operand correctly. Back to GenerateBytecodeBody(), in line 23, the VisitStatements() reads one AST node that represents a corresponding JavaScript statement, and generate one bytecode. If one AST node represents a JavaScript statement block, the VisitStatements() will recursively call itself to generate many bytecodes. Note: void BytecodeGenerator::VisitStatements(
    const ZonePtrList<Statement>* statements) {
  for (int i = 0; i < statements->length(); i++) {
    // Allocate an outer register allocations scope for the statement.
    RegisterAllocationScope allocation_scope(this);
    Statement* stmt = statements->at(i);
    Visit(stmt);
    if (builder()->RemainderOfBlockIsDead()) break;
  }
} Figure 1 shows the VisitStatements’s call stack. 2. Constant Pool function test(x)
{
 if(x>0)
 add3(x);//x=x+3
 dosomething........
} When compiling the function test(), the compiler infers that the add3 will be called, and the add3 address should be pushed into the constant pool. But, there’s a question here. We have no address if the add3() is not compiled yet. In this case, the compiler reserves a slot for the add3 in the constant pool and fills it once the add3() is compiled, namely DeferredConstants. Let’s give a glance at FinalizeBytecode(). 1.  Handle<BytecodeArray> BytecodeGenerator::FinalizeBytecode(
2.      Isolate* isolate, Handle<Script> script) {
3.    DCHECK_EQ(ThreadId::Current(), isolate->thread_id());
4.    AllocateDeferredConstants(isolate, script);
5.    if (block_coverage_builder_) {
6.      info()->set_coverage_info(
7.          isolate->factory()->NewCoverageInfo(block_coverage_builder_->slots()));
8.      if (FLAG_trace_block_coverage) {
9.        info()->coverage_info()->Print(info()->literal()->GetDebugName());
10.      }
11.    }
12.    if (HasStackOverflow()) return Handle<BytecodeArray>();
13.    Handle<BytecodeArray> bytecode_array = builder()->ToBytecodeArray(isolate);
14.    if (incoming_new_target_or_generator_.is_valid()) {
15.      bytecode_array->set_incoming_new_target_or_generator_register(
16.          incoming_new_target_or_generator_);
17.    }
18.    return bytecode_array;
19.  } In line 4, AllocateDeferredConstants creates the DeferredConstants. 1.  void BytecodeGenerator::AllocateDeferredConstants(Isolate* isolate,
2.                                                    Handle<Script> script) {
3.    // Build global declaration pair arrays.
4.    for (GlobalDeclarationsBuilder* globals_builder : global_declarations_) {
5.      Handle<FixedArray> declarations =
6.          globals_builder->AllocateDeclarations(info(), script, isolate);
7.      if (declarations.is_null()) return SetStackOverflow();
8.      builder()->SetDeferredConstantPoolEntry(
9.          globals_builder->constant_pool_entry(), declarations);
10.     }
11.     // Find or build shared function infos.
12.     for (std::pair<FunctionLiteral*, size_t> literal : function_literals_) {
13.    //omit..................
14.     }
15.     // Find or build shared function infos for the native function templates.
16.     for (std::pair<NativeFunctionLiteral*, size_t> literal :
17.          native_function_literals_) {
18.   //omit..................
19.     }
20.     // Build object literal constant properties
21.     for (std::pair<ObjectLiteral*, size_t> literal : object_literals_) {
22.       ObjectLiteral* object_literal = literal.first;
23.   //omit..................
24.       }
25.     }
26.     // Build array literal constant elements
27.     for (std::pair<ArrayLiteral*, size_t> literal : array_literals_) {
28.   //omit..................
29.     }
30.     // Build class literal boilerplates.
31.     for (std::pair<ClassLiteral*, size_t> literal : class_literals_) {
32.   //omit..................
33.     }
34.     // Build template literals.
35.     for (std::pair<GetTemplateObject*, size_t> literal : template_objects_) {
36.   //omit..................
37.     }
38.   } From lines 4 to 10, the loop-for store global variables in the constant pool. From lines 11 to 35 store the sharedfunction, constant property, and other stuff in the constant pool. 3. FinalizeBytecode The FinalizeBytecode() is the last step for generating bytecode. Let’s give a glance at line 13 of the FinalizeBytecode, the reorganizes the bytecodes and returns a bytecode array that holds a bytecode array representing your JavaScript code. What is “reorganize”? It is that just put bytecode, constant pool, etc. together because all the stuff is already done well before ToBytecodeArray. builder()->ToBytecodeArray 1.  Handle<BytecodeArray> BytecodeArrayBuilder::ToBytecodeArray(Isolate* isolate) {
2.    DCHECK(RemainderOfBlockIsDead());
3.    DCHECK(!bytecode_generated_);
4.    bytecode_generated_ = true;
5.    int register_count = total_register_count();
6.    if (register_optimizer_) {
7.  //omit
8.    }
9.    Handle<ByteArray> handler_table =
10.        handler_table_builder()->ToHandlerTable(isolate);
11.    return bytecode_array_writer_.ToBytecodeArray(
12.        isolate, register_count, parameter_count(), handler_table);
13.  } Line 9 gets out the handler_table, namely the dispatch_table that I have talked about. Line 11 stores the handler_table address into a bytecode array. Remember that each bytecode has only one handler that is the actual worker and is held by the dispatch_table. Let’s give a glance at line 11 bytecode_array_writer_.ToBytecodeArray. 1.  Handle<BytecodeArray> BytecodeArrayWriter::ToBytecodeArray(
2.      Isolate* isolate, int register_count, int parameter_count,
3.      Handle<ByteArray> handler_table) {
4.    DCHECK_EQ(0, unbound_jumps_);
5.    int bytecode_size = static_cast<int>(bytecodes()->size());
6.    int frame_size = register_count * kSystemPointerSize;
7.    Handle<FixedArray> constant_pool =
8.        constant_array_builder()->ToFixedArray(isolate);
9.    Handle<BytecodeArray> bytecode_array = isolate->factory()->NewBytecodeArray(
10.        bytecode_size, &bytecodes()->front(), frame_size, parameter_count,
11.        constant_pool);
12.    bytecode_array->set_handler_table(*handler_table);
13.    return bytecode_array;
14.  } Line 7 generates constant_pool, and line 9 generates a bytecode array that the source code is below: 1.  Handle<BytecodeArray> Factory::NewBytecodeArray(
2.      int length, const byte* raw_bytecodes, int frame_size, int parameter_count,
3.      Handle<FixedArray> constant_pool) {
4.    DCHECK_LE(0, length);
5.    if (length > BytecodeArray::kMaxLength) {
6.      isolate()->heap()->FatalProcessOutOfMemory("invalid array length");
7.    }
8.    // Bytecode array is AllocationType::kOld, so constant pool array should be
9.    // too.
10.    DCHECK(!Heap::InYoungGeneration(*constant_pool));
11.    int size = BytecodeArray::SizeFor(length);
12.    HeapObject result = AllocateRawWithImmortalMap(size, AllocationType::kOld,
13.                                                   *bytecode_array_map());
14.    Handle<BytecodeArray> instance(BytecodeArray::cast(result), isolate());
15.    instance->set_length(length);
16.    instance->set_frame_size(frame_size);
17.    instance->set_parameter_count(parameter_count);
18.    instance->set_incoming_new_target_or_generator_register(
19.        interpreter::Register::invalid_value());
20.    instance->set_osr_loop_nesting_level(0);
21.    instance->set_bytecode_age(BytecodeArray::kNoAgeBytecodeAge);
22.    instance->set_constant_pool(*constant_pool);
23.    instance->set_handler_table(*empty_byte_array());
24.    instance->set_source_position_table(*undefined_value());
25.    CopyBytes(reinterpret_cast<byte*>(instance->GetFirstBytecodeAddress()),
26.              raw_bytecodes, length);
27.    instance->clear_padding();
28.    return instance;
29.  } Line 12 allocates memory from the V8 heap, and then lines 15–25 copy the bytecode array into the memory, and line 28 returns the bytecode finally. Takeaways Constant pool holds constants, function addresses, etc. If a variable name is known, it as a constant is stored in the constant pool; Note: Constant pool always is at bytecode array[0]. In other words, given a bytecode array, the first element of the array is always the constant pool; A bytecode array is a heap object and its type is FixedArray. Okay, that wraps it up for this share. I’ll see you guys next time, take care! Please reach out to me if you have any issues. : qq9123013 : WeChat Email v8blink@outlook.com Also Published here

Understanding Chrome V8 - Chapter 22: Compiler Workflow Bytecode, Constant Pool

DevOps is a term, wide-reaching and applicable to many things that surround and support software development. big For the uninitiated, DevOps can be hard to understand. Is it a culture? A guideline for development processes? A set of tools? Someone's job? The answer to all these questions is yes. It's all the above. In this post, we take a surface-level look at the parts that make up DevOps. We explore the concepts, tools, and unique roles. DevOps as a concept In modern development, it's more important than ever to deliver software and updates as quickly and reliably as possible. The concept behind DevOps, then, is about removing barriers that get in the way of software delivery. Typical barriers include: Manual processes Office politics Counteractive support workflows The reason to overcome these barriers is simple: If the product is software, then all processes should work towards delivering that software, not against it. Let's look at what we must consider when thinking about a DevOps approach. DevOps as a culture In traditional organizations, developers and those that support them (such as operations or database teams) are often separate entities with little interaction. This means each team has its own beliefs, responsibilities, and priorities. While not at odds, often those priorities can clash in pursuit of doing the best for the business. Embracing DevOps as a culture means to remove that friction by introducing a shared purpose. The methods to achieve that can differ and depend on how entrenched people are in their ways of working. The aim, however, is to ensure all teams: Build trust through clear, honest communication and feedback Collaborate throughout the product's entire lifecycle Have what they need to make quick decisions Can take risks without fear of failure or blame Are collectively responsible for the product's success Review what worked and what didn't for the next lifecycle Why automation is important for DevOps In DevOps, if you can automate it, you should automate it. The benefits of automation are simple: Teams can focus more on the product Results are repeatable and predictable Teams can respond much quicker and flexibly to problems With that, it's no surprise that Continuous Integration and Continuous Deployment (CI/CD) are such a huge part of DevOps. DevOps in action: different approaches to DevOps Given DevOps' conceptual nature, there are many ways for organizations to approach its adoption. Alex Yates highlighted 2 approaches worth revisiting in his piece . On the naming of "DevOps Engineers" CALMS (Culture, Automation, Lean, Measurement, Sharing) CALMS is a framework explored in . The DevOps Handbook CALMS is an acronym where each letter describes the actions needed to adopt DevOps: - Remove silos and share responsibility Culture - Reduce time spent on manual tasks Automation - Streamline processes to reduce wasted time Lean - Collect and review data to find areas for improvement Measurement - Open and honest collaboration between teams Sharing CALMS is also the approach Atlassian took on its path to DevOps culture, using it to measure progress and success. You can read . how Atlassian uses CALMS on their DevOps site The Three Ways (flow, feedback, continual experimentation and learning) Featured in and DevOps novels and , The Three Ways boils DevOps down to 3 key principles: The DevOps Handbook The Phoenix Project The Unicorn Project Flow Feedback Continual experimentation and learning Let's take a quick look at what these principles mean. The First Way: Flow The First Way is about refining every process that takes place between the developer and the customer. This means: Not trying too much during one lifecycle by focusing on short sprints Removing arbitrary processes and giving the team what they need to keep things moving The Second Way: Feedback loops The Second Way is all about faster feedback. Faster feedback means faster reactions, so you can: Address problems before they become bigger problems deeply rooted due to iteration Troubleshoot with less reverse engineering Adjust processes to better meet the needs of customers and your teams The Third Way: Experiment and learn The Third Way is about recognizing that it's okay to take risks and that failure is an important part of learning. It's also about communicating the wins too. By recognizing this, it allows your teams to: Think outside the box Learn from mistakes Test the resilience of your DevOps culture DevOps roles Given the concept is about a unifying purpose, DevOps is really everyone's role. That said, it's still important everyone knows their responsibilities. Adopting DevOps means adding some specialist roles that sit alongside development staples like coders, QA, designers, and more. Let's take a quick look at some of the common extra roles needed in DevOps and what they do. Some of the naming conventions and finer responsibilities may differ between organizations. DevOps Engineer The exact function of DevOps Engineers is often debated. The consensus, though, is that the role exists so everyone else can perform theirs as smoothly as possible. Both conductor and problem solver, a DevOps Engineer oversees the bigger picture, managing: Workflows Infrastructure setup and config (including scaling) System permissions Needs of various teams Build Manager The Build Manager maintains the automation systems that make up Continuous Integration (CI). This means ensuring code gets compiled, built, tested, and handed off for deployment. Release Manager A Release Manager directs builds promoted for release. They communicate what's included in a release and plot their course through a pipeline's environments. This makes up what we call Continuous Deployment and Continuous Delivery (CD) Product Manager Product Managers are almost a conduit between developers and end users. Working with developers, they help ensure the product has the features and fixes customers’ need. Data Analyst A Data Analyst scours data to spot patterns and find areas for improvement. This helps you spot the things that impact user experiences, such as feature problems or product navigation. Example DevOps tools Most DevOps approaches agree a product lifecycle should look something like this: Plan Code Build Test Package Release Deploy Operate Monitor Rinse and repeat! It's important to find the right tools to help you manage each phase. Let's look at a few examples. Planning The planning stages are the most important for the direction and future of your product. Effective planning helps you focus on the improvements and features users get at the end of the next lifecycle. Given a lot of that process can get pretty conceptual, there's a range of tools that allow collaboration and project management. and are the main tools, but other popular options include: Slack Zoom Teams Confluence monday.com Trello Code repositories and source control Source control is important to any development team. Source control tracks and checks every new piece of code and file change. Most developers use for source control. Git is both a system and philosophy, allowing for distributed development and helping avoid risk through branching. Git Code repositories, then, are hosting services for Git-managed code. Popular options include: GitHub GitLab Bitbucket Build and test Build servers (also known as CI platforms) can save time by automating: Code compiling Code validation tests Package creation Popular options include: Jenkins GitHub Actions TeamCity Circle CI Atlassian Bamboo Azure DevOps We spent the first quarter of 2022 deep-diving into . Take a read and see if they're for you. Jenkins and GitHub Actions Package Software packaging tools turn your code into deployable artifacts. You host and deploy these artifacts from package repositories. Popular options include: Docker Hub JFrog Codefresh Red Hat Quay Releases and deployments Most code repositories and build servers let you manage releases and deploy to targets in some fashion. However, they don't solve the same problems dedicated release management or deployment tools do. Other popular options include: Atlassian Bamboo AWS CodeDeploy GitLab CI/CD DeployHQ ElectricFlow Operations Operations relates to the setup, running, and maintenance of infrastructure for a pipeline. Popular options include: Saltstack Chef Puppet Enterprise are also an important feature related to operations. Runbooks Monitoring Monitoring tools scrape your product and related systems for important data. This can inform decisions about performance and customer usage. Popular options include: New Relic Splunk Dynatrace Datadog Happy deployments! This article was first published here.

An Introduction to DevOps 

Every few years, we change browsers, and our web usage evolves, sometimes for the better, sometimes for the worse. The most recent switch to Google Chrome has had the greatest impact on daily web usage, owing to the incredible array of extensions. The browser is also light and stylish, with a variety of cool themes. Let’s take a look at some SEO chrome extensions. Google SEO Chrome Extensions, like Android Apps, are full of garbage with a few gold nuggets hidden amongst them. What are Chrome Extensions Extensions are small software programs that allow you to personalize your browsing experience. They allow users to customize Chrome functionality and behavior to meet their specific needs or preferences. An extension must serve a single purpose that is well-defined and simple to understand. As long as everything contributes to a common goal, a single extension can include different features and a wide range of functionality. What is SEO is an abbreviation for ‘Search Engine Optimization,’ and it refers to the process of obtaining traffic from free, organic, editorial, or natural search results in search engines. It aims to boost your website’s ranking on search engine results pages. Remember that the higher the website ranks in the search results, the more people will see it. A good SEO strategy includes a variety of activities, SEO Identifying relevant keywords with a high potential for search traffic. Creating high-quality, useful content that is optimized for search engines and users. Incorporating pertinent links from high-quality websites. Measuring the results. List of SEO Chrome Extensions SEO is a wide field, and it involves multiple aspects that need to be taken care of such as keyword research, link building, rank checking, and many more and there is an extension for each category. Here are the 10 most favorite SEO chrome extensions to increase your productivity. GrowthBar GrowthBar is a simple-to-use but potent SEO chrome extension tool. It is an easy-to-use Chrome extension that brings all of the features of an SEO tool suite right into your browser. Backlink analysis, keyword research, and rank tracking are all possible with it. It also provides you with actionable insights to help your online business succeed. Open SEO Stats Open SEO Stats, also known as PageRank Status, aids in gaining access to SEO information. It displays AlexaRank and Google PageRank and provides access to Whois, Geo IP Location, indexed pages, backlinks, and other information. This extension supports 30 languages and provides statistics on-page and link traffic, as well as page speed. It does not provide mobile testing. SEOStack Keyword Tool Scrapes autocomplete suggestions from Google, Youtube, Bing, Yahoo, Amazon, and eBay to generate thousands of keyword ideas from a seed keyword. If you’ve ever used Keyword Sheeter, you’ll recognize the concept. All data can be exported as a CSV file. SEO Peek SEO Peek is designed to meet all of your SEO requirements. It offers a snapshot of any website’s SEO factors. Provides a clear understanding of the various robot directives, HTTP status, Meta robots, and other annotations, as well as displaying relevant content elements. Because it is simple to use and understand, SEO Peek assists in gathering all information about a website’s SEO status and displaying any work that may have been overlooked by your clients. SEOInfo While navigating, SEOInfo alerts you to errors such as invalid canonical links, invalid AMP pages, invalid HRefLangs tags, pages or resources blocked by robots.txt, errors in images, CSS, or javascript files, mobile compatibility, and so on. It also includes an Audit module that makes use of Google’s Pagespeed Insights API. SEO Search Simulator by NightWatch It simulates Google searches from any location and determines whether a specific URL ranks in the top 100 results. It’s a great way to see how rankings for a query differ in different parts of the world. Nightwatch search simulator simplifies work for any SEO professional who wants to use a simplified solution for rank tracking to check website positions in SERPs (search engine response pages). Google Lighthouse Lighthouse is an open-source, automated tool for improving web page quality. You can use it on any web page, whether it is public or requires authentication. Audits your page and makes recommendations for improvements in performance, accessibility, developer best practices, and, of course, SEO. MozBar MozBar is one of the SEO tools that display SEO metrics. It takes up very little space in the browser and allows the user to create custom searches as well as check for the difficulty of ranking of the given keyword also displays page elements with the Analyze Page overlay, validates and previews markup, schema.org, and authorship. Also looks at social metrics for Google+, Twitter, and Facebook. SEO Analysis & Website Review by Woo Rank Woorank chrome extension provides a plethora of useful SEO tips for your website to help you improve your search engine rankings. It provides a detailed analysis of both on-page and off-page factors, allowing you to access the most up-to-date information when and where you want it. Alexa Traffic Rank Alexa Traffic Rank is a free SEO Chrome extension from Alexa Internet. Alexa Traffic Rank follows you as you surf, providing Alexa.com data about the sites you visit without interfering with your experience. Extension stipulates, Alexa Traffic Rank Traffic Rank in Country Sites Linking In Similar Sites Average Load Time Search Analytics Also Published here

10 Chrome Extensions to Transform Your SEO Game

, , , and similar items are very essential tools to enhance the user experience on an application regardless of platform. Snack Bars Alert Dialogs ProgressIndicators Intro Welcome to another exciting blog, this is the 8th part of the series . So, far we made , some like the app bar and bottom nav bar, and also implemented a global . In the last two sections, we created and wrote the for registration and sign-in methods to use with the Firebase project. In this section, we'll use flutters' features to give users visual feedback. Flutter App Development Tutorial Splash Screen global widgets theme UI backend You can find the source code . here Show Progress Indicator During Registration/Sign In Head over to the class and make some changes. auth_state_provider Create an enum outside of class, to toggle the Application Process State State. // Outside of any class or function
// Make an enum to togggle progrss indicator
enum ProcessingState {
  done,
  waiting,
} Inside of Provider class let's create a field of type and a  function that switches these values/states. ProcessingState ProcessingState _processingState = ProcessingState.done;
  // getter
  ProcessingState get processingState => _processingState;

  void setPrcState(ProcessingState prcsState) {
    _processingState = prcsState;
    notifyListeners();
  } We'll display the CircularProgressIndicator whenever the application is busy and remove it when done. For instance, after pressing the register/sign-in button we can show the progress indicator in place of the button and then remove it when firebase sends a response. So, let's first start by adding the function inside the register function. We'll make changes inside auth_form_widget files after this. Update Processing State In Register Function // Our Function will take email,password, username and buildcontext
  void register(String email, String password, String username,
      BuildContext context) async {
    // Start loading progress indicator once submit button is hit
    // #1
    setPrcState(ProcessingState.waiting);
    try {
      // Get back usercredential future from createUserWithEmailAndPassword method
      UserCredential userCred = await authInstance
          .createUserWithEmailAndPassword(email: email, password: password);
      // Save username name
      await userCred.user!.updateDisplayName(username);

      // After that access "users" Firestore in firestore and save username, email and userLocation
      await FirebaseFirestore.instance
          .collection('users')
          .doc(userCred.user!.uid)
          .set(
        {
          'username': username,
          'email': email,
          'userLocation': null,
        },
      );
      // if everything goes well user will be registered and logged in
      // now go to the homepage
      GoRouter.of(context).goNamed(APP_PAGE.home.routeName);
    } on FirebaseAuthException catch (e) {
      // In case of error
      // if email already exists
      if (e.code == "email-already-in-use") {
        print("The account with this email already exists.");
      }
      if (e.code == 'weak-password') {
        // If password is too weak
        print("Password is too weak.");
      }

    // #2
      setPrcState(ProcessingState.done);
    } catch (e) {
      // For anything else
      print("Something went wrong please try again.");
      // #3
      setPrcState(ProcessingState.done);
    }
    // notify listeneres
    notifyListeners();
  } In the registration function, we'll start to toggle the ProcessState to (which's by default) right before the Try block. waiting done We'll only switch it back to once the servers send an error response, at the end of the Catch block. We'll not switch it back at the end of the try block because, in case of success, we'll move on to the next page as well as display a Snackbar which we'll implement after this section. done Please make similar changes to the and the functions by yourself. Sign In Log Out Now, we'll need to tell the application, that when the processing state is in , display a progress indicator, and then remove the indicator once the processing state is done. To do so let's head over to . Right after we instantiate an AuthStateProvider, create a new variable of the type ProcessState whose value is equal to that of AuthStateProvider's process state. waiting auth_form_widget Widget build(BuildContext context) {
    final AuthStateProvider authStateProvider =
        Provider.of<AuthStateProvider>(context);

    // make new ProcessState var
    ProcessingState prcState = authStateProvider.processingState; After that, down where we have our elevation button, let's make it a conditionally rendering widget. if (prcState == ProcessingState.waiting) const CircularProgressIndicator(),
  if (prcState == ProcessingState.done)
              Row(
                mainAxisAlignment: MainAxisAlignment.end,
                children: [
                  TextButton(
                    onPressed: () {},
                    child: const Text('Cancel'),
                  ),
                  const SizedBox(
                    width: 20,
                  ),
                  ElevatedButton(
                    onPressed: () {
                      // call _submitForm on tap
                      _submitForm(authStateProvider, context);
                    },
                    child: Text(registerAuthMode ? 'Register' : 'Sign In'),
                    style: ButtonStyle(
                      elevation: MaterialStateProperty.all(8.0),
                    ),
                  ),
                ],
              ), The progress indicator will activate and deactivate at the right time. Display Information on Snackbar As mentioned earlier, we'll be using to display information like "Authentication Successful", "Welcome Back", etc. To do so let's again create a simple function on AuthStateProvider class. Snackbar // Right after setPrcState function
 //  create function to handle popups
  SnackBar msgPopUp(msg) {
    return SnackBar(
        content: Text(
      msg,
      textAlign: TextAlign.center,
    ));
  } This function will take a custom message and return a SnackBar to display on the screen. Snackbar works in conjunction with class. So, we'll pass this msgPopUp method in at the end of the block operations, and just before we call GoRouter. ScaffoldMessenger ScaffoldMessenger.of(context) try Inside Registration Function ScaffoldMessenger.of(context)
          .showSnackBar(msgPopUp("The account has been registered."));
// Before GoRouter
GoRouter.of(context).goNamed(APP_PAGE.home.routeName); Inside Login Function ScaffoldMessenger.of(context).showSnackBar(msgPopUp("Welcome Back")); With this when the authentication operation succeeds user will see a snack bar at the bottom of the application. Alert Error on Authentication With Alert Dialog The snack bar is useful because it subtly displays short messages. However, because of its subtlety, it is not very appropriate to be used in case of errors. In such cases, it's better to alert users using a dialog pop-up such as the widget. So, right after the msgPopUp function, let's create another function. AlertDialog // #1
  AlertDialog errorDialog(BuildContext context, String errMsg) {
    return AlertDialog(
      title: Text("Error",
          style: TextStyle(
            //text color will be red
            // #2
            color: Theme.of(context).colorScheme.error,
          )),
      content: Text(errMsg,
          style: TextStyle(
            //text color will be red
             // #3
            color: Theme.of(context).colorScheme.error,
          )),
      actions: [
        TextButton(
          // On button click remove the dialog box
           // #2
          onPressed: () => Navigator.of(context).pop(),
          child: const Text('OK'),
        ),
      ],
    );
  } Our method will take two arguments: an error message and a build context. Our alert box will have texts with the color red. On click, the dialog box will be removed from the screen. The build context is required here to call method. Navigator.pop On both registration and sign-in methods, we are only printing the error messages. Now, let's replace those print statements with errorDialog() and pass the same messages into the function. Inside Register Function's Catch Blocks on FirebaseAuthException catch (e) {
      // In case of error
      // if email already exists
      if (e.code == "email-already-in-use") {
        showDialog(
            context: context,
            builder: (context) => errorDialog(
                context, "The account with this email already exists."));
      }
      if (e.code == 'weak-password') {
        // If password is too weak
        showDialog(
            context: context,
            builder: (context) =>
                errorDialog(context, "Password is too weak."));
      }
      setPrcState(ProcessingState.done);
    } catch (e) {
      // For anything else
      showDialog(
          context: context,
          builder: (context) =>
              errorDialog(context, "Something went wrong please try again."));
      setPrcState(ProcessingState.done);
    } This is what the box will look like. Please, add these alerts to your sign-in method by yourself. Summary A few changes were made in AuthStateProvider and AuthFormWidget classes. We created an Enum to address process states. Progress indicators were implemented based on Processing Status. A simple snack bar function was built to display information for users when authentication succeeds. An error handler was created to display error messages on an alert dialog box. Support Us Was this an informative blog? If so, hit the like button, ask us some questions and share this with your colleagues. The next part is going to be on App Permissions and Google Places API. So, to get notifications. subscribe and follow us This is Nibesh Khadka from Khadka's Coding Lounge. We are a freelancing agency that makes websites and mobile applications. Also published . here

Improving the User Experience in Flutter

Creating and Releasing an App with Flutter: Part II

afeStake, a trust minimized middle-layer fostering the decentralization of ETH 2.0 Staking,  uses the underlying consensus protocol HotStuff — a more robust variant of the Byzantine Failed Tolerance (BFT) protocol  that helps improve security and minimize the validators’ slashing odds during ETH 2.0 Staking. S But what is the HotStuff protocol? Is it the same protocol that Facebook considered to achieve consensus with its LIBRA coin? In this post, we want to delve deeper into the HotStuff consensus mechanism, so that ParaState users who use our SafeStake product during ETH 2.0 Staking can learn more about the mechanism helping them to improve their passive income, and why it is part of our efforts at ParaState to bring the best of blockchain technology to our ecosystem. Byzantine Fault Tolerance (BFT): The Origin of HotStuff Byzantine Fault Tolerance (BFT) consensus mechanisms are implemented on blockchains as an alternative to Proof-of-Work (PoW) mechanisms, with the intention of achieving the same security that PoW offers without the need of the high energy consumption of the PoW protocol. Its general operation solves one of the great blockchain problems: the well-known problem of the Byzantine generals. If you want to know more about this topic, you can consult the of Leslie Lamport, Robert Shostak and Marshall Pease “ACM Transactions on Programming Languages and Systems”, July 1982. work In essence, an unfair actor in a decentralized network is considered 'Byzantine.'  One of the great achievements of Satoshi Nakamoto is to develop a consensus protocol, i.e., the PoW mechanism, to solve the Byzantine general problem. In the blockchain application, the solution to the Byzantine Generals problem is achieved by loyal commanders adhering to a pre-established pattern of behavior, known as consensus algorithms. A typical BFT algorithm solves this problem as follows: Each loyal lieutenant counts all the messages he has received from his companions. Some will say to attack, while others would  wait. Not knowing which ones are loyal or not, he chooses the message to attack (what action to take) based on the one that has received the highest number of votes. If all loyal lieutenants behave this way, there is a mathematical guarantee that they will make the right decision as long as less than ⅓ of the members are Byzantines. BFT essentially uses the simple formula “3f+1=x” to determine the ratio the system can tolerate of loyal/disloyal actors that make up the network. Thus, if f is equal to 1, at least 4 minimum nodes (lieutenants) are needed for the BFT consensus mechanism to work. When this protocol was originally conceived, the typical system size was n=4 or n=7, deployed on a local area network. However, the renewed interest in its use in blockchain networks calls for solutions that can scale to much larger s. n Since the introduction of Practical Byzantine Fault Tolerance (PBFT), the first practical solution to replicate BFT in the partial synchrony model, many derivations have been built around BFT based on its two phases, with the practical objective that a stable leader can promote a consensus decision in just two rounds of messaging: A first phase that guarantees the uniqueness of the proposal through the formation of a quorum certificate (QC) made up of (n-f) votes. A second phase ensures that the next leader can convince the replicas to vote for a safe proposal. In this system, as we have pointed out before, the algorithm for a new leader to collect information and propose it to the replicas — called view change * — is the epicenter of replication. But the paradigm-based view switch of the previous two phases is far from straightforward, error-prone, and incurs a significant communication penalty even for moderately sized systems. Even with only authenticators (digital signatures or message authentication codes), the transmission of a new proposal has a quadratic communication footprint [O(n^2)] in PBFT resulting in a scaling challenge in all PBFT-based derivations. Although there are several variations of PBFT implemented by various blockchain platforms, in this post we want to highlight the key characteristics of qBFT and iBFT, considered to be among the most robust within the family of BFT protocols and which has served as the point of reference for SafeStake in choosing HotStuff as its consensus mechanism. Immediate purpose: Only 1 block is proposed at a given chain height. Dynamic validator set Optimal Byzantine Resilience when operating in a partially synchronous network Message Complexity of O(n^2) QBFT is based on the BFT agreement protocol presented in IBFT-Moniz. These networks require ⅔ of the validators to be operational to create blocks and networks with these protocols, in which 3 or less validators can produce blocks but do not guarantee finality when operating in harsh environments. While they generate excellent returns compared to PoW, the time to add new blocks increases as the number of validators increases. The key difference between qBFT and iBFT is that the validators take turns creating the next block within a non-randomized dynamic validator set. iBFT removes chain forking and hence supports transaction finality. Both protocols are more convenient to use in private or consortium blockchain networks. In the case of iBFT, the algorithm is deterministic, leader-based, and optimally resilient, tolerating f faulty processes out of n where n≥3f+1. HotStuff: The First Partially Synchronous BFT Replication Protocol Asynchronous Byzantine Fault Tolerance has become one of the most popular ways to achieve consensus in the distributed ledger technology space. However, some critics argue that they are slow and expensive protocols to run, as well as being self-restricted given the existence of an upper limit on network delay. by researchers Maofan Yin, Dahlia Malkhi, Michael K. Reiter, Ittai Abraham, and Guy Gueta and proposed by VMware Research in March 2018, HotStuff introduces a number of substantial improvements to PBFT and its derivatives: Developed 1.- HotStuff removes some complexities from the asynchronous BFT protocols to make it more scalable, introducing a simple function of the number of nodes in the network instead of a function of the number of nodes squared. In addition, the simplicity of its code allows it to easily adapt to the demands of modern solutions to be deployed on the blockchain, solving the problem of state machine replication (SMR) in a few lines of code. 2.- HotStuff introduces a radical change in the PBFT mesh communication network by using a star communication network, shifting the communication burden to the leader which improves protocol response times. 3.- The protocol merges the view change process* with the normal process, reducing its complexity. 4.- The protocol for a new leader begins with the collection of new view messages from the replicas. The process for starting a new view after receiving a prepared message involves three phases in HotStuff instead of the two originally provided in PBFT: precommit phase, commit phase, and decide phase. 5.- Within the technological component of HotStuff are the threshold signatures to reduce the number of signatures in the consensus protocol. 6.- The last change applied to the PBFT by HotStuff refers to the pipeline of the consensus phases. Since all the phases have an identical structure, pipelining HotStuff can be achieved. You can see that the three confirmation phases of Basic HotStuff have the same structure: other nodes vote on a message, and the leader combines the votes and broadcasts them to other nodes. These phases can be represented in a uniform and segmented way. HotStuff has the advantage of being remarkably simple, due in part to its economy of mechanics: There are only two types of messages and simple rules to determine how each replica is treated. Security is specified through the voting and confirmation rules on the node graphs. SafeStake Leverages HotStuff to Improve Ethereum 2.0 Decentralization With the 'merger' on the horizon for the implementation of Proof-of-Stake in Ethereum, SafeStake is on track to avoid concentrating the staking service provider in the hands of a few operator nodes. To achieve this, its architecture is similar to that of the SSV Network, by using threshold cryptography to split the validation key of the staker/validator into several shares, each of which will be sent to one of the operators. The operator working together performs the relevant staking and validation task for the validators by running the threshold signature scheme. An adversary has to compromise a predefined threshold number of operators to launch a successful attack against the ETH staking service provided by the network. This entire process occurs under the HotStuff consensus protocol, which plays a key role in validating transactions and keeping operators within the system committed to offering the service for as long as possible to avoid penalties or cuts due to inactivity. In HotStuff's three-phase commit, the so-called voting refers to the signature of the threshold on a message from the replicas, which is then sent to the leader. The full signature is generated when the leader receives enough votes. The leader will attach this signature to the next phase message that it sends to the replicas, and the signature will be verified by them. With HotStuff, SafeStake governs the operator networks to determine the message content of the threshold signature scheme among the operator nodes. By implementing HotStuff on SafeStake, a node in the distributed system does not need to confirm the message; if "enough nodes want to make the view switch" before notifying the new leader, instead it can directly switch to the new view and notify the new leader. This not only makes the network more robust, but also generates higher performance than other BFT-based protocol derivations such as PBFT, Tendermint, or Casper, as HotStuff batches multiple operations on each node, thereby mitigating the cost of digital signatures by decision. HotStuff is a Byzantine fault-tolerant consensus protocol that has been gaining respect among the blockchain developer community for its robustness, security, and ease of implementation, which are among the reasons why Facebook's Libra whitepaper considered basing its LibraBFT protocol on HotStuff. Unlike Libra, SafeStake has the goal and intent, in contrast to Facebook's design, to implement open and permissionless HotStuff, which ensures greater decentralization for its ETH 2.0 Staking service. This not only favors the extensive development of the entire SafeStake ecosystem, but also avoids concentrating a lot of resources and attention on the performance and general security of the network. SafeStake has implemented HotStuff with a permissionless mechanism of a dual-chain consensus mode based on Ethereum 2.0 PoS, to ensure a more open and democratic structure, providing more decentralized staking. Thus, in SafeStake node operator and validator roles are separated. Node operator doesn't have to deposit Eth. Only validators need to deposit Eth on Beacon chain (32Eth) or your staking pool (2nd testnet stage for lowering staking threshold) to be a validator. Final Thoughts Despite adding a third phase to the protocol, HotStuff has been shown to achieve high performance. Thanks to its linear O(n) authenticator scheme to drive a consensus decision, the communication costs to reach a consensus are quite low compared to other BFT schemes. For these reasons and the optimistic response capacity together with the linear leader protocol without delay, SafeStake implements HotStuff for chain confirmation of the message content of the threshold signature scheme implemented during the ETH 2.0 staking service. ParaState is aware of the relevance of the Ethereum ecosystem and how vital it is to participate in securing the main network of decentralized applications on the market; so together with SafeStake, we are innovating with the best of available blockchain technology to together build a new decentralized world.

Stories, News & Mentions about SHOTSTACK

Shotstack’s Stories on HackerNoon

Shotstack's News & Mentions