If you&#x27;re seeing this message, that means JavaScript has been disabled on your browser, please enable JS to make <strong>Imgur</strong> work

Interrupção - GIF

Discover topics like unmuted, sound on, stand up comedy, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like countervail247365.

You see the yams on dis guy? - Album on Imgur

Discover topics like music, dump, metal, rock and roll, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like MudcrabMan.

Rock/Metal Dump XII - Album on Imgur

Discover topics like pridemonth, cosplay, pride, trans, cosplay girls, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like ...

Happy Pride - Paxiti Cosplay - Album on Imgur

Discover topics like cyberpunk 2077, evelyn parker, gaming, virtualphotography, evelyn, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much ...

"Cyberpunk 2077" Evelyn Parker - Album on Imgur

Discover topics like memes, unicorn, mildly interesting, wtf, brexit, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like ...

If one can't be a good example, they'll just have to be a terrible warning - Album on Imgur

Discover the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like WeAreOther.

Running out of sanity (more explanation) - Album on Imgur

Discover topics like makeup, eyes, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like DropDrop.

Eye makeup - Album on Imgur

Discover topics like meme dump, transgender, memes, pride, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like MermaidNinja.

Trans Meme Mini-Dump #4 - Album on Imgur

Discover topics like meme dump, funny, memes, dank memes, dump, and the magic of the internet at <strong>Imgur</strong>, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users like ...

Fresh daily dump for your coffee #247 - Album on Imgur

Imgur

imgur

So my previous 2 attempts at becoming a millionaire overnight have been resounding flops. Sure, I’ve managed to drum up a bit of excitement but I’ve not yet got a Ferrari sitting in the driveway of my own place yet… This third attempt will surely be a winner, right? I’m going to build a website that is able to rival that of popular image hosting site Imgur. This will be a highly scalable, resilient and low cost solution that will leverage in order to become a resounding success. More specifically, I’m going to use AWS’ Lambda offerings to host both the single page application and the series of endpoints that will make up the site. serverless technology In terms of monetization? I guess I’ll wait until I see the investments rolling in tomorrow from the hordes of angel investors looking for a bargain! I’ll be launching my own car into orbit soon enough… Traffic Projections. Requirements Ok, as always, we need to define a series of requirements that our project must adhere to in order for this to be considered a success. The site must be highly scalable from the outset. This will ensure that when it goes viral, it won’t crash. Thanks to our endpoints being based on Lambda, we don’t have to worry too much about scalability. The site must have some form of Authentication/Authorisation so that people can’t just start hitting an unauthenticated endpoint and uploading a million images and ruining our fun. The site must not cost me an arm and a leg to host! These are fairly achievable for a small project and shouldn’t take too long to implement, whilst also showing how powerful a combination of Cognito, Lambda and S3 could be. Stretch Tasks In future articles I’ll be extending this so that I can play about with services such as Rekognition and dynamodb as practice for certification exams. In a future article we’ll be looking at how I can use Rekognition to extract a series of tags from any images uploaded. These tags will then be stored within dynamodb and allow people to view certain categories of images. I also want to implement a comment and an upvote/downvote system so that users can comment and vote on their favorite content on the site. The Frontend in Vue.JS The frontend of my latest million dollar idea is going to be built using Vue.JS 2. This will allow users to login/register + upload new images to the site. It will also display any and all images available on the site in true imgur style. Version 0.01 will look like this: Our simple frontend Implementation Our frontend will utilise the and node modules in order to communicate with our AWS Cognito service. We’ll need to pass the following information into our frontend’s config file in order to connect to Amazon’s Cognito service: amazon-cognito-identity-js aws-sdk export default { region: 'eu-west-1',

IdentityPoolId: 'eu-west-1\_9IBAarCx9',

UserPoolId: 'eu-west-1:853957954650',

ClientId: '43duengi4ldb6jel18p84sgq22',

s3SignedUrl: 'https://rvv1a9to8j.execute-api.eu-west-1.amazonaws.com/dev/upload-node' } At first, the thought of having these included in my frontend code was somewhat terrifying. If you were like me then you may have thought — “If I expose these then people will be able to pretend they are me and rack up millions of dollars of expenses”. However, these values are only used to hit unauthenticated endpoints and *no* harm can be done by hackers with them. Our Cognito Service With the library we can easily create signup functions, authenticate and verification functions. I won’t post the full cognito service file but have a look at the configure and signup functions: amazon-congito-identity-js The full source file for this can be found here: https://github.com/elliotforbes/imgur-clone/blob/master/src/imgur-frontend/src/cognito/cognito.js With a little bit of setup in the AWS console, and 140-ish lines of code, we have a fully working user profile system. This is awesome when you consider how resilient, and fully fleshed out this is with very little work. I’ve not had to define a schema, set up a database, ensure database resiliency or anything like that. I simply configured a user pool and I was good to go. I’ll be doing an in-depth tutorial on my youtube channel: over the next few days on how you can implement your own Cognito user management system. https://www.youtube.com/tutorialedge Our Lambda Functions Now that we’ve got a simple frontend sorted, we need our lambda functions that will allow us to do cool things like upload files, and retrieve the links to all of the images in our bucket. In order for this to work, we’ll need 2 lambda functions to get us started. A function that will return a signed url that will allow us to upload to our s3 bucket A function that will return a simple JSON list of all of the items within our bucket. In order to deploy our functions we will again be using serverless.com’s cli. File Upload Lambda Function So our “upload” function will not actually be the one performing the upload of the image to our s3 bucket. It will essentially just fetch a signed url that will then be hit with a HTTP request in order to upload to S3. PUT A few key things to note here. We are creating an object that includes the bucket name we want to upload to, the key, which will be the file name, the content type, how long the signed url will remain valid for and the ACL. This ACL is the access control list and we need to set this to in order for people to be able to view the images within the bucket. s3Params public-read This then calls which returns the URL we subsequently upload to. Nice and simple. s3.getSignedUrl() List All Lambda Function Our list all lambda function will do the job of querying all of the objects within our s3 bucket and returning them as a json response. We could extend this further by associating all uploaded images with a key and a location in dynamo and then have the function return a paginated list of results from that, but in terms of a minimum viable product, this will do for now. Serverless Deployments Whenever it came to deploying my lambda functions, I could do so with ease using the serverless cli. Within this serverless.yml file that I can define the IAM permissions as well as the authorizors used to protect my lambda functions. These authorizors will ensure that people can’t just script something up and hit these endpoints 5,000 times per second. They will also need to have a registered account and have the appropriate Authorization header set with a valid token. Whenever I make a change to a function, I call and it deploys my serverless empire and provides me with the API endpoints that I can now hit with any HTTP requests. This deployment takes all of about 10 seconds for both of my functions so it is fairly quick. serverless deploy The Joys of Serverless One of the main tools underlying this entire project was the serverless cli. This greatly improved the way I was able to write my lambdas and subsequently deploy them. If you are interested in learning more about how you can manage all of your lambdas then I recommend checking out my other article titled, Managing your Lambda Empire with Serverless: _If you’ve been following me for a while, you will know that I am a huge fan of the concept of the new Serverless…_hackernoon.com Managing Your Lambda Empire with Serverless Full Source Code If you want to try this out yourself, the full source code for this project can be found here: _Contribute to imgur-clone development by creating an account on GitHub._github.com elliotforbes/imgur-clone Demo Link You can find the final product at this url: [ ] POSSIBLY? NSFW http://imgur-serverless-clone.s3-website-eu-west-1.amazonaws.com/#/ Test it out, register for an account and upload something Bear in mind though that I haven’t had the time to implement small things like password resets. appropriate! Conclusion This fairly simple project has a fully functioning user account system with email verification. It is also resilient and highly scalable and if you were to try and engineer something like this on dedicated servers you would require multiple servers in multiple data centers, you would have to set up load balancers and bake resiliency in to your underlying systems. By leveraging serverless and the Cognito service, we are able to build a system that would have taken months or even years of development work to meet this same standard on legacy infrastructure. Hopefully, you found this article entertaining and educational! If you liked this then feel free to let me know in the comments section below or by tweeting me . I am also on should you wish to connect! Elliot Forbes LinkedIn

Building a Highly Scalable Imgur Clone with Lambda and S3

Rails 5 Imgur Clone Welcome to the second and final part of the Rails 5 Imgur clone, in this part, we will build the image uploading, manage configurations and then deploy to Heroku. Be sure to check out the first part if you haven’t already. here First, signup and verify a new cloudinary account by going , it’s my referral link to cloudinary which will give me little bit extra space. here Let’s start by adding few more variables to our cloudinary configuration at config/initializers/cloudinary.rb Remember, the has to exactly in that folder, if you open the you’ll see in the comments written it says the config has to be there, it’s just how rails configuration works. cloudinary.rb config/application.rb Make sure your looks something like this. cloudinary.rb . . [ ] . [ ] . [ ] . . Cloudinary config do | | config config cloud_name = ENV 'CLOUDINARY_CLOUD_NAME' config api_key = ENV 'CLOUDINARY_API_KEY' config api_secret = ENV 'CLOUDINARY_API_SECRET' config cdn_subdomain = true config enhance_image_tag = trueend Go to cloudinary dashboard and you will the variables which we will use. The inclusion of increases image serving performance by getting the best cdn location and adds more functions to the built in rails method. Cloud name, API Key, API Secret cdn_subdomain enhance_image_tag image_tag Okay, here we are dynamically setting environment variables to our application. It will pick up variables for production and development environments. You really don’t wanna hard code this as it is not safe and wise to publicize your keys and other info as these are highly sensitive. This style is highly recommended for any of your rails project not just this. It’s just really a damn good practice. You might think its just a home project but if you work at a company this is really serious so, it’s better to practice from now. Secondly, and this is the most important step, setting up your variables in your environment. We are going to use the gem. Add this to your and run in your terminal to install the gem and then restart your server. dotenv Gemfile bundle gem 'dotenv-rails', groups: [:development, :test] This is by far the best, easiest and simple solution and works perfectly. Now go to project’s root directory and create a file called .env We are going to set our variables like this. export CLOUDINARY_CLOUD_NAME=export CLOUDINARY_API_KEY=export CLOUDINARY_API_SECRET= Just put the variables as it is without any quotes. And then add to your so that it doesn’t get pushed to your GitHub repo. We are going to add Heroku cloudinary add-on and set the variable in the production environment in their way. .env .gitignore Alright, our configuration part is done for now. If you wanna further know about the gem then go to their and read more. dotenv github page Finally, we are going to add code to our controller for uploading and saving images to the database. Go to your controller at and add to your & method like this. pics_controller.rb new create .**newend def @pic = Pic new def @pic = Pic**. ( . ( ). ( )) . create new params require :pic permit :image @pic saveredirect_to @picend What we are doing here is basically creating an called @ and adding the method to our model which we will use it in all over our project. Later, in the create method we are allowing it to add to the database, then save it and take it to that pic page after it’s been saved to our database. instance variable pic new Pic Now, create a & in directory new.html.erb _form.html.erb app/views/pics/ In your add this code. _form.html.erb <%= <%= <%= <%= <% <%= ** .submit %>**<% form_for @pic do | | %> f .label :image do %> f .hidden_field :image_cache %> f .file_field :image %> f end %> f end %> So, here we are creating a form for a pic to be added to our database and then we are going to display it in the web. Now, we going to implement the show page and controller. In your add this following code pics_controller.rb . ( [ ]) def @pic = Pic show find params :id end Here we are pulling the data from database by their id Create a new file in directory and this simple line to display the image to the public show.html.erb app/views/pics/ <%= image_tag(@pic.image.url, width: '100%', crop: "scale")  %> We are also scaling the image to fit in the display so, that it looks beautiful. Great, your image is displayed now we are going to display all the images to our index/ home page. Open add this following code to controller to get all the images from the database pics_controller.rb index . def @pics = Pic index allend In your add this to loop through all the images and display it to the index page. index.html.erb <% <%=  **image_tag( .image.url, width: '100%', crop: "scale")  %>**<% @pics.each do | | %> pic pic end %> Now, we are going to prettify our app a little by adding simple bootstrap css and a navigation. Go to , it’s a bootstrap material design flavor which I really like, you could use this or anything else you like. Grab the link add it to the of your right below the csrf meta tags bootstrap head application.html.erb <link rel="stylesheet" href="https://bootswatch.com/paper/bootstrap.min.css"> Now, create a named inside the directory, and add the navigation to our header partial _header.html.erb app/views/layouts/ <%= <%= <nav class="navbar navbar-default"><div class="container-fluid"><div class="navbar-header"> link_to 'Rimgur', root_path, class: 'navbar-brand' %> link_to 'New Post', new_pic_path, class: 'navbar-brand' %></div></div></nav> And then render the header to our main application page. Add this to your on top application.html.erb yield <%= render 'layouts/header' %> Now, as a final step we are going to take our app live by deploying it to heroku. For our application to work on production, we need to do some modification to our Move the to group and add for P** ** for Gemfile. gem ‘sqlite’ development pg ostgreSQL production. **group :development dogem 'sqlite3'end group :production dogem 'pg'end** Then do this following to install the gems bundle gem install bundler Git add and commit all of your files. git add .; git commit -m ‘project finished’ Create a new git repo in your account and our codebase to it. Follow the instructions on the screen to do . git remote add and push Now, we are ready to take it live. Create a new account for FREE and verify it. Then, create a new heroku app in your terminal and push it. heroku heroku create git push heroku master We are going to have migrate our schema to production for our app to work. We can do it like this. heroku run rails db:migrate Do to open the app in browser. heroku open Okay, now you are going to do the most important step for our image uploading to work. heroku addons:create cloudinary:starter Remember, we set up the environment variables for for machine, now we are going to do it for our production. cloudinary development Go to your heroku dashboard, then to your app, and then to the cloudinary dashboard to get our configurations. Now, in your terminal do this following to add the config variables. heroku config:set CLOUDINARY_CLOUD_NAME= heroku config:set CLOUDINARY_API_KEY= heroku config:set CLOUDINARY_API_SECRET= Try uploading a pic in your app and then check the cloudinary dashboard that it’s been added. Congratulations, you have successfully built a fully functional web app. Feel free to play around with it and make it more awesome. Here is the link to my github . All the code is hosted here for your reference. repo Thank you for reading it, hope you enjoyed it and will use this knowledge to build something awesome. Subscribe me if you would like to receive future posts to your inbox. LIKE & SHARE if you have liked my post. Please comment below any feedbacks, suggestions or requests. Also please visit my website .

Rails 5 Imgur Clone Tutorial…Part 2

Rails 5 Imgur Clone Handle image upload and painlessly with Carrierwave and Cloudinary and deploy on and share your photos fast with the world. management Heroku Hi everyone, today I’m going to show you how to do image share and upload in your Rails 5 application very easily. This is what I have learned by browsing several outdated tutorials, blog posts and a bunch of documentation. Most of the articles out there are older Rails version 4 so, I hacked together the steps for integrating into Rails 5 which I’m currently using on some of the projects and continue to use more. Plus in the process, we are going to build an imgur clone today by adding all the CRUD (Create, Read, Update, Delete) functionalities. It is going to be a 2 part series explaining every bits and pieces. This tutorial is going to very beginner friendly and also suitable to intermediate to advanced users. Now, let’s start some coding, shall we? First, let’s create a new rails project in your terminal by doing: rails new rimgur It’s always good practice to start using Git right away so, we are going to follow the same practice here: git init git add . git commit -m ‘project init’ Let’s test our app in the browser by using this command to boot up our server. rails server And then go to localhost:3000 in your browser. You will see this. Rails default home Great, your app is running successfully. Open the project in your favorite IDE or text editor. Now we are going to create a controller. rails generate Open file and an action pics_controller.rb index class PicsController < ApplicationControllerdef endend index Create a view file for our index action in app/views/pics/index.html.erb Open and write some dummy text like index.html.erb “Hello” Now, we are going to create a route to display the page. Go to and write this. config/routes.rb . . . Rails application routes draw doroot 'pics#index'end Okay, this line simply means you are setting path to controller and action. root pic index And you should or whatever you wrote. “Hello” Now, we are going to create our model like this. rails generate model Pic Awesome, now we have some routes and model setup and now we can start integrating our image handling system. First, open your and then put these right anywhere in middle. Gemfile gem 'carrierwave'gem 'cloudinary' Second, run in your terminal to install the gems and then restart your server. bundle Third, we will generate our image uploader like this. rails generate uploader Image Go to and add this inside our class. app/uploaders/image_uploader.rb include Cloudinary::CarrierWave And comment out everything else in the file. IMPORTANT! Now, create a new file in the directory name and then add this. config/initializers cloudinary.rb . . [ ] . [ ] . [ ]**end Cloudinary config do | | config config cloud_name = ENV 'CLOUDINARY_CLOUD_NAME' config api_key = ENV 'CLOUDINARY_API_KEY' config api_secret = ENV 'CLOUDINARY_API_SECRET' ** Okay, let me explain this. We are adding all the configuration variables dynamically and securely. I’ll explain later more in details. Open and then put this. app/models/pic.rb , class Pic < ApplicationRecordmount_uploader :image ImageUploaderend And then run to add our schema. Your should have something like this: rails db:migrate db/schema.rb , : . , : . , : create_table "pics" force :cascade do | | t t datetime "created_at" null false t datetime "updated_at" null falseend Now we’’ll generate a migration to add our to image uploader pic model. rails generate Open your and add this. <timestamp>_add_image_to_pic.rb :: [ ] , , class AddImageToPic < ActiveRecord Migration 5.0 def add_column :pics change :image :stringendend And then run to add it to our schema. rails db:migrate Great, our database part is done and now we going to add CRUD functionalities to our app. Open your file like this routes.rb resources :pics Let’s check our routes by doing in your terminal. And you should all the actions routes. rails routes Create all the action methods like this in your pics_controller.rb **class PicsController < ApplicationControllerdef end index def newend def end create def end edit def end show def end update def endend** destroy Thanks, everyone for reading it so far, hope you enjoyed it. In the next part, we will finish our application and also take it live. See you, in the next one.

Rails 5 Imgur Clone Tutorial…Part 1

This is a 1.0 story that I editedThanks to Tom de Ruijter, Steven Reitsma and Laurens Koppenol for proof reading this post. While playing Factorio the other day, I was struck by the many similarities with Apache Kafka. If you aren’t familiar with them: Factorio is an open-world RTS where you build and optimize supply chains in order to launch a satellite and restore communications with your home planet, and Kafka is a distributed streaming platform, which handles asynchronous communication in a durable way. I wonder how far we can take the analogy between Factorio and Kafka before it starts to break down. Let’s start from scratch, explore the core Kafka concepts through Factorio visualizations, and have some fun along the way. Why bother with async messaging? Let’s say we have three microservices. One for mining iron ore, one for smelting iron ore into iron plates, and one for producing iron gear wheels from these plates. We can chain these services with synchronous HTTP calls. Whenever our mining drill has new iron ore, it does a POST call on the smelting furnace, which in turn POSTs to the factory. From left to right: mining, smelting and producing — tightly coupled via synchronous communication This setup served us well, until there was a power outage in the factory. The furnace’s HTTP calls failed, causing the mining drill’s calls to fail as well. We can implement circuit breakers and retries to prevent cascading failures and message loss, but at some point we’ll have to stop trying, or we’ll run out of memory. Power outage at the factory If only there was a way to decouple these microservices... This is, of course, where Kafka comes in. With Kafka, you can store streams of records in a fault-tolerant and durable way. In Kafka terminology, these streams are called topics. Microservices decoupled by asynchronous messaging With asynchronous topics between services, messages, or records, are buffered during peak loads, and when there is an outage. These buffers obviously have limited capacity, so let’s talk about scalability. We can increase storage capacity and throughput by adding Kafka servers to the cluster. Another way is to increase disk size (for storage), or CPU and network speed (for throughput). Which of these options give you the best value for money is use-case specific, but buying bigger servers — unlike buying more servers — is subject to the law of diminishing returns. Kafka’s capacity scales linearly with each node added, so that’s usually the way to go. Vertical scaling — a bigger, exponentially more expensive server Horizontal scaling — distribute the load over more servers In order to divide a topic between multiple servers, we need a way to split a topic into smaller substreams. These substreams are called partitions. Whenever a service produces a new record, this service gets to decide which partition the record should land on. https://imgur.com/a/7lPAdVA A wagon producing records, a partitioner that puts messages on the right partition, and a topic with four partitions. The default partitioner hashes the message key and modulos that over the number of partitions: return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;That way messages with the same key always end up on the same partition. Note that messages are only guaranteed to be ordered within the context of a producer and partition. Records from multiple producers, or from a single producer on multiple partitions, can interleave. Now that we know how messages are put onto topics, let’s see how they are consumed. When you start listening to a topic, by default the records from all partitions are routed to you. It’s common though, to have multiple instances of a microservice running at the same time to achieve higher throughput or availability. If they all start listening to the topic, each record gets processed by each instance, which is usually not what you want. All microservice instances consume all messages Consumer groups allow you to evenly divide the partitions among multiple consumers. When a microservice instance joins the consumer group, Kafka will reassign some of the partitions to it. Likewise, when an instance crashes, or leaves the group for another reason, its partitions will be assigned to other instances. Kafka makes sure the partitions are always evenly divided among the consumers in each group. A single consumer group with three consumers If there’s a topic where the number of records per partition are skewed, you might be in trouble. An instance might not be able to keep up, because it was assigned the partition with many records, while other instances are idle. It’s up to you to make sure that there are no partitions that have vastly more records than others. Messages are piling up on a hot partition Each consumer keeps track of which records it has processed. Since records are processed in order, a simple offset is enough. Every once in a while (5 seconds by default), a consumer will commit its offset to Kafka. When a consumer leaves its group, its partitions are given to other consumer in the group. The new consumers will be able to start requesting records starting at the offset where the previous consumer stopped. It is possible that a record was processed, but not yet committed. You’ll either have to start at the committed offset, or start processing new messages and skip everything that’s not yet processed. This is why Kafka can only guarantee that messages are delivered at least once, or at most once. The analogy no longer really makes sense when we start duplicating data. With Kafka, we can process a single record multiple times. Multiple consumer groups can consume the same records. Topics can be stored with a replication factor of three for reliability. Topics can have a retention period after which records are deleted. All this is possible because data, unlike iron, can be duplicated easily. This is a good place to end this post. We’ve covered all the major concepts of Kafka and you should have a general understanding of how Kafka works. Let’s wrap up with a short recap. What we learned Kafka is a distributed streaming platform that stores records in a durable way through replicating records across multiple servers. Topics consist of partitions, that store records in order. Partitioners decide which records belong on which partitions. Consumer groups are optional, and help distribute partitions among consumers for scalability. Offsets are committed as checkpoints for when consumers crash. And that, in a nutshell, is how Kafka works. About the author Ruurtjan Pul (Twitter: @ruurtjan) is a data engineer at BigData Republic, a data science consultancy company in the Netherlands. We hire the best of the best in BigData Science and BigData Engineering. If you are interested in using Kafka and other data engineering tools on practical use cases, feel free to contact us at info@bigdatarepublic.nl. Learn more Apache KafkaFactorioAchieving stack and heap safety in recursive functionsFeature importance - what's in a name? This is a 1.0 story that I edited Thanks to Tom de Ruijter, Steven Reitsma and Laurens Koppenol for proof reading this post. Thanks to Tom de Ruijter, Tom de Ruijter, Steven Reitsma Steven Reitsma and Laurens Koppenol Laurens Koppenol for proof reading this post. While playing Factorio the other day, I was struck by the many similarities with Apache Kafka . If you aren’t familiar with them: Factorio is an open-world RTS where you build and optimize supply chains in order to launch a satellite and restore communications with your home planet, and Kafka is a distributed streaming platform, which handles asynchronous communication in a durable way. Factorio Apache Kafka I wonder how far we can take the analogy between Factorio and Kafka before it starts to break down. Let’s start from scratch, explore the core Kafka concepts through Factorio visualizations, and have some fun along the way. Why bother with async messaging? Let’s say we have three microservices. One for mining iron ore, one for smelting iron ore into iron plates, and one for producing iron gear wheels from these plates. We can chain these services with synchronous HTTP calls. Whenever our mining drill has new iron ore, it does a POST call on the smelting furnace, which in turn POSTs to the factory. From left to right: mining, smelting and producing — tightly coupled via synchronous communication From left to right: mining, smelting and producing — tightly coupled via synchronous communication This setup served us well, until there was a power outage in the factory. The furnace’s HTTP calls failed, causing the mining drill’s calls to fail as well. We can implement circuit breakers and retries to prevent cascading failures and message loss, but at some point we’ll have to stop trying, or we’ll run out of memory. circuit breakers Power outage at the factory Power outage at the factory If only there was a way to decouple these microservices... This is, of course, where Kafka comes in. With Kafka, you can store streams of records in a fault-tolerant and durable way. In Kafka terminology, these streams are called topics . topics Microservices decoupled by asynchronous messaging Microservices decoupled by asynchronous messaging With asynchronous topics between services, messages, or records , are buffered during peak loads, and when there is an outage. These buffers obviously have limited capacity, so let’s talk about scalability. records We can increase storage capacity and throughput by adding Kafka servers to the cluster. Another way is to increase disk size (for storage), or CPU and network speed (for throughput). Which of these options give you the best value for money is use-case specific, but buying bigger servers — unlike buying more servers — is subject to the law of diminishing returns . Kafka’s capacity scales linearly with each node added, so that’s usually the way to go. bigger more the law of diminishing returns Vertical scaling — a bigger, exponentially more expensive server Vertical scaling — a bigger, exponentially more expensive server Horizontal scaling — distribute the load over more servers Horizontal scaling — distribute the load over more servers In order to divide a topic between multiple servers, we need a way to split a topic into smaller substreams. These substreams are called partitions. Whenever a service produces a new record, this service gets to decide which partition the record should land on. partitions. produces https://imgur.com/a/7lPAdVA https://imgur.com/a/7lPAdVA A wagon producing records, a partitioner that puts messages on the right partition, and a topic with four partitions. A wagon producing records, a partitioner that puts messages on the right partition, and a topic with four partitions. The default partitioner hashes the message key and modulos that over the number of partitions: default partitioner partitioner return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; That way messages with the same key always end up on the same partition. return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; Note that messages are only guaranteed to be ordered within the context of a producer and partition. Records from multiple producers, or from a single producer on multiple partitions, can interleave. Now that we know how messages are put onto topics, let’s see how they are consumed . When you start listening to a topic, by default the records from all partitions are routed to you. It’s common though, to have multiple instances of a microservice running at the same time to achieve higher throughput or availability. If they all start listening to the topic, each record gets processed by each instance, which is usually not what you want. consumed All microservice instances consume all messages All microservice instances consume all messages Consumer groups allow you to evenly divide the partitions among multiple consumers. When a microservice instance joins the consumer group, Kafka will reassign some of the partitions to it. Likewise, when an instance crashes, or leaves the group for another reason, its partitions will be assigned to other instances. Kafka makes sure the partitions are always evenly divided among the consumers in each group. Consumer groups A single consumer group with three consumers A single consumer group with three consumers If there’s a topic where the number of records per partition are skewed, you might be in trouble. An instance might not be able to keep up, because it was assigned the partition with many records, while other instances are idle. It’s up to you to make sure that there are no partitions that have vastly more records than others. Messages are piling up on a hot partition Messages are piling up on a hot partition Each consumer keeps track of which records it has processed. Since records are processed in order, a simple offset is enough. Every once in a while (5 seconds by default), a consumer will commit its offset to Kafka . offset commit . When a consumer leaves its group, its partitions are given to other consumer in the group. The new consumers will be able to start requesting records starting at the offset where the previous consumer stopped. It is possible that a record was processed, but not yet committed. You’ll either have to start at the committed offset, or start processing new messages and skip everything that’s not yet processed. This is why Kafka can only guarantee that messages are delivered at least once, or at most once. The analogy no longer really makes sense when we start duplicating data. With Kafka, we can process a single record multiple times. Multiple consumer groups can consume the same records. Topics can be stored with a replication factor of three for reliability. Topics can have a retention period after which records are deleted. All this is possible because data, unlike iron, can be duplicated easily. replication factor retention period This is a good place to end this post. We’ve covered all the major concepts of Kafka and you should have a general understanding of how Kafka works. Let’s wrap up with a short recap. What we learned Kafka is a distributed streaming platform that stores records in a durable way through replicating records across multiple servers. Topics consist of partitions, that store records in order. Partitioners decide which records belong on which partitions. Consumer groups are optional, and help distribute partitions among consumers for scalability. Offsets are committed as checkpoints for when consumers crash. And that, in a nutshell, is how Kafka works. About the author Ruurtjan Pul ( Twitter: @ruurtjan ) is a data engineer at BigData Republic , a data science consultancy company in the Netherlands. We hire the best of the best in BigData Science and BigData Engineering. If you are interested in using Kafka and other data engineering tools on practical use cases, feel free to contact us at info@bigdatarepublic.nl . Ruurtjan Pul Twitter: @ruurtjan BigData Republic BigData Republic info@bigdatarepublic.nl info@bigdatarepublic.nl Learn more Apache Kafka Factorio Achieving stack and heap safety in recursive functions Feature importance - what's in a name? Apache Kafka Apache Kafka Factorio Factorio Achieving stack and heap safety in recursive functions Achieving stack and heap safety in recursive functions Feature importance - what's in a name? Feature importance - what's in a name?

Understanding Kafka with Factorio

*ting, ting* No, that is not your monthly paycheck coming in. It is Hackernoon’s socials notifications! If you say you don’t have any social account, you are either lying or being a kid who is too cool for popular stuff. Face it, there is no faster nor better way to connect than being present on social media. Getting the gist of it, Hackernoon is basically everywhere, from A to Z, covering most of the alphabet with our presence on social media platforms. Newest to the fam, HackerNoon welcomed one of the best heroes who came to rescue us from the bluebird disaster - ! Connect with us on Mastodon ! Mastodon here Here are all the links to these HackerNoon social media channels: Amazon Bloomberg BuiltIn CB Insights Changelog Clubhouse Crunchbase Cryptopanik Den DeepStash DevDojo ( ) Digg Official Blockchain Discourse Everipedia Facebook Feeder Flipboard GIPHY GitHub GitLab Glassdoor Google Books Google News Kaggle Imgur Instagram iTunes Learnerby Library of Congress Linkedin LunarCrush Minds Newsbreak ( ) OpenSea team Pinterest ProductHunt ReadWrite ( ) Reddit other subreddits Refind RSS Showwcase Spotify StartEngine Stitcher Tealfeed Tribel Twitch Twitter Unsplash Wikipedia Yahoo Finance Youtube If you ever got lost in this overwhelming Internet world, get back here and talk to Hackernoon whenever you want about everything tech crazy! Follow us on and also , , , , Twitter Linkedin, Google News, GIPHY, Instagram, iTunes Facebook, GitHub Youtube . Mastodon And of course, is where you can ALWAYS find us! Hackernoon.com

HackerNoon Social Media Presence: A Complete Guide

Presencia en las redes sociales de HackerNoon: una guía completa

HackerNoon Présence sur les réseaux sociaux : un guide complet

हैकरनून सोशल मीडिया उपस्थिति: एक संपूर्ण गाइड

HackerNoon ソーシャル メディア プレゼンス: 完全ガイド

Presença na mídia social do HackerNoon: um guia completo

Sự hiện diện trên mạng xã hội của HackerNoon: Hướng dẫn đầy đủ

HackerNoon 社交媒体存在：完整指南

-- , the technology publishing platform serving 4M readers per month, today announced a $1M strategic investment from , the San Francisco-based startup designed to help creators monetize content and provide a premium experience to consumers. Additionally, the companies kicked off a three year partnership focused on for contributing writers. Web Monetization is an open specification that is freely available to any monetization service provider who wants to support sites like HackerNoon. EDWARDS CO June 10, 2020 HackerNoon Coil Web Monetization “We want to become the best place for tech professionals to read, write and publish,” said HackerNoon CEO David Smooke. “To date, we’ve provided editorial resources, validation, and distribution. Now with this strategic partnership, HackerNoon will experiment with streaming micropayments to writers based on how much time Coil users spend reading their stories.” "HackerNoon is home to an incredibly dedicated community of technologists and software developers,” said Jonathan Greenglass, Coil’s Chief Growth Officer.  “We’re thrilled to be working with HackerNoon to provide its community with a simple way for readers and writers to participate in the value exchange between one another." Starting today, HackerNoon Writers can now add their Web Monetization meta tag to their stories on HackerNoon. Once enabled, Coil automatically streams micropayments to the writer’s wallet based on the amount of time Coil Members spend reading. We will also offer a way for writers to that are dedicated to making the Internet a better place. donate their earnings via Coil to charities Last year, HackerNoon (dubbed HackerNoon 2.0) to serve readers, writers and sponsors on their own terms. The company pivoted from a publishing platform to a software company and a publication, creating 100+ years worth of reading time since July 2019. launched their own content management system “Props to HackerNoon's editorial teams for editing my piece. Truly amazed at the editing quality and the effort they pour into *every* article that goes through their hands. Excited to see them grow even bigger!” tweeted Human+Business Founder and HackerNoon contributor Andy Chan. “Congratulations to David Smooke, Linh Dao Smooke, Dane Lyons,  Austin Pocus, and everyone else on the HackerNoon team. The new platform is well-built and progressing fast. Thanks for everything you all are doing for the global developer community,” tweeted FreeCodeCamp Founder Quincy Larson. Coil paid $11.35 per share, valuing the company at $11.5M pre-money. In HackerNoon’s previous , the share price was $8.20. The company remains entirely common stock. As part of the partnership, , Coil’s Chief Growth Officer will be joining David Smooke and Linh Dao Smooke on HackerNoon Board. Previously, he was CFO of Imgur, one of the internet’s largest communities, and a top 30 website in the US. equity crowdfunding round Jonathan Greenglass “In addition to implementing the Web Monetization standard, we will use this investment to expand the editorial, sales, and most importantly, software teams,” said David Smooke. About HackerNoon How hackers start their afternoons. We are an open and international community of 12,000+ contributing writers publishing stories and expertise for 4,000,000+ curious and insightful monthly readers. Founded in 2016, HackerNoon is an independent technology publishing platform run by David Smooke and Linh Smooke. HackerNoon is built for technologists to read, write, and publish. About Coil Coil was founded in 2018 to build a better business model for the web. Coil makes it easy for creators to monetize their content across the internet, and provides an alternative to advertising and site-by-site subscription models. As members enjoy content, Coil uses a proposed open web standard called Web Monetization to stream micropayments to creators in real time. Coil is designed to support a variety of creators, developers and content platforms. In 2019, Coil, in collaboration with Mozilla and Creative Commons, launched the to boost open, fair and inclusive standards and innovation in Web Monetization. Coil also invested $20 million in Imgur, the community-powered entertainment platform reaching 300 million people each month. For more information about Coil, please visit coil.com and follow on Twitter (@Coil). $100 million Grant for the Web

HackerNoon Raises $1M From Coil, Will Stream Micropayments from Readers to Writers via the Browser

⚠️UPDATE: is no more.  This article no longer applicable! ⚠️ Coil ⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️ Adding one line of HTML to your website can start making you money.  No one will see anything different on your site, no one will be tracked because of it. This one-liner is a meta tag that goes in your site header and looks something like this: <meta name = "monetization" content = " $ilp .uphold.com/LMQDwYFW6rA!" > Now, with this added, visitors of the site have the ability to indirectly fund whomever is running the website, without opening their wallets again.  Their paid membership along with the browser extension will make payments on their behalf, just by spending time on your site. This is How We Do It is providing a means to seamlessly distribute money to content providers on the internet.  The idea is to provide content creators with a free account with Coil, which they can connect to a wallet to receive money either via fiat or cryptocurrency. Coil The source of money fueling this comes from another sort of Coil account.  A Coil member account, which for $5 dollars a month, provides exclusive access and perks to websites partnering with Coil. comes with Coil member accounts, for example. Imgur Emerald To sign up for a free creator account go to and fill out your email address.  You'll be redirected to a verification page and receive an email with the code to use.  Afterwards, you will be asked to enter your credit card information, but you can simply go to to continue filling out your profile, where you will need to set a user name. https://coil.com/signup https://coil.com/blogs/creator-setup Finally, is where you can place your payment pointer if you have one already or sign up for an account with the different wallets supported by Coil.  I recommend Uphold because it supports the most currencies and has no fees. https://coil.com/blogs/creator-payout If you have an account already, the link at the top gives you a spot to enter your address.  There is a helpful link to each organization's help docs for location your payment pointer ("Get a payment pointer") After this, you can visit where Coil conveniently displays the meta tag with your payment pointer filled in for you.  This tag just goes into the <head> of your website.  That's all there is to it. https://coil.com/settings/monetize How Do They Do It? The magic that links consumers and creators is in the meta tag and either a or the The contents of the attribute, in this example shown here, is an address to an wallet.  This is where the money goes, regardless of who is actually serving up the website with this tag.  The browser extension will pickup on the meta tag and react to let you know you are supporting the content provider.  So, if you create a free Coil account and install the extension, then next time you enjoy some stream on for example, Coil will begin streaming money via micro-transactions to whomever is streaming to your browser at the rate of .  Pretty simple way to help people you are a fan of, no? Coil browser extension Puma Browser. content Uphold Twitch.tv 36 cents an hour Under the hood and , Coil makes use of a proposed JavaScript browser API called and an open protocol for sending payments to ledgers named to make this work.  All of this without ads popping up or any other shady practices going on (psst- maybe you'd like to learn about ?). via the tubes Web Monetization Interledger Privacy Badger This is a nice summary of Coil with pretty pictures. So, it makes no difference what the content is, you could put this on a site with recipes for , a or a y VR experience, if visitors are spending time on the site, these same visitors will be funding the creators of said content.  You just need to have this little meta tag with your wallet address and if visitors are using the Coil extensions or Puma browser, creators will get paid for having visitors merely viewing their content. kkakdugi Unity game Firefox Realit As more perks for paid members of Coil emerge, we can hope to see more people adopting this platform to support your corner of the internet. I'll close with this aphorism: “Money is like manure; it's not worth a thing unless it's spread around encouraging young things to grow.” -Thornton Wilder, The Matchmaker

Monetizing Your Hacker Noon Stories and Own Website with Coil [A How-To Guide]

Fetching Places From Google Maps with Flutter

Server admins commonly use SSH to log in to Linux servers. They will disable login via password due to the ease of brute force attacks these days. The recommended practice is to only allow login to the servers using private/public keys. It is also good to set up SSH notifications with IP geolocation for better security. Admins will receive real-time notifications whenever anyone logs in to the servers. Getting an immediate notification when someone logs in is beneficial from a security viewpoint. It increases the chance that you can prevent an authorized party from doing real damage to your server. If you see a login for a suspicious region or ISP, you can take remedial action ASAP. A fast response to intrusion will give the bad actor no chance to steal your data. Pre-requisites to setup SSH notification with IP geolocation Our script will use the Web Service to query geolocation data using the user’s IP address. WS4 has data about Country, Region, City and ISP; more than enough for our notification example. That said, you can use a different package if you require more geolocation data. To use the Web Service, you need an API key. If you don’t have one, ip2 is an option: https://www.ip2location.com/web-service/ip2location In addition to the API key, you must also install and in your Linux server. The package is required to call the Web Service while the package is required to parse JSON response from the web service. jq wget wget jq SSH Login Notification Script To enable the SSH notification script to run upon login, the following code should be copied and pasted into your file. Replace with the actual Web Service API key. ~/.bashrc IP2LOCATION_API_KEY If you want to enable Slack notification, then replace with the actual URL. Otherwise, comment out the Slack-specific lines of codes. SLACK_WEBHOOK_URL Lastly, edit with the email address that will receive the notification. NOTIFICATION_EMAIL API_KEY="IP2LOCATION_API_KEY" SLACK_WEBHOOK="SLACK_WEBHOOK_URL" EMAIL="NOTIFICATION_EMAIL" IP="$(echo $SSH_CONNECTION | cut -d " " -f 1)" if [ ! -z "$IP" ]; then RESULT="$(wget -q -O /dev/stdout 'https://api.ip2location.com/v2/?key='"$API_KEY"'&ip='"$IP"'&package=WS4')" CITY="$(echo $RESULT | jq -r .city_name)" REGION="$(echo $RESULT | jq -r .region_name)" COUNTRY="$(echo $RESULT | jq -r .country_name)" LOCATION="$(echo $CITY, $REGION, $COUNTRY | sed 's/^[, ]\+//g')" ISP="$(echo $RESULT | jq -r .isp)" HOSTNAME=$(hostname -f) NOW=$(date +"%e %b %Y, %a %r (UTC %Z)") # Slack notification wget -q -O /dev/null --no-check-certificate --header 'Content-Type: application/json' --post-data '{"username":"'"$HOSTNAME"'", "icon_url":"https://i.imgur.com/X2W00e2.png", "channel":"#general", "attachments":[{"title":"SSH Notification", "color":"#FDAE02", "mrkdwn_in": ["text"], "text": "*IP Address*: '"$IP"'\n*Location:* '"$LOCATION"'\n*ISP:* '"$ISP"'\n*Date:* '"$NOW"'"}]}' $SLACK_WEBHOOK # Email notification echo "SSH login from $LOCATION ($IP)." | mail -s "SSH Login Notification" "$EMAIL" fi Conclusion: SSH Login Notifications with IP Geolocation IP geolocation is useful to detect where your users are logging in from. If you see any locations that are in unexpected regions, you should be alert and perform a security audit of your system. A data breach is a very serious issue these days so it pays to be extra vigilant with your monitoring.

Stories, News & Mentions about IMGUR

Imgur’s Stories on HackerNoon

Imgur's News & Mentions