“Please pray for my son,” is my mother’s go-to phrase whenever she comes to visit my condo in San Diego. My home lab is strung up with a web of audio, visual, and power cords. Wandering eyes will find themselves crossing resistors, soldering irons, microcontrollers, drones, robot pets, and even more . It’s certainly no surprise why she’s a little worried. My mother’s opinion aside; in the developer community, personal projects can be seen as ways to reinforce bad practices and a poor investment of time and money. Even I notice that my might be misconstrued as a red flag. But I believe the personal projects I’ve made have helped me become a better, creative, and more resourceful engineer. robot pets Instagram posts on wire organization In-Plant Plant Monitor — https://www.hackster.io/nick-engmann/1a17f6 Personal projects are things that you invest time and effort in to create outside of your job. The purpose of the project could be anything — to fulfill or automate a task, to prove a political point, to improve your skill set, or to produce art. For example, in order to stop killing my beautiful house plants, I built a mesh network of devices to monitor their water levels called . There was also a point in my career where I wanted to better my software developing skills so I helped develop — a free web application that helps developers prepare for interviews using flashcards. However, there are plenty of ignoble personal projects in my collection including my , , and of course my . If anything, having a fully autonomous deployable nerf turret in my condo is a reason for my mom to pray for frail and slow-moving home intruders, not me. In-Plants Flashcards For Developers Tinder Auto Swiper cosplay projects Nerf Gun Turret Nerf Turret Defense System — https://www.hackster.io/quodcertamine/a50dd1 Both my practical and completely impractical projects are fueled by the same motivation: it’s a lot of fun. Thinking of cool, unique problems and architecting a solution to solve them in a few weeks is like running a marathon where you design the course. You start fresh and naive, not knowing that your limits of creativity and resourcefulness will be exhausted by the end of the first sprint. The experience or skill gaps you face become obstacles that you learn to swiftly avoid after doing the necessary research to get past them. Oftentimes, there is a supportive community both along the way and at the finish line when you present what you have learned or created. Best of all, you always win because you walk away with your one of a kind trophy — your personal project. And even then, the true reward isn’t the object you are left with at the end, but rather the improved skillset you gained along the way. Many individuals already get this sort of growth, freedom, and creativity at work, but personal projects provide a unique experience that can rarely be replicated. Over the past 3 years, I’ve worked various full time engineering jobs that have helped me develop the skills that I utilize to make some of the projects that I create. But jobs, especially engineering jobs, are incredibly 2-Dimensional. There are schedules that you have to stick to, budgets that you can’t break, and many times you are limited to the amount of personal creativity that you can add to a project. I currently work on a Research and Development team that is tasked with thinking up and creating unique cybersecurity tools. But at the end of the day, if the ideas that my team has are helpful to the general populous but aren’t marketable, the company can discourage us from working on them. Personal projects give the power back to the individual. The goals that you have for your ever-changing masterpiece can take as long as you like and can cost as little as you like. Flashcards for Developers — https://flashcardsfordevelopers.com One of the most frequent questions I get from friends is “How do I get started?”. Here is a list of helpful resources that can help out, most of which have helped me immensely: For Hardware Project Ideas: — excellent site to not only purchase hardware but also find professionally made project tutorials. Adafruit — great website full of community-sourced tutorials on how to replicate fun personal projects. Hackster — another tool to find instructions on how to replicate personal projects. Instructables — daily blog posts on hardware projects. Ranging from cybersecurity to arcade machines. Hackaday Youtube Channels For Hardware Projects: Greatscottlab Simone Giertz Eevblog ElectroBOOM Michael Reeves William Osman Novaspirit Tech HackerShack Instructional Literature For Hardware Knowledge Mims III, Getting Started in Electronics Platt, Make: Electronics Geier, How to Diagnose & Fix Everything Electronic Kybett & Boysen, All New Electronics Self-Teaching Guide For Software Project Ideas: — lets users discover and share new projects. http://producthunt.com/ — a crowdsourced collection of tutorials from across the web for learning languages and libraries (ignore sponsored stuff, look at upvotes). https://hackr.io YouTube Channels for Software Projects: — His videos are a bit dated but they provide a solid foundational mindset when it comes to solving problems. The New Boston — Good Understanding of Web Development in 2019. https://www.youtube.com/watch?v=UnTQVlqmDQ0 Instructional Literature For Software Projects: — Resources for everything you would ever want to learn about software Awesome Lists — a good introduction to HTML/CSS/JS and Node.js or Django. https://developer.mozilla.org/en-US/docs/Learn — curriculum including HTML/CSS/JS, React, Node.js, Express, and MongoDB. https://www.freecodecamp.org — curriculum providing a strong basis in JavaScript https://javascript.info — quick reference sheets for the syntax of many different languages (generally not sufficient on their own for learning something, but very helpful). https://learnxinyminutes.com — Everything PHP. https://pastebin.com/gfBPg24A Communal Learning Environments: : If you are a high school or college student, these are tailored for you! Major League Hacking hackathons . There are plenty of organizational hackathons that exist outside Major League Hacking and Meetup is a great resource to find them. Meetup: Hackathons initiative. Many people aren’t comfortable with investing in a project without practical implementations. Code For America is a national organization with developers, designers, and idea-makers who want to get together to create projects to help their local community. Code for America Brigade Raspberry Pi Zero W Network Access Storage Device— https://www.hackster.io/135000/3f47be These resources and communities that I have listed above have all helped me. Over the past three years, I’ve met incredible people, learned a lot, and have been able to complete over 30 personal projects. While I’ll admit my first few projects were something only a mother could love (or ), I believe that I’m becoming a better engineer the more I create. So I’d encourage you to take that first step and start creating, regardless of what you want to make. And know that if your loved ones begin to pray for you and your legion of robot pets, that you are not alone. DJ Khaled

Instagram

Mozilla

YouTube

Check Out My Projects

Read My Stories

Playing With Yourself: The Power of Personal Projects

A super secure decentralized file sharing application powered by Web 3.0 uses protocol called proxy re encryption to secure and permission data efficiently. Hako Hako Logo Unlike other file sharing services such as Dropbox and Google Drive, . The user is always in control of your keys and your data. Hako does not place your keys in the hands of a large companies Centralized storage has a single point of failure (the company) and require a user to be online (connected to the central servers) in order to transfer data and delegate access. PLEASE VOTE just signup here and then upvote the two projects: https://coinlist.co/register NCIPFS - CoinList Hako - CoinList The future of the web? Web 3.0 protocols Hako leverages Web 3.0 peer to peer protocols so users can directly share data, as well as a live distributed nucypher network — or a federated network (like on run on a local machine) or run by some central body you trust (a bank, a school, a 3rd party you pay for). Hako has no single point of failure and allow non-trusted 3rd parties to delegate access to data — while never being able to decipher the original message. Y.O.E.O not Y.O.L.O Aside from the added network resiliency and cryptographically secured network. Hako follows a Y.O.E.O (you only encrypt once) methodology which saves huge amounts of data. Instead of duplicating and encrypting a message for every user you’d like to share with. You only encrypt the data once, and do transformations on the keys to allow other users access. Example data flow provide by NuCyphers Heartbeat Demo https://github.com/nucypher/nucypher/blob/master/examples/heartbeat_demo/heartbeat_demo.md How’s it work? If you were to share a 1 TB file with 5 people, you would save 4 TB of encryption time and 4 times the amount of computational run time. Hako allows for secure and efficient data sharing. It relies on 100% open source protocols that encrypt data with full transparency as well as leveraging content identifiable addressing and zero knowledge proofs for data integrity. Why it was built? Hako was born out of the NuCypher + Coinlist hackathon. The project is actually two parted, and consists of a library called NCIPFS and the application Hako. Hako is powered by NCIPFS and is provides a REST interface and web-app to the underlying methods of NCIPFS. The architecture It is important that we break down Hako into its smaller pieces to get a full understanding of what it enables. It’s not that useful to know Hako was built (whilst traveling through Japan, on trains/planes and in wifi permitting coffee shops) but it was built the way it is. how why First people ask, why another file sharing app? Well this is easy — with the developments of more powerful/resilient communication protocols it seems obvious that we would port our most useful applications to a peer to peer web. One with no central body, one that is offline first, is cryptographically secure and people powered. Second people ask, what is a ? Futari Futari actually means “two people” in Japanese and seemed like an apt name for the Hako user (a combo of an Alice and Bob role). Futaris have the ability to encrypt data before they place it on IPFS and also can receive data from other Futaris and subsequently download/decrypt the data from IPFS. Alright so… Futari is simply a fancy name for a user. Yes true — but it means exactly what is it, and in Hako we don’t have users. We have Futaris. Now that were familiar with Hako participants — we need to understand what people are sharing. Since data on IPFS resides on all the locations that is it’s requested we don’t want to send a file, we simply want to share the location (and content hash) with individuals of interest. However the paying attention reader will notice that the CID doesn’t really share anything of value — all the CID points to is an encrypted bucket of bits. Not useful for anything but wasting space on your computer. So we need a way to pass the information that would allow a person to request the appropriate re-encryption keys or in human terms — all the info needed to unscramble the bits into the original file. The data needed for is 4 parts: CID + POLICY KEY + SIGNING KEY + FILENAME We concatenate all these values together with _'s and create what we call in Hako the NUCID Wait whats a NUCID? It’s simply a special identifier we use in Hako to fetch a file and the respected policy for that file. If a person has been granted access by the file’s originator (the Futari who uploaded it) then they should be able to decrypt the file with their own keys. This is the magic of NuCyphers proxy re-encryption in play. It allows us to encrypt the data onetime and then create policies where we allow others to decrypt the data with their own keys (unbelievable; I know) but like any other magic trick — with a peep under the hood we can see all the beautiful mathematics that make this work. Shout out to the geniuses at NuCypher for developing these cryptographic primitives. 🎉 The NUCID is simply a serialized string that contains all the data we want in a short (double clickable way). The NUCID is purposely encoded using Base58 so user can double click any part of the NUCID and the device will automatically highlight the whole NUCID (this would not be the case with higher base encoding). Wrapping it up So we’re now familiar with the Futari, the NUCID and Hako in general. We see the value of a distributed file sharing app and a simple way to interact with IPFS, and NuCypher without knowing anything about the underlying workings. We can share files with who we want, how we want. We can secure data with time tested symmetric encryption algorithms We have an easy to use web-app that works on our browser and mobile device. We have a developer friendly library (NCIPFS) to interact with IPFS and NuCypher easily. We have extendable concepts that can be reused in other implementations of distributed data stores and privacy layers. (the Futari and NUCID). We have a two part submission for the hackathon. One dev tool and one application. Who cares If your itching to try out the demo via cloning the repo, jump to the bottom of the article, however we might want to better understand who will use this app and how they can use it. What benefits it brings to those users and short examples of their usage patterns. The Demo To start, make a new Futari by clicking create when the app loads. Login if you have an account Next the page should and now we can login. reload Same user/pass as you created above Now we’re at the home screen and can do a few different things! If someone has already shared a file with you and added a policy to allow your newly made Futari access you should have a NUCID ready to download and decrypt. You can enter that NUCID in the nucid bar at the top of the app and click GET to retrieve your file. Enter your NUCID on the other hand, if you do not have a file to download and want to upload your own encrypted files to IPFS just click the 💾 symbol and a drag-and-drop box will appear. Landing spot for your files You can add files up to 4GB (at the moment — chunking for larger files is yet to be added, this is a limitation of the core symmetric encryption algorithm) On a desktop you can drag and drop on mobile just tap that grey pane and you will get a menu like the following native iPhone photo access Once a photo or file is selected, a series of behind the scenes processes will start. The photo well use for this article File is converted to a multipart upload and is sent to the Hako server The file is encrypted with the current Futari’s Alice encryption keys. The encrypted file will be stored in IPFS. In return a CID will be sent to the application. Hako now re-encrypts the data for your Bob keys. This way even if we change pier Alice keys, we can access the data. up up up loading to Web 3.0 Yay 🙌 we now have the file successfully encrypted and stored in IPFS, as well as access to a decrypted copy of the file (since we granted ourselves permission). We can see our files in the file list now and have a few options on what we do next. The file list — with example file Theres a lot happening on this table so we’ll look at each column individually. — is the human readable file name — we truncate it for the view but if you hover over it, you can see the full filename. Label — this is the content identifiable address, kinda like a URL but the identifier in IPFS. Anyone can fetch the bytes at that address but if you do not have access it will return a bunch of encrypted gibberish. Try it yourself and aceess that CID at your favorite IPFS gateway like:https://cloudflare-ipfs/ipfs/YOUR_CID_HERE and you can see what I mean about it being gibberish. CID — is the human readable file size once it has been encrypted and placed on IPFS. You’ll notice that this file size is slightly bigger than the original file. There is a small amount of overhead needed to allow for NuCyphers re-encryption schema. Size — now we get to the clickable options. Here we can open and view the file! You’ll be able to see the decrypted file contents (assuming you have access) Open — This is how we add policies and viewing permission to a file. When you click on this 🔓 you’ll be prompted to choose or add a Futari to share the file with (covered in more detail below) Permission — is how you can see who the file has been shared with. If you uploaded the file, you should see your Futaris name since Hako automagically allows yourself viewing access to your own files. Policies Sharing Now that we have our files added and we understand the file table, we’ll want to start sharing files with friends, coworkers, or maybe even our doctors. In order to share a file with someone- we have to first add their Futari’s recipient keys (or their Bob keys in NuCypher world) to our contact list. Once we add them as a contact we can share any file with them we want. Just click on the 🔑 symbol on the top bar to copy your keys to your clipboard. Note we also copy the username so others know what to call your Futari Once copied you can send them however you want to another person (email, slack, carrier pidgin). Note: these keys are *public* so no worries about accidentally sharing them. The only thing someone can do with these keys is allow you access to their Hako secured file. We want to share the photo we just uploaded, so we click on the 🔓 symbol and are prompted to add a contact. So far we have no contacts so we’ll copy some keys over from a demo Futari’s I made earlier. All we need to do is copy their keys (and username) they sent into the input box and press add. Yay 🎊, now we have a contact whom we can allow access. Cool. 🤙 when we grant access a multi step process is kicked off behind the scenes. A new policy is made for the file in question. The policy is sent to the NuCypher network and distributed across the nodes. Hako created a NUCID for the policy including the files CID and human readable file name. Hako updates the UI to reflect the new Futari who we’ve granted permission. If we click on their name, it will copy the NUCID to our clipboard and we can share this with them. Once again this can be publicly shared and has no value unless you are that Futari. You’d need their Bob private keys to make use of the NUCID. Done ✔️✔️✔️ We’ve now successfully added a file, granted ourselves access, viewed it in the app, added a friend and granted them permission as well. We’re all done with our part. For completeness we’ll login to our demo Futari we just shared the file with and retrieve, download and decrypt it. We login as Abc We’re in and now we just need to copy that NUCID over and paste it into our NUCID bar. Simply press GET. And voila ✨ the files is added to this Futaris file list At a quick glance you can see the file sizes are the same size and have the same filename/CID. Looking good. We can ensure that the data was correctly decoded by clicking the 📖 button and viewing the image. We now have seen an end to end process of securely sharing data using Hako! Not to hard, and we keep the highest level of security, we share the data in a peer to peer manner and have no security leaks 👏 What if someone else sees that NUCID? To show the complete security, we also will login as another user and try to decrypt the file from that NUCID shared above. Since the photo originator didn’t grant this user access they should not be able to see the decrypted photo. We login as a user named xyz. Using that same NUCID that Abc decrypted — we copy that into our NUCID bar A file is returned…. But its tiny — 17 bytes. (this is actually a nullified response we get when we try to request re encryption keys from NuCypher.) Opening the file shows the same CID on IPFS, but the user cannot actually see anything. They are not granted access which causes their decryption attempt to fail. Great! We’ve successfully shared our file with one person, but only encrypted it on IPFS once Note: since IPFS is immutable — the same CID proves that we only encrypted the data once. User stories Sharing large encrypted files with many people One great example of a user would would benefit from Hako, is a person who wants to share a large secured files. Since Hako only needs to encrypt the data and store it once. It removes storage redundancy for each person you share it with. Instead of having many copies of the same data encrypted with each of the recipients keys. We just store it once. Yay to saving huge amounts of storage space. Think about the implications of a 5gb file, or a 100 TB file! That’s full hard drives amounts of savings. Sharing sensitive data with multiple participants Another great example, and a primary focus of other NuCypher apps it the ability to selectively share private data with who you want. Take medical data for instance. Let’s say you store a copy of your DNA sequence in Hako. This is very sensitive data that you only want certain people to have access to. Let’s say you move to a new city and want to share your data with your new doctor. Traditionally you would have to have a copy of the DNA (not encrypted) and then when you find out your new doctors public keys you could encrypted that data, store it somewhere and then they should be able to decrypt it. However when using Hako, the data is already encrypted and stored on IPFS, all you need to do is add your new doctors public keys to your files permissions, this would grant then access to retrieve your DNA sequence from IPFS and decrypt it. No need to move data, no need to re encrypt and you get to share the data how you want when you want. Sharing data offline Another awesome use of Hako allows users to share data offline. IPFS is an offline first protocol that does not require a persistent connection to the rest of the web. If one opts to use a locally hosted IPFS gateway — and run a small NuCypher federated network locally. They can achieve all of the security and data sharing benefits that an online Hako user would have. The offline user could be in an area where internet is unstable or not available. This could be in a country where governing bodies attempt to censor data transfer. In both cases Hako could be used to share, permission and view data between parties. Hako is built to be offline first, to preserve security and data integrity and be easy to use. No matter your connectivity to the rest of the world- you should be able to share information securely. There are many other uses that Hako enable and the user stories above are just the start. There are many developments in the underlying technologies that enable Hakos awesome features. NuCypher for instance is working to release NU tokens — which will enable a truly distributed network for access control. IPFS is soon to be native in browsers like Chrome and is already available in Brave (as a dev branch, not publicly released yet) so accessing IPFS files will be as easy as any http site. What’s next The future is bright for distributed web technologies and Hako is just the beginning of powerful ways to share data. Hako was made in less then 4 weeks and is still in alpha stage for the hackathon. There is a lot of work to be done to make Hako the premier data sharing app for Web 3.0 and development will be continued after submission. Please help us make Hako more awesome! Check out the demo, leave a comment, clone the repo and make some pull requests. Hako is by the the people and for the people! GIF demo and how you can try it In order to access Hako fully, please clone the repo and run the app on your computer. This way you’ll be in charge of all your own keys and you can even run Hako offline. Python library is the library that handles NUCIDs and Futari’s. It connects to IPFS and the NuCypher network of choice. This is a developer friendly library that can be integrated into many applications. https://github.com/drbh/ncipfs Hako Application is the web app that provides the UI and sever endpoints for Hako to run. This could be hosted or run locally. Hako also handles all the keys related with a Futari so it should be kept in a secure running location. https://github.com/drbh/hako Note: You’ll need NCIPFS to run hako .Hako expects to find the NCIPFS library in the same directory as Hako, however this can be easily changed in the server file. A demo video The above GIF was created from this youtube video. Check out the video if your interested in scanning the demo at your own pace.

BUNCH

Dropbox

Fetch

Google

Slack

Super

Hako

A High-Throughput, Low-Latency Public Blockchain Platform for Decentralized Economies As blockchain technology marches forward in its quest for mass adoption, the biggest hurdle it faces is arguably scalability. More and more merchants the world over have warmed up to the possibility of , particularly Bitcoin and Ether, as means of payment. But if crypto is to be considered a serious contender for a borderless payment solution, then it has to go head-to-head with the prevalent electronic payment services like Visa and Mastercard. And as things stand today, crypto is lagging far behind. Bitcoin can handle less than 10 TPS, while Ethereum can manage up to about 40 TPS on a good day. Compare this to Visa’s daily 2,000 TPS (which can theoretically scale up to ~50,000 TPS), and you have an idea of the gap that has to be bridged. accepting crypto Various crypto projects have tried out different techniques to solve the scalability bottleneck. Ethereum has been trying to implement a of its own for what seems like an eternity now. Increasing network throughput typically involves making the network more centralized or sacrificing on security. This is known as the classical — when it comes to Decentralization vs Scalability vs Security, you have to choose 2 out of 3. One scalability technique that doesn’t compromise on either security or decentralization is called ‘ ’. Sharding involves fragmenting the available dataset horizontally into smaller datasets, called ‘shards’. Centralized systems have already been using this technique for a long time now, but it has caught the attention of the decentralized world fairly recently. scaling solution Blockchain Trilemma sharding is possibly the poster boy of sharding when it comes to blockchain projects. A ‘high-throughput public blockchain’, Zilliqa has achieved 2828 TPS in their testnet. It should be noted here that in sharding solutions, the TPS increases with the number of nodes. Zilliqa has achieved this with 3600 nodes. Over the last few months, a handful of projects have come up working on the same sharding principle. is one of the most exciting projects working in this space now. A next generation public, scalabale blockchain infrastructure for the decentralized economy, Harmony promises to connect 10 billion people seamlessly by applying “10x innovations in networking, systems and innovations”. But what is Team Harmony doing differently to make sure that they live up to their promise? And what sets Harmony apart from Zilliqa and other predecessors? Let’s take a closer look here. Zilliqa Harmony Website: click here Telegram: join here Whitepaper: read here State Sharding Technique Though both Zilliqa and Harmony employ sharding, Zilliqa implements what is known as and , while Harmony goes one step further and implements . To put it simply, Zilliqa splits its network into multiple shards, each with hundreds of nodes (network sharding). This allows different transactions are processed by different shards concurrently (transaction sharding). However, in order to be able to process transactions, each node in the shard has to store the entire blockchain state information. This prevents machines which have limited resources from being part of the network — not an encouraging sign for decentralization! network sharding transaction sharding state sharding Harmony, on the other hand, shards the blockchain state as well (state sharding). They have christened this unique sharding technique as , wherein they shard not only the transactional layer but also the layer where consensus happens. Nodes can securely locate other nodes that are relevant to the specific transaction, and transact with only those specific nodes in the consensus process. This increases the level of parallel processing, leading to extremely high network throughput. Moreover, since all nodes do not need to store the entire blockchain state, simple personal computers can join the network as nodes too, leading to a greater amount of decentralization. Deep Sharding PBFT Consensus Mechanism Zilliqa uses a consensus mechanism known as (Practical Byzantine Fault Tolerance) protocol. In this algorithm, one node is elected as ‘leader’ and the other nodes become ‘validators’. Each consensus round has 2 phases — and . In each phase, the leader node broadcasts a proposal to all validator nodes, and the validator nodes on their part broadcast their votes to every other node. The votes of all validators need to be counted by every other validator. This increases the communication complexity of PBFT to O(n*n), where n is the total number of nodes. Needless to say, this algorithm becomes unrealistic for networks with hundreds of nodes. PBFT prepare commit Harmony uses a modified consensus algorithm called FBFT, or Fast Byzantine Fault Tolerance. In this method, all validator nodes do not broadcast their votes. Instead, the leader node collects the validators’ nodes through a multi-signature signing process, and then broadcasts it. The multi-signature size is O(1), which reduces the communication complexity from O(n*n) to O(n) — a humongous leap from Zilliqa’s communication complexity. The validators are chosen based on a (Proof of Stake) mechanism, where nodes who wish to become validators have to stake a certain minimum number of tokens. PoS is much less energy extensive than (Proof of Work), which traditional sharding-based blockchains like Zilliqa use. Larger the amount of tokens being staked, greater is the chance to be selected as a leader node. Moreover, validators with larger stakes have a greater incentive of following the protocol and not engaging in any malicious behaviour, since any such activity will result in loss of the entire cache of staked tokens. PoS PoW Distributed Randomness Generation Sharding systems generally employ a randomness-based approach when it comes to assigning nodes to shards, in order to reduce the risk of any attack. A random number generation scheme is used in these cases to assign nodes randomly to shards. This random number should be completely unpredictable and unbiased, so that a malicious node has no way of figuring out which shard it will be assigned to. Moreover, the random number generated should also be verifiable at any given time, and the generation algorithm itself should be scalable. Let us consider some of the sharding-based public blockchain solutions which Harmony has used as case studies. OmniLedger uses the RandHound protocol for distributed randomness generation (DRG), which involves dividing the nodes into groups of size c each. The problem with this approach is that the complexity is O(n*c*c), which is too slow to be feasible for large decebtralized networks. RapidChain, on the other hand, uses a simpler and faster approach, where each node can perform Verifiable Secret Sharing (VSS). Unfortunately, this protocol is fast but not completely secure. Algorand, on its part, uses a Verifiable Random Function (VRF) based cryptographic procedure. And Ethereum’s PoS-based scaling solution ( ) is proposing a Verifiable Delay Function (VDF), where the reveal of the random number would be delayed so as to lessen the chances of any attack. Ethereum 2.0 Harmony utilizes the strengths of both VRF and VDF. The validator nodes each create a VRF-based random number, and send it to the leader node. The leader collects this set of random numbers, and generates the final random number from this data set. VDF is used to delay the reveal of this random number, which makes this approach more secure. Furthermore, the algorithm complexity is only O(n) in this case, which makes it extremely scalable as well. Kademlia Cross-shard Communication For any sharding-based system, the protocol for cross-shard communication is extremely important. Zilliqa depends on the main chain itself for initiating transactions across different shards. In OmniLedger, messages between shards are exchanged through clients. And RapidChain relies on the nodes in the shards themselves to initiate communication with nodes in other shards. Harmony has adopted RapidChain’s shard-driven approach. Typically, the communication complexity for this shard-driven approach is O(n), but Harmony uses the routing protocol to reduce this complexity to O(log(n)). Kademlia In Kademlia routing protocol, each node in the network maintains a routing table that contains the distance of all other shards. This distance is a function of the shard ID. When nodes in a certain shard want to send a message to another shard, instead of broadcasting the message to all neighbouring shards, the nodes will send the message only to the shard with the closes ID. By this method, messages only travel over O(log(n)) shards before reaching their intended destination. Beacon Chain Harmony’s sharding architecture consists of a beacon chain alongwith the shard chains. Each shard chains processes a subset of the total transactions. The beacon chain is a special shard chain with 2 extra responsibilities — generating the random number for DRG, and accepting the stakes from validators. The beacon chain also increases the security of the whole ecosystem. After every new block is added to a shard chain, the block header is sent to the beacon chain (via Kademlia routing). The beacon chain checks if the block header is valid, and broadcasts it to the entire network. Each shard maintains a list of valid block headers, which is used to check if transactions for other shards are valid. With the beacon chain being present, attackers have to corrupt both the shard and the beacon chain if they want to insert an invalid block in the chain. Moreover, since the beacon chain broadcasts all valid block headers itself, the communication complexity is just O(n) once again. If each shard had to separately broadcast the list of valid block headers to all other shards, then the complexity would have been O(n*n). As it turns out, the beacon chain is an extremely important entity in Harmony’s ecosystem both in terms of security and scalability. Parting Thoughts Harmony dreams big, but big dreams are what you need if you want to achieve something monumental. In their testnet, they have achieved 118,000 TPS with around 44,000 nodes. To be honest, bringing forward the product from the testnet to the real world is the actual challenge; and definitely possesses the talent to implement their vision. Moreover, they are a team which listens to the community. You can always find their developers available on Telegram and Discord, ready to answer your queries and listen to your feedback. Finally, this article has been written based on their succint , and I encourage all readers to give the WP a read too. Team Harmony technical whitepaper

Mastercard

Visa

2022 - HackerNoon Contributor of the Year - Blockchain

2022 - HackerNoon Contributor of the Year - Decentralized Internet

Welcome to Insta !

Nominated for 2022 - HackerNoon Contributor of the Year - Blockchain

Nominated for 2022 - HackerNoon Contributor of the Year - Decentralized Internet

Harmony — A High-Throughput, Low-Latency Public Blockchain Platform for Decentralized Economies

This is a summary of , the anonymous payment system coming up as a replacement of Zcoin’s current protocol Zerocoin. Lelantus hides transaction amounts and participants without a trusted setup and relies on simpler cryptographic primitives compared to something like Zcash. It also removed the need for spending or creating fixed denomination coins present in the current and the need for RSA accumulators corresponding to each denomination. Lelantus Zerocoin protocol The basic idea in Lelantus is similar to Monero (and some others) where the spender while spending a coin hides “his” coin among a set of coins (not necessarily “owned” by him) and proves to the ledger (blockchain) that he “owns” a coin in this set and the coin has not been spent yet. This set of coins is called the for that particular spend. anonymity set The reason why Lelantus sounds more attractive that Monero is that the anonymity set can be much bigger (10s of thousands) without incurring any significant performance cost as the table from their paper shows (Benchmarks on Intel Core i7 4th Gen) . Primitives Most of the cryptographic primitives Lelantus relies on are fairly common and well understood, but the 2 most important techniques it relies on are relatively newer. They are One-out-of-Many Proofs for proving ownership of a coin and Bulletproofs for proving that the coin being created does not have a negative value or a large enough value to cause an overflow. However both of these techniques are modified and the paper contains security proof for these modifications and i am unable to understand whether those are correct ;). . Pedersen commitments are of the form gᵐ.hʳ where m is the message being committed to and r is the randomness chosen by the committer. g and h are publicly known generators. (m, r) are called the opening of the commitment. Coins in Lelantus are Pedersen commitments due to the homorphically additive nature of Pedersen commitments, meaning if you have 2 commitments C1 = gᵐ.hʳ and C2 = gⁿ.hˢ to m and n, multiplying the 2 commitments together gives you C1 * C2 = gᵐ⁺ⁿhʳ⁺ˢ, a commitment to m+n.Sometimes randomness r is also called the blinding factor of the commitment. Lelantus uses double blinded commitments meaning that in a commitment to m, there are 2 blinding factors r1 and r2 like gᵐhʳ¹jʳ². Commitments . This technique is used to prove that given n commitments, the prover knows an opening of one of the commitments and that commitment commits to 0 without revealing which one. Hence given n commitments, C1, C2, …Cn, the prover proves that he knows that one of these commitments Ci open to (0, r), i.e Ci = g⁰.hʳ without revealing i. The prover does not need to know the opening to any other commitment than Ci. The proof size for “One-out-of-Many Proofs” is not linear in n but logarithimic in n. It was introduced by Jens Groth et.al in and later made more efficient by J. Bootle et.al in . Since Lelantus’s coins are double blinded commitments, a modified version of this technique is used where the prover proves that one of the commitments, Ci = g⁰hʳ¹jʳ² without revealing i. One-out-of-Many Proofs this paper this paper . This technique lets a prover prove that a committed value lies in a range, also called a range proof. Hence, a prover can prove that in commitment C1 = gᵐ.hʳ, 0≤m<2ⁿ for some n. Previous range proof techniques require a proof size, proving time and verification time proportional to n but with Bulletproofs, these fall down to lg(n). Again Lelantus modifies it to work with commitments of form gᵐhʳ¹jʳ². Bulletproofs In Zcoin’s old protocol Zerocoin, each coin was a commitment that was added to an RSA accumulator (a constant size set), in Lelantus every few blocks, a new set is created which contains commitments to the coins created in those blocks; each of these sets can be referenced with a unique id. Lets call each such set as a “coin set”. The ledger will have a “transparent” currency meaning any transaction involving it will reveal amounts and participants. The currency supporting anonymous and confidential transactions is called “shielded” and such transactions as “shielded” transactions. The ledger. I did not find this improvement published anywhere but learned this from Aram Jivanyan , the author of Lelantus. Coins and transactions Unlike Zerocoin where coins have fixed denominations 1, 10, 25, 50, 100, Lelantus allows arbitrary value in a coin. Each coin hasi) a value Vii) a spending private key q and corresponding public key Q. The private key q is used to sign a transaction where the coin is spent and the ledger verifies the signature with public key Q.iii) a unique serial number S derived from Q by hashing, S = Hash(Q). The serial number is revealed when a coin is spent. The ledger records the serial number in a set and it only allows a spend if it has not seen the serial number before, hence S prevents double spend.iv) a random number R.The coin is represented by the commitment C = gˢh1ⱽh2ᴿ, g, h1 and h2 being 3 publicly known generators. Lelantus has three kinds of transactions 1. Mint. This transaction generates a new coin with a certain value. Assume that the Lelantus was deployed Bitcoin like ledger and someone owned V satoshis. If he wanted to convert his “transparent” V satoshis to a “shielded” coin that he can later use anonymously, he wouldi) Generate a spending private key q and public key Q and serial number S = Hash(Q)ii) Generate a random value R.iii) Generate the coin commitment C = gˢh1ⱽh2ᴿ. He can now use the mint transaction to publish C to the ledger with a proof of ownership of V satoshis where the ledger will consume the V satoshis and add C to a “coin set”. But here he could cheat by creating the commitment with a value greater than V, say C = gˢh1ⱽ⁺¹⁰h2ᴿ rather than C = gˢh1ⱽh2ᴿ. To prevent this, the transaction also contains a Schnorr proof of the relation C/h1ⱽ = gˢh2ᴿ which the ledger will verify as it already knows C and V. Note that the ledger only learns C and V but not S or R. Mint 2. . Spend This transaction does the opposite of as it consumes 1 or more “shielded” coins (commitments) and gives back the “transparent” currency (satoshis) of equal value to the spender of the coins. During Spend, the spender reveals the serial numbers of the coin being spent as well as the total value of the coins being spent The ledger will allow the spend only if a coin with the same serial number has not been spent before. utilizes the homomorphic additive property of Pedersen commitments and One-out-of-Many Proofs to prove that there exist coins with the serial numbers that the spender is revealing and the spender knows V and R for such coins. To ensure spender is not spending already spent coins, the ledger ensures it has not seen the serial numbers before. An additional Schnorr proof is needed to ensure that the spent coin value is the same as the “transparent” currency value generated. Mint however which coins (commitments) are being spent is never known. Spend To spend a coin C = gˢh1ⱽh2ᴿ with value V, the spenderi) Chooses an anonymity set to hide his commitment in. The spender downloads these commitments from one or more “coin set”(s) from the ledger and then assembles a list L of n commitments C1, C2, C3, … Cn with some Ci being equal to C.L = [C1, C2,….. ,…Cn]= [gˢ¹h1ⱽ¹h2ᴿ¹, gˢ²h1ⱽ²h2ᴿ²,…. , …. gˢⁿh1ⱽⁿh2ᴿⁿ].The transaction will contain reference(s) to the “coin set”(s) used so that the ledger can create the same list L.ii) Creates a commitment D = gˢh1⁰h2⁰ and multiplies each element of L with D. L now becomes[gˢ¹⁻ˢh1ⱽ¹h2ᴿ¹, gˢ²⁻ˢh1ⱽ²h2ᴿ²,….. ,….gˢⁿ⁻ˢh1ⱽⁿh2ᴿⁿ]=[gˢ¹⁻ˢh1ⱽ¹h2ᴿ¹, gˢ²⁻ˢh1ⱽ²h2ᴿ²,….. ,….gˢⁿ⁻ˢh1ⱽⁿh2ᴿⁿ]iii) Uses One-out-of-Many Proofs with above list L to create a proof P that one of the commitments in L opens to 0 and he knows which one and the opening for it.iv) Creates a Schnorr proof that the “transparent” value generated is equal to V.v) Finally he signs the transaction with coin’s spending key q and reveals the public key Q. Ci gˢh1ⱽh2ᴿ gˢ⁻ˢh1ⱽh2ᴿ g⁰h1ⱽh2ᴿ The ledger willi) verify the signature with Q andii) recreate the list L with the references to the “coin set”(s) present in the transaction.iii) Will use the public key Q to create S and then create the same D as the spender did.iv) Modifies L as the spender did to get the sameL = [gˢ¹⁻ˢh1ⱽ¹h2ᴿ¹, gˢ²⁻ˢh1ⱽ²h2ᴿ²,….. ,….gˢⁿ⁻ˢh1ⱽⁿh2ᴿⁿ]iv) Uses One-out-of-Many Proofs to verify the proof P.v) Verifies the Schnorr proof to ensure correct V. g⁰h1ⱽh2ᴿ 3. JoinSplit. transaction can be considered a combination of and transactions as they can spend old coins and create new coins and optionally create some “transparent” value Vₜ. A transaction spending N old coins and creating M new coins should prove thati) He knows S, V and R for each of these N coins and none of them have been spent before. This is achieved by the same method as the used in the transaction .ii) All new coins M have a positive value and are not big enough to cause overflows. This is done with Bulletproofs.iii) The sum of value of all N old coins = (sum of value of all M new coins + any “transparent” value Vₜ). This is done with a Schnorr proof. JoinSplit Mint Spend JoinSplit Spend with the difference that the value V is not revealed for any coin The verifier will now verify the N signatures and N One-out-of-Many Proofs like transaction, M Bulletproofs and a Schnorr proof to ensure that the prover did not create money out of nothing. Spend Note that the verifier only learns that the sum of the value of old coins equals the sum of the value of new coins but not the individual value of any old or new coin. The spender should aim to use more than one such “coin set”s (created far apart in time) in a Spend or JoinSplit to avoid timing based correlation. Anonymous payment transfer Until now, the transactions described do not show how one party can pay another. This is achieved by the sender creating a coin such that only the recipient knows the spending private key corresponding to that coin. To spend a coin C = gˢh1ⱽh2ᴿ, the tuple (S, V, R) needs to be known. Since S can be learnt by hashing the public key Q and Q can be created with private key q, the coin can be spent with knowledge of (q, V, R).Say Alice wants to pay Bob a value V.i) Bob chooses a secret x and sends Alice gˣ.ii) Alice chooses a value y and creates gˣʸ which becomes the public key for that coin. The coin’s private key is now x*y.iii) Alice now creates a coin C = gˢh1ⱽh2ᴿ where S = Hash(gˣ*ʸ). Alice cannot spend this coin since it does not now x but Bob can if he knows y.iv) Alice can now create a transaction where she spends a coin she owns and creates a new coin in the above manner.v) Alice now communicates y, V and R to Bob either on a private channel or it can publish them alongside this transaction encrypted with Bob’s public key gˣ (Elgamal encryption seems appropriate). JoinSplit Note that Alice can learn when Bob spends the above coin C since it knows the public key gˣ*ʸ and hence serial number S. To prevent Alice from learning Bob can swap the coin for a new coin (in a + ) immediately. Spend Mint Performance The paper describes several optimizations like the ledger batching multiple One-out-of-Many Proofs that were created over the same anonymity set. The ledger can do some precomputations to verify these proofs faster. For the prover (spender), Bulletproofs support efficient aggregation which makes transactions with more than 1 new coins to have shorter proof size.However, the prover does have to download (and probably store) commitments equal to his anonymity size so for an anonymity size of 2¹⁶ commitments, the prover has to download 2¹⁶ -1 commitments and do computation (and spend time) proportional to 2¹⁶. JoinSplit It is yet to be seen how efficient such large computations are on light clients like mobile devices. Thanks to the paper’s author Aram Jivanyan for answering questions regarding Lelantus.

Alongside

Intel

ZCoin’s new anonymous payment system, Lelantus

Polkadot

Analytica

Apple

Coinbase

Dapp

Facebook

Ledgers

Stripe

Twitter

What Bitcoin Did [the podcast]

Amber Baldet on Surveillance Capitalism

fees

Southern

Andrew Poelstra on Bitcoin Math & Research

AirBnB

Progressive

Routine

Will These UX Trends Stick or Fade Away?

Neural Networks that represent a supervised learning method, requires a large training set of complete records, including the target variable. Training a deep neural network to find the best parameters of that network is an iterative process, but training deep neural networks on a large data set iteratively is very slow. So what we need is that by having a good optimization algorithm to update the parameters (weights and biases) of the network can speed up the learning process of the network. The choice of optimization algorithms in deep learning can influence the network training speed and its performance. In this article, we will discuss the need for improving the gradient descent optimization technique and we will also discuss the different variants of gradient descent optimization algorithm. Citation Note: The content and the structure of this article is based on the deep learning lectures from One-Fourth Labs — Padhai Rest of the article is structured as follows, Gradient Descent Momentum Gradient Descent Nesterov Accelerated Gradient Descent Mini-Batch & Stochastic Gradient Descent Adaptive Gradient Descent — AdaGrad Root Mean Square Propagation Gradient Descent — RMSProp Adaptive Moment Estimation — Adam Gradient Descent Gradient Descent is one of the most commonly used optimization techniques to optimize neural networks. Gradient descent algorithm updates the parameters by moving in the direction opposite to the gradient of the objective function with respect to the network parameters. Gradient descent algorithm for single sigmoid neuron works like this, Parameter update rule will be given by, Initialize the parameters randomly and and iterate over all the observations in the data. w b Calculate the value of the gradient for each parameter (i.e. In which direction we need to move such that loss is reduced). Update the value of each parameter based on its gradient value. Continue doing step 2 and 3 till loss function gets minimized. defines the size of steps we take to reach the minimum. In other words, it controls how fast or slow should we converge to the minimum. Learning rate Can we come up with a better update rule? To understand the gradient descent update rule in a better way, I have taken some toy data and iterated over all the data points for 1000 epochs and computed loss for different values of and Once I got the loss values for all possible combinations of and , I was able to generate an animation that shows the gradient descent rule in action. w b. w b Gradient Descent Rule in Action (Animation) The darker shade of red color in the graph indicates the area where the loss value is a high and darker shade of blue in the valley indicates the lowest point of loss. The points at the bottom indicate the different combinations of & (parameters) and the points on the contour indicate the loss value for the corresponding parameter values. w b From the gradient descent loss animation, you would have observed that for initial iterations while the curve is still on the flat light red surface, the and values at the bottom of the contour plot change very little for every epoch. It means that we are taking very smaller steps towards our goal but once our loss curve enters into the valley (light blue area) for every epoch the values of and are changing drastically (black bots are far from each other) w b w b Small Steps towards Loss (Left) & Large Steps towards Loss (Right) In short, in the gentle areas of the loss surface, the movement is very slow and in the steep areas the movement is fast and again at the gentle areas at the bottom, the movement is very slow. But Why? We have made a pretty interesting observation about the gradient descent update rule but we don’t know why the movement is slow in some parts and fast in some parts of the contour surface. Let’s take a simple function shown below, As you can see from the figure, the curve is a combination of very steep region and gentle region. The slope (derivative) of the curve in the steep region evaluates to a value of 3 (change in y = 3, change in x = 1) similarly, the slope of the curve in the gentle region evaluates to a value of 1. So we infer that when the slope is very steep the derivative(gradient) is high and when the slope is gentle the derivative is low. The derivatives at the steep slopes are larger in magnitude, whereas for the gentle slopes they are smaller in magnitude. Now relating this information to the gradient descent animation, when we are at the plateau with a gentle slope the derivatives at that area will be small. If the derivatives are small then the update of the parameter will also be small. Hence in the areas where the curve is gentle, the updates are small whereas in the areas where the curve is steep the updates are large So What? We have figured out the reason why the gradient descent update is moving slowly in some regions and faster in some regions of the contour. what is the problem with this kind of movement in the update rule? The problem is that in gradient descent, we are initializing parameters randomly and if that so happens that we initialize and in a place where the surface is very gentle then we need to run many many epochs just to come out that flat surface and enter into the slightly steep region from there you will start seeing some improvements in loss function. In order to avoid this kind of scenarios, we need to improve our update rule. w b Contour Maps Before we proceed to see some improvements to the gradient descent update rule first we will take a small detour and understand how to interpret contour maps. Contour maps help us to visualize 3D data in two dimensions using contours or color-coded regions. Contour maps tell about the movement of the slope along the particular direction based on the distance between the corresponding contours in the 2D contour map. A small distance between the contours indicates a steep slope along that direction.A large distance between the contours indicates a gentle slope along that direction. Visualize Gradient Descent using Contour Map The 3D representation of gradient descent error surface looks like this, Gradient Descent Convergence in 3D Gradient Descent Animation (Contour Map) We can notice that from the above figure as the curve reaches the steeper surface it takes larger and larger steps towards convergence represented with sparse points. Once the curve reaches the dark blue flat region the slope in that direction is gentle that means the distance between the consecutive contours in the dark blue region would be less. As you can see from the animation once the curve reaches the cluster of light blue contours where the distance between consecutive contours is less it takes larger and larger steps towards convergence. Momentum-based Gradient Descent From our discussion on gradient descent, it is clear it takes a lot of time to navigate regions having a gentle slope because the gradient in these regions is very small. How can we overcome this? Let’s consider a situation where you are going to a recently opened shopping mall in an unknown area. While you were trying to locate the mall you have asked multiple people about the location of the mall and everyone directed you to go towards the same location. Because everyone is pointing you towards the same direction you will go faster and faster in that direction with more confidence. Now we will use the same intuition in momentum based gradient descent, Momentum GD In Momentum based gradient descent update rule, we also included the history component vₜ it stores all the previous gradient movements till this time t. Exponential Weighted Average At every time step or every update instead of just moving by the value of the current gradient like the Vanilla GD. In Momentum GD, we are moving with an exponential decaying cumulative average of previous gradients and current gradient. Now let’s see how Momentum GD would perform on the same error surface, Momentum-based GD Animation From the animation, it is evident that Momentum based gradient descent is moving faster than the vanilla GD on the flat surface because it is not only using the current gradient value to update the parameters but also a history of gradients value till that point. By the time it has taken around 5 to 6 steps on the flat surface, it’s history would have grown up to a certain extent, allowing the Momentum gradient descent to take larger and larger steps on the flat surface. Is moving fast always good? To analyze the limitations of Momentum based gradient descent on a different error surface, I have taken some other toy data set and used single sigmoid neuron to find the loss value at different combinations of and to generate 3D loss surface which is having narrower minima surface. w b Momentum-based GD convergence Momentum-based gradient descent oscillates in and out of minima because it would have accumulated more history by the time it reaches minima resulting in taking larger and larger steps evidently leads to overshooting the objective. Despite the u-turns, it still converges faster than the vanilla gradient descent. Nesterov Accelerated Gradient Descent How can we avoid the overshooting of minima?. In Momentum based gradient descent we are making one movement in the direction of gradient history and another in the direction of the current gradient because of this two-step movement we are overshooting the minima. Instead of moving two steps at a time, why not first move a little in the direction of gradient history, compute the gradient at that point and update the parameters. In essence, what we are doing in Nesterov Accelerated Gradient Descent is that we are looking forward to see whether we are close to the minima or not before we take another step based on the current gradient value so that we can avoid the problem of overshooting. Comparison between NAG(Red curve) & Momentum(Black curve) We can see that all the oscillations made by the Nesterov Accelerated Gradient are much smaller than that of Momentum based Gradient Descent. Looking ahead helps NAG in correcting its course quicker than Momentum based Gradient Descent. Hence the oscillations are smaller and the chances of escaping the minima valley are also smaller. Mini-Batch and Stochastic Gradient Descent Should we use entire data for computing the gradients? Consider that you have a huge dataset with million data points if we use the batch gradient, the algorithm will take a million calculations (compute derivates for each of the million points and accumulate all these derivates) and then make one tiny update to our parameters. Can we do Something better? Instead of looking at all data points at one go, we will divide the entire data into a number of subsets. For each subset of data, compute the derivates for each of the point present in the subset and make an update to the parameters. Instead of calculating the derivative for entire data with respect to the loss function, we have approximated it to fewer points or smaller batch size. This method of computing gradients in batches is called the . If we have one million data points with a batch size of 10, we will have 100,000 batches and we would have made 100,000 updates to the parameters. Mini-Batch Gradient Descent If we take the idea of mini batch Gradient descent to the extreme and look one point at a time, compute the derivative and make an update to the parameters for every point. This method is called the . Stochastic Gradient Descent Stochastic Gradient Descent Convergence In Stochastic we update the parameters for each point, the points are not working in consensus. Every point is making the greedy decision and updates the parameters best suitable for that point alone which may be the case for the remaining points. We can reduce the problem of oscillations is by considering a subset of data or batch of data and update the parameters after calculating the derivates for each point within that batch. Comparison of Stochastic (Blue Curve) & Mini-Batch GD (Red Curve) We can see the oscillations have reduced in Mini-Batch gradient descent and oscillations are completely contained within the blue curve. In practice, we choose the batch size equal to 16, 32 and 64. Whether we are using Mini-Batch or Stochastic gradient descent key point to note is that we are approximating the derivatives with respect to loss function, we are not computing the true derivatives of the loss function. Number of updates (steps) made to the parameters in one epoch, Batch Gradient Descent — 1 Stochastic Gradient Descent — N(Number of data points) Mini-Batch Gradient Descent — N/B(B = Batch Size) AdaGrad AdaGrad — Gradient Descent with Adaptive Learning Rate The main motivation behind the AdaGrad was the idea of Adaptive Learning rate for different features in the dataset, i.e. instead of using the same learning rate across all the features in the dataset, we might need different learning rate for different features. Why do we need Adaptive Learning rate? Consider a dataset that has a very important but sparse variable, if that variable is zero in most of the training data points the derivative proportional to that variable will also be equal to zero. If the derivative is equal to zero then the weight update is going to be zero. If our parameters (weights) are not moving towards the minima then the model will not make optimal predictions. To aid such sparse features we want to make sure that whenever that feature value is not zero whatever is the derivative at that point it should get boosted by a larger learning rate. AdaGrad Intuition If a particular parameter is getting updated frequently that means the derivative is not zero most of the times, for such parameters we want to have a smaller learning rate in contrast if we have a sparse parameter which will be off (zero) most of the times that mean the derivative will be zero most of the times. For such feature whenever this feature is on (not zero) we want to boost the gradient update with a higher learning rate. In AdaGrad, we are dividing the learning with the history of gradient value until that point. Non-sparse features will have a large history value because they would be getting frequent updates, by dividing the learning rate with the large history, the effective learning rate would be very small. In the case of sparse features, gradient history value would be very less leading to large effective learning rate. Consider that we have data with two features (sparse feature) and which means undergoes fewer updates. Now we will compare AdaGrad update rule with Vanilla GD (black curve), Momentum GD (red), NAG (blue) and AdaGrad (green) w b w Comparison of different GD variants From the above plot, we can see that AdaGrad is taking larger steps in the direction of even though it is a sparse feature in contrast to the other variants because of its gradient value is boosted by the learning rate. The disadvantage of AdaGrad is that the learning rate decays very aggressively as the denominator grows which is not good for parameters corresponding to dense features. As gets updated, again and again, denominator increases a lot and the effective learning rate becomes close to zero, as a result, AdaGrad is no longer able to move in the direction of and gets stuck close to the convergence. w b b RMSProp RMSProp — Root Mean Square Propagation To prevent the denominator from growing rapidly why not decay the denominator and prevent its rapid growth? RMSProp uses this intuition to prevent the rapid growth of the denominator for dense variables so the effective learning rate doesn’t become close to zero. RMSProp Equation In RMSProp history of gradients is calculated using an expoential decaying average unlike the sum of gradients in AdaGrad, which helps to prevent the rapid growth of the denominator for dense features. RMSProp Brown Curve Next to AdaGrad Green Curve Since the denominator for dense features ‘ ’ is not decaying as aggressively as AdaGrad, RMSProp is able to move in the direction of eventually leading to convergence. b b Adam The name Adam is derived from adaptive moment estimation In Momentum based gradient descent we are using the cumulative history of gradients to move faster in the gentle surfaces and we have seen RMSProp which is also using history to decay the denominator and prevent its rapid growth. The way these algorithms uses the history is different, In Momentum GD, we use history to compute the current update whereas in case of RMSProp history was used to adjust the learning rate (shrink or boost). Adam combines these two separate histories into one algorithm. Adam maintains two histories, ‘mₜ’ similar to the history used in Momentum GD and ‘vₜ’ similar to the history used in RMSProp. In practice, Adam does something known as bias correction. It uses the following equations for ‘mₜ’ and ‘vₜ’, Bias Correction Bias correction ensures that at the beginning of the training updates don’t behave in a weird manner. The key point in Adam is that it combines the advantages of Momentum GD (moving faster in gentle regions) and RMSProp GD (adjusting learning rate). Adam Optimizer (cyan color curve) Because Adam has a property of Momentum GD, we can see that it overshoots the minima and comes back, makes few u-turns before convergence. Summary In this post, we have looked at the batch gradient descent, the need to develop new optimization techniques, and then we briefly discussed how to interpret contour plots. After that, we have looked at behind six different optimization techniques and three different data strategies (batch, mini-batch & stochastic) with an intuitive understanding which helps to know where to use any of these algorithms. In Practice Adam optimizer with mini-batch of sizes 32, 64 and 128 is the default choice, at least for all the image classification tasks which deal with CNN and large sequence to sequence models. What’s Next? Backpropagation is the backbone of how neural networks learn what they learn. If you are interested in learning more about Neural Networks, check out the by Abhishek and Pukhraj from . This course will be taught using the latest version of Tensorflow 2.0 (Keras backend). Artificial Neural Networks Starttechacademy

The Graph

Target

Data Science Courses Holiday Sale

Demystifying Different Variants of Gradient Descent Optimization Algorithm

In this post, I show an example of simple application based on the . The application uses Amazon Appsync, Lambda, DynamoDB, , to demonstrate how to use Amazon DAX as caching layer in front of AWS Appsync Resolvers. Serverless framework Amazon DynamoDB Accelerator (DAX) Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available, in-memory for that delivers up to a 10x performance improvement — from milliseconds to microseconds — even at millions of requests per second. cache DynamoDB A simple serverless CMS The example application in this post is a simple CMS. I use Serverless framework templates to simplify the setup for Appsync. The sections in template that create the Appsync, Dynamodb, Cognito identity pool and user pool, DAX cluster, roles, security groups, and subnet groups, and you can use them for automatic and repeatable deployments. The following diagram illustrates application solution: Getting started with Serverless Framework To start, you need to have the Serverless Framework installed: $ npm install -g serverless Now, let’s create a new serverless project: $ sls create --template aws-nodejs --path simpleCMS
$ cd simpleCMS
$ npm init The directory that is created includes two files — handler.js is the Lambda function. The serverless.yml file contains the configurations of the backend. Add Plugins You need to add two plugins of Serverless framework: and : serverless-appsync-plugin serverless-pseudo-parameters $ npm install serverless-pseudo-parameters
$ npm install serverless-appsync-plugin Edit the serverless.yml file and add the plugins to section: plugins plugins:
  - serverless-appsync-plugin
  - serverless-pseudo-parameters Define the AWS AppSync Schema Let’s look at the schema. Schema files are text files, usually named schema.graphql, the sample scheme should look like as below: <a href="https://medium.com/media/5cf9c8c080c05375eb1fd89c8c6f206c/href">https://medium.com/media/5cf9c8c080c05375eb1fd89c8c6f206c/href</a> Create Mapping Templates Now we have our schema defined we need to add resolvers for it, The mapping template files should be located in a directory called mapping-templates relative to the serverless.yml file. Pipeline resolvers AppSync team released resolvers last year, Pipeline resolvers offer the ability to serially execute operations against data sources. pipeline In this example, there are two functions attached Query and Mutation APIs , The pipeline resolver to run the following logic: Connect to the Lambda datasource, get/put item from/to DAX. If the Lambda function raises an error, connect to DynamoDb datasource, get/put item from/to DynamoDb directly. Let’s create the request template for adding post mutation in a file called mapping-templates/Mutation-addPost-request.vtl , note that it is a resolver. It will save the argument to context stash and invoke the attached pipeline functions. pipeline $util.qr($context.stash.put("id", $util.autoId()))
$util.qr($context.stash.put("title", $ctx.args.post.title))
$util.qr($context.stash.put("createdAt", $util.time.nowISO8601()))
$util.qr($context.stash.put("content", $ctx.args.post.content))
{} Then create template for the first pipeline function mapping-templates/Function-addPostViaDax-request.vtl which will invoke a lambda function to insert post to . DAX {
    "version": "2017-02-28",
    "operation": "Invoke",
    "payload": {
        "field": "addPostViaDax",
        "arguments": $utils.toJson($context.stash)
    }
} Next let’s create the second pipeline function template mapping-templates/Function-addPostToDB-request.vtl which will insert post to DynamoDB directly if previous lambda function execrated failed. #if($ctx.prev.result.errorMessage)
{
    "version" : "2017-02-28",
    "operation" : "PutItem",
    "key": {
       "id" : $util.dynamodb.toDynamoDBJson($context.stash.id)
    },
    "attributeValues" : {
        "title" : $util.dynamodb.toDynamoDBJson($context.stash.title) ,
        "content" : $util.dynamodb.toDynamoDBJson($context.stash.content),
       "createdAt":  $util.dynamodb.toDynamoDBJson($context.stash.createdAt)
    },
}
#else
  #return($ctx.prev.result)
#end Define Dependent Services Now, I need a DynamoDB table, Appsync, Amazon Cognito user pool and identity pool, DAX in a Vpc, Security group and some IAM roles to hook them all together. These are placed in the section of the serverless.yml. resources Define the AppSync GrpahQL API Let’s define the AWS AppSync resource, Add the following example config to the section of serverless.yml custom <a href="https://medium.com/media/765d21742279f0a537037d3b1ee293fd/href">https://medium.com/media/765d21742279f0a537037d3b1ee293fd/href</a> As you can see, each query and mutation has an entry in the mapping templates. Define the DynamoDB table The next section of this code example creates a DynamoDB table. PostsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.postsTable}
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 1
      StreamSpecification:
        StreamViewType: NEW_IMAGE This table has only a single hash key. The are kept low because DAX serves most of the read traffic. DynamoDB is called only if DAX has not cached the item. ProvisionedThroughput ReadCapacityUnits Define the DAX cluster Now let’s create DAX cluster, the template sets up following services: DAX cluster with one t2.small node. Security group with a rule to allow Lambda to send traffic to DAX on TCP port 8111. An IAM role allow DAX access to DynamoDB. A VPC refers to this security group. Security groups control how network traffic is allowed to flow in a VPC. <a href="https://medium.com/media/8d3a78df7bdef234caa4e5034f7a01b1/href">https://medium.com/media/8d3a78df7bdef234caa4e5034f7a01b1/href</a> Define VPC into Lambda To properly connect AWS Lambda to DAX, I need to deploy your lambda functions into the same VPC of DAX, Configure section in serverless.yml file: provider provider:
  vpc:
    securityGroupIds:
      - Fn::GetAtt: [DaxVpcSecurityGroup, GroupId]
    subnetIds:
      - Ref: daxSubnet1
      - Ref: daxSubnet2 The other parts of the AWS configuration like authentication, schema, and data sources etc. you can find them in My . GitHub repo The code The lambda function code is in . The Lambda handler uses environment variables for configuration: is the name of the table containing post data, and is the DAX cluster endpoint. These variables are configured automatically in the serverless.yml. handler.js POSTS_TABLE_NAME DAX_ENDPOINT When there is mutation or query request, Appsync would invoke lambda function data source to insert or get data from DAX, If Lambda function raises an error, Pipeline resolver will go to next pipeline function and connect to DynamoDb datasource, get/put item from/to DynamoDb. <a href="https://medium.com/media/9a63674059e79e70365f2f6037a6beca/href">https://medium.com/media/9a63674059e79e70365f2f6037a6beca/href</a> Deploy your API To deploy your API, run the following: $ serverless deploy -v Conclusion In this post, I showed how to use Serverless framework to create Appsync lambda data source that uses DAX and DynamoDB to implement simple quey and mutation. I hope you have found this article useful, the full code example is available on my GitHub repo .

Amazon

How to Add a Cache Layer to Serverless GraphQL AppSync API

Enough with the Chainsplaining (Part 1): How to Talk Blockchain Empathetically Chainsplaining starts innocuously enough. Maybe you’re at a cocktail party or a neighborhood gathering, and blockchain happens to bubble up in casual conversation. “Blockchain? That’s like Bitcoin right?” Or “Yeah, I’ve read a bit about Crypto but I still don’t get it…” Or “Didn’t everyone lose a bunch of money on ICOs?” You might be bursting at the seams to explain, describe, and demystify the layers of blockchain. And in the beginning, your audiences will probably greet this with a mixture of feigned curiosity, FOMO, surveillance unease, and disdain for yet another over-hyped techno-fad. But — before you know it — you’re twenty minutes deep into a tirade about the illusory root value of fiat currency, the inefficiencies of government bureaucracy, and the promises of a crypto-utopian future… that seems so close you can almost touch it. Meanwhile, your previously just-being-polite audience is looking for the exit. Sound familiar? Yep. You’ve been chainsplaining. Or maybe this is just me… 🤔😅 Whether you’ve been a fellow-chainsplainer or a victimized chainsplainee, it’s not a great place to be. It’s tempting to try and share everything you’ve ever learned about blockchain in the time that it takes the ice in your drink to melt… but for the crypto-curious or the crypto-skeptical, chainsplaining is ineffective and awkward. Chainsplain : to mansplain cryptocurrency and/or blockchain technology. However, blockchain come up in social settings these days. And it deserves a thoughtful dialogue since it’s so new and often counterintuitive. How might we describe blockchain without someone excusing themselves to refresh their beverage…again? does While I can’t claim to have mastered the art of explaining blockchain, along my own adventures down the rabbit hole I’ve stumbled across a few non-technical ways to help describe things so that people actually have a fighting chance of lighting the lightbulb. This is a big topic, but here are a few top-level lessons-learned to help keep things on the rails. So take a read and keep me honest. Empathize with your audience, and try to remember they’re new to all this Chainsplainers can be unbearable because they launch straight into blockchain technology — name-dropping new terms and complicated concepts — to an audience who often doesn’t have the slightest idea what they’re talking about. And an audience that feels overwhelmed and disoriented will soon want to get the heck outta dodge. That’s why before I get into , I try my best to lead with here’s how this grand machine works why any of this might be interesting to you. Forget the tech, instead start with a story People learn through stories. Our brains naturally seek out a narrative structure with a beginning, a middle, and an end. Stories are how we receive new information. Stories provide the metaphorical maps that enable us to navigate new experiences. And stories are also how we recall information to share with others moving forward. When I start to introduce the complex concept of blockchain, I do my best to do so in a familiar narrative with relatable concepts to make my audience feel a little more comfortable. Use an example they already understand For better or worse, analogies involving money tend to work well, as most people naturally associate cryptocurrencies with blockchain. Simple props help. For instance, I might take my wallet out of my back pocket and explain, “In the fiat world, if I want to give you $20.00, I take a twenty dollar bill out of my wallet, hand it over to you, and then you put that money inside your wallet. In this world, the money moves. It changes hands. It can be in transit. And critically it changes possession in a tangible, physical sense.” At this point in my conversation, people’s heads are nodding slowly as they begin to wonder why I’m bothering to be precise over something so familiar. Which is the perfect opportunity to surprise them. Knock them off balance “Nothing we’ve just said is true — or even possible — in the world of blockchain. My crypto wallet will never ‘contain’ $20.00 worth of cryptocurrency in the same way that my physical wallet contains this $20.00 bill. I literally cannot ‘hand’ cryptocurrency to you. And your crypto wallet will never have $20.00 worth of cryptocurrency inside of it.” My crypto wallet will never ‘contain’ $20.00 worth of cryptocurrency in the same way that my physical wallet contains this $20.00 bill. At this point, folks tend to lean in, squint their eyes, and maybe tilt their heads. “Wait… what?” This is not where they expected the conversation to start. And, significantly, they have just asked their first exploratory question. Pro-tip: the presence (or absence) of curious questions is the easiest and most immediate method to gut-check if you might be mansplaining, chainsplaining, or any other *splaining. “In the crypto world, the ‘money’ is fixed in time and space. Sort of like bricks in a wall. Imagine that each brick is worth $20.00. I can prove that I control a specific brick. And I can transfer control of that brick to you. Now it is your brick. However, that brick was never in my wallet. And it will never be in your wallet. It’s just a brick on a wall that you can prove that you control. When you transfer control over it, it switches to the next person almost instantly. So… in the crypto world, the money doesn’t move. Only the permissions move. I can transfer the value, but it was never inside my wallet, and it will never be inside of your wallet. This one concept has profound implications: at a relatively fundamental level, blockchain technology enables digital ownership but explicitly does not allow any notion of possession. Ownership without possession. This changes almost everything.” I’m watching for folks to look away, defocus their eyes a bit, and say something along the lines of “Whoa… nothing I’ve read about so far mentions anything like this.” Acknowledge that the learning journey is difficult Learning about blockchain is hard because everyone has expectations about how things work. The word ‘wallet,’ for example, has an existing meaning. Most people think of it as a container that holds cash and cards. In blockchain, the word ‘wallet’ means something else entirely — and new learners are confronted with countless counterintuitive terms like this. But before explaining a new concept, it’s much easier if we first acknowledge and begin to break down preconceived notions and baseline assumptions. That’s one of the hardest parts of the blockchain journey: in order to make heads or tails of this new world, people must first a whole bunch of deeply held notions. Notions that are so second-nature as to barely be recognized as assumptions at all. And that’s really, really tough. unlearn In order to learn make heads or tails of this new world, people must first unlearn a whole bunch of deeply held notions. But this is where the ocean meets the shore. And where the descent down the blockchain rabbit hole needs to begin if new users are going to be interested enough to explore further, and persistent enough to work through the inevitable moments of frustration and confusion that come with this territory. Interested in upcoming chapters in this narrative? At this point in their journey, my audience is usually both curious and confused… and starting to be compelled to stick around to learn more. In future posts, I hope to share additional narrative techniques that I’ve found help continue the blockchain conversation in language and metaphors that bring the key concepts alive and help people light the lightbulb. In the meantime, give it a shot and let me know how it goes. I’d also love to learn about the stories that you find effective when talking blockchain. . You can find me on Twitter via @mcutler Big thanks to Sarah Mills , Taylor Monahan , Seth Goldstein , and the Blocknative team for providing feedback on earlier drafts of this post! Originally published at blog.blocknative.com .

Fomo

Instantly

Transit

Enough with the Chainsplaining (Part 1): How to Talk Blockchain Empathtically

Good UX design is all about putting the user first. Any aspect of a website or app that doesn’t consider the user’s needs fails. “People ignore design that ignores people.” — Frank Chimero, Designer by Nike90 Store Balraj Chana Space Odyssey by MadeByStudioJQ Yomes — Renters Transparency Initiative by Claudio Guglieri Work hard by JON NorthFace by Eddie Lobanovskiy Surf Article Interaction by Bradley Ziffer Divers App Concept by Dannniel by Charity Website Riko Sapto Dimo Motel | Food App by Johny vino™ Youtube concept design — profile by for Yi Li Hiwow Marble Interactive TV App — Case Study by for Afrian Hanafi Paperpillar Experimental Work by Divan Raj Photography Blog by Charles Patterson Movies Site by Aji Darmawan by HQ Food Delivery Homepage Aleks Faure Iphone X Instagram Concept by Ivan Martynenko #Exploration | Website Learn to Surfing by for Dwinawan Paperpillar MH home page by Anton Skvortsov Fashion Butique — slider animation by kreativa by Mixing — Online music service website concept Anmol Arora Thanks for watching this… Top UX Inspirations of all time!

Inspiration for UX Designers: Putting the User First

HackerNoon Tech Stories Archive April 7th, 2019

Playing With Yourself: The Power of Personal Projects

@cyengmann
Nick Engmann

Hako

@david.richard.holtz
David Richard Holtz

Harmony — A High-Throughput, Low-Latency Public Blockchain Platform for Decentralized Economies

@falconite
Rohit Chatterjee

ZCoin’s new anonymous payment system, Lelantus

@loveshharchandani
Lovesh Harchandani

Amber Baldet on Surveillance Capitalism

@whatbitcoindid
Peter McCormack

Andrew Poelstra on Bitcoin Math & Research

@whatbitcoindid
Peter McCormack

Will These UX Trends Stick or Fade Away?

@miklosphilips
Miklos Philips

Demystifying Different Variants of Gradient Descent Optimization Algorithm

@NKumar
NiranjanKumar

How to Add a Cache Layer to Serverless GraphQL AppSync API

@yia333
Yi Ai

Enough with the Chainsplaining (Part 1): How to Talk Blockchain Empathtically

@mcutler
Matt Cutler

Inspiration for UX Designers: Putting the User First

@AbhaySalvi
Abhay Salvi

HackerNoon Tech Stories Archive April 7th, 2019

@cyengmannNick Engmann

@david.richard.holtzDavid Richard Holtz

@falconiteRohit Chatterjee

@loveshharchandaniLovesh Harchandani

@whatbitcoindidPeter McCormack

@whatbitcoindidPeter McCormack

@miklosphilipsMiklos Philips

@NKumarNiranjanKumar

@yia333Yi Ai

@mcutlerMatt Cutler

@AbhaySalviAbhay Salvi

@cyengmann
Nick Engmann

@david.richard.holtz
David Richard Holtz

@falconite
Rohit Chatterjee

@loveshharchandani
Lovesh Harchandani

@whatbitcoindid
Peter McCormack

@whatbitcoindid
Peter McCormack

@miklosphilips
Miklos Philips

@NKumar
NiranjanKumar

@yia333
Yi Ai

@mcutler
Matt Cutler

@AbhaySalvi
Abhay Salvi