Hydrais in the Database Technology Industry

Hydra

hydra

In my last post I wrote about the first session of from two weeks ago. The twitch stream was an all day event where the first half of the day consisted of a variety of workshops around creative coding topics, while the second half featured performances from various audio visualization artists and creative coders. Unfortunately I could not attend all the events, but I wanted to write in detail about both the Hydra event and P5JS event. You can . Now let’s dive into some live coding visuals! SpacyCloud’s Live Twitch stream read the P5JS post here The Hydra tutorial on SpacyCloud was taught by Zach Krall, a graduate student at Parsons School of Design with an impressive portfolio of projects. Though I had been experimenting with creative coding since college and knew about , the language that P5JS was ported from, I had never heard of Hydra before. Just the fact that it was something new peaked my interest, but when I saw the home page for the I was pretty much sold. Every time you load Hydra, a different visualization appears on the screen, with the code that wrote to make it overlaid on top. You can copy and paste the code, so in a way each new visualization is like its own mini tutorial. Processing Hydra-editor Zach Krall’s personal website It turns out that all of the coding for Hydra happens in the browser, and the background of the entire browser window changes to display the product of your code. Personally I prefer this over the two panel system that most web coding editors use, because when it comes to visualizations you want to be able to see them in as large a display as possible. However I could see some people not liking this, because the code is a bit harder to read, even though it does have a background color applied. Hydra was created by who wanted to build a visualization engine that took its inspiration from analog televisions. It did that and a lot more, because with Hydra you can connect to other machines and each output your own video stream that can then be modified by others. Olivia Jack Probably the hardest thing about starting out with Hydra is wrapping your head around some of the paradigms, which are pretty different from your typical application. In Hydra, you typically start with a basic visual preset or texture, like noise, voronoi, or oscillation. Check out these basic visuals below. Note that while these screenshots are static, within Hydra all of these are moving visualizations. A basic oscillation visualization is drawn to the browser with osc() A basic voronoi visualization is drawn to the browser with voronoi() A basic noise visualization is drawn to the browser with noise() You can also pass values into the function to change it. For example, if I write instead of just the gray matter gets much smaller, like specks of dust rather than blobs. If you pass the specks of dust will start moving around the screen MUCH more quickly. The same can be said for voronoi and oscillation. First number defines the density of shapes, the second defines the speed of movement. Be careful passing in large numbers for the speed, it can be quite painful on the eyeballs. noise(100) noise() noise(100, 100) In order to execute the code you need to hit on the keyboard. You might have noticed the code inside the screenshots include a second function chained on called . This function is basically telling the browser to output everything in front of it in the chain. If you remove nothing will render to the browser and you will only see a black page. Shift + Ctrl + Enter out() out() We’ve covered voronoi, noise, and oscillation. There’s one more basic render and that is . Drawing a shape in Hydra is simple enough. The number you pass into the function defines the number of sides for the polygon. So, is a triangle while is a rectangle, and so on. shape() shape() shape(3) shape(4) You can also specify how large each shape is and how blurred its edges are by passing in 2 more numbers into the function. You might be wondering, what could one possibly do with a simple shape in the middle of the screen? That is hardly interesting to look at. I also thought it was a little bit odd that you couldn’t place multiple shapes or define that border and size of the shape like you can do in most creative coding languages. However, I was pleasantly surprised after some experimenting, as hopefully you will be too. One of the easiest things to do is create a tile pattern with the shape. You can do this by chaining a function, where the numbers you pass into the function define how many times the shape is repeated. repeat() If you write like in the screenshot above, you get the shape repeating ten times both in the vertical and horizontal directions. If you write then you will have the shape repeat ten times in the horizontal direction, but not vertical. This function is one of the geometry functions, which you can . repeat(10,10) repeat(10) read more about in the documentation So how might you apply color to these shapes? If you were using voronoi, noise, or the other automatically generated textures, its very easy. You just chain a function where you pass 3 values corresponding to the amount of red, green, and blue. color() A pure red oscillating texture Because you cannot apply color directly to a shape, the workaround is to use a blending function with and applying color within the blending function. For example, you can blend the red oscillator above with the rectangular tiles in the other screenshot. shape() Now you can see the rectangles are overlaid on top of the oscillating red texture. There are multiple blending functions, and each one has a different effect. I won’t go into detail on all of them because this post is getting lengthy and I am wary of the cognitive burden, Suffice it to say there are 6 blending functions in total, . The other 5 are add, diff, layer, mask, and mult. If you’ve ever experimented with layer effects in Photoshop, some of these should sound familiar. Depending on the complexity of your visualization, these operators will sometimes output the same result. You are most likely to notice differences when using a range of color and texture. called operators in the documentation Let’s take the our oscillator and jazz it up a bit. Rather than using the function to apply a simple red color, you can actually pass 3 values into the function directly. The first still specifies the number of oscillating rows, while the second specifies the speed they move across the screen, and the third specifies the range of color. Let's say we use the operator and also tweak our rectangles by making them a bit larger and blurrier. What might that look like? color() osc() diff() Now we’re cooking with gas. Just a few extra functions and things are much more interesting. There are many variables we can tweak to experiment even with this relatively simple visualization. For example, what happens if we change the oscillator to a voronoi or a noise generator? Alright, so it looks like we lost the cool colors but got a more interesting texture in return. Are there other ways to bring back color besides the ones I showed? Absolutely! The function which brings all sorts of psychedelic fun. It’s the last function I wanted to demonstrate because it can spice up pretty much any visualization, and is probably the quickest to get interesting results with. colorama() I hope by now you have the hydra-editor open in several tabs and have virtually lost interest in this post because you are too busy experimenting. Hydra is seriously one of the most absorbing and exciting creative coding tools I’ve had the pleasure of working with, and the goal of this post was to give you enough knowledge that you can hit the ground running. Of course there is tons of material I couldn’t cover, and for that I want to leave you with a few references: Hydra book is a very detailed guide that goes into pretty much every function Hydra has to offer, with lots of screenshots to help you along the way: https://naotohieda.com/blog/hydra-book/ Olivia Jack’s documentation is also nothing to shake a stick at, and has lots of coding examples that you can copy and paste to experiment with. There are also more Hydra tutorials listed here: https://github.com/ojack/hydra#Getting-Started The Github repo includes a section with a whole list of resources which you can check out here: https://github.com/ojack/hydra/blob/master/examples/README.md If its community you’re craving Hydra also has a facebook group with over 500 members: https://www.facebook.com/groups/1084288351771117 I hope you enjoyed getting your feet wet with live coding visuals. Good luck and happy creative coding! If you enjoyed this article, consider following me on Twitter @nadyaprimak or if you need more tips on breaking into the tech industry, you can read my book “Foot in the Door” in paperback or Kindle now. Previously published at https://www.nadyaprimak.com/blog/programming/getting-a-ui-developer-job/

Hydra Tutorial For Live Coding Visuals

The Hydra is a multi headed serpent from Greek mythology that can regrow any head that was cut off Vitalik Buterin recently made a massive series of where he described his line of thinking regarding Proof of Stake (POS). It was quite technical and filled with a bunch of jargon, and I doubt that the average person interested in cryptocurrencies can decipher a fifth of what he was talking about. The goal of this post is to explain in simple term where his logic falls apart and why POS doesn’t work. tweets As correctly pointed out, there are only three tweets ( ) in Vitalik’s tweet storm that is relevant to the POS discussion. I will summarize here for clarity purposes: Vitalik believes that by imposing 2 additional requirement to Ethereum clients, he can make POS work. The requirements are that “clients log on at least once every four month”, and that new clients download the blockchain “from some trusted source when you sync for the first time”. The 2 requirements are needed to prevent “long range attacks”. In this attack, an alternate blockchain which appears completely legitimate to a POS node is created in order to mislead the user. For those that want a better understanding of this attack, I’d suggest reading Ethereum’s POS FAQ sections on the and . Hugo Nguyen 9–11 nothing at stake problem weak subjectivity Vitalik believes that these 2 additional requirements, which do not exist in Proof of Work blockchains, are not a “big deal”. The purpose of this article is to explain why they are a big deal, by outlining exactly how governments seeking to destroy Ethereum will exploit this weakness. Yes, [we will not find a solution to political problems in cryptography,] but we can win a major battle in the arms race and gain a new territory of freedom for several years. Governments are good at cutting off the heads of a centrally controlled networks like Napster, but pure P2P networks like Gnutella and Tor seem to be holding their own. - Satoshi Nakamoto Satoshi states that governments are good at cutting off the heads of centrally controlled networks. But governments are also good at cutting off heads in a decentralized network. The State will happily play an endless game, where they spend multiple decades and billions/trillions of dollars, trying to cut off as many heads as they can. The war on drugs and the war on terror are good examples of this fact. Governments are generally terrible at killing the thing that they are fighting against but this does not stop them from trying. They can cut off a thousand heads but a robust system will just regrow a thousand new heads (i.e. put a drug dealer in jail and the market creates a new one). Think of a proper decentralized system as the mythical creature Hydra. The Hydra is a mythical many-headed serpent with the ability to regenerate any head that is cut off. A centralized system is just a snake; cut off 1 head and it is dead. To kill the Hydra, you must cut off all of its head, and prevent it from regenerating new heads. In order for a cryptocurrency to be like the Hydra, not only must it have many nodes, it must be easy to create new nodes to replace ones that have died. I will now explain the method that governments would take to attack a POS system, which is quite simple. Firstly, they will start cutting off heads by shutting down individual nodes (this can be done in many ways such as physical raids, DOS attacks, network censorship, etc..). Secondly, they will try compromising various trusted entities in the Ethereum space such as Vitalik, ethereum.org, and Joseph Lubin so that they are either silenced or forced to promote an alternate version of the Ethereum blockchain. And thirdly, they will try to create mass confusion by creating many fake Ethereum websites, social media accounts, and fake news that will lead users onto the alternate Ethereum blockchain. Now when a Ethereum users tries to replace the nodes that were shut down, they will find the task to be impossible because of the two requirements that Vitalik created. If the user has to start a node from scratch without any blockchain data, he cannot because his trusted source has been compromised and he has no way to tell which blockchain is the real one. The same applies to a user who was thrown into jail for 4 months and he has to restart his Ethereum node which is missing 4 months of blockchain data (if in the future, you receive a 4 month jail sentence for running an Ethereum node, you can blame Vitalik for choosing that duration). If new nodes cannot replace the ones that were killed, the system will eventually die. Ethereum has consistently made design decisions that hinders the ability of users to run full nodes so the 2 POS requirements are not unique in this regard (for further reading on this topic, see ). As Ethereum continues along this path, it will eventually be revealed that the system is just a Siamese snake not a Hydra; cut off all of its head and it will be dead because it cannot grow new ones. StopAndDecrypt’s article on the Ethereum blockchain size

Proof of Stake and Ethereum: Hydra or Siamese Snake?

Polkadot has allocated 2 million DOT tokens, equivalent to $14.4 million, to its leading DeFi project, Hydration. The funds are aimed at enhancing liquidity and trading efficiency on Hydration's single-sided liquidity provisioning platform, Omnipool. The allocation of the DOT tokens will be split into two parts. One million DOT will be used over the course of a year to attract new liquidity to the Polkadot ecosystem. Initial rewards for Liquidity Providers (LPs) are expected to exceed 200% APY, with various assets being incentivized, including native stablecoins, DOT, BTC, and several ecosystem coins. The single-sided LP design of the Hydration Omnipool allows users to start receiving rewards after providing a single asset and joining an incentives farm. The remaining one million DOT will be directly injected into the Hydration Omnipool to support the creation of a robust and accessible layer of native liquidity. This liquidity layer is intended to benefit the broader Polkadot 2.0 ecosystem over time, as outlined in Treasury Proposal #730. The capital injection adds to the existing 690k+ DOT (and 560k+ vDOT) already provided as liquidity in the Hydration Omnipool. While the funding is expected to drive significant advancements in liquidity provision and management within the Polkadot ecosystem, it is important to consider the potential implications and challenges that may arise from this initiative. Firstly, the injection of such a large amount of capital into a single DeFi project raises questions about centralization and the concentration of power within the ecosystem. The Polkadot community must ensure that the allocation of funds is transparent, fair, and aligns with the principles of decentralization. Secondly, the sustainability of the high initial rewards for LPs should be carefully evaluated. While attractive APYs can draw in new liquidity, it is essential to consider the long-term viability of these rewards and their potential impact on the overall health of the ecosystem. Moreover, the success of this initiative will depend on the adoption and utilization of the Hydration Omnipool by the Polkadot community. The platform will need to demonstrate its value proposition and attract a significant user base to justify the substantial investment from the Polkadot treasury. Despite these challenges, the allocation of funds to the Hydration project represents a significant step towards the growth and development of the DeFi ecosystem within Polkadot. The single-sided liquidity provisioning model offered by Hydration's Omnipool has the potential to revolutionize how liquidity is provided and managed within the shared blockchain network. As the DeFi space continues to evolve and mature, initiatives like this showcase the commitment of blockchain networks to foster innovation and support the growth of decentralized finance. However, it is crucial for the Polkadot community to actively monitor the progress of this initiative, ensure transparency, and work towards maintaining a balanced and decentralized ecosystem. Final Thoughts While the allocation of $14.4M to the Hydration project is a notable development for the Polkadot DeFi ecosystem, it is essential to approach it with cautious optimism. The success of this initiative will depend on various factors, including the effective utilization of funds, the adoption of the Hydration Omnipool, and the ability to maintain a decentralized and sustainable ecosystem. As the Polkadot network continues to evolve, it will be interesting to observe how this investment shapes the future of DeFi within its ecosystem. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Polkadot has allocated 2 million DOT tokens, equivalent to $14.4 million, to its leading DeFi project, Hydration. The funds are aimed at enhancing liquidity and trading efficiency on Hydration's single-sided liquidity provisioning platform, Omnipool. The allocation of the DOT tokens will be split into two parts. One million DOT will be used over the course of a year to attract new liquidity to the Polkadot ecosystem. Initial rewards for Liquidity Providers (LPs) are expected to exceed 200% APY, with various assets being incentivized, including native stablecoins, DOT, BTC, and several ecosystem coins. The single-sided LP design of the Hydration Omnipool allows users to start receiving rewards after providing a single asset and joining an incentives farm. The remaining one million DOT will be directly injected into the Hydration Omnipool to support the creation of a robust and accessible layer of native liquidity. This liquidity layer is intended to benefit the broader Polkadot 2.0 ecosystem over time, as outlined in Treasury Proposal #730. The capital injection adds to the existing 690k+ DOT (and 560k+ vDOT) already provided as liquidity in the Hydration Omnipool. While the funding is expected to drive significant advancements in liquidity provision and management within the Polkadot ecosystem, it is important to consider the potential implications and challenges that may arise from this initiative. Firstly, the injection of such a large amount of capital into a single DeFi project raises questions about centralization and the concentration of power within the ecosystem. The Polkadot community must ensure that the allocation of funds is transparent, fair, and aligns with the principles of decentralization. Secondly, the sustainability of the high initial rewards for LPs should be carefully evaluated. While attractive APYs can draw in new liquidity, it is essential to consider the long-term viability of these rewards and their potential impact on the overall health of the ecosystem. Moreover, the success of this initiative will depend on the adoption and utilization of the Hydration Omnipool by the Polkadot community. The platform will need to demonstrate its value proposition and attract a significant user base to justify the substantial investment from the Polkadot treasury. Despite these challenges, the allocation of funds to the Hydration project represents a significant step towards the growth and development of the DeFi ecosystem within Polkadot. The single-sided liquidity provisioning model offered by Hydration's Omnipool has the potential to revolutionize how liquidity is provided and managed within the shared blockchain network. As the DeFi space continues to evolve and mature, initiatives like this showcase the commitment of blockchain networks to foster innovation and support the growth of decentralized finance. However, it is crucial for the Polkadot community to actively monitor the progress of this initiative, ensure transparency, and work towards maintaining a balanced and decentralized ecosystem. Final Thoughts While the allocation of $14.4M to the Hydration project is a notable development for the Polkadot DeFi ecosystem, it is essential to approach it with cautious optimism. The success of this initiative will depend on various factors, including the effective utilization of funds, the adoption of the Hydration Omnipool, and the ability to maintain a decentralized and sustainable ecosystem. As the Polkadot network continues to evolve, it will be interesting to observe how this investment shapes the future of DeFi within its ecosystem. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program . HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: business blogging program business blogging program

Polkadot Allocates $14.4M to Support DeFi Growth through Hydration Project

পোলকাডট হাইড্রেশন প্রজেক্টের মাধ্যমে ডিফাই গ্রোথকে সমর্থন করার জন্য $14.4M বরাদ্দ করেছে

Polkadot stellt 14,4 Millionen US-Dollar zur Unterstützung des DeFi-Wachstums durch das Hydration-Projekt bereit

Polkadot asigna 14,4 millones de dólares para respaldar el crecimiento de DeFi a través de un proyecto de hidratación

Polkadot alloue 14,4 millions de dollars pour soutenir la croissance de DeFi grâce à un projet d'hydratation

पोलकाडॉट ने हाइड्रेशन प्रोजेक्ट के माध्यम से DeFi ग्रोथ का समर्थन करने के लिए $14.4M आवंटित किया

ポルカドット、ハイドレーションプロジェクトを通じてDeFiの成長を支援するために1440万ドルを割り当てる

Polkadot은 수화 프로젝트를 통해 DeFi 성장을 지원하기 위해 1,440만 달러를 할당합니다.

Polkadot aloca US$ 14,4 milhões para apoiar o crescimento do DeFi por meio do projeto de hidratação

Polkadot выделяет 14,4 миллиона долларов на поддержку роста DeFi посредством проекта Hydration

Polkadot, Hidrasyon Projesi Yoluyla DeFi Büyümesini Desteklemek İçin 14,4 Milyon Dolar Ayırdı

Polkadot phân bổ 14,4 triệu đô la để hỗ trợ tăng trưởng DeFi thông qua dự án hydrat hóa

Polkadot 通过 Hydration 项目拨款 1440 万美元支持 DeFi 增长

Scientific American Supplement, No. 365, December 30, 1882 by Various, is part of the HackerNoon Books Series. You can jump to any chapter in this book . RECENT HYDRAULIC EXPERIMENTS. here RECENT HYDRAULIC EXPERIMENTS. At a late meeting of the Institution of Civil Engineers, the paper read was on "Recent Hydraulic Experiments," by Major Allan Cunningham, R.E. This paper was mainly a general account of some extensive experiments on the flow of water in the Ganges Canal, lasting over four years—1874-79. Their principal object was to find a good mode of discharge measurements for large canals, and to test existing formulæ. There are about 50,000 velocity, and 600 surface-slope measurements, besides many special experiments. The Ganges Canal, from its great size, from the variety of its branches abounding in long straight reaches, and from the power of control over the water in it, was eminently suited for such experiments. An important feature was the great range of conditions, and, therefore, also of results obtained. Thus the chief work was done at thirteen sites in brickwork and in earth, some being rectangular and others trapezoidal, and varying from 193 ft. to 13 ft. in breadth, and from 11 ft. to 7 in. in depth, with surface-slopes from 480 to 24 per million, velocities from 7.7 ft. to 0.6 ft. per second, and discharges from 7,364 to 114 cubic feet per second. For all systematic velocity measurements, floats were exclusively used, viz., surface floats, double floats, and loaded rods. Their advantages and disadvantages had been fully discussed in the detailed treatise "Roorkee Hydraulic Experiments"—1881. They measured only "forward velocity," the practically useful part of the actual velocity. The motion of water, even when tranquil to the eye, was found to be technically "unsteady;" it was inferred that there is no definite velocity at any point, and that the velocity varies everywhere largely, both in direction and in magnitude. The average of, say, fifty forward velocity measurements at any one point was pretty constant, so that there must be probably average steady motion. Hence average forward velocity measurements would be the only ones of much practical use. To obtain these would be tedious and costly, and special arrangements would be required to obviate the effects of a change in the state of water, which often occurred in a long experiment, as when velocities at many points were wanted. As to surface-slope its measurement—from nearly 600 trials—was found to be such a delicate operation that the result would be of doubtful utility. This would affect the application of all formulas into which it entered. The water surface was ascertained, on the average of its oscillations, to be sensibly level across, not convex, as supposed by some writers. There were 565 sets of vertical velocity measurements combined into forty-six series. The forty-six average curves were all very flat and convex down stream—except near an irregular bank—and were approximately parabolas with horizontal axes; the data determined the parameters only very roughly; the maximum velocity line was usually below the service, and sank in a rectangular channel, from the center outward down to about mid-depth near the banks. Its depression seemed not to depend on the depth, slope, velocity, or wind; probably the air itself, being a continuous source of surface retardation, would permanently depress the maximum velocity, while wind failed to effect this, owing to its short duration. On any vertical the mid-depth velocity was greater than the mean, and the bed velocity was the least. The details showed that the mid-depth velocity was nearly as variable from instant to instant as any other, instead of being nearly constant, as suggested by the Mississippi experimenters. The measurement of the mean velocity past a vertical was thought to be of fundamental importance. Loaded rods seemed by far the best for both accuracy and convenience in depths under 15 ft. They should be immersed only 0.94 of the full depth. The chief objection to their use, that—from not dipping into the slack water near the bed—they moved too quickly, was thus for the first time removed. A double float with two similar sub-floats at depths of 0.211 and 0.789 of the full depth would also give this mean with more accuracy and convenience than any instrument of its class; this instrument is new. Measurement of the velocity at five eighths depth would also afford a fair approximation. One hundred and fourteen average transverse velocity curves were prepared from 714 separate curves. These average curves were all very flat, and were convex down stream—over a level or concave bed—and nearly symmetric in a symmetric section. The velocity was greatest near the center, or deepest channel, decreased very slowly at first toward both banks, more rapidly with approach to the banks or with shallowing of the depth, very rapidly close to the banks, and was very small at the edges, possibly zero. The figure of the curve was found to be determined by the figure of the bed, a convexity in the bed producing a concavity in the curve and , and more markedly in shallow than in deep water. Curves on the same transversal, at the same site, and with similar conditions, but differing in general velocity, were nearly parallel projections. At the edges there was a strong transverse surface flow from the edge toward mid-channel, decreasing rapidly with distance from the edge. The discussion showed that it was almost hopeless to seek the geometric figure of the curves from mere experiment. vice versa Five hundred and eighty-one cubic discharges were measured under very varied conditions. The process adopted contained three steps: (1) Sounding along about fifteen float courses, scattered across the site in eight cross sections; time, say four hours. (2) Measurement of the mean velocities through the full depths in those float courses, each thrice repeated; time, say four hours. (3) Computation, say two hours. This process was direct and wholly experimental; each step was done in a time which gave some chance of a constant state of water. From an extended comparison of all results under similar conditions, it appeared that the above process yielded, under favorable circumstances, results not likely to differ more than 5 per cent. The sequel showed that in a channel with variable regimen, a discharge table for a given site must be of at least double entry, as dependent on the local gauge-reading, and on the velocity or surface-slope. Special attention was paid to rapid approximations to mean sectional velocity. The mean velocity past the central vertical, the central surface velocity, and Chézy's quasi-velocity—i.e., 100 × √( R × S ) where R=the hydraulic mean depth, and S=surface slope—were tried in detail; thus 100, 76, and 83 average values thereof respectively were taken from 581, 313, and 363 detail values. The ratios of these three velocities to the mean velocity were taken out, and compared in detail with Bazin's and Cutter's coefficients. Other formulæ were contrasted also in slight detail. Kutter's alone seemed to be of general applicability; when the surface slope measurement is good, and the rugosity coefficient known for the site—both doubtful matters—it would probably give results within 7½ per cent. of error. Improvement in formulæ could at present be obtained only by increased complexity, and the tentative research would be excessively laborious. Now the first two ratios varied far less than the third; thus their use would probably involve less error than the third, or approximation would be more likely from direct velocity measurement than from any use of surface slope. The connection between velocities was probably a closer one than between velocity and slope; the former being perhaps only a geometric, and the latter a physical one. The mean velocity past the central vertical was recommended for use, as not being affected by wind; the reduction coefficient could at present only be found by special experiment for each site. Three current meters were tried for some time with a special lift, contrived to grip the meter firmly parallel to the current axis, so as to register only forward velocity, and with a nearly rigid gearing wire. No useful general results were obtained. Ninety specimens of silt were collected, but no connection could be traced between silt and velocity; it seemed that the silt at any point varied greatly from instant to instant, and that the quantity depended not on the mean velocity, but probably on the silt in the supply water. Forty measurements of the evaporation from the canal surface were made in a floating pan, during twenty five months. The average daily evaporation was only about 1/10 in. The smallness of this result seemed to be due to the coldness of the water—only 63 deg. in May, with 165 deg. in the sun and 105 deg. in shade. Lastly, it must suffice to say that great care was taken to insure accuracy in both fieldwork and computation. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Various (2006). Scientific American Supplement, No. 365, December 30, 1882. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/18763/pg18763-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

RECENT HYDRAULIC EXPERIMENTS

Scientific American Supplement, No. 799, April 25, 1891 by Various, is part of the HackerNoon Books Series. You can jump to any chapter in this book . THE POWER OF WATER, OR HYDRAULICS SIMPLIFIED. here THE POWER OF WATER, OR HYDRAULICS SIMPLIFIED. By G.D. Hiscox. CURRENT WHEELS FOR POWER AND RAISING WATER. The natural flow of water in a current is probably one of the oldest and cheapest of the methods for obtaining power, or the lifting of water within moderate elevations, for a supply for irrigation and domestic purposes; and we propose, apart from the current wheel, to treat only of self-water-raising devices in this chapter. Water wheels of various forms for this purpose have been used from time immemorial in Europe, Asia and Egypt, where the record gives examples of wheels of the noria class from 30 to 90 feet in diameter; the term having been applied to water wheels carrying buckets for raising water; the Spanish having buckets on an endless chain. noria noria Records of a Chinese noria, of 30 feet diameter, made of bamboo, show a lifting capacity of 300 tons of water per day to a height of ¾ of the diameter of the wheel—velocity of current not stated. For less quantity and greater elevation, these forms of wheel may have pumps attached to the shaft, by crank, that will give a fair duty for a high water supply. For power purposes, as in the plain current wheel, Fig. 23, there are two principal factors in the problem of power—the velocity of the current and the area of the buckets or blades. Their efficiency is very low, from 25 to 36 per cent., according to their lightness of make and form of buckets. A slightly curved plate iron bucket gives the highest efficiency, thus ( to the current, and an additional value may also be given by slightly shrouding the ends of the buckets. The relative velocity of the periphery of the wheel to the velocity of the current should be 50 per cent. with curved blades for best effect. The most useful and convenient sizes for power purposes are from 10 to 20 feet, and from 2 to 20 feet wide, although, as before stated, there is scarcely a limit under 100 feet diameter for special purposes. In designing this class of wheels special attention should be given to the concentration and increase of the velocity of the current by wing dams or by the narrowing of shallow streams; always bearing in mind that any increase in the velocity of the current is economy in increased power, as well as in the size and cost of a wheel for a given power. The blades in the smaller size wheels should be 1/4 of the radius in width, and for the larger sizes up to 20 feet, 1/5 to 1/6 of the radius in width and spaced equal to from 1/4 to 1/3 of the radius. They should be completely submerged at the lowest point. For obtaining the horse power of a current wheel, the formula is Area of 1 blade × velocity of the current in ft. per sec.
----------------------------------------------------------
                          400 × by the square of difference of velocities of current and wheel periphery = the horse power; or in which A equals the area of blade in square feet, V and v velocities of current and wheel periphery respectively, in feet per second. Thus, for example, a wheel 10 feet in diameter with blades 6 feet long and 1 foot in width, running in a stream of 5 feet per second—assuming the wheel to be giving as much power as will reduce its velocity to one half that of the stream—the figures will be horse power of the wheel. The total power of the stream due to the area of the blade equals the Square of the velocity of the stream
  ------------------------------------ ×
         Twice gravity (64.33) volume of water in cubic feet per second × 62.5 (weight of 1 C') = the value or gross effect in pounds falling 1 foot per second. This sum divided by 550 = horse power. Thus, as per last example, due to the area of the blades of the water wheel. For the efficiency of this class of wheel, with slightly curved and thin blades, divide the horse power of the wheel by the horse power of the current area, equals the percentage of efficiency. As in the last case, 0.468 /  1.32 = 0.35½ per cent. efficiency of the water wheel. With higher velocities of stream and wheel the efficiency will be from 2 to 3 per cent. less, although the horse power will increase nearly with the increase in velocity of the current. For details of application of various forms of current wheels for power purposes see illustrated description Yagn's and Roman's floating motors in SCIENTIFIC AMERICAN SUPPLEMENT, No. 463. A very good example of a floating motor of the propeller class is Nossian's fluviatile motor, illustrated and described in SCIENTIFIC AMERICAN SUPPLEMENT, No. 656. Fig. 24 represents a very complete floating motor, in which the floats are wedge shaped at the stem, for the purpose of increasing the current between them, the wheel being an ordinary current wheel, as shown in Fig. 23, with a curved shield or gate in front, which can be moved around the periphery of the wheel for the purpose of regulating its speed or stopping its motion by cutting off the stream from the buckets. The float, rising and falling with the stream, is held in position by a braced frame swinging on anchorages within the mill on shore, and parallel with a swiveled shaft. Tide wheels and tidal current wheels have been in use for more than 800 years, and were largely in use in Europe and the United States during the first half of the present century. No less than three were running in the immediate vicinity of New York, in 1840, for milling purposes. Their day seems to be past, except in some special localities. We will also pass them, and illustrate some of the SELF-ACTING WATER-RAISING DEVICES. The tympanum derives its name from its similarity to a drum as made by the Romans, but its origin was Egyptian. It is a current wheel with frame like Fig. 23, to the outside of which a set of chambers or tubes are fixed, radiating spirally, so as to lead the water to the shaft as the wheel revolves, as shown in Fig. 25. It has a lift of a little less than half its diameter, and answers an excellent purpose for the irrigation of rice and cranberry fields, or on streams running through low lands in arid districts. It is still one of the Nile irrigating wheels. The building of these wheels is within the scope of the carpenter and the tinsmith. A short wooden shaft made square or octagonal, as convenient, with gudgeons in the ends and arms of wood bolted across each of the sides of the shaft, or as shown in the cut, will form a frame work upon which a rim may be fastened, to which the blades and tubular buckets can be attached. The directions in regard to the current wheel, Fig. 23, may be followed as to number and form of blades, which must be made in length and width proportional to the velocity of the stream and the quantity of water to be lifted by each tubular arm. The tubes may be made of galvanized sheet iron and attached to the outside of the wheel, as shown in Fig. 25. THE NORIA OR BUCKET WHEEL. This is a simple current wheel with pot buckets, rigid or swinging, arranged on the rim of the wheel, to carry up and discharge the water nearly at the top of the wheel, and through the long ages that it has been in use for irrigation, village water supply, and even for private establishments, has assumed a variety of forms in detail of construction ranging from the bamboo wheels of the Chinese to the light iron wheels of modern construction. We illustrate the most simple of these forms in Figs. 26 and 27, in which the first is a series of boxes or chambers in the rim of the wheel with side openings in the forward part of the box as the wheel revolves, and a lip extending from the inner edge of the opening to direct the outflow into the trough. Another form, Fig. 27, is arranged with swing buckets or pots, pivoted just above their centers, and with the catch trough so fixed as to tip the buckets at the highest point, thus giving this wheel the greatest possible advantage as to height of discharge for a given diameter. The power value of these wheels for raising water is a matter of computation as nearly reliable as for other devices for the same purpose, when the velocity of the current is known at the point of contact with the blades. The horse power of the wheel may be computed as for the current wheel, Fig. 23, and, as the horse power is equal to 33,000 pounds raised one foot high per minute, we may assume a construction of wheel that will allow of discharging at 8 feet above the stream; then 33,000 / 8 = 4,125 pounds of water discharged at 8 feet elevation per horse power per minute. As the net power of the wheel in the last example, for Fig. 23, was 0.468 of a horse power, then 4,125 × 0.468 = 1,930 pounds of water raised 8 ft. per minute by the size of bucket and velocity of current in that case. From this a deduction of 20 per cent. should be made for loss by spill and imperfect construction, so that 1,500 pounds or 176 gallons per minute would be the probable output—over 253,000 gallons per day; or, for irrigating purposes, equal to a rainfall of over 1¼ inches in depth on 50 acres in one week. The proportion of capacity of the lifting buckets for such a wheel becomes of as great importance as its efficiency. If the buckets are too large, the wheel will stall, and if too small, the wheel will not give its full duty. For obtaining the approximate capacity of the lifting buckets, assuming the example as above computed, a 10 foot wheel with the velocity at periphery of 2½ feet per second is 150 feet per minute, or five revolutions per minute, nearly. Then 1,930 lb. per m. / 5 revolutions = 386 pounds water capacity for all of the buckets on the wheel. If such a wheel is constructed with 16 blades and 16 buckets, one between each blade, then 386 / 16 = 24 pounds for each bucket, or 38 / 100 of a cubic foot. The spill from this capacity of bucket being sufficient to compensate for the friction of the shaft journals. The lifting buckets of the noria class, Figs. 26 and 27, can be made of positive dimensions to suit the computations as above; but those of the tympanum class, Fig. 25, should be made of dimensions to conform with the required capacity at the moment of leaving the water, as the water at this point flows into the arm. ( .) To be continued About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Various (2004). Scientific American Supplement, No. 799, April 25, 1891. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/11649/pg11649-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

THE POWER OF WATER, OR HYDRAULICS SIMPLIFIED

Scientific American Supplement, No. 717, September 28, 1889, by Various, is part of the HackerNoon Books Series. You can jump to any chapter in this book . THE GIRARD HYDRAULIC RAILWAY here THE GIRARD HYDRAULIC RAILWAY. We give herewith some illustrations of this railway which has recently excited so much technical interest in Europe and America, and which threatens to revolutionize both the method and velocity of traveling, if only the initial expense of laying the line can be brought within moderate limits. A short line of railway has been laid in Paris, and we have there examined it, and traveled over the line more than once; so that we can testify to the smoothness and ease of the motion. Sir Edward Watkin examined the railway recently, and we understand that a line two miles long is to be laid in London, under his auspices. He seems to think it might be used for the Channel tunnel, being both smokeless and noiseless. It might also, if it could be laid at a sufficiently low price, be useful for the underground railways in London, of one of which he is chairman. We are favorably impressed by the experiments we have witnessed; our misgivings are as to the cost. The railway is the invention of the well known hydraulic engineer, Monsieur Girard, who, as early as 1852, endeavored to replace the ordinary steam traction on railways by hydraulic propulsion, and in 1854 sought to diminish the resistance to the movement of the wagons by removing the wheels, and causing them to slide on broad rails. In order to test the invention, Mons. Girard demanded, and at the end of 1869 obtained, a concession for a short line from Paris to Argenteuil, starting in front of the Palais de l'Industrie, passing by Le Champ de Courses de Longchamps, and crossing the Seine at Suresnes. Unfortunately, the war of 1870-71 intervened, during which the works were destroyed and Mons. Girard was killed. After his death the invention was neglected for some years. A short time ago, however, one of his former colleagues, Mons. Barre, purchased the plans and drawings of Mons. Girard from his family, and having developed the invention, and taken out new patents, formed a company to work them. The invention may be divided into two parts, which are distinct, the first relating to the mode of supporting the carriages and the second to their propulsion. Each carriage is carried by four or six shoes, shown in Figs. 3, 4, and 5; and these shoes slide on a broad, flat rail, 8 in. or 10 in. wide. The rail and shoe are shown in section in Fig. 1. The rail is bolted to longitudinal wooden sleepers, and the shoe is held on the rail by four pieces of metal, A, two on each side, which project slightly below the top of the rail. The bottom of the shoe which is in contact with the rail is grooved or channeled, so as to hold the water and keep a film between each shoe and the rail. The carriage is supported by vertical rods, which fit one into each shoe, a hole being formed for that purpose; and the point of support being very low, and quite close to the rail, great stability is insured. It is proposed to make the rail of the form shown in Fig. 2 in future, as this will avoid the plates, A, and the flanges, B, will help to keep the water on the rail. Figs. 3, 4, and 5 show the shoe in detail. Fig. 3 gives a longitudinal section, Fig. 4 is a plan, and Fig. 5 is a plan of the shoe inverted, showing the grooves in its face. Fig. 3 shows the hollow shoe, into which water at a pressure of ten atmospheres is forced by a pipe from a tank on the tender. The water enters by the pipe, C, and fills the whole of the chamber, D. The water attempts to escape, and in doing so lifts the shoe slightly, thus filling the first groove of the chamber. The pressure again lifts the shoe, and the second chamber is filled; and so on, until ultimately the water escapes at the ends, E, and sides, F. Thus a film of water is kept between the shoe and the rail, and on this film the carriage is said to float. The water runs away into the channels, H H (Fig. 6), and is collected to be used over again. Fig. 3 also shows the means of supporting the carriage on the shoe by means of K, the point of support being very low. The system of grooves on the lower face of the shoe is shown in Fig. 5. So much for the means by which wheels are dispensed with, and the carriage enabled to slide along the line. The next point is the method of propulsion. Figs. 7 and 8 give an elevation and plan of one of the experimental carriages. Along the under side of each of the carriages a straight turbine, L L, extends the whole length, and water at high pressure impinges on the blades of this turbine from a jet, M, and by this means the carriage is moved along. A parabolic guide, which can be moved in and out of gear by a lever, is placed under the tender, and this on passing strikes the tappet, S, and opens the valve which discharges the water from the jet, M, and this process is repeated every few yards along the whole line. The jets, M, must be placed at such a distance apart that at least one will be able to operate on the shortest train that can be used. In this turbine there are two sets of blades, one above the other, placed with their concave sides in opposite directions, so that one set is used for propelling in one direction and the other in the opposite direction. In Fig. 6 it is seen that the jet, M, for one direction is just high enough to act against the blades, Q, while the other jet is higher, and acts on the blades, P, for propulsion in the opposite direction. The valves, R, which are opened by the tappet, S, are of peculiar construction, and we hope soon to be able to give details of them. Reservoirs (Fig. 6) holding water at high pressure must be placed at intervals, and the pipe, T, carrying high pressure water must run the whole length of the line. Fig. 6 shows a cross section of the rail and carriage, and gives a good idea of the general arrangements. The absence of wheels and of greasing and lubricating arrangements will alone effect a very great saving, as we are informed that on the Lyons Railway, which is 800 kilometers long, the cost of oil and grease exceeds £400,000 per annum. As Sir Edward Watkin recently explained, all the great railway companies have long tried to find a substitute for wheels, and this railway appears to offer a solution of that problem. Mons. Barre thinks that a speed of 200 kilometers (or 120 miles) per hour may be easily and safely attained. Of course, as there is no heavy locomotive, and as the traction does not depend upon pressure on the rail, the road may be made comparatively light. The force required to move a wagon along the road is very small, Mons. Barre stating, as the result of his experiments, that an effort amounting to less than half a kilogramme is sufficient to move one ton when suspended on a film of water with his improved shoes. It is recommended that the stations be placed at the summit of a double incline, so that on going up one side of the incline the motion of the train may be arrested, and on starting it may be assisted. No brakes are required, as the friction of the shoe against the rail, when the water under pressure is not being forced through, is found to be quite sufficient to bring the train to a standstill in a very short distance. The same water is run into troughs by the side of the line, and can be used over and over again indefinitely, and in the case of long journeys, the water required for the tender could be taken up while the train is running. The principal advantages claimed for the railway are: The absence of vibration and of side rolling motion; the pleasure of traveling is comparable to that of sleighing over a surface of ice, there is no noise, and what is important in town railways, no smoke; no dust is caused by the motion of the train during the journey. It is not easy for the carriages to be thrown from the rails, since any body getting on the rail is easily thrown off by the shoe, and will not be liable to get underneath, as is the case with wheels; the train can be stopped almost instantly, very smoothly, and without shock. Very high speed can be attained; with water at a pressure of 10 kilogrammes, a speed of 140 kilometers per hour can be attained; great facility in climbing up inclines and turning round the curves; as fixed engines are employed to obtain the pressure, there is great economy in the use of coal and construction of boilers, and there is a total absence of the expense of lubrication. It is, however, difficult to see how the railway is to work during a long and severe frost. We hope to give further illustrations at an early date of this remarkable invention.— Industries. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Various (2006). Scientific American Supplement, No. 717, September 28, 1889. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/17755/pg17755-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

THE GIRARD HYDRAULIC RAILWAY

HYDRAULIC ELEVATORS AND MOTORS

HYDRAULIC MACHINERY.

, known for its expertise in off-chain trading, just released the , a Layer 3 trading platform that integrates multiple off-chain protocols to enable trustless, cost-effective, and near-instant cross-chain trading. Hydranet Hydranet DEX The Hydranet DEX made its official debut as a mainnet beta on September 23, 2023, following more than a year of dedicated development. The release signifies a major milestone in the project's history, whose future promises even more groundbreaking developments. I am really proud of us. We have evolved from a small project to a currently #700 mature DeFi project with a great 2023/24 roadmap ahead. - Joe Park, project manager of Hydranet Hydranet presents their DEX as a Layer 3 trading platform that connects the Bitcoin and Ethereum ecosystems (including all off-chain compatible networks), allowing users to trade seamlessly between them. Ethereum Virtual Machine (EVM) By their use of off-chain protocols, such as the Lightning and Connext Vector technologies, crafted into a single solution that allows these protocols to interact, Hydranet has made it possible to trade between fundamentally different blockchains with virtually no fees, near-instantly, and, most importantly, in a trustless manner, as users will always retain full control of their funds. Off-chain protocols play a pivotal role in this solution and refer to a set of rules, specifications, and frameworks that define the standards and procedures for conducting transactions outside the blockchain using what is commonly known as state channels. Unlike on-chain transactions, which necessitate miners’ confirmations, off-chain transactions are distinguished by their instant transaction times, minimal-to-no transaction fees, high level of privacy, and trustless operations. These attributes make off-chain transactions particularly useful for microtransactions, frequent interactions, and scenarios where real-time responsiveness is crucial. Recognizing the benefits of off-chain protocols, Hydranet has undoubtedly taken advantage of them in creating a solution that will meet the long-standing demand for a cost-effective trading platform capable of bridging fundamentally different blockchains without compromising security, integrity, and scalability. This trading platform is now available as a downloadable desktop client from Hydranet’s . official website The platform comprises a self-custodial wallet, a state channel management interface, and an order book for trading on its Layer 3 exchange. Trades on the exchange are secured using Hashed TimeLock Contracts (HTLC) which guarantees that they are completed in accordance with what is agreed upon, or not completed at all if either one of the trading parties attempts to manipulate the trade. Below is a video showcasing a mainnet Bitcoin to Ethereum trade using the latest version of the Hydranet DEX. https://www.youtube.com/embed/ZgbZOjo5WT4?showinfo=0 The Hydranet DEX is currently hosting a set of four trading pairs to start off with: aETH - BTC BTC - aUSDT HDN - aUSDT ETH - BTC These trading pairs effectively bridge the Bitcoin, Ethereum, and Arbitrum (denoted as 'a') blockchains and highlight the capabilities of the trading platform. Adding support for other trading pairs and networks is said to be as easy as adding a few lines of code. With all the excitement surrounding this new type of trading platform and how it will develop in the future, Hydranet emphasizes that this is only the beginning of the Hydranet DEX cross-chain future and more news is coming. Users can learn more about Hydranet and the Hydranet DEX at . Hydranet’s official website Users can Join Hydranet’s and to stay updated on their announcements. Discord Twitter About Hydranet Hydranet is a building the world’s first Layer 3 off-chain decentralized exchange. With a commitment to trustlessness, efficiency, and scalability, their vision extends beyond the confines of traditional on-chain exchanges. Hydranet is active on Discord, Twitter, Telegram, Facebook, Medium, YouTube, and Instagram. Decentralized Autonomous Organization Contact Nico McFinity Hydranet contact@hydranet.ai This story was distributed as a release by Chainwire under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author

Hydranet Announces Layer 3 DEX: Redefining Trustless Cross-Chain Trading

Ory Hydra is a popular open-source OAuth2 and OpenID Connect that provides secure authentication and authorization for applications. One of the key challenges in building a scalable and performant OAuth2 server is managing the persistence layer, which involves storing and retrieving data from a database. server Motivation Ory was approached by a popular service provider to optimize performance in their auth system at high load. They often struggled to cope with the huge influx of authorization grants during peak times (over 600logins/sec). Looking for another solution, they started to evaluate Ory Hydra, investigating if it can handle this amount of grants. After reaching out to the team, we started to investigate ways to improve the overall performance to make Ory Hydra faster and more scalable than ever before. The key to success was to re-engineer parts of Hydra's persistence layer to reduce write traffic to the database, moving to a transient OAuth2 flow. Moving to a transient OAuth2 flow One of the core parts of this work was moving a large chunk of the transient OAuth2 flow state, which is exchanged between the three parties involved in an OAuth2 flow, from the server to the client. Instead of persisting transient state to the database, the state is now passed between the parties as either AEAD-encoded cookies or AEAD-encoded query parameters in redirect URLs. AEAD stands for authenticated encryption with associated data, which means that the data is confidential and also can't be tampered with without knowing a secret (symmetric) key. The flow is then only persisted in the database once when the final consent is given. This change has several benefits. First, it reduces the amount of data that needs to be stored in the database, which in turn reduces write traffic. Second, it eliminates the need for multiple indices on the flow table that were previously used during the exchange. The OAuth2 authorization code grant flow in detail The relevant part of the OAuth2 flow that we wanted to optimize is an exchange between the client (acting on behalf of a user), Hydra (Ory's OAuth2 Authorization Server), and the login and consent screens. When a client requests an authorization code through the , the user will be redirected first to the login UI to authenticate and then to the consent UI to grant access to the user's data (such as the email address or profile information). Authorization Code Grant Below is a sequence diagram of the exchange. Observe that each UI gets a as part of the URL parameters (steps 3 and 12) and then uses this as a parameter to retrieve more information (steps 4 and 13). Finally, both UIs either accept or reject the user request, usually based on user interaction with the UI (from steps 6 to 8 and 15 to 17). This API contract keeps Ory Hydra headless and decoupled from custom UIs. CHALLENGE CHALLENGE Optimization: Passing the AEAD-encoded flow in the URL parameters To reduce database access we now pass as the , , , and an AEAD-encoded flow. This way, we rely on the parties involved in the OAuth2 flow to pass the relevant state along. LOGIN_CHALLENGE LOGIN_VERIFIER CONSENT_CHALLENGE CONSENT_VERIFIER Before After The login and consent challenges and verifiers are random UUIDs stored in the database. The login and consent challenges and verifiers are the AEAD-encoded flow. Accepting or rejecting a request from the UI involves a database lookup for the specific challenge. Accepting or rejecting a request from the UI involves decrypting the flow in the challenge and generating an updated flow as part of the verifier. Implementation Since Ory Hydra is open source, you can review code changes in the Ory GitHub repositories. This is the relevant . commit Here is where we encode the flow in the specific challenges and verifiers: // ToLoginChallenge converts the flow into a login challenge.
func (f *Flow) ToLoginChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginChallenge)
}

// ToLoginVerifier converts the flow into a login verifier.
func (f *Flow) ToLoginVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginVerifier)
}

// ToConsentChallenge converts the flow into a consent challenge.
func (f *Flow) ToConsentChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentChallenge)
}

// ToConsentVerifier converts the flow into a consent verifier.
func (f *Flow) ToConsentVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentVerifier)
} In the persister (our database repository) we then decode the flow contained in the challenge. For example, here's the code for handling a consent challenge: func (p *Persister) GetFlowByConsentChallenge(ctx context.Context, challenge string) (*flow.Flow, error) {
    ctx, span := p.r.Tracer(ctx).Tracer().Start(ctx, "persistence.sql.GetFlowByConsentChallenge")
    defer span.End()

    // challenge contains the flow.
    f, err := flowctx.Decode[flow.Flow](ctx, p.r.FlowCipher(), challenge, flowctx.AsConsentChallenge)
    if err != nil {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.NID != p.NetworkID(ctx) {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.RequestedAt.Add(p.config.ConsentRequestMaxAge(ctx)).Before(time.Now()) {
        return nil, errorsx.WithStack(fosite.ErrRequestUnauthorized.WithHint("The consent request has expired, please try again."))
    }

    return f, nil
} Let's look at the impact of the changes when compared to the code without optimizations: The flows are now much faster and talk less to the database. Improved indices lead to further performance improvements By introducing a new index on the table, we were able to increase throughput and decrease CPU usage on PostgreSQL. The screenshot below shows the execution of the benchmarks without the improved indices where CPU usage spikes to 100%, and with improved indices, where CPU usage stays below 10%. hydra_oauth2_flow With the newly added indices, CPU usage (green bars) is removed, which reduces the likelihood of BufferLocks and related issues: Results The code and database changes reduced the total roundtrips to the database by 4-5x (depending on the amount of caching done) and reduced database writes by about 50%. Benchmarks Benchmarking the new implementation on Microsoft Azure with the following specifications: Services Configuration Total max SQL connections Notes Ory Hydra Consent App OAuth2 Client App rakyll/hey (http benchmark tool) 3x Standard_D32as_v4; South Central US 5x Standard_D8s_v3; South Central US 512 Every VM ran all the processes mentioned. PostgreSQL 14 in HA configuration Memory Optimized, E64ds_v4, 64 vCores, 432 GiB RAM, 32767 GiB storage; South Central US RAM beats CPU. Ory can perform up to at the peak and in the above configuration. This is possible by making the flow stateless and optimizing indices in frequently used queries. 1090 logins per second 800 logins / second consistently Conclusion The performance optimization work done by the Ory team has resulted in a significant improvement in Hydra's performance and scalability. By reducing write traffic to the database and improving the codebase and dependencies, Hydra is now faster and more responsive than ever before. By improving the indices, Hydra now scales much more efficiently with the number of instances. In the future, we will continue to optimize Ory's software to handle even more traffic. We believe that it's possible to get 5x more throughput on a single PostgreSQL node with data model optimizations. If you're building an OAuth2 server, we highly recommend giving Ory's fully certified OpenID Connect and OAuth2 implementations a try: Ory OAuth2 – our running on the global Ory Network, based on Ory Hydra – already utilizes the optimizations described in this article and setting it up only takes few minutes! fully managed service open source Also published . here

Scaling Ory Hydra to Two Billion Monthly OAuth2 Flows on a Single PostgreSQL DB

Escalamiento de Ory Hydra a dos mil millones de flujos mensuales de OAuth2 en una única base de datos de PostgreSQL

Mise à l'échelle d'Ory Hydra à deux milliards de flux OAuth2 mensuels sur une seule base de données PostgreSQL

सिंगल पोस्टग्रेएसक्यूएल डीबी पर ओरी हाइड्रा को दो बिलियन मासिक OAuth2 फ्लो तक स्केल करना

Ory Hydra を単一の PostgreSQL DB で月間 20 億の OAuth2 フローに拡張

Dimensionando o Ory Hydra para dois bilhões de fluxos OAuth2 mensais em um único banco de dados PostgreSQL

Mở rộng quy mô Ory Hydra lên hai tỷ luồng OAuth2 hàng tháng trên một cơ sở dữ liệu PostgreSQL

在单个 PostgreSQL 数据库上将 Ory Hydra 扩展到每月 20 亿个 OAuth2 流量

When we think about data networks and systems that use mechanical mechanisms to send data across a distance, we think about this concept as a pretty modern one that only existed as a practical idea in the late 1900s. However, what if I told you that one of the earliest data networks to exist predated the modern internet as we know it? Heck, this ancient data network predates modern digital computers and analog computer systems. The modern electric telegraph is still younger than this particular early data network and most mechanical computers have not been around as long as this old-school data network. And when I say old-school, I mean truly old-school, as this data network was around before the Medieval Times and was developed by one of the oldest civilizations in written history. There is an ancient device known as the telegraph. This particular device was created to use water and devices in order to send information over long distances. The earliest version of this creation was produced back around 350 BCE by an early inventor and ancient Greek writer by the name of Aeneas Tacticus. It was produced by Tacticus as a device to be used for some long-form data communications during wartime & includes a very complicated system. The telegram worked by having identical containers on separate hills that are not connected to one another with each container being filled with water & having a vertical rod floating within both containers. The rods would have a bunch of predetermined codes put on them at various points along these poles. There would be a spigot at the bottom of these containers and water could be drained out of the device until a desired code was reached. Using a torch or signal, the person sending the code would be able to indicate how long it takes for the water to drain out to the desired code, allowing the receiver to know what signal/code is being communicated. hydraulic mechanical How these particular telegraph devices work is even explained by the Greek historian Polybius in his book : The Histories Then we are to fill the vessels with water and put on the corks with the rods in them and allow the water to flow through the two apertures. When this is done it is evident that, the conditions being precisely similar, in proportion as the water escapes the two corks will sink and the rods will disappear into the vessels. When by experiment it is seen that the rapidity of escape is in both cases the same, the vessels are to be conveyed to the places in which both parties are to look after the signals and deposited there. Now whenever any of the contingencies written on the rods occurs he tells us to raise a torch and to wait until the corresponding party raises another. When both the torches are clearly visible the signaler is to lower his torch and at once allow the water to escape through the aperture. Whenever, as the corks sink, the contingency you wish to communicate reaches the mouth of the vessel he tells the signaler to raise his torch and the receivers of the signal are to stop the aperture at once and to note which of the messages written on the rods is at the mouth of the vessel. This will be the message delivered, if the apparatus works at the same pace in both cases. What is so important about these primitive methods of sending data? Well, while it was limited in codes, it was a great early attempt at long-range telegraphic communication. It came as an early telegraph system that even comes before mechanical optical telegraphs and even inspired a British version that, while never made commercially available, was in 1838 by British civil engineer Francis Whishaw using pipes and more modern mechanical pressure systems. All of this came before and helped to inspire, the electric telegraphs that would -eventually - become the basis for the electric systems of communication we use today for the modern internet. So, to put a long story short, you better thank the Greeks and their system of sending codes long distances using containers and water for the modern communications that we get to enjoy today! developed

The Hydraulic Telegraph: The First Data Network All The Way From Ancient Greece!

Yet Another Lightning Hydra Template for ML Experiments

There has never been a better or worse time to be online. Humanity's endless ingenuity in digitizing more parts of our lives and finances has empowered and connected people and communities. It also made us helplessly dependent on our machines, so the act of losing or breaking your computer is more horrible than losing your wallet and keys at the same time. Having your computer compromised by a hacker – that's some of the worst luck of all. We are all familiar with the global extortion ransomware attacks have inflicted on small to mid-sized businesses – but what about DDoS attacks? Are they still a concern? Link11, a leading European IT security provider "in cyber-resilience," puts out a bi-annual report on DDoS Attacks. They recently released the . DDoS Report for 2021 TL;DR – DDoS attacks have gotten trickier, more powerful, more frequent and are often part of a more complex attack. According to the report, attacks grew 40% YOY vs. 2020, there were many high volume attacks, and 71% of DDoS attacks were identified as multi-vector. By multi-vector attack, they mean the hacker entity used multiple access paths, methods, and protocols making the DDoS attack. They compare this attack to a mythical hydra, which conveys the near hopelessness of fighting a diversified attack. Multi-vector DDoS attacks are more complex to run but far more challenging to catch and defend against. ** ** The report also showed an increase in carpet bombing -- a high-volume DDoS attack that runs a low volume of traffic from many IP addresses so they can avoid detection. Perhaps worst of all, DDoS attacks are being used as a smokescreen to hide more complex attacks. A mid-sized business' poor IT team will be wiping their brows having restored an online site or service, thinking they have been successful when they are , in fact, due for a ransomware attack a few months later. We spoke with Ken MacIntyre, Sales Director and Marc Wilczek, COO of Link11, on the state of DDoS attacks. Why are DDoS attacks such a big concern today? "So straight after ransomware, your number two problem as an organization these days is DDoS. And what DDoS basically does is create a data tsunami that is unleashed and takes an organization offline for hours, potentially for days. We're thinking about remote work at scale. Everyone is working from home… With a DDoS attack, the entire organization could be taken offline for hours, causing reputational damages, causing productivity losses, revenue losses, and more," Wilczek said. Why the growth in DDoS attacks? "I think there are at least two major driving forces – one is public cloud adoption. APIs are being compromised. The utility power and bandwidth of global providers is being misused or abused in order to produce massive amounts of DDoS attacks. And on the second part of that equation is the massive emergence of the Internet of Things with billions and billions of all these beautiful IoT devices. But unfortunately, to a large extent, they're not protected, and they can be put together to form a bot army to produce these attacks. All of that opens up and widens the attack surface… So you have a sort of a perfect storm. A perfect environment that lets DDoS attacks thrive and grow exponentially," Wilczek said. What about the threat of state actors, like hacker groups in China, North Korea, and Russia? Do they pose a special threat? "There's lots of attention on state-backed actors... But the fact of the matter is that 90%+of attacks are still financially driven. So, it's really financially-motivated cybercrime that is driving everyday attacks. The state-backed actors contribute to the problem, but they tend to go after more prestigious targets such as the critical infrastructure providers. That said, some of these cyber-criminal gangs go after the big hosting companies, and domino effects can happen because if they're taken offline, it will trickle down... They represent a significant challenge today and in the future," Wilczek said. What made Link11 focus specifically on DDoS? "Originally, Link11 was a hosting company in the gaming industry and we were subject to an attack. Consequently, we pivoted from dealing with DDoS as a way of protecting our own business to a shift into becoming a leading DDoS mitigation company. That's how we began. And since then, the trend has been growing," Wilczek said. How can you guard against a DDoS attack? "The rules-based approach we've taken is to look at customers' profiles – what we call their fingerprints – and look at the shape of the data. And then any delta, any change from the norm, we'll look at a lot closer. If it's obviously a malicious attack, then we mitigate it. So the technology approach is an evolution from the traditional approaches many organizations take, which is manually looking at the attacks and trying to work out what's going on… We do it using machine learning and we're looking at it in real-time," Wilczek said. What's the real danger around a DDoS attack beyond having your site or service down? "DDoS attacks are often used to masquerade or camouflage other attacks. If somebody wanted to infiltrate a corporate network with ransomware, it's so easy to simply unleash a DDoS attack in order to keep the IT team busy, so they don't really notice what's happening... And unfortunately, it can take an organization months, if not years, to figure out that what appeared to be a DDoS attack was actually part of something much bigger… We could be looking at what looks like a DDoS attack, and obviously, it's there, but it's masking something else. And that's a really high risk, because that could literally be the backdoor to try and take banking details, credit details, and things like that. And that's not uncommon, and obviously, it could also be the basis of a ransomware attack," Wilczek said. Are there any industries that have struggled particularly with the issue? "Banking has undergone a massive transition because it is no longer a retail conversation. It's no longer brick and mortar – it's about apps and mobile. It's digital, online, and 24/7. And every DDoS attack can cause major drama financially, but it also can erode trust. DDoS attacks are used to inflict chaos and possibly produce a bank run in the banks. And that's what DDoS can do, because it causes panic. If everything is offline, if you can no longer access your mobile apps, it causes a massive wave of chaos. It's interesting because you will see sectors that are being attacked at the same time or groups of organizations being attacked at the same. Utilities, for example, in the last few months… It's very well structured. It's very well thought through. And the unfortunate ones who haven't the proper protection are brought down," Wilczek said. Are DDoS attacks becoming more dangerous? "The attacks are becoming a lot more sophisticated. And what can send an organization reeling is not just a large-scale frontal attack. There are more sophisticated forms of attacks. Think about Carpet Bombing, for example. Instead of having a massive wave of bandwidth, you actually end up with a tiny little spike in traffic per individual IP address, but because everything happens at once, it overwhelms the entire system and brings everything to a standstill. And unfortunately, less sophisticated tools really struggle to detect these traffic patterns because they look like or appear to be regular traffic," Wilczek said. How do you protect companies against DDoS attacks? "When our customers come in, we want to get a sense for their pattern of traffic. And that can be quite detailed. And then we do our work and machine learning does what it does in real-time, tracking those profiles. We check-in against algorithms, patterns of traffic from similar customers, according to industry, and so on. And once we're certain that we've detected a malicious attack, then we basically block the traffic for customers," MacIntyre said. So if you had one piece of advice for companies that might be vulnerable, what would it be? "First, identify your mission-critical IT assets. When was the last time that you audited or tested your defense? Often, companies procured a solution many, many years ago, and then they fail to run fire drills. They fail to practice. They fail to do audits, they fail to pentest, and they simply believe everything is perfectly up and running until there is an attack, and unfortunately, they learn the hard way. That is why I would strongly suggest asking, when did you audit it? When did you pentest it? When was your last fire drill, and it really needs practice, and practice, and practice, because otherwise, maybe there's a couple of months or half a year without any attack. You think you're in a good spot, and it all comes down in the middle of the night," MacIntyre said. Image by from Elchinator Pixabay

Fighting the Hydra of DDoS Attacks (Spoiler: They Got Worse)

Look at the picture below. Look at how many more dots are on the right than on the left. Then look at how many more lines are on the right than on the left. As you add pieces to an interconnected system, it gets more complex, and fast. Conventional wisdom in the tech industry is that you manage complexity like weeds in a garden. You put aside a fixed amount of time (let’s say a couple days) at a fixed interval (let’s say every month) to do some documentation here, some code refactoring there. Many don’t do that. Many want to, but are so busy churning out new features that they end up treating their product’s complexity the way one treats Boo, that Mario villain who slyly follows you while your back is turned and stops moving when you look at him. In the real world, many settle for “build really fast and sometimes acknowledge that complexity exists.” Deep down, in places they don’t talk about at parties, every engineer knows these approaches are broken. Software complexity is like a hydra, a monster from Greek myth that grows two new heads each time you cut one off. The more you build, the more it grows. Making your team build faster to make up for the slowdowns that complexity can bring is the opposite of what you should do, and exactly what many teams do. In fact, the bigger the gap between the pace at which you build and the pace at which you address your complexity, the more trouble you’re creating for your future self. The hydra’s heads generally attack from three places: The languages and frameworks you use The libraries and apps you use The product you’re building The languages and frameworks you use The programming language I use the most is called TypeScript, which was built on a better-known language called JavaScript. TypeScript is now both a “frontend” and “backend” language. This means, in a nutshell, that you can use it not just on devices to make app screens and webpages show up, but also on the faraway computers, called the servers, that handle the app’s data and send the right data to each user’s device. Being able to use one language for most of your job is nice. Being able to use one language for all of your job is pretty much impossible. The vast majority of web developers need to be multilingual by default. To be a frontend engineer, you can’t just know some flavor of JavaScript. You need HTML and CSS. HTML lets you decide what goes where — a heading here, an image there, a block of text yonder. CSS is how you style it with colors and fonts and such. JavaScript adds the logic and data. When I click on something, what happens? When I ask the server for some data and it arrives, how does it get handed off to the HTML? Most backend engineers know more than one language too, in large part because so many more exist, and because they have such different strengths. Many were around long before the web was, like C++ and Java. Some are newer but still largely meant for other disciplines, like Python for data science. And completely separate from whichever of these language(s) you use, there are also the ones you specifically need for database querying, be it SQL or MongoDB or whatever else. Especially for those languages that weren’t built for backend web development at all (and for some that were), tools called “frameworks” have sprung up to make backend web development easier. There’s Django and Flask for Python, NestJS and Express for Typescript, Rails for Ruby, and a lot more. A lot of impressive frontend frameworks have sprung up, too, that make Javascript easier to use with HTML and CSS, and more generally, that make it easier to build big web apps. React, Vue, Angular, and Svelte are a few big ones. They can have major philosophical differences — React, for example, favors a “top down” approach where data is handled and dispersed from some central source, while Vue is friendlier to the idea of user actions and events causing data to “bubble up” in a tree of components. Going further, each framework has an entire ecosystem around it. React, for example, has a spinoff called Preact that’s smaller and faster, and a related framework called React Native for building mobile apps. They also tend to have constellations of specific third-party libraries that work especially well with that framework. Which brings us to… The libraries and apps you use This section is going to be a lot of lists. This is on purpose. It’s a shock-and-awe tactic. There are a ton of things a codebase might (and should) want to “outsource” to an open-source library, if that library has already done the job well. Here’s a small subset of examples from the Javascript world alone: Date formatting (Dayjs) Creating forms for users to fill out (Final Form) Validating different types of form info like emails, phone numbers, etc. (Yup) Running automated tests to save time when doing quality assurance (Jest) Giving specific users of your app the permission to do specific things (CASL) Letting your frontend and backend communicate more easily (GraphQL) Letting your backend and database communicate more easily (TypeORM) Combining your app’s many files, which you separated for readability, into much bigger files that work better when parsed by actual web browsers (Webpack) When a library for something is particularly hard to build, you often have to integrate with another service entirely, sometimes paying for it, and usually getting some kind of web dashboard on the side. These also tend to cover pretty universal needs, like: Hosting your app (Heroku) Payment processing (Stripe) Letting people log in and out of your app (Auth0) Hosting landing pages for your app (Webflow) Hosting random files your app needs to access (Amazon S3) Making sure new code doesn’t break your app (CircleCI) Logging the errors your site has (Sentry) Logging what actions your users are taking (Mixpanel) And then there are libraries from both categories — open-source and paid — that are much more domain-specific, depending on the needs of your company, like: Rendering charts and graphs (D3) Allowing drawing and image processing (Fabric) Sending emails (Sendgrid) Sending texts and making calls (Twilio) Hosting chat services (Pusher) Managing customers and customer support tickets (Salesforce) Workforce management (Humanity) Medical record integration (AllScripts) Rendering math text (MathJax) Hosting alternative databases for data that doesn’t do well in rows and columns, like time-series data (InfluxDB) And then, of course, there are tools that don’t require a library that works with your codebase, but still involve adding some kind of product to your workflow that you have to get familiar with: Data dashboards (Looker, Tableau) Product management (Jira, Asana) Documentation (Slab, Google Docs) And finally, you have your apps. Every day, when I open my computer, there are at least ten apps from my launch bar that get opened without a second thought. I have to use them all pretty regularly. There’s: VS Code, for writing code Chrome, for testing code and looking stuff up Postman, for testing backend work without needing the frontend Figma, for looking at the designs of anything visual I’m building Slack, for communicating over text Zoom, for communicating over video Datagrip, for inspecting data in my database GitHub, for looking at changes over time in the codebase A text editor, for random notes, planning things out, and drafting long messages Terminal, for starting up the app I’m working on, switching between projects, and committing my work As numerous as these tools are, they’re almost always well-documented — your end users are the developers using them, after all. Inside companies, there’s less incentive for this. The code itself, and most of the nooks and crannies of how your product works, aren’t what you’re showing to the user. It’s the “happy path,” the core experience of your product. But the code itself, and its many edge cases, take up a huge percentage of your developers’ time. This discrepancy leads to whole new cans of worms. The next section is the big kahuna — it produces the most hydra heads by far. The product you’re building If you’ve read the book “The Mythical Man Month” by Fred Brooks, you already have an idea of the gist of this section. (If you haven’t, and if you work in a trade where teams of people do knowledge work, give it a read.) The premise, which I’ve seen validated too many times, is this: when you double the number of people working on a software project, you do not cut the time it takes to finish that project in half. Dream on. With each person you add, there’s extra communication you need to do regarding how the system works (or should work), and extra planning regarding when things should get worked on relative to each other and by whom. This takes time. Beyond writing the code that builds the thing, you need the following pieces in place, at a minimum: Are you “Agile,” as they say? Or do you “waterfall” instead? What kinds of meetings do you have where you decide what to build, how to build it, and how long it will take? Who does the work for each piece below, and when? A process. This can be data from app usage or results of user testing. If you’re constantly going on hunches, you’ll find yourself scrapping work, wasting money, and demoralizing your people. It’s hard to be sure of that, and it’s hard to get right — here’s good general rule: Reasons for building stuff. For a team to build something, the requirements of that thing need to be thought through. The internal consistency and external consequences of those requirements need to be considered. Then they need to be written down and set in stone for others. Changing the finish line after you start the race causes chaos and makes the end result sloppy, and often so sloppy that it needs to be redone. An ounce of prevention is worth a pound of cure. Specs. You need to know how, when you finish building something, everyone will come to agree that the finish line has been crossed. This means checklists that people can run through. This means automated tests that your machines can run through, both to save people time and to help ensure that you didn’t break old stuff, which inevitably happens in complex systems. Testing. You should have some kind of design system that doesn’t let engineers make user interface and user experience decisions while they’re writing code, for they will get those decisions horribly wrong. Design systems. Important pieces of the product, important decisions made in your project’s past, and important ways in which your product is modeled and represented and stored in your codebase and database — all of these should be written down as much as possible. The shortest pen is better than the longest memory. New engineers should also get some kind of documentation about how to start building things for your product, and on the tools and frameworks and libraries you use. Documentation. Virtually no one overestimates how much time it will save to be methodical about these. On the contrary, most of us underestimate that all the time. Is the fault in our tools? Clearly there’s a dizzying array of tools and processes you now need to understand in order to make software that’s creative and profitable and better than alternatives. This, as mentioned at the beginning, is where real complexity comes from. Some of this increasing complexity, it’s fair to say, is out of teams’ control. Software is eating the world, and that complexity . has to live somewhere Many software ecosystems, especially Javascript, are understood by those within them to be bloated and crowded, if incredibly versatile. When presented with the statement “building Javascript apps is overly complex right now,” an astonishingly low percentage of developers — — say they disagree. (Note that this percentage is also among people who like JavaScript enough to take a survey about it.) 34.4% That number is improving modestly over time, thanks to awesome work by awesome people building awesome tools. You might say “But wait! You said those tools can create even more opportunities for complexity!” They can. But not because of the tools themselves. It’s our understanding of them and of how we use them. The tools are incredible. Through them, a staggering amount of complexity has already been wrangled for us. Others have packaged up so much math and logic and physics and electrical engineering and computer science over the centuries, and especially over the last few decades, that you can now literally type some characters into some text files and pop out apps like magic. These tools are not hammers or screwdrivers, and it’s a mistake to think we can just cobble them together in increasingly complex ways and leave it at that. Brains process information like “how to ride a bike” or “how to build a chair” (this is called “implicit memory”) fundamentally differently from information like “the capital of North Dakota” or “the name of that script you use to fill your database with dummy data” (“explicit memory”). Every product, app, and library you use as a developer is a little language unto itself, with new challenges for your explicit memory. New vocabularies of jargon. New lists of functions to remember (or at least remember that they exist). The things in each of these categories are also typically nested — built on top of other tools — and they evolve and get updated frequently in ways that hammers don’t. The constant recall and lookup of these changing facts means there’s no real upper limit to the demands that they can impose on your brain’s . Chronically bumping up against the limits of your cognitive load is a powerful source of stress, and a lot of companies have very smart people fine-tuning the amount you experience while using their apps to make sure this doesn’t happen. cognitive load Teams can do this for themselves, too. Much still rides on the way you manage, or fail to manage, the complexity that you build upon and create for yourself. What’s within a team’s control, then, is not the nature of software tools, but the nature of the product that it’s building, and the ways in which you marshal different resources to build it. The solutions Solutions? You still want solutions? Ugh. Come back later. Tired from explaining the problems. Okay, obviously not. But it’s also tempting to skip solutions because they’re so infuriatingly, maddeningly simple. Most people could throw out uninformed suggestions that, if actually enacted, would beat the status quo. Anyway, fine. Here. Hire more people, not just to build, but to wrangle complexity Spend increasing amounts of time on product quality, not quantity That’s it. It’s really just up to teams whether they want to do it or not. The “increasing” part is annoying, but important. A fixed amount won’t cut it. As you build, you need to allocate more time and resources to reducing what complexity you can, and documenting what you can’t. Remember that the biggest productivity gains come from laying groundwork for others to walk on, whether that’s documenting complicated code, streamlining the ways developers test it, or anything in between. That’s why people build useful software in the first place. Too many teams understand this perfectly when it comes to selling easy-to-use software to their customers, and not at all when it comes to making their own product easier, or sometimes even possible, for their own team members to work on effectively. Many companies want to have their cake and eat it too. They want to build on top of, and make money from, the many ways in which others have wrangled complexity for them. They very rarely want to worry about wrangling it themselves. They want to build, build, build. They want to be feature factories. (Read for a great deep dive into why that’s not the only approach startups should consider.) this thread Again, the more of a feature factory you are, and the faster your complexity grows without being managed, the more damaging it becomes. And the harder it is to stay in front of it. And the less feasible it is, after a while, to even get in front of it. It’s a sneaky, vicious cycle, and the concrete symptoms are ugly. Projects start blowing through their time estimates. Unnecessary work is done. Necessary work is redone. Team communication suffers. Employees burn out. Improving each individual employee’s ability to handle complexity, even if that’s a good goal to have, is not a substitute for the real solutions. Your product’s internal complexity will still catch up to people’s cognitive load, even if they’re very smart. You have to strengthen your team’s collective ability to wrangle any part of the product’s complexity. For a product to be worked on by a team, where people come and go, and frequently communicate with each other, you have to make the product itself, and the many ways people work on it, inherently less complex and easier to understand. The people who can do this legwork — document, understand, and wrangle complexity — are out there. Many people, as crazy as it sounds, actually thrive on it. There are some roles for it already, with job titles like “Technical Product Manager,” and luckily they’re growing more common, but they’re not common enough. Instead, practically all of the complexity I introduced at each level of this post is currently distributed across a mishmash of engineers, product managers, and designers, rather than being owned in full-time roles. The collective need for changing this fact can manifest itself in many places — retrospective meetings, Slack discussions, 1:1s with managers, and offhand comments during daily standups, to name a few. When it does, don’t let it be swept under the rug. Don’t let people make excuses for it or kick the can down the road. Take note and take action. If you do so, you won’t just keep the creeping vines of complexity at bay. You’ll build better features, and you’ll build them more quickly, and you’ll build them with a much, much happier team.

Defeating Hydra: The Growing Problem of Complexity in Web Development

Ever since its inception in the year 1089 by Guido Van Rossum, the programming language Python has along far away. Sheldon did its creator knew that Python would in today's world be utilized for a variety of purposes such as research, development, scripting, among many others. Built as a successor in the ABC language, Python does not just find its applications in software development but also in research. Through the means of models based on Machine learning, Python is helping the world transform into a better place. Apart from these, the programming language is being used for web applications, graphic user interface applications, software development apps, numeric apps, etc. It is also being used for developing some business and 3D gaming applications. There are many reasons that Python is a . Be it the flexibility, abundance of robust libraries, strong mathematical foundation, ease of coding, platform independence, readability, etc. Today, Python stands as the world's most widely used language with over around the globe. developer's favorite language 8 million developers Even though its usability as an enterprise-level development tool is questionable, it's giving JAVA a fierce competition. Try to understand it from the fact that JAVA was once the world's popular development language. One of the best virtues of Python comes from its openness. It is a general-purpose, open-source programming language that can be used for the broadest number of applications. Having said this, you can find all popular organizations creating their versions of the word. Be it Microsoft's Cognitive Toolkit popularly known as CNTK, Google's Tensorflow or Facebook's PyTorch. Each is designed to serve a specific purpose and aid developers around the world achieve greater success with Python development. With new libraries and frameworks in the picture, it is becoming quite convenient for developers to build sophisticated applications and harness the best of Python. One such framework that has been launched recently launched by the online networking giant Facebook is . Hydra is fundamentally designed to aid complex computations and application building in Python. With this, Facebook will provide the other open-source framework in Python after . Hydra PyTorch Other popular open-source frameworks developed by organizations have specific purposes of serving too. Microsoft's CNTK is a deep learning framework that describes neural networks in Machine Learning as a structured series of computational steps that are directed by a graph. is primarily a mathematics library used for neural networks along with a plethora of other tasks. While PyTorch aims at assisting applications in computer vision and natural language processing, among others. TensorFlow Facebook's latest Hydra, on the other hand, is a framework for elegantly configuring complex applications. Even though Python might seem like a quintessential programming language, it becomes sluggish with increasing complexity. Some of the high-level programming concepts such as multi- threading work poorly with Python Development india
. In a step to resolve such underlying issues with the programming language, the framework Hydra is brought into the picture. To get a more in-depth insight into what Hydra holds for the future of Python, let's take a look at some of its features- Focus on the Problem One of the best qualities of Hydra is that it enables users to focus on the problem. Python users are already familiar with wasting time on boilerplate code such as logging, command-line flags, loading configuration files etc. Hydra eliminates all of this and helps you concentrate on the critical issues. This approach aids developers in saving a lot of time and increasing their efficiency when it comes to writing the code. In other words, Hydra may be the key to work efficiency for developers. Powerful Configuration Hydra lets the user compose a configuration dynamically. It means that developers can get a correctly built configuration for all runs individually. Hydra also aims at making test runs faster. When creating an application, developers devote a significant chunk of their time in experimentally testing it. However, Hydra facilitates overriding everything from the command line, making it easier to test and modify applications. Apart from this, one of the main tasks of Hydra is to remove the need to maintain multiple configuration files. Its robustness reflects in the way that it enables faster decision making and quick implementation. Pluggable Architecture With its rock-solid architecture in place, Hydra enables its integration with the user's infrastructure. Even though it is in its nascent stages right now, Facebook aims at taking the pluggable architecture a notch higher with time. In more versions to come, some Hydra plugins will enable launching a user's code directly on the Amazon Web Services or other similar cloud technology platforms straight from the command line interface. Again. Not only will this feature save a lot of execution and testing time for applications, but it also enables a more unified approach to programming. Being an open-source framework, Hydra promises to eliminate the need to write significant portions of the code again and again. This means it is much easier for Python developers to add particular functionality to projects for new use cases. Not only does it aim at reducing errors but also prototype complex research projects. It is already being used at Facebook to accomplish this task with ease. With Hydra, configurations are composed and overridden just before the application runs. This resolves one of the underlying issues in Python - maintaining multiple configurations or modifying them. Conclusion All these matter because it picks some of the most fundamental issues of Python that have been troubling developers for a long time. It targets those pain points by making development much faster, integrated and smoother. features of Hydra While a piece of code needs to accept and progress with new requirements in the applications, Python was the opposite. It turned slow-paced on complex applications, giving users a reason to switch to other languages such as Google's Go. But with Hydra in the picture, there reduced chances of bugs and a platform for the code to evolve more naturally with changing requirements. For the future of Python, it means a more problem-focused and more straightforward implementation approach. https://www.aegissofttech.com/en-au/python-development-services.html

What Does Facebook Hydra Mean For The Future of Python

Okay – I know thats an excessively dramatic title, but I wanted to emphasize a simple but important concept. I believe that normalizing remote work isn’t a single problem to be solved, but rather a number of distinct issues that need to be tackled separately. The benefits of adopting remote working practices are widely understood. Having flexibility around hours and geography allows you to engage the best talent no matter where they are located or what their current circumstances are. The drawbacks may be more contested, but regardless of whether you believe having a remote worker is 90% as effective as having that person physically present in your office or 65% as effective, it is hard to argue that it is just not as good. There are a lot of reasons for this, like the lack of face to face interaction, the need to adapt tools and processes, the difficulties around cultural engagement. The folks over at Zapier do a great job of breaking out and discussing these topics in their . The key point I'm trying to illustrate is that in order to close the delta between remote and in-person work, by 5% say, you probably need to think about improving 5 separate aspects by 1%. guide to remote work Rather than hoping that you can get that 5% bump by "fixing" a single issue. Hence the titular reference to the multiple headed beastie! That said, if you are to slay a hydra, you have to start somewhere, and I'd hate to turn you loose without some actionable insight. A quick internet search will bring up of ~80 companies that have a completely remote workforce. In the last month or so, I have been lucky enough to speak to leaders at 16 of the better known ones. a list One comment from a VP of Engineering really resonated with me, and I wanted to share. He said “The key piece of advice I give new employees is that you cannot over-communicate your status.” It is worth remembering that a lot of the tooling we use was largely built with an in-office experience in mind. Now while new( ) tools like Slack and Zoom have made it easier for remote workers to engage with their in-office counterparts, without a regular face to face interaction we still don’t have a good mechanism to communicate ongoing team status. ish By status I mean some way of indicating both project progress and team health. Remote standup tools can be helpful, but they have a couple of shortcomings, like a myopic focus on individual tasks, inconsistent reporting and no real way to gauge team sentiment. If you ask a remote worker “How are things going?” They will typically answer with the dreaded “eff” word - “ ”  “Fine” is a dreaded answer because fine might mean fine, or it might mean everything else up to the point where something is on fire. Fine! This is important because for the most part it is easier to prevent a fire than it is to extinguish one! We need a better way to indicate status, and there are actually some “old school” models that might help. I’m thinking of the concept of a or an . Niko Niko (sort of) means “smiley” in Japanese and the idea is that in a physical office, the team would have a prominent chart upon which they would indicate their mood – either via a happy, indifferent or frowny face on a chart when leaving the office each day, or some kind of traffic light system.

But both approaches have the same intent and benefit. Niko Niko calendar Information Radiator(/BVC) Seeing the “Mood” of the team can act as an early warning system for real problems. Given that team members in the same physical location are able to convey their mood at the water cooler or similar locations, it behooves those of us with remote teams, to make additional effort to make sure we understand our remote colleagues mood and respond to it. Of course, I’m not an impartial observer to this phenomena, and offers a tool that “solves” this problem.  Rather than mood, tracks your team’s confidence. That is the collective confidence of the entire team in achieving their goal. Genchi Genchi Tracking the team's confidence as it ebbs and flows through the life of a project helps highlight problems as they are encountered and makes it easier to manage dependencies. Also, because we involve every member of the team, we always have the best most current view of the team’s status, avoiding that rather abrupt shift from "green" to "red" that so often occurs in the last two weeks of a project. Being aware of team status (or mood) is 10 times more important for remote teams than for those that are co-located. So, I strongly encourage you to either give Genchi a try of find some other way to actively keep your fingers on the pulse of your team. It is a little additional effort that can pay enormous dividends. (Get a Free Trial by Clicking on my Writer Ad on my Hacker Noon User Profile) ( : ) Disclosure The Author is the Founder of Genchi

Slaying the Hydra of Remote Work Woes - One Head at a Time.

The Little Kernel That Could(n’t) After working around a minor bug in the , elucidated how I lost a week discovering and working around the hydra bug for my KVM/vf based virtual machine management prototype. explained how I lost another week making the iPXE intelxvf driver more robust. Introduction Part I Part II Everything was working, but I was annoyed that I didn’t actually fix the hydra bug and worked around it instead. I may not be able to control what kernel gets booted. It therefore makes sense to prevent hangs on the iPXE side, but I didn’t know exactly why it fixed the problem, and that bothered me. I had cheated and it felt dirty. I couldn’t rest until I had rectified the problem. Read on for the real solution. Months have passed since I last worked on this, so my first step is to reproduce the original issue. I revert my workaround and try to boot kernel 3.8. Same issue. I make sure to pull the most recent release of UEK3 just in case the bug has been fixed. No luck. I need to know where the kernel is hung. My first attempt is to try to enable the via the kernel command line so I can force it to stack trace. Unfortunately, no matter how much I mash print-screen, there is no response from the hung kernel. Perhaps I’m too early in the boot process? sysrq key It looks like I’m going to have to get down and dirty with gdb to get a stack trace. I download the debuginfo packages for the guest kernel and start qemu stopped with a debug listener ( and options). I remotely connect gdb, load the debug-ready kernel image, and type for continue. -s -S c All of the online help talks about debugging the kernel by booting it directly with qemu. When qemu starts my kernel isn’t loaded. It jumps through the BIOS, iPXE and grub before finally reaching the kernel code. I therefore assume that I’m going to have to figure out where the kernel is loaded in memory and tell gdb. Once I’m sure the guest is hung, I hit Ctrl-C, and to my surprise I see source code. Upon further reflection this makes sense, as the early kernel boot is going to load its routines into the same place in memory. Or maybe gdb is just magic. Looking at the backtrace it seems to be stuck in a in kvm_steal_clock. Ok time to investigate how the kvm clock works. while loop It appears that the kvm clock communicates with the host by allocating some shared memory that the host side writes to and the guest reads from. After a quick check to see if any of the code in that function had been changed in UEK, I do notice that it is wrapped in an , but the function is definitely being called so that is a red herring. #ifndef CONFIG_DTRACE Just to verify that the problem is at this location, I try to disable the completely. Passing in on the qemu command line changes the hang location slightly but doesn’t seem to solve the problem. I reason that the memory used by the kvmclock is somehow conflicting with iPXE, and maybe removing it is just corrupting something else. kvmclock cpu=host,-kvmclock Reading the early memory slab allocation in the kernel source, it looks like it is allocating a large chunk that includes part of the reserved area (this was the value for that was printed out way back in . Despite that, the kvmclock code prints out the memory locations that it is using, and, while they are close to the iPXE reserved region, they don’t seem to be directly conflicting. NODE_DATA Part I I decide to step through the kvm clock initialization to see if it can shed any more light. I restart the debugging session, type in followed by and wait for the breakpoint to hit. I reach the hang without hitting the breakpoint. What the? Is kvmclock_init getting skipped? No, the console shows the debug statement from the middle of the function, so it definitely ran. b kvmclock_init c I search the internet for qemu breakpoints and stumble across a that mentions that hardware breakpoints may work better than software breakpoints. I recall having issues with software breakpoints when I was developing a bootloader using qemu long ago. Something about being in 16-bit mode confused the debugger. I try a hardware breakpoint instead. It stops this time but gives me the cryptic message . page Remote ‘g’ packet reply is too long Brief Aside: GDB and Kernel Debugging All x86 processors start in 16-bit real-mode for compatibility. When intel introduced 32-bit processors, 32 bit instructions were hidden in . Even modern 64-bit processors start in real-mode and call specialized instructions to eventually enter a mode where 64-bit instructions are available. When you connect to qemu before the vm executes (using ), gdb detects that the processer is in real-mode and configures itself appropriately. protected-mode qemu -s -S Unfortunately, gdb doesn’t with mode switching during a remote debugging session. Software breakpoints are lost, and it doesn’t decode packets from the remote side properly. It is possible that a more recent version of gdb and/or gdbserver would solve this problem, but a couple of workarounds are sufficient to get moving again: deal well Use a hardware breakpoint instead of a software breakpoint. After the packet reply error, connect a new instance of gdb to the server The packet reply issue is solved by a reconnect and I’m able to dump memory through gdb and through a couple of qemu . There definitely seems to a value in the location that the kvmclock is using to communicate with the host, but it isn’t obvious what data is there and why it might be breaking. console commands I do another run with a newer kernel to see if I can spot any differences. I notice that the broken kernel refers to instantiating the “boot clock”, whereas the working kernel refers to the “primary clock”. Hmm, the code for instantiating the primary clock seems to be in a different location. Perhaps it is actually failing the initialization in the newer kernel but not hanging for some reason? I spend far too much trying to track down the difference in the primary clock. After a couple of hours of being frustrated in gdb, I finally discover that I’m looking at an older version of the code. In the actual working kernel code, the duplicate initialization of the primary clock has been removed and the name of the clock is changed from “boot clock” to “primary clock”. So the new kernel is actually running the same code. What a red-herring that was! Next I go down the rabbit hole of attempting to understand the early memory allocation process. The memblock allocator seems to find the highest available address by looking at the end of the available memory reported by the kernel and then slicing off a chunk. As far as I can tell by reading the code, any reserved sections are only taken into account later when the page table is constructed. I start to think that there is a bug in the early boot process where allocations before the page table is initialized could conflict with BIOS-reserved memory. There are a couple of problems with my theory: It seems unlikely that a massive bug like this would have made it through 10+ major versions of the kernel without getting noticed. The working kernel reports allocating the kvmclock in the same location as the broken kernel and the specific address seems ok. If the bug is definitely in the kvmclock code, maybe the memblock allocation is ok after all. I take another stab at verifying the bug is definitively in the kvmclock. I notice a couple of command-line parameters that seem to control the initialization of the clock from the guest side. Trying various combinations of these parameters, I discover that if i specify and then the guest actually boots. This looks like hard evidence that the problem is in kvmclock. no-kvmclock no-kvmclock-vsyscall In the end I get lucky. Scouring the net for various function calls in kvmclock_init, I stumble across a very promising . The details refer to a hang because kvmclock is reading unitialized memory when there is more than one processor. As a test, I lower the number of processors to 1. The kernel boots! This bug report is definitely about our bug. bug report The for the bug is fairly trivial. It was a brief regression introduced in 3.8 and fixed in 3.10. Other distros had backported it. I check the UEK source and it has not been backported. The last step is to verify that the fix actually solves our hang. fix I’d prefer not to do a full kernel build, so instead, I fire up gdb and set a breakpoint where the offending memory is allocated. After the breakpoint, I manually call memset to clear the memory, and continue the boot. Eureka! I have a working 3.8 kernel with kvmclock enabled. I send a bug report to the UEK team, confident that the next release of the 3.8 kernel will be free from this tenacious little bug. So it turns out that the kernel memblock allocation is just fine. It just so happens that it allocates some previously used memory for the kvmclock shared area. Due to a minor change in the implementation, the memory for the kvmclock for secondary processors is not initialized, so it attempts to read a garbage value and loops forever. The workaround in was moving the kvmclock to memory that happened to be initialized to zero as a side-effect. This is good to know, because the workaround will fail if something in the boot process (BIOS, iPXE, or grub) happens to write data at the new location. In other words, my workaround mostly works by chance. Part I So what is the moral of the story? Search for existing bug reports earlier! I had dreams of finding an interesting problem in kernel memory allocation, but if I had just googled more directly about some of the lines of code around the actual hang, I would have discovered the key bug report much sooner. Is a missed backport a fitting conclusion to three weeks worth of debugging? Perhaps not. But at least I will sleep easier knowing why more poor little kernel couldn’t boot. No bug left behind!

The Hydra Bug: Part III

iPXE — 60% of the Time, it Works Every Time After working around a minor bug in the , elucidated how I lost a week discovering and working around the hydra bug for my KVM/vf based virtual machine management prototype. Introduction Part I I had a build of iPXE that worked… the first time. I stretched the truth slightly in . The failure to reboot wasn’t a totally a surprise. I had run into the bug a few times during prototyping, but I would fix it by using a different vf or rebooting the host machine. Read on for part II of the adventure. Part I On the second boot of a vm, iPXE fails to communicate via iSCSI. Turning on debugging for the intelxvf driver reveals that no packets are being received. Once the vf is in this state, it will remain broken until the host is rebooted. Clearly there is some persistent state in the virtual function that is not being cleared. Brief Aside: Intel NIC Configuration Configuration for the Intel 82599 NIC is performed by writing to specific DMA locations. Individual locations control different features and are referred to in the data sheet as “registers”. For example, there is a register to control where packet data should be read from. Some registers can only be written to via the physical function (e.g. on the hypervisor host), but other registers can be written via the virtual function. This is how the virtual function driver sets up transmit and receive queues. If a bug is persisting across multiple instantiations of a kernel/driver, it must be because some register value has been changed. The driver in iPXE sets register values itself so it can communicate, but the linux driver later reconfigures the registers once the kernel has taken over. I read through the Intel trying to understand what might be going wrong. I find the for setting up the registers to see if it is failing to reset something. That’s odd, the code is quite clearly sending a message to the physical device asking for a reset. Why wouldn’t it be resetting? datasheet iPXE code The for the physical function (that’s the driver for the physical device on the hypervisor side), I am surprised to discover, doesn’t reset everything when it receives a vf reset message. While some values (like mac address) are reset, others are not (like send and receive queues). This gives me a hypothesis: The vf driver in linux must be setting some up some registers, and when iPXE runs the second time it is using incorrect values. driver code I peruse the code, looking for a host-side workaround to perform a complete reset of the vf. Eventually I find one buried in . I can now reset the queues by using ethtool to change the ring parameters of the parent device. This is fantastic for testing as it means I don’t have to reboot, but, unfortunately, it resets all virtual functions at once. That means resetting the NIC for one vm when it is created will reset all of the other vfs on the same box. That’s not very friendly! So I’m going to have to modify the iPXE side to reset properly. ethtool Initially, I try comparing the linux vf code to the iPXE vf code to look for differences, but they are stylistically different enough to make it difficult. I wonder if there is some way for me to view the current setting of the vf registers. Fortunately, Intel provides source for , a tool that can dump the current register values. ethregs My goal is to get the registers of a NIC after iPXE has reset it to be the same in the before linux and after linux scenarios. With ethregs and diff, i find a difference in the configuration of the second RX queue. The register for the RX queue contains a DMA location. This is where the NIC will write the contents of packets to be read by the driver. On the first boot the second RX queue has the register set to zero — it’s disabled. On the second boot, there is a value there. Diving into the code, I see that iPXE has only configured a single RX queue. A picture starts to become clear. The linux driver is configuring multiple RX queues and telling the hardware where its receive rings are located in memory. On reboot, iPXE sets the first queue to its receive ring, but the second RX queue is left with a memory location that is no longer in use. The NIC is happily sending some of the packets to the second queue and nothing is reading it! I prefer to keep the iPXE code simple; it doesn’t need multiple read queues. I therefore add a bit of code to reset all of the RX and TX queues before it does its existing configuration. The first boot with the new code still works perfectly, but the second still fails. Apparently there are more registers that need to be reset. Through meticulous diffing, I discover a few more differences. The main offending register seems to be one called INTELXVF_PSRTYPE, which the datasheet tells me configures how the packet will be split between header and data. The NIC can do some of the header processing, which linux takes advantage of. This advanced setting just confuses iPXE, which is expecting the whole packet to show up in its queue. There are also some flags that get changed by linux in the SRRCTL register. These don’t seem likely to be causing problems, but I reset them anyway. With my new changes, the registers in each case are as close as I can get them: there are minor differences in allocated memory locations or counters, but the stable values are identical. All that remains is to see if the second boot actually works. I switch to my virtual console to check the output from iPXE. For a moment i think it has stalled again, but it makes it through and I’m greeted by grub. Success! I run through a third and fourth boot for good measure. Everything is smooth. It appears my prototype is complete. I compile a patch for an upstream . I move on to other things. My prototype turns into a system that is almost ready for production. Before we ship, it might be nice to fix that kernel bug that I worked around in . That adventure is detailed in . proposal Part I Part III

Stories, News & Mentions about HYDRA

Hydra’s Stories on HackerNoon