Thanx is a leading provider of digital guest engagement for offline businesses

Thanx

thanx

Mission Generation Using Classic Machine Learning and Recurrent Neural Networks in Zombie State

জম্বি রাজ্যে ক্লাসিক মেশিন লার্নিং এবং পুনরাবৃত্ত নিউরাল নেটওয়ার্ক ব্যবহার করে মিশন জেনারেশন

Missionsgenerierung mit klassischem maschinellem Lernen und rekurrierenden neuronalen Netzwerken im Zombie-Zustand

Generación de misiones utilizando aprendizaje automático clásico y redes neuronales recurrentes en estado zombie

Génération de mission à l'aide de l'apprentissage automatique classique et des réseaux de neurones récurrents dans un état zombie

ज़ोंबी स्टेट में क्लासिक मशीन लर्निंग और रीकरंट न्यूरल नेटवर्क का उपयोग करके मिशन जनरेशन

ゾンビ状態での古典的な機械学習とリカレントニューラルネットワークを使用したミッション生成

좀비 상태에서 고전적인 기계 학습과 순환 신경망을 사용한 미션 생성

Geração de missão usando aprendizado de máquina clássico e redes neurais recorrentes no estado zumbi

Генерация миссий с использованием классического машинного обучения и рекуррентных нейронных сетей в состоянии зомби

Zombie State'te Klasik Makine Öğrenimi ve Tekrarlayan Sinir Ağlarını Kullanarak Görev Oluşturma

Tạo nhiệm vụ bằng cách sử dụng mạng lưới thần kinh định kỳ và học máy cổ điển ở trạng thái Zombie

在僵尸状态下使用经典机器学习和循环神经网络生成任务

Part 1: Odell vs Saylor It was only last week, exact date and time unknown given how well-protected is the individual data. A Nostr client and Cypherpunk going by ODELL, unleashed a seething post from the depths of his Post Office. The post started by saying it was not controversial, in capital letters. Then proceeded to say something controversial. This, in addition to other, capital-lettered posts. In brief, Odell claims that Michael Saylor, a prominent Bitcoin whale and renowned hodler, is against the funding of Bitcoin developers. Let Bitcoin “ossify”, believes Saylor (says people on his side). So the code can become something so usual humans sing it in an anthem. Like, unchanged. *** Meanwhile, Mr. Saylor hasn't said anything to either deny or verify the aforementioned claims nee sayings in his name. The Advantage of Not Funding Bitcoin Developers Aside from the snide remarks and memes on X comparing BTC Devs to busybodies, the biggest threat of the ‘Dev Funding Race’, goes the argument, is that it incentivizes fiat overlords to bulldoze Bitcoin into the shape and form that they want. Masked as “maintenance” and “upgrades”, perpetual changes to Bitcoin's code could leave vulnerabilities. Currently, Bitcoin is still struggling, and the price won't stop dancing around $69k. It is barely a trillion-dollar asset. Meanwhile, the fiat world boasts hundreds of trillions of dollars, and Bitcoin ETF companies alone are bench-pressing north of 100 billion dollars. And they want to dangle all those carrots in front of our developers. Evil. But it might be true. Jack Dorsey mentioned names. Bitcoin developers have lived for years like the Fremen of Dune. Modestly quenching their financial thirst with drops of donated water-satoshis. Recycling each sat they get their hands on while leveraging more circularity than a hamster on a new wheel. Yes, while we all reap the benefits. Now, the promise of big money raining down on their well-honed skills is a big threat to the core code. Like lawyers, they are itching to change the Constitution. But hey, what is a developer who does not develop new tools? The devs in the wider cryptocurrency community must be cracking their ribs with laughter. And counting their lucky altcoins, grateful for the fact that they can buidl. A Developer Shall Develop. What do you call a developer who does not develop on a whim? A Watcher. A Waiter. Waiting for the call to help. Like a Lone Ranger in a Western. But that is painfully lonely. And sounds segregative af if we enforce it on other people. Honestly? Developing is a form of free speech. So no one can actually stop developers from speaking their speech. Go ahead, develop. It is the highest form of individual autonomy known to humans in this digital age. Paid or Not paid. For if you can develop, you can maintain your digital privacy (which is just privacy), maximum autonomy, and making a few bucks for people who want features isn't too bad either. Win-win-win. I dare say, if we could all develop (We cannot. It is a vocation some of us shall only inch towards. Slowly), this would not even be an issue. It would be strange for Saylor, anybody, to even suggest that Bitcoin developers try not to develop too much. However, Bitcoin being what it is, a global community project of extreme importance, actually constrains hacker ethos so much there won't be any cypherpunks like the pre-Bitcoin OGs. Forget flipping it to centralized systems anymore by engineering some arcane codex. Now, it is a game of sitting and waiting and watching. Praying that the price stops dancing around $69k. Saylor and his crew imply that the price of freedom is to follow the majority. And the majority says just sit, wait, and watch. But do they? The conversation is still evolving. Here is more data so that you have an informed opinion. How Odell and His Crew Are Right Here is a listicle of all the good points supporting Mr. Odell and his band of merry devs. The First Bitcoiner was a Bitcoin Developer. He was called Satoshi Nakamoto. You might have heard of him. 2. The Last Bitcoiner will be a Bitcoin Developer. When the Bitcoin is finally dead (a nuclear holocaust will end it), only a developer listening for its faint signals in the cold dark nuclear winter shall help keep the flame of freedom alive. 3. When in doubt, we ask Bitcoin Developers. Don't know why Michael Saylor is so confident that the US government cannot hack his bitcoins? Ask a Bitcoin Developer. 4. Developer Wars birth great stuff. Like the Lightning Network. Once Upon a Time, Bitcoin Developers fought for 2 bitter years over how much storage space should be held by Bitcoin. In the end, they doubled the storage space from 1MB to 2MB, invented Segwit which kept fees very low for a long time, and work on Segwit helped birth the Lightning Network. When developers fight, good things happen. When politicians fight, … 5. If ETF people want arcane-looking wallets with features from the year 3000, they have a right to pay for them. I think the key problem in this battle between Odell and Saylor is the fear that deep-pocket ETF funds may corrupt most Bitcoin developers. Otherwise, Bitcoin developers have been funded forever, and no one has been complaining. Indeed, Ark Invest should tell us what they need to pay OG Bitcoin developers some eye-popping sums of money for. If they want features, they could pay this to anybody else who is into UI. Bitcoin core devs are OUR lawyers, and they unravel weird shit like weird fee hikes and why LN nodes need a balance before any money can be received at all from other nodes. Bullshit. OG Bitcoin core devs have a right to build what they want and that does not mean they are corrupt. If a job developing UIs pays more than one developing arcane zk proofs, I'm taking the UI job. Let's not put evil intent into people's praxeologic actions. Give Ark and Bitcoin devs a chance. Then judge them when they mess shit up. Coz it could be magic. It can always be unmessed up. It is software, and we have all the commits in the GitHub repo. (It is not the same as giving Ark your money. That is on you. DYOR). Basically, let's all have some balls and break some dev rules. 6. ETFs paying Developers does not mean we shall be bought as well. Point 5 above feels like if the OG Bitcoiner developers are bought, doom and gloom for the rest of us non-dev plebs. But I assure you, if Bitcoin drops its 21 million coin supply cap for any reason, I will not follow it. That will not be Bitcoin but some silly hard fork. Call it BFD - Bitcoin For Devs. It will be worth exactly 0 US dollars after a few days. 7. Developing is greater speech than English. Last for today, but certainly not least. We should all aim to be Bitcoin developers, AI developers, Quantum Developers, and Math gurus creating an alternative universe free from the greed and psyop pervasive in this universe. But on the same planet earth. Orthogonal to this one in cyberspace. Bitcoin has done it for money. But planting a seed and watering it are two different things. And it will start by paying devs and supporting the right to pay devs. 100 years from today, Bitcoin should look like it is 100 years from today. Developing is how that happens. If developers break something we can always cancel them on X. Then unbreak it back. Part 1: Odell vs Saylor It was only last week, exact date and time unknown given how well-protected is the individual data. A Nostr client and Cypherpunk going by ODELL, unleashed a seething post from the depths of his Post Office. The post started by saying it was not controversial, in capital letters. Then proceeded to say something controversial. This, in addition to other, capital-lettered posts. In brief, Odell claims that Michael Saylor, a prominent Bitcoin whale and renowned hodler, is against the funding of Bitcoin developers. Let Bitcoin “ossify”, believes Saylor (says people on his side). So the code can become something so usual humans sing it in an anthem. Like, unchanged . unchanged *** Meanwhile, Mr. Saylor hasn't said anything to either deny or verify the aforementioned claims nee sayings in his name. The Advantage of Not Funding Bitcoin Developers Aside from the snide remarks and memes on X comparing BTC Devs to busybodies, the biggest threat of the ‘Dev Funding Race’, goes the argument, is that it incentivizes fiat overlords to bulldoze Bitcoin into the shape and form that they want. Masked as “maintenance” and “upgrades”, perpetual changes to Bitcoin's code could leave vulnerabilities. Currently, Bitcoin is still struggling, and the price won't stop dancing around $69k. It is barely a trillion-dollar asset. Meanwhile, the fiat world boasts hundreds of trillions of dollars, and Bitcoin ETF companies alone are bench-pressing north of 100 billion dollars. And they want to dangle all those carrots in front of our developers. Evil. But it might be true. Jack Dorsey mentioned names. Bitcoin developers have lived for years like the Fremen of Dune. Modestly quenching their financial thirst with drops of donated water-satoshis. Recycling each sat they get their hands on while leveraging more circularity than a hamster on a new wheel. Yes, while we all reap the benefits. Now, the promise of big money raining down on their well-honed skills is a big threat to the core code. Like lawyers, they are itching to change the Constitution. But hey, what is a developer who does not develop new tools? The devs in the wider cryptocurrency community must be cracking their ribs with laughter. And counting their lucky altcoins, grateful for the fact that they can buidl. A Developer Shall Develop. What do you call a developer who does not develop on a whim? A Watcher. A Waiter. Waiting for the call to help. Like a Lone Ranger in a Western. But that is painfully lonely. And sounds segregative af if we enforce it on other people. Honestly? Developing is a form of free speech. So no one can actually stop developers from speaking their speech. Go ahead, develop. It is the highest form of individual autonomy known to humans in this digital age. Paid or Not paid. For if you can develop, you can maintain your digital privacy (which is just privacy), maximum autonomy, and making a few bucks for people who want features isn't too bad either. Win-win-win. I dare say, if we could all develop (We cannot. It is a vocation some of us shall only inch towards. Slowly), this would not even be an issue. It would be strange for Saylor, anybody, to even suggest that Bitcoin developers try not to develop too much. However, Bitcoin being what it is, a global community project of extreme importance, actually constrains hacker ethos so much there won't be any cypherpunks like the pre-Bitcoin OGs. Forget flipping it to centralized systems anymore by engineering some arcane codex. Now, it is a game of sitting and waiting and watching. Praying that the price stops dancing around $69k. Saylor and his crew imply that the price of freedom is to follow the majority. And the majority says just sit, wait, and watch. But do they? The conversation is still evolving. Here is more data so that you have an informed opinion. How Odell and His Crew Are Right Here is a listicle of all the good points supporting Mr. Odell and his band of merry devs. The First Bitcoiner was a Bitcoin Developer. The First Bitcoiner was a Bitcoin Developer. He was called Satoshi Nakamoto. You might have heard of him. 2. The Last Bitcoiner will be a Bitcoin Developer. When the Bitcoin is finally dead (a nuclear holocaust will end it), only a developer listening for its faint signals in the cold dark nuclear winter shall help keep the flame of freedom alive. 3. When in doubt, we ask Bitcoin Developers. Don't know why Michael Saylor is so confident that the US government cannot hack his bitcoins? Ask a Bitcoin Developer. 4. Developer Wars birth great stuff. Like the Lightning Network. Once Upon a Time, Bitcoin Developers fought for 2 bitter years over how much storage space should be held by Bitcoin. In the end, they doubled the storage space from 1MB to 2MB, invented Segwit which kept fees very low for a long time, and work on Segwit helped birth the Lightning Network. When developers fight, good things happen. When politicians fight, … 5. If ETF people want arcane-looking wallets with features from the year 3000, they have a right to pay for them. I think the key problem in this battle between Odell and Saylor is the fear that deep-pocket ETF funds may corrupt most Bitcoin developers. Otherwise, Bitcoin developers have been funded forever, and no one has been complaining. Indeed, Ark Invest should tell us what they need to pay OG Bitcoin developers some eye-popping sums of money for. Ark Invest If they want features, they could pay this to anybody else who is into UI. Bitcoin core devs are OUR lawyers, and they unravel weird shit like weird fee hikes and why LN nodes need a balance before any money can be received at all from other nodes. Bullshit. OG Bitcoin core devs have a right to build what they want and that does not mean they are corrupt. If a job developing UIs pays more than one developing arcane zk proofs, I'm taking the UI job. Let's not put evil intent into people's praxeologic actions. Give Ark and Bitcoin devs a chance. Then judge them when they mess shit up. Coz it could be magic. It can always be unmessed up. It is software, and we have all the commits in the GitHub repo. (It is not the same as giving Ark your money. That is on you. DYOR). Basically, let's all have some balls and break some dev rules. 6. ETFs paying Developers does not mean we shall be bought as well. Point 5 above feels like if the OG Bitcoiner developers are bought, doom and gloom for the rest of us non-dev plebs. But I assure you, if Bitcoin drops its 21 million coin supply cap for any reason, I will not follow it. That will not be Bitcoin but some silly hard fork. Call it BFD - Bitcoin For Devs. It will be worth exactly 0 US dollars after a few days. 7. Developing is greater speech than English. Last for today, but certainly not least. We should all aim to be Bitcoin developers, AI developers, Quantum Developers, and Math gurus creating an alternative universe free from the greed and psyop pervasive in this universe. But on the same planet earth. Orthogonal to this one in cyberspace. Bitcoin has done it for money. But planting a seed and watering it are two different things. And it will start by paying devs and supporting the right to pay devs. 100 years from today, Bitcoin should look like it is 100 years from today. Developing is how that happens. If developers break something we can always cancel them on X. Then unbreak it back.

Odell Vs Saylor - Should Bitcoin Devs Dev?

Elon Musk's X Lawsuit Against Anti-Hate Group, CCDH, Alleging Speech Suppression Tactics 

Hey everybody! I'm Mike Roks, and this is the second part on solving Dynamic Programming problems. , we mentioned the base principle of solving all DP problems:
Finding a 'question' to 'ask' for input (X) - large X so that: In the first part Answer is "Given" (obvious, known) Or can be always calculated using answers for (x) - , where (x) is an input "smaller" (simpler, in the context of a given task, logically ordered before) than (X). We saw this method working with an example of Fibonacci numbers, and now, let's use it to solve a more complicated task. small x - Unique Paths ( ) Medium https://leetcode.com/problems/unique-paths/ We have a robot, which starts at the top-left corner. They can move down & right. We need to calc all the unique ways for the robot to get to the right-bot corner. Remember, we need to find such a 'question' so the answer is either obvious or can be formed from previous answers. We want to know the number of ways the robot can get into the FINISH cell. The robot can get to FINISH either from the cell one higher than it or one lefter. So, to get to cell 20, the robot HAS to go either from cell 19 or through cell 13, And there is only one way to go from 19 to 20 (make a step right), and only one way to get from cell 13 to the cell 20 (make a step down). So all the possible paths to cell 20 == (all the possible paths to cell 13) + (all the possible paths to cell 19). We can get to cell 19 from cells 18 & 12 in the exact same manner. To cell 13 - from 12 & 6 (you can already see 12 being used several times, so it's a good idea to cache it) To cell 6 we can only get from cell 5, so the total amount of ways to cell 6 is the same amount as to cell 5 (as there is only one way to get from 5 to 6 - just go righter from 5). And so on until we consider cell 1 & cell 7. We can get there only from the cell 0. The robot starts at cell 0. How many ways are there to get from cell 0 to cell 0? This is a ridiculous question, so we just need to figure out the number which would make sense. If we say, "there are 0 ways to get to cell 0", so by our calculation, the number to get to cell 1 would be 0, to cell 7 == also 0, to cell 8 == (0+0), so also 0, and all the cells would have "0 ways to get to them". Saying there is > 1 way is unreasonable, as cell 0 is the initial cell, and we are given no information that there were prior branches in paths to get to it. Saying there is exactly one way to get to cell 0 would mean there is one way to cell 1, 1 to cell 7, and two to cell 8, and it makes sense. So our `base case` is - there is 1 way to get to cell 0, and we can get the number of paths to the current cell by looking up & left. If cell exists, we += its number of paths, else not. We have all of the necessary parts to solve this problem. The question is - "what is the number of ways to i, j (row, column)"? The answer is: if i == 0 and j == 0, then 1

else if i < 0 or j < 0, then also 0

else sum(by i-1, j; by i, j-1) Let's code it out: class Solution {
    int[][] g;
    public int uniquePaths(int m, int n) {
        g = new int[m][n];
        g[0][0] = 1;
        return paths(m-1, n-1);
    }


    //paths i, j
    //  if oob ret 0
    //  if in g > 0 -> ret g[i][j]
    //  else return paths i-1, j + paths j, i-1


    int paths(int i, int j) {
        //if oob, then 0
        if(i < 0 || j < 0) return 0;
        //if cached - from cache
        if(g[i][j] > 0) return g[i][j];
        //if not in cache - calc
        int res = paths(i-1, j) + paths(i, j-1);
        //record calced to cache
        g[i][j] = res;
       
        return res;
    }
} We could enhance the Space Complexity of this solution, but it's not necessary for the purposes of this article - we managed to decompose a complex problem statement to a question by the generic pattern & solved it. **That's it, folks, like, share, subscribe! :D

Grokking Dynamic Programming (Part 2)

Hi everybody, my name is Mike Roks, and today I'm going to try to give you a comprehensive understanding of what Dynamic Programming actually is, how are the tasks regarding it structured, and what is the general approach to solving these tasks. The base principle in all dynamic tasks is finding a way to get the return value for input (X) - so that: programming large X Answer is "Given" (obvious, known) Or can be always calculated using answers for (x) - , where (x) is an input "smaller" (simpler, in the context of a given task, logically ordered before) than (X) small x That's pretty much it, but it requires clear examples to be understood in practice, let's check them out: - Fibonacci numbers Easy In the example of Fibonacci numbers, the "Given" answers are for inputs 1 (== 0) and 2 (== 1), the "logically larger" input, in this case, is a larger number on an axis [1, +inf), and to calculate answer for X, you need answers for X-1 and X-2. So the "direction" in which the X grows is - towards +inf on an integer scale, and the way to get the return value is: If 1 or 2 ret 0 or 1 If > 2 ret fib(X-1) + fib(X-2) This way, getting return values for "Larger X" can always be done using values of "smaller x” and values for "the smallest x" are known. So, we have a valid solution to this problem using the base principle of dynamic programming. The non-optimized but obvious code is: public class FibFinderBad {
    public static void main(String args[]) {
      int numToFindFibFor = 8;
      System.out.println(fib(numToFindFibFor));
    }
    
    private static int fib(int x) {
        if(x < 1) return -1;
        //the "obvious" answers - "base cases"
        if(x == 1) return 0;
        if(x == 2) return 1;
        
        //the answer which uses "smaller" x's, but not "larger" x's
        return fib(x-1) + fib(x-2);
    }
} As you can see, in each call of , we either 1) return straight away if x is < 0, 0 or 1, or make 2 other calls of the same method . So if is 0, we make a total of 1 calls for , if is 1, we also make a total of 1 call. If is 2, we make 1 call for 2 & 1 call for both and , so the total of 3 calls.
If is 3, we make 1 for 3, 1 call for 2, 2 calls for 1, 1 call for 0.
Here is a picture of calls for calculating fib fib numToFindFibFor fib numToFindFibFor numToFindFibFor 1 0 numToFindFibFor fib(5) But in this graphic, you can notice we are doing the same operation multiple time, for example in the same run of calculating of 5, calculating of 2 comes up a numerous amount of time! Same for fib of 3. You can imagine if the initial becomes a huge number, we will calculate of 2, 3, 4… thousands or millions of times, completely unnecessary, as of 3 is the same no matter how many times you calculate it! fib fib x fib fib So, we introduce an , which is not necessary for a task to be solved in a , but which often helps us to drastically reduce the Time Complexity of our solution - . optimization mechanism Dynamic Programming manner Caching The number of calls of in the initial solution was of the order of , which is already a monstrous number when goes beyond 40, and to calculate of 300 with this code in your lifespan, you'd need a supercomputer 😀. Let's cache the previously calculated values of instead of just throwing them away. A cache can be any data structure, it just needs to allow us to save the currently computed result & try to find a previously calculated result fib 2^numToFindFibFor numToFindFibFor fib fib public class FibFinderBadWithCaching {
 final static Map<Integer, Integer> fibCache = new HashMap<>();


 public static void main(String args[]) {
   int numToFindFibFor = 7;
   System.out.println(fib(numToFindFibFor));
 }


 private static int fib(int x) {
   if(x < 1) return -1;
   //the "obvious" answers - "base cases"
   if(x == 1) return 0;
   if(x == 2) return 1;


   if(fibCache.containsKey(x)) return fibCache.get(x);
   //the answer which uses "smaller" x's, but not "larger" x's
   final int res = fib(x-1) + fib(x-2);
   fibCache.put(x, res);
   return res;
 }
} And now, when you are running , you'll have to call ~900 times in total. That's linear complexity! fib(300) fib This code can be even further optimized to require a constant Space instead of requiring as many items in as there are unique calls to , but this optimization it's not necessary for the purpose of this tutorial. We have already gone from TC of O(2^N) to , which is a qualitative difference, as before we could not run on anything above 100, and now we can.
That's it for now, in part 2 we will see a more complicated task & use the same base principle in solving it! fibCache fib TC of O(N) fib

Do Not Fear Dynamic Programming (Part 1)

is an ML expert with over six years of hands-on experience in ML engineering and product management. As an ML Tech Lead at Deliveroo, Gusev developed a proprietary internal experimentation product from scratch as the sole owner. As part of the innovative stream of Yandex Music transforming the product to add podcast listening experience to the service, he built a podcast recommendation system from scratch as an at Yandex and achieved a remarkable 15% target metrics improvement. Additionally, as Head of Recommendations at SberMarket, his tech-driven roadmap elevated AOV by 2% and GMV by 1%. Petr Gusev ML Engineer Can you provide examples of specific ML projects that you have managed successfully? How did you ensure the project's success? I led a project to personalize the ranking of products featured on the main page of a grocery delivery service. Our main task was to increase business metrics through personalized product offerings. To make sure the product would be successful, I did: Interviews with users, with the intention of understanding what is valuable to them and the type of products they would like to see on the main page. In this project, I also invited ML engineers to participate in the interviews so that they could get context by listening to the users. Checked whether the hypotheses that arose after interviews with users were confirmed by the data we gathered Recorded the metrics that we expected to reach as a result of developing this functionality Before starting actual development, we developed the model and made sure that it gives us the necessary quality and that we can achieve the necessary results in the A/B test. This is a very important point because developing an is a research project that may not give us the results we expect. And if development has already started, we will waste the entire team’s time without achieving the necessary results. ML model Assembled a cross-functional team of analysts, developers, and ML engineers and made sure that all people working on the project were on the same page throughout every stage of product development. What are the specifics of managing an ML team? A key aspect of managing an ML team is defining the project scope with precision, given the exploratory nature of ML projects. This requires setting clear, structured tasks. For instance, to delineate tasks accurately, I often use a template like "Implementing X will impact metric Y by Z.” This approach helps us set tangible goals and measure progress accurately. Talent acquisition and skills development are also vital. Building a team with the right mix of skills is essential, and continuous learning should be encouraged to keep up with the rapidly evolving field of machine learning. This includes nurturing both technical skills and soft skills, such as problem-solving and teamwork. Cross-functional collaboration is another important element. Especially when integrating and building a new ML model, the ML team needs to work closely with other departments, such as data engineering, product management, and marketing, to ensure that the ML model goes live. Lastly, in this field, risk management and experimentation are crucial. Managing an ML team involves balancing and mitigating the risks associated with new and untested models while encouraging innovation and experimentation. It's about creating a safe space for trial and error, learning from failures, and continuously improving the models and techniques used. How do you delegate tasks and responsibilities within your ML team to maximize efficiency and productivity? Structuring Work by Product Functions: I organize responsibilities based on different product functions such as consumer, operations, and marketing. This approach ensures that team members can specialize and develop deep expertise in specific areas, leading to more efficient and effective problem-solving. Rotating Team Members Between Streams: Periodically, I rotate Machine Learning Engineers (MLEs) between different streams. This not only prevents monotony but also broadens their experience and perspective. For instance, an MLE who has completed an initial project in the consumer stream may be moved to work on improving the model in a different stream. This rotation fosters a well-rounded skill set and promotes adaptability within the team, and reduces the so-called “bus-factor.” Assigning Tasks Based on Seniority and Growth Potential: Senior MLEs are often tasked with more complex, end-to-end projects or major research initiatives, such as building initial model prototypes. Meanwhile, junior MLEs are assigned to support these projects, providing them with opportunities to learn and hone their skills. This mentor-mentee dynamic accelerates the growth of junior team members and ensures effective knowledge transfer. Setting Clear Objectives and Expectations: For each task or project, I clearly define the objectives, expected outcomes, and timelines. This clarity helps team members understand their roles and responsibilities, enabling them to work more autonomously and efficiently. What strategies do you employ to ensure effective communication and coordination between data scientists, engineers, and other team members? Scheduled regular meetings where team members from different functions can update each other on their progress, discuss challenges, and brainstorm solutions. Ensured all work, whether code, data analysis, or research findings, is well-documented and accessible on shared platforms. Established and agreed with team members on clear channels for communication specific to different needs – such as urgent queries, brainstorming, or status updates. Involved all relevant team members in goal-setting and project-planning sessions. How do you handle challenges that may arise during the development and deployment of ML models? What potential bottlenecks can occur when managing an ML team? Often, the key factor is the model's execution speed. Initially, it's crucial to understand the specific constraints needed, as these can vary across different segments of a product. For instance, a model might be required to deliver results in less than X milliseconds or operate on less powerful machines. To effectively manage this, a rapid prototype should be developed to evaluate the model's performance in a production setting. If meeting these constraints is impossible, then it’s important to negotiate with stakeholders to determine which aspect we should prioritize: the model's accuracy or its execution speed and resource consumption. Another common challenge in , particularly those with a stronger focus on science and mathematics rather than engineering, is optimizing the model for high performance in a production environment. Teams with a predominantly scientific and mathematical orientation might excel in creating robust models but may lack the skills to enhance their operational efficiency. Therefore, incorporating diverse profiles into the team is vital, as this ensures that we have both engineering and scientific backgrounds. ML teams Additionally, managing the required accuracy constraints of the models is a critical aspect of the process. Tasks often necessitate careful balancing of precision and recall. For instance, in scenarios where false positives have significant implications, prioritizing precision might be more crucial. Conversely, in cases where missing true positives is costly, focusing on recall becomes essential. Tailoring the model to effectively manage these trade-offs according to the specific requirements of the task is fundamental to the successful deployment of machine learning models. Can you share an example of a time when you had to make tough decisions regarding resource allocation within your ML team? with a fast-growing user base, our team ran out of resources to store personalized recommendations, which were generated offline. Problem: As an immediate measure, I ceased generating personalized recommendations for users with insufficient history feedback. Recognizing that the quality of these recommendations would not be optimal due to the lack of data, I’ve replaced them with non-personalized recommendations. This approach leaned more towards popular recommendations, utilizing a different ML model to manage the immediate storage issue. Short-Term Adaptation: To sustainably address the problem, my team developed an online recommendation system. Unlike the previous method, this new system did not require storing recommendations for every user. Instead, it generated recommendations dynamically, in real-time, when a client requested them. This innovative solution effectively resolved the storage constraints and adapted the recommendation process, and made it more scalable and responsive to user demands. Long-Term Solution: How do you ensure that the ML team's work aligns with the business goals and objectives of the organization? Ensuring that the ML team's work aligns with the organization's business goals and objectives involves a strategic and proactive approach: I have established a process for consistently sharing the relevant business context of our project with the team. This involves frequent updates and discussions about the company's broader goals, market trends, and customer needs, ensuring that the team's work remains aligned with the overarching business objectives. Regular Business Context Sharing: In collaboration with the team, we set quarterly Objectives and Key Results (OKRs) that are directly derived from and support the company’s own OKRs. This ensures that our projects and initiatives are contributing to the company's strategic objectives. Setting Quarterly OKRs in Alignment with Company Goals: To monitor our progress towards these OKRs, we conduct bi-weekly check-ins. These sessions allow us to assess our current standing, identify any challenges early, and make necessary adjustments to stay on course for achieving our objectives. Bi-Weekly Progress Tracking: I proactively engage with stakeholders, providing them with monthly progress updates. This not only keeps them informed about the ML team's contributions and advancements but also fosters transparency and trust. It allows stakeholders to provide feedback and ensures that our efforts are in sync with the company's evolving priorities and needs. Monthly Stakeholder Updates: How do you measure and track the impact of ML projects? Prior to the actual development of the ML model, I conducted a thorough data analysis to estimate the potential impact of the project. This step involves evaluating historical data and predictive analytics to forecast the expected outcomes and benefits of the ML model. Pre-Development Impact Estimation: I implement A/B testing to empirically assess the effectiveness of the ML model and its impact on business metrics. Before initiating the experiment, I identify the key business metrics that are most likely to be influenced by the model. These metrics are carefully selected based on their relevance to the project goals and are analyzed to gauge the model's performance. A/B Testing: Post-deployment, I actively monitor a set of dashboards that track various company and proxy ML-model metrics. This includes, but is not limited to, metrics like the number of purchases facilitated by the recommendation system or distribution of travel time model predictions. Regular monitoring of these dashboards, typically on a weekly basis, allows for real-time assessment of the model's impact on business operations. Dashboard Monitoring: To further understand the value added by the ML models, I conduct quarterly degradation tests. These tests involve temporarily disabling the ML models in specific parts of the business flow or across the entire application for a randomly selected, small user group. This helps in quantifying the by observing changes in business metrics in its absence. However, it’s very important to align with all the departments involved, so everyone is aware and actively agrees on performing such a test. Quarterly Degradation Testing: impact of the ML systems What steps do you take to foster a collaborative and inclusive team environment where all members feel motivated and valued? I emphasize the importance of every team member's perspective and actively encourage open communication. It's crucial not only to state that all opinions are valued but to demonstrate this in practice by giving equal attention and consideration to everyone's ideas and suggestions. Valuing Every Team Member's Opinion: I prioritize building and maintaining a diverse team. This includes diversity in skills, backgrounds, experiences, and perspectives. A diverse team is more likely to generate innovative ideas and solutions, and it also fosters a richer, more inclusive work culture. Maintaining Team Diversity: During team meetings, I ensure that everyone has the opportunity to speak and contribute. I actively facilitate discussions in a way that every member feels comfortable sharing their thoughts. This might involve directly inviting quieter team members to share their views or setting up meeting structures that ensure balanced participation. Inclusive Meeting Management: Organizing team-building activities and informal networking opportunities helps in building rapport and understanding among team members. Encouraging Team Building and Networking: I support the professional development of each team member through opportunities like personal development plans, workshops, and conferences. Providing Opportunities for Professional Growth: Regularly acknowledging and celebrating the achievements and contributions of team members. Recognizing and Celebrating Contributions: How do you encourage continuous learning and professional development within your ML team? Are there any specific initiatives or programs you have implemented? We have established a reading club where team members take turns to present and discuss a machine learning science paper relevant to our work. This not only keeps the team updated on the latest research but also encourages critical thinking and knowledge sharing. Reading Club: Every two weeks, we dedicate a day to R&D, where each team member explores a research project or tests new hypotheses. This initiative is designed to foster innovation and practical application of new ideas that could benefit our daily work. Biweekly Research and Development Days: To stimulate creativity and team bonding, we organize quarterly ML hackathons. These events span a few days and are focused on working on exciting, out-of-the-box projects. They provide an opportunity for the team to experiment with new technologies, collaborate in different team configurations than usual, and offer a fun and stress-free environment to innovate. Quarterly ML Hackathons: How do you handle conflicts or disagreements within the ML team? Can you provide an example of a conflict that you resolved successfully? Initially, I conducted one-on-one meetings with the involved parties. This allows me to understand each person's perspective and the core issues at hand. It's important to listen actively and empathetically during these discussions. Following the individual meetings, I arranged a joint meeting with both parties. Here, I share my observations, facilitate a constructive discussion, and encourage both individuals to express their views. This step often helps in clearing misunderstandings or mismatched expectations. Most conflicts are resolved at this stage. However, if the disagreement persists, I involve HR to assist in finding a resolution, ensuring that all parties feel heard and that a fair solution is reached. For technical disagreements, I advocate for a to conflict resolution. I encourage team members to support their arguments with measurable metrics or empirical evidence. For example, a conflict arose over which algorithm would perform faster for a particular application. To resolve this, I asked both parties to provide benchmarks and empirical data supporting their claims. This approach not only resolved the conflict but also fostered a culture of evidence-based decision-making. data-driven approach How do you ensure that your team is also up-to-date with the latest technologies and techniques? I arrange for experts from other companies within the industry to visit and share their experiences with our team. This exposure to external expertise provides valuable insights into new practices, tools, and methodologies being used in the industry. It's an opportunity for the team to learn from the successes and challenges faced by others in similar fields. I actively encourage team members to both attend and present at relevant conferences. This serves a dual purpose. Attending conferences keeps them informed about the latest trends and breakthroughs in machine learning while presenting their own work fosters a deeper understanding of their areas of expertise and enhances their professional profiles. This involvement in the professional community not only benefits individual team members but also brings fresh perspectives and ideas back to our team.

Interview with Petr Gusev: How To Manage an ML Team the Right Way

Initial We will have a charity event that will last for 3 days(let’s say from Friday to Sunday). We want to collect $100M and there will be 10 charities so each user of our system may decide where he/she wants to donate the money. We may assume that the front-end part is ready and also we have an agreement with some 3d party payment system so there is no need to handle money transfers from one account to another. Also, we need to collect all money in a single bank account and after the event will be finished we need somehow to spread the collected money among all 10 charities. The last thing to mention is that we need to handle any failure we may face while processing the payment. Functional Requirements Process transactions using 3d party service (payment service providers like Stripe, Braintree, etc) Keep track of which donations belong to a particular charity (to be able to spread all donated money among charities) Non-Functional requirements Since we are going to work with payments our system should be , we need to handle stale data. highly consistent Also since will use some 3d party service that will handle our transactions we need to provide some reconciliation process (a process that will ensure that our stored data is consistent). Our system should be and reliable fault-tolerance. Usually, a plan for a system design interview looks like this: Functional requirements Non-functional requirements APIs Data structures High-level design Design deep dive Wrap up But in this article, we will focus on the high-level design to discuss probable issues with it and follow-up questions we might face during the interview. High-level design The user submits the form on the UI(form contains credit card information, charity ID, and amount of money the user wants to donate to this charity The web server calls PSP to proceed payment PSP responded with an acknowledgment that payment proceeded successfully The web server saves data to the DB Issues with that design: the user clicked on submit button twice, what will happen in that case? Does the system proceed with 2 different payments? working with sensitive data, how to handle credit card information? how to make the system reliable and fault-tolerance? Click on “submit” button multiple times We can handle this issue in several ways: we may have some javascript restrictions, but it won’t help if the user disables javascript in the browser. This is also not so reliable since users may try to change your javascript code(rather than you providing some obfuscation to your code to make this action almost impossible) provide idempotent API with two-step processing. Let’s go ahead with the second approach. First, we provide a page with a dropdown of 10 charities, the amount user wants to donate, and the button “next”: After that, we will generate UUID from PSP and store it in our DB, after data was saved we redirect the user to the payment page: The whole flow will look like this: UUID was used to make our request . In other words, if we will call our PSP with a method that proceeds transactions several times for a single UUID, the result will remain the same. idempotent Hosting Credit Card pages provided by PSP Storing credit card information is not an easy task since we need to follow complex regulations like Payment Card Industry Data Security Standard(PCI DSS) in the United States. Most companies avoid storing credit card information and instead host payment pages provided by PSP. Here is an example of Stripe checkout: https://stripe.com/payments/checkout?embedable=true PSP systems like Stripe got internal mechanisms of retry in case the first attempt to transfer money failed for some reason. Keeping that in mind and the fact that we will not store any sensitive data in our storage we can say that , what we can do is check the result of payment and depending on the result make a record about this transaction consistent. For that purpose, we need a reconciliation mechanism. if for some reason the payment will fail we cannot automatically retry to do it once again Reconciliation Here is the whole flow: In step 4 our record in the transactions table will look like this: id(pk) charityID(fk) amount status updated uuid123 1 500(String) in progress timestamp We will have a status field that indicates the current status of the transaction. Notice that the amount field got type String, not double. The reason for this is that double is usually not a good choice because either software, protocols, etc. may support different numeric precisions in serialization or the number could be extremely big or extremely small. On step 8 PSP calls our webhook and provides an acknowledgment that the transaction was completed successfully. After that, we may change the status to . “completed” id(pk) charityID(fk) amount status updated uuid123 1 500(String) completed timestamp If something went wrong between , then we just not saving any data to the DB and our DB remains consistent. steps 1-4 If something went wrong between , then we may provide a reconciliation process that will periodically read the transactions table and check all transactions with status in progress and timestamp older than X and push such records to a dedicated queue so the finance team may handle it manually. Or if PSP provides some API to check transactions by ID we can make it automated. steps 4-8 The last thing worth mentioning here is a Great approach working with accounting, the idea behind this is that you got 2 records for each transaction, with debit and credit respectively. But in our design, this might be overkill. double-entry bookkeeping system.

System Design Interview - Designing Payment Systems; Follow-up Questions and Probable Issues

In , we reported on the popularity of Reddit’s Avatar marketplace, which launched in July. It seems others were paying attention too. Reddit Avatar trading volume has more than doubled, with over $3M in sales volume since last week. The market cap of the collection is now at around $100M. Read more . 🟠 Reddit NFT's total sales volume has more than doubled over the past week. last week’s newsletter here The rumor was ignited by a by Jane Wong, who is well-known for reporting on to-be-launch tech features. 🐦 Twitter is rumored to be working on a crypto wallet prototype. tweet Apple’s new App Store rules, which were introduced last week, impose restrictions on NFTs & crypto exchanges. The guidelines, which limit NFT utility, are aimed to protect Apple’s revenue models & ensure exchanges are properly regulated. Read more . 🍏 New App Store guidelines tighten web3 restrictions. here The international money transfer company has filed trademark applications relating to virtual currency exchange and transfer, commodity & crypto trading & brokerage, issuing tokens, and more. Read more . 💸 Western Union files crypto-related trademark applications. here DC3 involves dropping NFT comics on a weekly basis. The first collectible is Superman #1, which sold for $9.99. DC3 is designed to merge the real-world collectible comic book experience with the digital & expand DC’s digital community. Read more . 🦹🏽 DC announces DC Collectible Comics (DC3). here Loyalty 3.0 enables restaurants to easily level up & customize their loyalty programs, without damaging margins. The launch is designed to help restaurants more readily compete with loyalty leaders like Starbucks & Chipotle. Read more . 🥗 Thanx launches Loyalty 3.0 platform for restaurants. here Nissan plans to release up to 15 NFTs in collaboration with Grease Monkey Games, an Animoca Brands subsidiary. The car NFTs will have utility in the upcoming game, Torque Drift 2. Read more . 🚘 Nissan to release NFTs alongside a new game. here UmbroID is an upcoming NFT collection influenced by football culture & Umbro’s influence in the sport, ahead of the upcoming World Cup. NFT holders will receive rewards, such as physical and digital giveaways. Read more . ⚽️ Umbro launches smart digital collectibles. here Fans can enter ShimmerVille and kit out their avatars with Claire’s jewelry, also available in real-life. Claire’s is also partnering with similar brand partners through limited-edition merch and digital experiences. Read more . 💍 Claire’s launches Halloween-themed Roblox experience. here The brand has formed an exclusive partnership with metaverse specialist, Infinite Reality. The partnership will enable Warner Bros to create immersive experiences for sports fans powered by Web 3. Read more . 🤾🏾‍♂️ Warner Bros Discovery Sports forms metaverse partnership. here The launch of Gucci Vault Land makes Gucci the first major fashion label with a space dedicated to web3 products in The Sandbox. Gucci purchased the plot in February. The hub will feature Gucci NFTs, real-world fashion pieces, and more. The experience will run until November 9th. Read more . 🔴 Gucci opens Gucci Vault Land in the Sandbox metaverse. here The Voice in the metaverse will give fans the opportunity to engage with the show in a whole new way. The experience will include social games, events, unique content, a web3 loyalty program, and the development of a metaverse-based competition. Read more . 📣 ITV Studios’ The Voice enters the metaverse. here Google’s Blockchain Node Engine offers fully-managed node hosting for Ethereum applications. Running nodes is a data and resource-intensive process. The new centralized service moves the onus of running nodes to Google, allowing builders to focus their efforts on building applications. Read more . 🟡 Google launches Ethereum node service. here Nokia was one of the first companies to manufacture a consumer-grade mobile phone. The metaverse, powered by virtual reality and augmented reality glasses, will allow users to interact with new experiences. 📶 Nokia believes the metaverse will replace smartphones. Chief Strategy and Technology Officer Nishant Batra confidently stated that Nokia believes mobile devices will be overtaken by a metaverse experience in the decade's second half. Read more here . TFKT is the Nike-backed NFT studio behind the CloneX avatars. They dropped a new collection on October 27th with LVMH’s luxury luggage manufacturer Rimowa. 🧳 RTFKT has seen successful trading volume since its launch. The collection has seen 1,123 ETH ($1.7M) of volume on OpenSea alone. The 888 NFTs can be redeemed for cabin luggage and 2,222 “robot collectibles” were sold through a virtual raffle. Read more . here ELSEWHERE IN THE METAVERSE… The findings are a result of a survey by the financial services giant and signal significant long-term demand for crypto assets. . 🟢 Almost half of the young people want crypto in 401(k)s. More announced the release of METAVERTU. . 🟢 Vertu Launches first ever web3 phone. Last week, the phone, which has a starting price of $3300, More . A Fidelity survey shows that institutional appetite for digital assets has not been deterred by the market downturn. . 🟢 74% of institutions plan to purchase crypto More . NFT Marketplace LooksRare has followed in others’ footsteps and made NFT royalties optional. LooksRare instead is giving creators 0.5% of the 2% that previously went to LOOKS stakers. . 🟢 LooksRare follows suit and drops mandatory NFT royalties More

Weekly Web3 Tracker: Twitter Crypto Wallet, DC Comics Announces NFTs, and More

Weekly Web3 Tracker: Twitter Crypto Wallet, DC Comics anuncia NFT y más

Suivi hebdomadaire Web3 : Twitter Crypto Wallet, DC Comics annonce des NFT, et plus encore

साप्ताहिक वेब3 ट्रैकर: ट्विटर क्रिप्टो वॉलेट, डीसी कॉमिक्स ने एनएफटी की घोषणा की, और बहुत कुछ

ウィークリー Web3 トラッカー: Twitter の暗号ウォレット、DC コミックスによる NFT の発表など

Rastreador Web3 semanal: Twitter Crypto Wallet, DC Comics anuncia NFTs e muito mais

Trình theo dõi Web3 hàng tuần: Ví tiền điện tử Twitter, DC Comics thông báo NFT, v.v.

每周 Web3 追踪器：Twitter 加密钱包、DC Comics 宣布 NFT 等

This is Part 1 of a longer post. Hopefully worth the read whether you are completely new to the decentralized finance (DeFi) concept, or already have a decent understanding of how the sector works. In Part 1, I will sell you the idea of DeFi by pointing to some of the conventionally mentioned advantages DeFi holds over traditional finance (TradFi). Then I will add in my philosophical inputs. Towards the end, I will offer my take on a Crypto Investing 101 course to prepare you for an in-depth view of how DeFi is used to generate income in Part 2. Motivation My motivation for writing this and giving out, what I believe to be, an insane amount of value for free, is tied to my own interest in gaining more knowledge about DeFi. I am by no means an expert, but have been studying DeFi intensely over the last couple of weeks and months and found that trustworthy, accurate, and adequate information is hard to identify. I sincerely believe there should be more university courses dedicated to the topic of DeFi across academic disciplines such as law, economics, business, computer science, IT engineering, mathematics, philosophy, social science, and more. At this point, information about DeFi too often falls under the domain of “internet hustle culture” where conmen and fake gurus, stand on every corner to sell you a dream while they take your money. Such was the case with , where it came forth during the ongoing court proceedings that the CEO had recklessly played monopoly with user funds and left a billion-dollar hole in their balance sheet. We are reminded why Bitcoin was invented in the first place: to avoid dealing with shady custodians and profit-seeking third parties. This is what DeFi is all about, and hopefully, the direction it's heading long-term as the market matures. Celsius Advantages of DeFi Decentralized finance (DeFi) can eloquently be defined as this : [1] “An ecosystem of protocols, applications, investors and traders that use smart contracts to perform financial transactions in an open, permissionless and transparent way.” The TL; DR description of this section could be that DeFi is much faster, cheaper, and more accessible compared to corresponding services in TradFi. I will not dig deep into the fundamentals of DeFi here, but merely highlight a few key points to make sure we are on the same page: -      The main quality of DeFi is, as the name suggests,” ”. There is no intermediary company standing between transactions to collect user data or to profit from transaction fees. decentralization -      DeFi uses cryptocurrencies instead of state-issued fiat money. Cryptocurrencies are code with a set of rules embedded within them or for a fancier term they are “ The code behind serves a similar function to legal documents in a limited holding company. programmable money”. smart contracts -      Smart contract code is and auditable while transactions on DeFi applications are public. In sharp contrast to traditional financial institutions which usually operate behind closed doors. transparent -      Another buzzword to describe DeFi is “ ”. Due to the open-source nature of blockchains, DeFi applications can be built and used over, under, sideways or parallel to each other “like Lego bricks”. This makes DeFi more flexible and interoperable compared to TradFi or FinTech solutions. composability -      Additionally, fully decentralized DeFi projects (DAOs) are based on an which means that any stakeholder can participate in important decisions about the future of a project. Anyone with a stake in the project can initiate or vote on protocol changes via governance tokens, thereby participating in decisions that would usually be carried out by the board of directors in a traditional company. open governance model What are the disadvantages of investing in DeFi then? As the DeFi asset management platform writes in bold on their website: Yearn.finance “These are not risks for ants”. Whatever that means exactly, the connotation is clear: Investing in DeFi can lead to massive profits but it’s also a high-risk game. We will get much more into that in Part 2. Advantages of the DeFi Sector If we look at DeFi on a sector level, what is the major advantage it holds over TradFi? Most importantly; anyone with a stable internet connection, a PC or smartphone, and a crypto wallet can access it. In TradFi, asset management companies like Blackrock and Vanguard are only available for the wealthiest percentage of the global population. In DeFi, anyone can buy a few tokens and act as their own asset manager, regardless of their situation, where they are from, or what their personal history is. According to , the major advantage of DeFi is that older people don’t understand it. Vinay Gupta in a Twitter thread from early 2022 The boomer generation (1946-1964) sits heavily on real estate, stocks, venture capital, media distribution, retail, and patents. Meanwhile, the younger generations are generally left with scraps in exchange for hard labor. Due to the effect of compounding interest, the younger generations have a hard time catching up since they started out with less money. They lack the capital to take life-changing loans or buy into the older generations' resource pool. The numbers speak for themselves. Gupta points to the to illustrate the wealth gap between generations. following chart from Washington Post analyzed income trends in nine European countries between 2008-2017. The results show that income of the younger age groups stagnated or declined in most countries since 2008 (around the global financial collapse), while income of the older population increased. A study from November 2021 of income trends in the Netherlands shows that the boomer generation stands out both in terms of income and wealth. More than 2/3 of total net wealth was held by households with a main-income earner older than 55 years, and 42% was held by households over the age of 65 - excluding pension assets. The current generation of 35–45-year-olds in the Netherlands owns on average EUR 100,000 less than the current generation of 45–55-year-olds when they were the same age. In the UK, shows that Millennials (1981-1996) are the first generation in history that is poorer than the previous generation. A pre-pandemic study a report from 2019 In the United States, finds that the average 64-75-year-old American is 94% wealthier than the average 35-year-old, while today’s American 40-year-olds own half the wealth of older generations when they were the same age. Millennials only possess 5% of the US total wealth. And here is another interesting fact: a study from 2021 [2] “If we look closer at the wealth distribution of millennials, it was recently revealed that Mark Zuckerberg, who has an estimated net worth of $97 billion, single-handedly owns 2% of all millennial wealth.” Mark Zuckerberg and other tech billionaires have made their fortunes from creating addictive web platforms which users pay for with personal data instead of money. The daily operation of these platforms is carried out automatically by computer code, instead of humans as we are used to. Those are the facts. Whether you think that Web 2.0 services like Facebook or Google are good for the world or not, I think it’s safe to say that they have outsmarted the system. The blockchain movement represents a rebellion against all powerful, centralized institutions - whether banks or Big Tech monoliths - by emphasizing values that are more in tune with the digital age. The advantages DeFi holds over TradFi are the same as those Web 3.0 holds over Web 2.0: decentralized/centralized, open source/proprietary, permissionless/ limited access, transparent/opaque, publicly owned/privately owned. DeFi has the potential to disrupt wealth distribution by creating an entirely new group of young, tech-savvy millionaires. Like Bitcoin did with long-term holders in 2013, 2017, and again in 2021. The only barrier to entry is the technical complexity of DeFi and the fast-moving market – both in terms of price and innovation. Let’s jump into DeFi with a Crypto Investing 101 course. I am still undergoing a steep learning curve myself, so any constructive feedback or comments are welcomed. Trading Crypto Overarchingly, there are two types of exchanges for trading crypto: Centralized exchanges (CEXes (that’s what they are called)) such as Binance and Coinbase, and decentralized exchanges (DEXes) such as Uniswap and SushiSwap. CEXes act as intermediaries to transactions like regular banks. On the plus side, they are easy and convenient to use - at least once you are fully onboarded and have gone through the rather tedious KYC processes. CEXes oversight of customer data is "against the spirit of crypto", but also necessary to comply with financial regulations. The biggest minus with using a CEX is that you essentially delegate the control of your crypto to the exchange. illustrates perfectly how bad it can turn out. The recent Celsius freeze DEXes run on smart contract code and the exchanges are therefore not governed by an intermediary company that takes a cut for deposits or transactions. As a result, DEXes are generally cheaper to use compared to CEXes. The flip side is that you cannot just deposit your money on a DEX like you would on a bank account or a CEX. They may require a bit of studying and technical flair to use. It's important to note that DEXes are unregulated or at least a deeply gray area of the law. In the true spirit of Bitcoin, a DEX offers full privacy to customers without questionnaires or registering processes. On the other hand, the biggest risk for a user is a hacking attack on the platform which could put all customer holdings at risk. In such a worst-case scenario, there would most likely not be a legal remedy or any way to recover the funds since there is no one to hold accountable on behalf of the DEX (in theory). Storing Crypto There are two main options when it comes to storing your crypto: hot wallets or cold storage Hot wallets are digital wallets. They can either be custodial or non-custodial depending on whether you have ownership of your . Your private key is a string of numbers and letters, a kind of password that you use to sign off on transactions from your wallet. If you hold your crypto on a CEX such as Coinbase or Binance, your wallet is custodial, since you hand over the custody of your private keys to the exchange. Metamask is one example of a non-custodial wallet where you can access your private keys. You will typically need a non-custodial wallet to communicate with DeFi applications. private key Cold storage means that you store your public and private keys offline. Your wallet is safe from cyberattacks, while you remain in full control as Bitcoin intended you to be. The most commonly used methods for cold storage are paper wallets or hardware wallets. A paper wallet is simply a piece of paper that contains your private key and public address. They are usually generated for free via a website. If they are stored securely for example in a bank box, hidden away from thieves and wear and tear, they should be completely safe as such. However, as they are typically generated on a third-party website, you have to trust the web operator. Relying on trust in the crypto space is never a good thing. And if your computer is infected with malware during the creation process, your private key could be exposed. For these reasons, calls paper wallets an obsolete technology and warns against using them blockchain expert Andreas Antonopoulos Hardware wallets are not actually wallets, but like paper wallets, a method of storing your private keys offline. They are considered to be the safest choice, since they are not connected to the internet unless you make a transaction, thus minimizing the risk of attacks. The most popular hardware wallets are made by Ledger and Trezor. They are relatively easy to set up. for setting up a Ledger Nano X wallet. Here is an excellent guide On the downside, you have very limited options to earn passive income from your holdings, once your is crypto stored on the hardware wallet. However, hardware wallets can be integrated with non-custodial hot wallets. For example, Ledger can be integrated and . with MetaMask WalletConnect By remaining in control of your crypto, you are also the guardian of it, which means that if you lose access to your private keys or give them out to another party by mistake, you are in deep trouble. One measure to prevent you from losing access to your crypto in such a situation is by having a . multi-signature wallet Basic Investment Strategies I see three basic investment strategies for “traditional” crypto investing: short-term holding, long-term holding, and dollar cost averaging. Short-term holding is essentially the same as day trading. Investors try to “catch the waves” of market movements by buying and selling assets on a daily or very frequent basis. Timing the crypto market is extremely difficult. Long-term holding (HODL’ing) is about buying and holding assets over a longer time frame, at least one year. Particularly in crypto, investors have to stomach abrupt price swings from time to time. They have to resist the temptation to sell at a loss even when the market . drops by 50% or more in a matter of days Dollar-cost-averaging (DCA) is similar to putting money into a weekly or monthly savings account. At regular intervals, you invest a predetermined amount of money (or crypto). DCA has been highlighted as a good way to mitigate market volatility. But still, when the market price is either very high or very low, the desire to sell can overcome the confidence to keep investing for most people. Before the emergence of DeFi, the buy-hold-sell model was the main option to earn income from crypto. But over the last few years, really since 2018, and taking off during “the DeFi summer” of 2020, the space has been rapidly evolving. Today the DeFi space offers a wide and fast-growing range of revenue streams for crypto-HODLers that mimics how the rich get richer in traditional finance but with the added advantages of DeFi. DEFIYIELD App (2021), The Wall Street Era is Over. The Investor's Guide to Cryptocurrency and DeFi, pg. 17. [1] Also Published Here

DeFi's Opportunities and Dangers

Introduction Reference counting (RC) has rather been a minor party to the other garbage collection (GC) algorithms in functional programming in the last decades as, for example, and use non-RC GC. However, several recent papers, such as and , showed the efficiency of highly optimized RC GC in functional languages with sacrifice or restriction of some language features like circular references. The latter paper introduced an efficient RC GC algorithm called Perceus which is basically RC. OCaml Haskell Counting Immutable Beans: Reference Counting Optimized for Purely Functional Programming Perceus: Garbage Free Reference Counting with Reuse all-in-one In this post, I describe my experience and some caveats about implementing and gaining benefits from the Perceus RC. I've been developing and implemented a large part of the Perceus RC there. I hope this post helps someone who is implementing the algorithm or even deciding if it's worth implementing it in their own languages. a programming language called Pen Overview of Perceus The Perceus reference counting algorithm is a thread-safe ownership-based reference counting algorithm with several optimizations: Heap reuse on data structure construction and deconstruction (pattern matching) Heap reuse specialization (in-place updates of data structures) Non-atomic operations or atomic operations with relaxed memory ordering for heap blocks not shared by multiple threads Borrow inference to reduce reference count operations By implementing all of those optimizations in , they achieved GC overhead much less and execution time faster than the other languages including OCaml, Haskell, and even C++ in several algorithms and data structures that frequently keep common sub-structures of them, such as red-black trees. For more information, see . the Koka programming language the latest version of the paper Implementing the algorithm What I've implemented so far in Pen are two core functionalities of the Perceus algorithm: In-place updates of records on heap This corresponds to heap reuse specialization described above. Relaxed atomic operations on references not shared by multiple threads Due to some differences in language features between Koka and Pen, I needed to make some modifications to the algorithm. First, Pen doesn't need any complex algorithm for in-place record updates with heap reuse specialization because it has , and its lowered directly into its mid-level intermediate representation (MIR) where the RC algorithm is applied. syntax for record updates Secondly, although I've also implemented generic reuse of heap blocks that matches their frees and allocations in functions initially, I've reverted it for now since I realized that it won't improve performance much in Pen because of the lack of pattern matching syntax with deconstruction and another optimization of small record unboxing planned to be implemented later. In addition, the implementation doesn't include borrow inference yet as it had the least contribution to the performance reported in a previous paper. The main part of the algorithm is implemented in the source files of a compiler itself and an FFI library in Rust listed below: https://github.com/pen-lang/pen/blob/d44df6d9cdcbe97fcdd5ac14c4de30f4897664ff/lib/mir-fmm/src/reference_count/pointer.rs https://github.com/pen-lang/pen/blob/d44df6d9cdcbe97fcdd5ac14c4de30f4897664ff/lib/ffi/src/arc/arc_block.rs Counting back synchronized references to 0 In this section, I use the term "synchronized" to mean "marked as shared by multiple threads." In Koka and Lean 4, they use the term "shared" to mean the same thing but I rephrased it to reduce confusion. In the Perceus reference counting GC, memory blocks have mainly two and states represented by positive and negative counts respectively. Heap blocks are before they get shared with other threads and are never reverted back to states once they get synchronized. But you may wonder if this is necessary or not. If we have a memory block with a reference count of 1, that also means it's not shared with any other threads anymore. So isn't it possible to use a common count value of 0 to represent unique references and reduce the overhead of some atomic operations potentially? un-synchronized synchronized synchronized un-synchronized The answer is no because in that case, we need to synchronize memory operations on those references back with drops of those references by the other threads with release memory ordering. For example, let's consider a situation where a thread shares a reference with the other thread: un-synchronized Thread A shares a reference with thread B. Some computation goes on... Thread B drops the reference. Thread A drops the reference frees its inner memory block. and Or, thread A reuses the memory block for heap reuse optimization mentioned in the earlier section. So if references can be back, we always need to use atomic operations with acquire memory ordering at the point (4) above to make all side effects performed by thread B at the point (3) visible for thread A. Otherwise, thread A might free or rewrite memory locations thread B is trying to read! So as a result, we are rather increasing the overhead of atomic operations for references never before. un-synchronized synchronized Benefitting from the algorithm In general, to get the most out of heap reuse in the Perceus algorithm, we need to write codes so that data structures filled with old data are updated with a small portion of new data. Pen's compiler previously had a performance bug where a relatively old data structure was merged into a new one of the same type. As a result, the code to merge two pieces of data was taking almost double in time although the logic was semantically correct. Recursive data types When your language has record types and syntax for record field access, things might be a little complex. Let's think about the following pseudo code where we want to update a recursive data structure of type in place (The code is written in but assume that we implemented it with Perceus.): A Elm type alias A =
  { x : Maybe A
  , y : Int
  }

f : A -> A

-- From here, assume that we are in a function scope rather than a module scope.
foo : A
foo = { x = Nothing, y = 42 }

bar : A
bar =
  -- Here, we want to reuse a memory block of `foo`...
  { foo |
    x = case foo.x of
      Nothing -> Nothing
      -- There are two references to `x` on evaluation of `f x` here!
      Just x -> f x
  } At the line of , the program applies a function to a field value which originates from . However, at this point of the function application, we are still keeping the record value itself and the value of has two references! Therefore, heap reuse specialization (i.e. in-place record update) cannot be applied there. In order to update the value of in place instead, we need to rather deconstruct into its inner values first as follows. Just x -> f x f x foo foo x x foo bar =
  let { x, y } = foo
  in
    { x =
      case x of
        Nothing -> Nothing
        Just x -> f x
    , y = y
    } Note that, even if languages do not support record deconstruction, for self-recursive types, dropping fields containing the types themselves is possible in most cases in practice because otherwise such types' values cannot exist at runtime unless they are dynamically generated in functions or thunks in those fields. When I look at , it seems to support record types but I couldn't find out how it handles this specific case yet. It's also an option to expose the compiler's details and allow annotations to enforce in-place updates for end-users while it might not be the best option in a long term. the Koka's documentation Benchmarking Here, I'm excited to show some benchmark results and their improvements. Details of their configurations are in a section later. Note that Pen still lacks some basic optimizations to reduce heap allocations (e.g. lambda lifting, unboxing small values on heap.) So the eventual performance improvements by Perceus would be lower than those results. Since I've never implemented the other GC methods like mark-and-sweep for Pen before, this is not a comparison of RC GC vs non-RC GC but rather a proof of how performant traditional thread-safe RC can be adopting Perceus on functional programming languages. Atomic RC (seconds) Perceus RC (seconds) Improvement (times) Conway's game of life 2.136 1.142 1.87 Hash map insertion 0.909 0.255 3.57 Hash map update 1.935 0.449 4.31 Configuration Conway's game of life The implementation uses lists to represent a field and lives so that it causes many allocations and deallocations of memory blocks on heap. Field size: 20 x 40 Iterations: 100 Implementation Hash map insertion/update The map update benchmark includes the time taken by insertion for map initialization as well. A number of entries: 100,000 Key type: 64-bit floating point number Data structure: Hash-Array Mapped Trie (HAMT) Implementation Insertion Update Conclusion In my experience so far, implementing the Perceus algorithm appears to be fairly straightforward compared with the other non-RC GC algorithms while there are a few stumbling blocks especially if you are not familiar with low-level concurrency and atomic instructions. I'm pretty happy having the algorithm implemented in my language and seeing it performing well despite its simple implementation. The Perceus RC can be a game changer in functional programming as it outperforms traditional GC on several common patterns in functional programming. However, it's definitely not for everyone and most likely affects language design. Finally, thanks for reading! I would appreciate your feedback on this post and . The language's new release has been blocked by but the ticket had some progress in the last few weeks. So I can probably release v0.4 of it soon... the Pen programming language LLVM 14 adoption in Homebrew Also, special thanks to the developers of for answering my questions on the algorithm! That was really helpful! Koka Appendix Raw benchmark results Environment: > uname -a
Linux xenon 5.13.0-1033-gcp #40~20.04.1-Ubuntu SMP Tue Jun 14 00:44:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

> lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU @ 2.20GHz
Virtualization features:
  Hypervisor vendor:     KVM
Caches (sum of all):
  L1d:                   64 KiB (2 instances)
  L1i:                   64 KiB (2 instances)
  L2:                    512 KiB (2 instances)
  L3:                    55 MiB (1 instance)

> lsmem
Memory block size:       128M
Total online memory:       4G Conway's game of life > hyperfine -w 3 ./life-old ./life-new
Benchmark 1: ./life-old
  Time (mean ± σ):      2.136 s ±  0.033 s    [User: 1.699 s, System: 0.392 s]
  Range (min … max):    2.097 s …  2.206 s    10 runs

Benchmark 2: ./life-new
  Time (mean ± σ):      1.142 s ±  0.035 s    [User: 1.087 s, System: 0.009 s]
  Range (min … max):    1.102 s …  1.203 s    10 runs

Summary
  './life-new' ran
    1.87 ± 0.06 times faster than './life-old' Hash map insertion > hyperfine -w 3 ./insert-old ./insert-new
Benchmark 1: ./insert-old
  Time (mean ± σ):     909.0 ms ±  14.2 ms    [User: 605.7 ms, System: 250.6 ms]
  Range (min … max):   890.7 ms … 932.8 ms    10 runs

Benchmark 2: ./insert-new
  Time (mean ± σ):     254.8 ms ±   5.1 ms    [User: 189.1 ms, System: 15.0 ms]
  Range (min … max):   247.0 ms … 263.4 ms    12 runs

Summary
  './insert-new' ran
    3.57 ± 0.09 times faster than './insert-old' Hash map update > hyperfine -w 3 ./update-old ./update-new
Benchmark 1: ./update-old
  Time (mean ± σ):      1.935 s ±  0.032 s    [User: 1.405 s, System: 0.476 s]
  Range (min … max):    1.879 s …  1.976 s    10 runs

Benchmark 2: ./update-new
  Time (mean ± σ):     448.6 ms ±  14.4 ms    [User: 381.5 ms, System: 15.1 ms]
  Range (min … max):   430.9 ms … 484.3 ms    10 runs

Summary
  './update-new' ran
    4.31 ± 0.16 times faster than './update-old' Also Published Here

How to Implement the Perceus Reference Counting Garbage Collection

Say it with me: data engineers are not a data catalog. You would be hard-pressed to find “answering multiple Slack messages every week about which tables are good to use for this report,” in their job description, but it happens nonetheless. Data analysts aren’t psychic. Yet, they are often placed in the position of having to intuit if the data being piped is trustworthy. This misalignment has arisen as data teams are pushed to move faster, weave themselves across the , and enable increasingly self-service data platforms. data mesh It’s the data team’s equivalent of the classic document version control issues that have plagued knowledge workers for decades. What starts as a tight pitch deck evolves into: A million people making and sharing ad-hoc slides; Massaging content on those slides until it becomes an echo of its original intent; and Creating copies labeled V6_Final_RealFinal. The same thing happens across the data team. Everyone is trying to do the right thing (i.e., support your stakeholders, generate insights, pipe more data, etc.), but everyone is also moving fast. One day you look up and notice you have 6 different models with slight variations essentially doing the same thing…and no one knows which one is most up-to-date or even which field to use. This creates real operational problems downstream including: Inefficient cycles of redundant “traffic control;” Lower data quality; Time spent resolving problems created by analysts using improper/problematic data; Lower data trust across the organization; and Increased data downtime When you don’t trust your data or you have lower data reliability, organizations often pad the margins of error in their forecasts. As highlighted by Peleton’s , poor forecasting can be especially problematic during the pandemic when uncertainty across demand, supply chains, and the overall business environment is at an all-time high. recent production halt More data discovery, more problems Data discovery is a new approach to understanding the health of your distributed data assets in real-time, and it’s an essential part of the solution. Data discovery provides a domain-specific, dynamic understanding of your data based on how it’s being ingested, stored, aggregated, and used by a set of specific consumers. As with a data catalog, governance standards and tooling are federated across these domains (allowing for greater accessibility and interoperability), but unlike a data catalog, data discovery surfaces a real-time understanding of the data’s current state as opposed to its ideal or “cataloged” state. It is especially useful when teams take a distributed approach to that holds different data owners accountable for their data as products, which allows data-savvy users throughout the business to self-serve from those products. governance But as data becomes more accessible, how can downstream stakeholders determine what data sets have been served, transformed, and approved by a given domain’s data team? How can one domain be sure a common set of are being upheld across the organization? data quality standards, ownership, and communication processes One of my customers, a leading media company with a mature data organization, was facing these exact questions. As a result, we have been working with them and several others to implement a data certification program. What is data certification? is the process by which data assets are approved for use across the organization after having met mutually agreed-upon , or service-level agreements,  for data quality, observability, ownership/accountability, issue resolution, and communication. Data certification SLAs Similar to the concepts of data quality, data validation, or data verification, data certification layers on critical processes that align people, frameworks, and technology to central business policies. Data certification requirements vary based on the needs of the business, the capacity of the data engineering team, and the availability of data, but typically incorporate the following features: Data certification programs increase scalability by leveraging a consistent approach applied across multiple domains. They also increase efficiency by facilitating more trustworthy exchanges of information between domains with clear lines of communication. Here’s how it works. Step 1: Build out your data observability capabilities Implementing data observability–an organization’s ability to fully understand the health of the data in their system–is an important first step in the data certification process. Not only do you need insight into your current performance to set a baseline, but you also need a systemic end-to-end approach for proactive incident discovery, alerting, and triaging. If within the pipeline breaks–and it will break–you will be the first to know. This head start, along with a detailed understanding of the data ecosystem, will reduce time to detection and resolution by pinpointing where errors occur. anything Knowing what systems and data sets have a tendency to create the largest or most frequent problems downstream also helps inform the process of writing effective data SLAs (Step 4). Additionally, understanding the upstream dependencies of your most important tables or reports helps data teams understand what data to give the most attention. The bottom line is that a table or data set should be closely monitored for anomalies  (ideally continuously learning and evolving via machine learning) to be considered certified. Step 2: Determine your data owners Each certified data asset should have a responsible party across its lifecycle from the ingestion to the analytics layer. Some data teams may choose to implement a , others may build it directly into the specific SLA along with the expected communication procedures and resolution times. RACI (responsible, accountable, consulted, informed) matrix Step 3. Understand what “good” data looks like By asking your business stakeholders the “who, what, when, where, and why,” you can understand what data quality means to them and which data is actually the most important. This will enable you to develop key performance indicators such as: : Freshness Data will be refreshed by 7:00 am daily (great for cases where the CEO or other key executives are checking their dashboards at 7:30 am). Data will never be older than X hours. : Distribution Column X will never be null. Column Y will always be unique. Field X will always be equal to or greater than field Y. : Volume Table X will never decrease in size. : Schema No fields will be deleted from this table. : Lineage 100% of the data populating table X will have upstream sources and downstream ingestors mapped and include relevant metadata. (or availability) Data downtime : The number of incidents is multiplied by (the time to detection + time to resolution). An example of a data downtime SLA could be, table X will have less than Y hours of downtime a year. SLAs that measure each of the components of data downtime can be more actionable. Examples include: we will reduce our incidents X%, time to detection X%, and time to resolution X%. Query Speed: Our friends at : “Average query run time is a good place to start, but you may need to create a more nuanced metric (e.g., X% of queries finish in <Y seconds). Locally Optimistic suggest Ingestion (great for keeping external partners accountable): Data will be received by 5 am each morning from partner Y. This process also enables you to configure granular alerting rules tailored to what matters most to the business. Step 4: Set clear SLAs for your most important data sets Setting SLAs (service level agreements) for your data pipeline is a major step toward increasing your data reliability and is essential to a data certification program. SLAs need to be specific, measurable, and achievable. Not only do SLAs describe an agreed-upon standard of service, but they also define the relationship between parties. In other words, they outline who is responsible for what during normal operations as well as when issues occur. , a Senior Data Scientist with Red Ventures, suggests that an effective SLA is realistic. Simply saying “having reliable data at all times” is too vague to be useful; instead, Brandon suggests, that teams should set SLAs that are focused. Brandon Beidel “Good SLAs are specific and detailed. They will describe why it’s important to the business, what the expectations are when those expectations need to be met, how they will be met, where the data lives, and who is impacted by it.” Beidel includes within his SLAs how the team should respond if the SLA isn’t met. For example, “the data in table X will be refreshed every day by 7:00 am” will transform into, “Team Z will ensure the data in table X will be refreshed every day by 7:00 am. Within 2 hours of an anomaly alert, the team will verify, communicate to affected parties, and begin a root cause analysis of the issue. Within one business day, a ticket will be created and the wider team will be updated on the progress made toward resolution.” To achieve this level of specificity and organization, teams should align early – and often – with stakeholders to understand what good data looks like. That includes within the data team as well as the business. A good SLA needs to be informed by the realities of how the business operates and how your users consume the data. I take a slightly different approach and differentiate between what I consider the SLA of “table x will be updated by 7 am” and the SLO (Service Level Objective) of “we will aim to meet this SLA 99% of the time.” However you decide to approach it, I’d recommend against boiling the ocean. Most of my customers are implementing their data certification programs as “go forward” first and cleaning up older assets in a second wave. In fact, many of the best data teams will start certifying the most critical tables and data sets: the ones that add the most value to the business, have the most query activity, the number of users, or dependencies. Some are also implementing tiers of certification–bronze, silver, gold–that convey different levels of service and support. Step 5: Develop your communication and incident management processes Where and how will alerts be sent to the team? How will the next steps and progress be communicated internally and externally? While this may seem like table stakes, clear and transparent communication is essential to creating a culture of accountability. Many teams opt to have alerts and incident triage discussions take place in Slack, PagerDuty or Microsoft Teams. This enables rapid coordination while giving full transparency to the wider team as part of a health incident management workflow. It’s also important to consider how to communicate major outages to the rest of the organization. For example, if an alert turns out to be a huge production outage, how does the on-call engineer inform the rest of the company? Where do they make that announcement and how frequently do they provide updates? Step 6: Determine a mechanism to tag the data as certified At this point, you have created SLAs with measurable objectives, transparent ownership, clear communication processes, and strong issue resolution expectations. You have the tools and proactive measures in place to empower your teams to be successful. The final step is to certify and surface the approved data assets for your stakeholders. I recommend decentralizing the certification process. After all, the certification process is designed to help make teams faster and more scalable. Having centralized regulations, enacted at the domain level will achieve these goals and avoid creating too much red tape. For the certification process, data teams will tag, search and leverage their tables appropriately either using data discovery solutions, a home-grown tool, or some other form of data catalog. Step 7: Train your data team and downstream consumers Of course, just because tables are tagged as certified doesn’t guarantee analysts will stay inbound. The team will need to be trained in the proper procedures, which will need to be enforced as necessary. Fine-tuning the level of alerts and communication is important as well. Occasionally receiving alerts that don’t require action is healthy. For example, you may have a table that grows significantly in size, but it was expected because the team added a new data source. Nothing is broken and in need of fixing, but it’s still helpful for the team to know. After all, “expected” behavior to one person might still be newsworthy and critical to another member of the team – or even another domain. However, alert fatigue is real. If the team is starting to ignore alerts, it can be a sign to optimize your approach by either adjusting your monitors or bi-furcating communication channels to better surface the most important information. When it comes to your data consumers, don’t be shy! You have put in an incredibly robust system for data quality aligned to their needs. Help them move from a subjective to objective understanding of how your team is performing and start giving them the vocabulary to be part of the solution. It’s all about proactive communication Data certification can be a beautiful process to see in action. The data engineer tags the table as certified along with the owner of the data set and surfaces it within the data warehouse for an analyst to grab it and use in their dashboard. And voila! No more (or at least, a whole lot less) data downtime. At its core, this process underscores that without the proper processes and culture in place, certifying reliability and building organizational trust in your data is extremely difficult. Technology will never be a replacement for good data hygiene, but it certainly helps. Also published here.

Certify Your Data Assets to Avoid Treating Your Data Engineers Like Catalogs

Why do you want to invest in cryptocurrency? Many people answer straight away – I want to earn money. So, the question is why do you want to achieve this using crypto? To answer this question, we will go through the 1st pillar of the , which is . Cryptocurrency Investing Framework Portfolio Management We will go through the six steps starting with creating our vision. End-To-End Investment Process, Vision - What is the Goal? are about the results that we want to achieve. Goals What is your ? Defining it as being wealthy isn’t very precise. People understand it differently: Financial Goal •	I want to have X money when I retire. •	I would like to get X of monthly cash flow. •	Average returns from capital is higher than X. •	I don’t want to have a worse return than benchmark X. Where X is up to you. It could be any figure. I like to define financial goals as the (prevent capital loss) and (increase purchasing power) . For better understanding, we need to define wealth. ability to preserve compound wealth to assure financial security & freedom Wealth What is wealth? For me, it’s . freedom of being independent of limiting factor money, which is a vehicle to transfer wealth is wealth related to money, including promises to pay money and the capacity to sell things for money. Financial wealth is relevant to because it confers spending power. Financial Wealth inflation is wealth related to the real goods and services that people want, including the capacity to produce those goods and services. Real wealth is relevant to inflation because it's how spending power is fulfilled. Real Wealth To better understand the difference between the financial and real economy, I would recommend you check this video. In our section we mentioned that we should be able to preserve our wealth. Let’s then check what type of enemy do we have. Financial Goal , The Reason for Investing We have one main reason for investing – Inflation. The best way to understand it is to look at the below charts, how money is changing its value in time (100% is currency to the gold ratio in the first year): 1.	How are vs gold across 400 years? reserve currencies devaluating 2.	How did currencies are devaluating vs gold in the last century? Dalio, Ray. 2020. The changing world order where we are and where we’re going. The trend is clear, but if this does not convince you, check also how is related to some expenses across the time: CPI Even if we wouldn't have inflation (Broad CPI), we have to compound our wealth to maintain at the same level. purchasing power Portfolio Assessment -> Where am I Now? We know what we want to achieve now. It’s also worth understanding where we are standing right now. Let’s then check our current financial situation: •	What do we have? financial assets •	How our real and financial income is changing? In this article, we will focus on financial assets. The best way to do this assessment is to create an inventory of what you have. This inventory will be used later to manage our portfolio, so it’s worth spending some time on it. There are several tools available to do so, but what works best for me is Excel or Google Sheets, whatever you prefer. Therefore, I created an example spreadsheet for you, which you can adjust according to your needs. You can download it . HERE Portfolio Tool In the first tab, you have a description of all proposed columns to describe your portfolio. You can add more later if needed. Now, what should be considered as a financial asset? Let's use the below rules to evaluate what can we add there: •	Value – This should be valuable for others (we have for this asset) demand • – There should be a for it, which allows you to sell it. I am not adding my home, car, or kidney however they have value for others and probably I can find a market for it. If you are trading cars, let's add them. If you have many houses, add them. I like to think about it, that if someone will give me the value in cash for any listed item with little premium (it should also include potential taxation, but let’s keep it simple), I will sell it. Liquidity market The main to consider are: Assets Classes •	Cash •	Bonds •	Equities •	Commodities •	Precious metals •	Real Estates • P2P loans •	…And cryptocurrencies Finally, Later, we will consider when it makes sense to add cryptocurrency to your portfolio and when you should focus on other assets. we get into crypto! Once you are done with your portfolio assessment, you’ll see where you stand. Let’s then precisely define our goal. Portfolio Goals -> Where Do I Want to Be? To better define our financial goal, we will use this simple equation: Salary + Capital Changes = Assets and Liabilities – money contribution given in exchange for your time in the real economy (for most people is just a job) Salary (Real Economy) - your capital can be decreased by wrong investment decisions or specific market situations. It’s also decreasing over time due to inflation. Successful investment by the other way is increasing your capital Capital changes (Financial Economy) – Real and financial things that increase your capital over time (we just added them to portfolio) Assets - Things that do not generate your interest or are illiquid (due to your decision or nature) Liabilities So, in order to increase your wealth (assets and liabilities), you can increase your salary or income from wealth. Let’s dive deeper into this. To increase salary, you can get a promotion or change job, etc. The main here to consider is . This is not part of this article, but you can find more details on the website. KPI salary/h 80000 Hours To increase the rent from the capital, you need to have more assets, as liabilities don’t produce cash flow and you can’t sell them for a higher price in normal market conditions, or get a better return from them. We also need to consider an additional measure which is time. You can also automate what you are doing now (maybe just buy instead of spending time finding the best stocks) to decrease time, and ultimately increase . ETF financial income/h Investor’ Path Now the questions arise: 1.	Should you invest to increase your capital rent vs focusing on your job to increase your salary/h? 2.	How much should you invest in crypto? The first question is the hardest. We can compare and simulate changes in your salary or capital rent, but you should also consider less . Again, is the best source to start. Assuming that investment is for you, we can move to the second question. quantifiable factors 80000 Hours . Each of them has different parameters and this is the reason why they are distinguished within different classes. Crypto is just one of many assets classes Some of the asset’s parameters to consider (list is not exhausting and can be adjusted to current needs): 1. – How do changes in economic growth impact your returns? Economic Growth 2. – How do inflation and deflation impact your returns? Inflation 3. – What are assets’ issuance policy and transaction finality? Can it be manipulated by a third party? Trustlessness 4. – What is the liquidity rate of the asset? Liquidity 5. – Is the return tightly coupled with any region or area? Region By comparing them, you will find the best assets to contribute to your goal, or even better – a mix of the assets. We will dive deeper into this in the section from the 2nd pillar of the . Assets Allocation Cryptocurrency Investing Framework If the investor path is for you, let’s discuss our strategy. Portfolio Strategy -> How do I Get There? Now we know why we are doing this, where we are, and what do we want to achieve. Let’s move to create a . This is the 2nd pillar of Framework – , but let’s highlight it here. Strategy Research & Development . They differ in strategy to achieve them. To achieve your goals, you will need: Winners and losers have the same goal •	Time •	Money •	Skills Many of these parameters are interchangeable, depending on your choice and current situation. To create and implement your strategy, you will need to have some specific skills, depending on previously discussed factors. You can acquire this by spending your time or using money to accelerate learning. Probably you will also need some data, reports, tools, etc. . Many of your choices can be evaluated after a few years. This is the reason why it's hard to stick to your strategy or follow one of the and Lose Your Money (don’t look for this book – I’ve created this name). Investing is both simple and difficult at the same time Famous Investor Strategies To Invest Guess, what will happen if you invest your money in stocks and a few weeks later the economy collapses? You will have to wait many years when stock prices return to the same level. How this will impact your further plans? Of course, you don’t know what will you do and how this will affect you. Think more about the potential scenario. Can I delay buying my home by X years? What if you invest in cryptocurrency and we will have another bear market where a good project will lose up to 80%? Their values are going almost to zero. You can check the performance of old projects on . For many of them, the trend is clear. They are going to zero. CoinMarketCap In the 2nd pillar of the Framework, we will also be using to mitigate some (hopefully all) of the potential risk. Let’s consider now how to choose an . Risk Management Investment Strategy Different strategies We can start with defining two parameters, which will be used to evaluate our strategy: 1. - Determine needed from invested capital Return Objective average return 2. - As low as practical to achieve the objective. You can also specify and parameters (I am not a big fan of doing it, because the market is a ). Risk Appetite kurtosis skewness complex system We have a huge impact on these parameters, mainly due to . You can see this on the chart below, where we have returns from different asset classes for the last 200 years. Assets Allocation https://modelinvesting.com/content/uploads/2015/06/Total-Real-Return-by-Asset-Class.jpg You can find many similar charts on the web with different returns for those asset classes, depending on their definition and time period. You can also check it by yourself, using and playing with different parameters. Portfolio Visualizer Anyway, the conclusion is clear – Stocks (including ) give you the highest returns with the highest variability. This is in line with . REITs Effective Market Hypnotehis Bitcoin and other cryptocurrencies have a shorter track record, so we have to shorter our time period to add it. Let’s check the below chart comparing returns for the best-performing investments. The image is from here . Again, high volatility, high returns. Ask yourself - What is the highest variability that you can tolerate alongside minimal return that can help you achieve your goals? This will tell how aggressive you should be, what should be your starting capital, how much you can withdraw or should deposit to achieve your goals. Of course, we cannot predict the future, but this exercise will tell you how achievable your goal is. •	Is it possible to achieve goals only by investing in stocks? •	Can I tolerate the risk of putting 50% of my money into cryptocurrency? •	When I will achieve my goals assuming that I will invest 10% of my salary? Take your time and think about it because it will vastly impact your strategy. What I will recommend to you in the next articles is to diversify across many assets, especially combining stocks with crypto. Now assuming that we have our strategy, it’s time to stick to it and take all needed actions. Strategy Implementation - How to Do This? Implementation is the 3rd pillar of the . This part is simple if you like to teach someone, but hard to practice same as trying to stay fit. Almost everyone knows what to do, or at least what not to do. After all, execution is a much harder part. Cryptocurrency Investing Framework The most helpful thing here will be your experience and strategy (fortunately we have a dedicated pillar for it): 1.	The first piece of advice is to (we never know whether this works if, but having a good one significantly increases our chances). educate yourself to get confidence, so that your strategy can work 2.	The next advice is to lower the stake. . Therefore, start with the money that you can afford to lose, where you can also experiment to better understand your behavior. Most of us tend to understatement our Risk Tolerance can be helpful for this. You can put there all your financial decision with explanations – Why did I do this? What I was expecting? How did I feel? Investor Journals This will later be helpful to evaluate and track your progress. We will also have to learn a bunch of different skills, frameworks, and that will help you achieve your goals. It’s time to dive into psychology. Mental Models Investor Psychology As an investor, . . we don’t control returns, especially in short term, but we have an impact on risk and our behavior We are deciding on a balance between losing money and opportunism (gains) : To win in this game, we have to better predict the future than other investors and be aware of our Mental Errors 1. (too little information, incorrect information, wrong analytical process) Analytical/Intellectual 2. (greed and fear, ego and envy, willingness to suspend disbelief and skepticism, the drive to pursue high returns through risk-bearing, tendency to overrate one's foreknowledge, extrapolating past) Psychological/Emotional . This will allow us to answer below questions: The best way to minimize the impact of Mental Errors is to be aware of the fact that all of us are affected and we have frameworks to reduce our biases 1.	How to decide when buying something? 2.	Should we sell with a penalty or buy more, if our investment is going in the opposite direction? 3.	How to calm down ourselves upon watching a huge drop in our portfolio value? 4.	How to prevail against and not check our investment every day or even a few hours…? FOMO We will learn all of this later, but now let’s check how to evaluate our strategy. Strategy Evaluation - Did I Get There? How to check that you have a good strategy, or maybe it needs some updates? Or that you possess the required skills to achieve your goals? . We need an Evidence-Based Evaluation that tells us if we are progressing in the chosen direction We will evaluate our performance and what we learnt. This will be the input to our and . Investment Strategy Learning Path How can we define our end goals to be able to track them and later evaluate our strategy? We need to know there we are going in a good direction, so I propose you to create some towards our goal. milestones Let’s now move to the evaluation process. Evaluation Process The review process can be triggered by time or external events. At each point in time, we will have a specific amount of wealth. The difference between these time intervals gives us a return from the portfolio (assuming no deposit and withdrawals). This is a key metric that we will be tracking. Another is . We can define it in many ways, but it should tell how changes in our portfolio differ from this annual return. variability We should also have many different evaluation periods which can be adjusted to our needs and current situation. You can change this later, but I am proposing you to start with: •	Annually Changes related to taxation Strategy Evaluation Goal Evaluation •	Monthly/Quarterly Deposit/Withdrawal Portfolio Monitoring •	Weekly Portfolio Monitoring (Fast-changing assets like crypto) I am creating a new sheet for each review period in my , to have a snapshot of past data, to analyze it later. tool We will go deeper into each of those reviews processes in the 4th pillar of the . Cryptocurrency Investing Framework What Next? We just review topics related to portfolio management to be able to: 1.	Define our expectations regarding investment. 2.	Asses our financial situation. 3.	Define our portfolio goals. 4.	Understand why we need a strategy, how to use it, and evaluate. In the next article we will create our , so be sure to have your portfolio up to date. You can start with . Investment Strategy this template Stay tuned and follow me on to get a sneak peek of what is coming! Twitter

How to Manage your Cryptocurrency Portfolio

Introduction: What Does a Product Owner Do? A Product Owner is a widely recognized role in agile software development methodologies. In this article, I'm going to tell you who a Product Owner is and how to become a better one.  There will be practical tips and general advice. While the role is sometimes mistaken with the classic position, the Product Owner's job, in general, doesn't include as many responsibilities related to people management. team lead This article is a summary of what I've learned about a Product Owner role while working with one of my employers. It was written based on a series of interviews with other Product Owners, therefore it is an opinionated set of tips and suggestions. Contents of this article do not apply to every project at the but certainly can inspire you and educate you on the general responsibilities of a Product Owner in a software environment. services I'm going to use a PO acronym for Product Owner from now on. I've collected this set of thoughts in order to pass the experience of being a PO to new and existing work colleagues but I'm sure an online community can benefit from it as well. I've been in the role multiple times and got a lot of feedback. It's time to share. Disclaimer: Product Owner role described here is not a direct implementation of Scrum Product Owner. The tips are targeted but not exclusive to service-type work. I do not hold any Scrum certificate. Let's start with some definitions. Most of the publications agree on the following statement about the role. A (Scrum) Product Owner is accountable for maximizing the value of the product resulting from the work of the (Scrum) Team. How this is done may vary widely across organizations, (Scrum) Teams, and individuals. ~ scrum.org The multifaceted nature of the role is also widely acknowledged. But to do this, an agile product owner takes on several roles, including business strategist, product designer, market analyst, customer liaison, and project manager. In short, agile product owners are an integral part of any scrum team. ~lucidchart The variety of responsibilities usually causes new POs to be lost. The struggle to balance the interests of customers, management and development teams is real. They have to deliver product increments on a typically tight schedule. When communicating with everyone they have to stay assertive while having limited decision-making power. Considering the above, candidates for the PO role are expected to be experts in the field and possess strong domain knowledge accompanied by maturity in communication and willingness to learn the new responsibilities. Who Is a Product Owner? I have held multiple conversations with other Product Owners and asked "who is a PO? what does she do?". There were too many answers to try to make a coherent block of text out of them. responsible person contact person team lead business analyst psychologist coach development team member proxy developer / scientist user perpective in mind converts wishes into actionable items maximizes value requirement collection martyr not a tester - but defines testing criteria That's a lot. Let's try to have a look at the role from four different perspectives: company's, team's, client's and a technical one. From a the Product Owner is someone who . She is responsible for backlog and minds the definition of done. At the company I use as a case study for the article, in smaller projects, at least 20% of a person's time spent should be spent on the role if one also maintains heavy technical engagement. Ideally, one person should "own" one product. company perspective, leads the contract From , the PO is the guide on the path forward. This is viewed by many as the toughest part of the job. She should also be a servant, catering to team's needs. The PO usually is also responsive for team composition. She decides who's needed to successfully deliver the product. Teams usually expect POs to be scribes, maintaining notes from relevant meetings. POs are not supposed to micromanage but maintain functional product backlog. She doesn't have to know every single detail. The PO should have a vision for the product, not the project. She should be a proxy and minimize organizational overhead for the development team. a team perspective expect POs to be their point of contact, . The PO is their proxy, understands them and cares for the success of their project. Additionally, the PO negotiates features with customers when a budget is limited (which is always). Product Owners should also coach inexperienced clients in software development methodology in order for them to understand consequences of certain design choices. Clients the responsible one At this particular employer, leadership role candidates are sourced from individuals with higher competence levels. That said, good leadership doesn't come with expertise and requires additional knowledge and experience. , the PO should follow the company's SDLC (Software Development Lifecycle) or a set of standard practices in the absence of one. POs are advised to apply Continous Testing - "we think project goes smoothly because we don't test, we don't test because we don't want to know it's bad". Lastly, the Product Owner should facilitate smooth and constant deliveries. From the technical perspective Other responsibilities of a Product Owner include: Risk management Converting stories into tasks Have a responsibility when it becomes blurred Maintain less than X backlog items (100 was pointed out as a good number) Product Ownwers in the Project Lifecycle The Product Owner should be present in the process from the beginning of the project development process. A typical process includes: Pre-sales Sales Picking a team Project development Support Sales The presence of a PO at different stages is debatable and is highly dependent on a company structure. It is however a consensus that mostly indirectly by execution of their job and maintaining contact with clients. every employee sells On each of those stages, one should remember a truth repeated numerously in the interviews I conducted: ** BACKLOG  is THE tool of PO** As a Product Owner, you should be aware of the key points of concern in the Software Development Lifecycle, agile processes and your role in them. I have tips for some of the meeting types you're going to take part in. Kickoff meeting Here are key tips: send an agenda before the meeting state what is the purpose of the meeting introduce the team go over the scope organisational items: status meetings, communication channels, work result sharing, task management, resource access leave an opening for their ideas for the meeting propose several dates Planning The goal is to specify the value increment that will be developed in the next Sprint. Remind near and long term goals and priorities to the team Split the tasks into ones fitting in one sprint Don't get into details All attendees engaged TIMESLOT - respect it, productivity sharply decreases after one hour Status meetings By status meeting, I mean catchup with a client. start with a light topic form - not that important no need to prepare perfect slides every time stick to a fixed agendas send summaries the whole team is not necessary General meeting tips be prepared be the first one stick to the defined time slots ask for any additional time limitations small talk is okay some include client's logo in materials don't ask unprepared people for things you think they're not ready for, especially with bigger audience be yourself / human Cultural differences Keep those in mind. In the international setting, people are usually educated in intercultural communication or at least accustomed to it, so they will adjust their social behaviours to you. But it should work both ways. If you're starting a project with a client from a cultural or regional background you know little about, do some light reading. Closing thoughts: What Is a Product Owner? The first time as a product owner can be stressful. But as long as you're sensible and communicate what you really mean you should be fine. We're all human. Don't be afraid to ask for help. Your manager and business developers will likely help you with anything client contact-related. And tech leaders are there for you to reach out and consult larger technical decisions. Use your environment to your advantage. Cover photo by fauxels from Pexels

Agile Development: What Is a Product Owner?

At , our developers are busy. HackerNoon They are so busy, that our marketing department is having a hard time keeping up with all of the awesome new features they are creating and improving. They also seem to like deploying something and releasing it as ‘yours to discover.’ In other words, very low-key and covert. Which is adorable, but also much too discreet. The rest of the team at wants to change that up a bit. HackerNoon We want to honor them, and show gratitude that we appreciate their hard work. But also to make sure our users (and certain non-dev team members) are up to date with everything that is happening. About once a month or so, we will publish a list of new (or occasionally some we forgot) product features, and highlight some of the biggest ones at the top. Please be sure to follow this account to keep yourself posted, or check out the . About Section's: Software by HackerNoon Small disclaimer: You may already be aware of some of the ones below, please allow an adjustment period for this new process. Thank you! The Notable 4 The about page is the second most visited page on the average site (after the homepage). With this user about page, we are validating writers’ expertise and perspective, and giving readers a page they can be proud of on HackerNoon. So far, 6,635 About Pages have been created. . 1. Individual User About Pages : Some people are very psyched Thank you for your patience. 2. Editor 3.0 has YouTube Embeds! 3. The shop.hackernoon.com launch! We partnered with Stackcommerce to build an online shop to sell HackerNoon branded merchandise, and provide users with discounts on valuable tech tools and online courses. Users can now subscribe to different types of emails. Check out more information . 4. The Email Subscribe Options are as Nuanced as You Are: here Other Awesome Improvements/Creations With a few simple clicks, users can now select a topic/prompt of choice (such as , , or knowledge on a niche such as . They then will be directed to a draft page on and simply start answering the questions. Since its launch, this initiative has resulted in 483 quality stories from all corners of the web. Writing Prompts Templates : writers-exclusive interview opinion on the news data science) app.hackernoon.com NFTs in Blog Posts: Have you heard? You can embed NFTs in your blog posts! Read more about that here. The HackerNoon Pixel Font 2.0 is now open-sourced and available on and . HackerNoon Pixel Font 2.0: brand.hackernoon.com GitHub Read more about it here. of the backend: We migrated our database for the Noonies to Firebase, both for current and previous years to ensure our award links remain evergreen. The front end got a new lick of paint, a dark mode, and several speed improvements over past iterations. Last but not least, we resolved several SEO shortfalls from last year to boost organic traffic. A total revamp Noonies Voting Software Story URLs now no longer have random strings of letters at the end, which makes them more user-friendly and good for SEO. Cleaner URL Structure: was built into the editor system. The Plagiarism Checker get the love they deserve! Users who left a draft open for longer than x days now get an email from Natasha reminding them to take care of their mental health AND finish the draft. Abandoned Drafts Elements such as the welcome message, and the ability to drag and drop convos around have been improved and/or implemented. and counting! Slogging Back-End Improvements: 277 stories are currently published A more colorful HackerNoon FTW! Users can choose from 4 default color palettes. Users who do this stay x time longer on site. Read more about that Choose Your Color : here. : Certain trusted, highly qualified users are now able to bypass the editorial process and publish their own stories. We are working up to our desired rate of publishing by empowering the top contributors to earn and leverage the publish button. We are flattered that some users still choose to submit to the editors anyway because 'we have better headline practices' <3. Examples and . Verified Writers (invite-only, for now) here here We're proud to announce a revamped Editor Dashboard and in-app workflow to match our evolving and expanding editorial team.  It has WoW stats on stories published, scheduled, rejected, and more! Editorial Dashboard 2.0! Read more about the overall growth of HackerNoon in The State of the Noonion: The Year of Profitability and Product Renaissance This post was written and compiled by Linh Dao Smooke and Ellen Stevens.

Explore Some of the Latest (Major) HackerNoon Product Updates! 

Merkle trees, also known as , are a type of . They can be used to . They are used in both blockchain and non-blockchain-related projects. binary hash trees binary tree efficiently verify the integrity of large sets of data Merkle trees are build from the bottom up: The leaves of the tree contain the hash of each element in our dataset. This dataset could represent a list of transactions, parts of a file, etc. Each internal node is the result of the concatenation of the hashes of its two children. hashing Outline Hash functions Properties of a Merkle tree What problems can a Merkle tree solve? Implementing a Merkle tree in JavaScript Known vulnerability Sparse Merkle Trees Conclusions Hash functions Hash functions take as input any piece of data (a string, an image, etc.) and produce a fixed-length output. An example of a hash function is SHA-256, which takes data of arbitrary size and produces an output that is 256 bits long. Good hash functions have the following properties: . They produce evenly distributed outputs in the possible output range. In the SHA-256 example, this range would go from 0000...0000 to 1111....1111. Uniformity . The same input will always generate the same output. Deterministic . Fast to compute need additional properties to make them more secure: Cryptographic hash functions . Given an input x, they generate an output h(x). From an output h(x), it is computationally infeasible to find the input it came from, x. One-way functions . The likelihood of 2 different inputs x,y producing the same output h(x) = h(y)  should be negligible. Collision resistant . Given an output h(x), it is infeasible to find an input such as h(x) = h(y). Resistant to preimage attacks y A will produce a . This is known as the . minor modification in the input completely different output avalanche effect Properties of a Merkle Tree The properties of a Merkle tree derive from its structure and its use of hash functions: . The root of the tree will change if any node in the tree is modified. Even the same dataset, with its elements presented in a different order (ex: [T2, T1, T3, T4] instead of [T1, T2, T3, T4]) will produce a different tree. Data integrity . We do not need the full tree to verify whether a transaction belongs to it. We only need a path that starts in one of the leaves and goes all the way up to the root (proof of inclusion). Partial verification ceil(log2(N)). The function ceil(x) maps x to the smallest integer greater or equal than x. Ex: ceil(3.14) = 4 and ceil(3) = 3. The height of a tree of N leaves is What problems can a Merkle tree solve? Introduction to Blockchain I will use to illustrate how Merkle trees can verify if a transaction belongs to a block. Bitcoin In case you aren’t familiar with blockchains, we can oversimplify things imagining that a blockchain is a of elements called . contain a list of that stores . Part of this metadata is the root of the block’s Merkle tree, called . distributed linked list blocks Blocks transactions and a header metadata Merkle root , to ensure immutability. If the previous block in the chain changes, even by one bit, its hash would change and it would break the chain. Each block points to the hash of its predecessor Imagine you are running an app that needs to verify if a transaction belongs to a block . In Bitcoin, store a block’s Merkle root along with all its transactions. For them, it is easy to verify if belongs to . Also, they can generate proofs for any transaction in a block on demand. T B full nodes T B What if the app runs on a device with limited disk, computing power, and bandwidth? To determine whether belongs to , the device needs information. It can retrieve this information from other participants in the network. T B Problems The following problems arise: How can we ? Data might be corrupted and other nodes in the network can send fake data. trust that the data we receive from other nodes is correct Given the limited resources, how can we ? minimize the amount of information we need to receive from the network Merkle trees can help to solve these problems. From the properties of Merkle trees we covered above: solves problem 1. Let’s assume we receive different pieces of information from a series of participants in the network. We have , which contains its Merkle root. The data we receive from the network is used to build a Merkle tree whose root must be the same as ‘s Merkle root. Otherwise, we cannot trust the data that others have shared with us. Data integrity B B and having a solve problem 2. Hashing all the transactions would also work to verify data integrity. However, that does not scale well, because proof generation and verification would be O(N) operations. Partial verification height of ceil(log2(N)) commonly known as , do not store full blocks. They interact with the network to request the information they need to verify transactions. They can assess whether a transaction belongs to a block requesting an and the block header, where the is stored. Simplified Payment Verification(SPV) nodes, lightweight nodes or light clients inclusion proof Merkle root Implementing a Merkle Tree in JavaScript The algorithm to generate a Merkle root from a list of transactions is as follows: The leaves of a Merkle tree contain the hashes of the transactions that belong to a block. In the image above, [T1, T2, T3, T4, T5, T6] represent transactions. From them, we generate the leaves of the Merkl tree using a hash function. If the number of transactions is odd, we duplicate the last node. We take 2 consecutive leaves and hash the concatenation of their hashes. This generates their parent. In the image, T1’s and T2’s parent is equal to hash(hash(T1) + hash(T2)). We keep repeating this process with all the transactions at the same “level”. These parents will be siblings that will form a new level in the tree hierarchy. We keep repeating this process until we only have one element left, the root of the tree. You can find my implementation of a Merkle tree. It can verify a Bitcoin block’s Merkle root and generate and verify proofs of inclusion. here Known vulnerability Bitcoin’s Merkle tree duplicates the last node in levels with an odd number of nodes. Also, if Bitcoin finds a block that is not valid, it caches its root to avoid trying to mine it again. This combination makes the tree susceptible to : for an input x, we can find a second input y, different from x, such that h(x) = h(y). second preimage attacks How does this design suffer from this? Let’s go back to our first example, based on the tree represented in the image below: The leaves of the tree come from a block containing these transactions [T1, T2, T3, T4, T5, T6]. B1 The level right above the leaves is [hash(hash(T1) + hash(T2)), hash(hash(T3) + hash(T4)), hash(hash(T5) + hash(T6))]. Since we have an odd number of nodes, we duplicate the last one. Hence, the next level changes to [..., ..., hash(hash(T5) + hash(T6)), hash(hash(T5) + hash(T6))] This is equal to the level produced by a block with the following transactions [T1, T2, T3, T4, T5, T6, ]. B2 T5, T6 contains duplicate transactions, making the block invalid. Nodes would cache this root so that next time they find they discard it. Both B1 and B2 produce the same root. Thus, nodes will reject , even though B1 might be a valid block. B2 a block with this root B1 For more details, have a look at and in Bitcoin’s source code. this post this comment There is another known vulnerability in Bitcoin’s original Merkle trees (already fixed). Understanding it requires more knowledge of how Bitcoin builds transactions. Since this is an introductory article, I have decided to leave it out. However, I might cover transactions in more depth in future articles and revisit this vulnerability. Sparse Merkle Trees A Sparse Merkle Tree (SMT) is a variation of a Merkle Tree, where every leaf node has an index along with its value. All leaves are initialized to an empty (null) value. By design, SMTs have 2^256 leaves. The Merkle tree I described above can verify that a transaction is part of the block by exploring the path from its corresponding leaf to the root of the tree. However, they cannot be used to prove that a transaction does not belong to a block. And in general, to prove that elements do not belong to a dataset. SMTs can do this. SMTs can produce proofs of non-inclusions in an efficient manner. The idea is that we know the index of every element. Thus, you know which leaf contains an element. If an element is not part of the tree, the leaf where the element would be stored will be null. Therefore, to produce a proof of non-inclusion, we produce a proof of inclusion generated from that node initialized to null. Proofs of inclusion work in the same way as Merkle trees’. Updating a leaf an SMT is also efficient because we only need to modify the path that goes from that leaf to the root. Since SMTs have 2^256 leaves, only 256 nodes need modification. SMTs deserve their own article, but I wanted to at least introduce the basic idea behind them. Conclusions Now you should have a clearer idea of what Merkle trees are and what they are used for. If you want to learn more, I suggest you investigate how: Git uses them to keep track of changes in files.DynamoDB uses them to identify any inconsistencies between nodes. I hope you found this article helpful. Share it, because you could help someone pass their exams or get a job. You can access more of my content on and my . my Twitter blog Previously published at https://www.yourdevopsguy.com/merkle-trees-and-bitcoin/

About Bitcoin and Merkle Trees

Statement We have to search for a value in a sorted matrix . If exists, then return its coordinates , else return . x M x (i, j) (-1, -1) Let us consider the above matrix as an example. In this example, we are going to search for the value Since 12 is present in the matrix, the algorithm should return its coordinates ) 12. (2, 1 Simple Approach A simple approach is to traverse all the values in the matrix and check if it is equal to 12. The worst case time complexity of the above algorithm will be when Time Complexity O(n x m) = O(n²) n = m Efficient Approach The above algorithm behaves worse for large values of n and m. Let us look into the efficient algorithm now. Algorithm Start from Top Right position ( in the matrix 0, m - 1) M If the value is equal to return x (0, m - 1) Move one row down if the current value is less than x Move one column left if the current value is greater that x Let us apply this algorithm into our matrix We are going to search for the value in M. 12 M 1. Start from the Top Right value at is less than 12, so 12 should be somewhere in the bottom of the matrix since all row and column values are sorted in ascending order. So we move one row down. 5 M [0][4]. 5 2. The value at is also less than 12, so we move one row down 10 M [1][4] 3. The value at is greater than 12, so 12 should be somewhere in the left of the matrix, so we move one column left. 15 M [2][4] 4. The value at is also greater than 12, so we move one column left 14 M [2][3] 5. The value at is greater than 12, so we move one column left 13 M [2][2] 6. The value at is equal to 12, so we return its index M [2][1] (2, 1) Time Complexity The worst case time complexity of the above algorithm will be when O(n + m) = O(2n) = O(n) n = m, because we will be iterating all rows and all columns once if x is at the bottom left position (n - 1, 0) You can find the Java solution for this algorithm in the Github repo. https://github.com/ganeshkumarm1/DSAlgo/tree/master/src/com/dsalgo/Algorithms/BinarySearch2D Thank you 🙏 👋🤘 | Linkedin Github

How To Search An Element In Sorted Matrix In Linear Time

Why do we so often ask teams to estimate in detail how much effort is required to build a product, and to that upfront? There’s evidence that this lead us to output-oriented thinking, premature optimization, bloat and generally a commitment to suboptimal solutions that are designed with too many assumptions and a lack of clarity on the problem space. Yet the main reason, I’d argue, is that estimation is an entirely rational risk mitigation strategy. Some of the reasons might be that: 1) we’re promising to build a product for price X, when it’s going to cost us more than X-M, where M is our minimum acceptable profit margin. Maybe we don’t want to build this thing. 2) there are explicit and irreversible costs to not meeting a deadline and we want to minimize the risk of not doing so, so that we can organize and dedicate appropriate resources to it. 3) we are uncertain of the returns it will yield, so we want to be sure that we’re not over-investing vs what we have in the bank. That’s all fine, but I argue that diminishing returns to estimation happen pretty early in the effort scale, and discount your ability to respond to future ambiguity. Teams should focus on simple, transparent budgeting, asking themselves a few key questions: how much runway do we have to build this? E.g. a start-up may need to get a particular feature out the door in about a month to test market appetite. A scale-up might need to deliver a big product stream within a year to capture the returns of that investment in the next financial exercise. how much of that runway do we want or need to bet on this particular problem? Is it a bet the company move? Are we hedging our bets with 3/4 major plays? Are we just seeking initial validation? Do we believe we need to acquire additional resources in order to fund these goals? Why? Is there anything we could drop instead? Do we know enough to make the decision? Clarifying Runway Teams should be formed and funded to solve a specific problem or need, not given a fully shaped answer already. Understanding and negotiating the right runway to answer a given question upfront is an important part of the process. Follow-up: How much funding do we need to figure it out? Can we meet in a few weeks to talk about what we’ve learned? Do we know the problem we’re solving well enough? F: How do we break this down? What’s our first goal and how much budget do we want to allocate to it? Does this look too big for our team? F: Are we likely to run out of runway to tackle it or can we solve with the current constraints? Have we discovered a major hurdle or roadblock? The right level of constraint fosters creativity, productivity, and agility, so managing your burn rate towards your expected outcomes should be a fundamental part of your day to day discussions. When questions such as the ones above arise, when you can’t solve them within the budget you have, there’s a frank conversation to be had with your investors (either internal or external), to get a portfolio take on it and use your voice in a forum where you can advocate for the right resource structure given your current level of knowledge. as a team Betting Now when you’ve worked out your current runway (give us 3 months to work on this), it’s up to you as a team to decide how best to use it. People should share a stake in outcomes, so that they’re accountable and responsible for their achievements. For simplicity’s sake let’s take money out of the picture. You have 5 highly qualified team members and 12 weeks of runway. Each team member gets 12 coins, which they can spend as they please. They can use it for experimentation. They can use it to solve technical debt. They can use it for foundational work with a longer payback period. They can manage their ROI horizon so long as there is team buy-in. As a team, your communication becomes about whether you’re using your runway the right way given the problem you have to solve, how much do we believe in our solution, how much evidence we have to back it up, and how to we maximize our personal time investment. A great product manager, I’d say, maximizes this ROI function for their team. They devise the right investment strategy proposal, offer their teams the right level of information so they make informed decisions and good trade-offs, and continuously evaluate and communicate said strategy in the presence of new information — did we get results back from users? Are we becoming more confident about the value this represents? They also let others take the lead and own solutions, emergent problem exploration or resource allocation plays, devoting their time to synthesising the outcomes into a good map of where we’ve been, what we know, and where we’re headed. Are we getting bang for our buck? Funding Finally, acknowledging that you need funding as early as possible in the process, whether it’s in the shape of time, money or both, is important. “Gotcha!”, you’d rightly say right about now. In order to know whether you need funding, you’d have needed to estimate whether your runway is sufficient to meet your goals. I don’t disagree. I argue though, that accuracy gains in product development estimates are quickly offset by the time and cost of those gains, particularly for complex problems where you have too many unknowns and need to make up assumptions to move forward. You should be making a continued assessment based on risk and reversibility while reducing unknowns and assumptions using a diverse set of viewpoints and tactics. Set your ego aside and ask honest questions, be open to change, be confident enough to bet. Reviewing and Restructuring Once you start budgeting and funding, as you become better at breaking problems down, identifying key assumptions, incorporating new information, assessing risks, potential returns, modeling your predictions and even improving your productivity, deep and detailed estimation will become much less important, while systematic flow will grow in relevance. We’re able to define bite-sized chunks of our larger meal, and use what we’ve learned to move forward in the right direction, gaining momentum. Keeping a healthy bets portfolio and shipping consistently against it becomes ever more important and reviewing your investment strategy often, understanding the shape of each of your investment’s returns, is a fundamental exercise. At a portfolio level, organizing teams in relation to the structure of your needs, your runway, and your risk level becomes the better lens through which to look at the world and gives individuals the authority to go out and discover the best solutions to your current problems, especially with healthy and regular feedback mechanisms in place. Previously posted at . Building Fast and Slow: Budgeting as a Product Team

Product Teams: Stop Estimating, Start Budgeting

Simple Framework for Alerting anomalies, spikes and other patterns from data in elasticsearch. We use ElasticSearch, Log stash, Kibana for managing over increasing amount of data and logs, where kibana is great visualizing & querying data, but it need companion tool for alerting when inconsistencies in our data — where Elastalert is created. Overview — Reliable , Modular ,Ease Set Up & Configure. — in event of restart elastic search or in case of unavailable elasticsearch. Why is it Reliable? ElastAlert— (when it started & resume previously stopped). save its state to elasticsearch If Elasticsearch is , ElastAlert waits until it recovers before continuing. unresponsive ElastAlert (alerts) which throws Errors may be . automatically retried for Period of time Why is it Modular? — Main due to 3 Components. :  Responsible for Processing Data returned from elasticsearch(basically by initializing the rule configuration & parse data -return data by querying elasticsearch). Rule Types : Responsible for Taking action based on match. Alerts : Global Configuration file (config.yaml) — defines several aspects of its operation.(Ex: es_host,es_port,type) Configuration with two types of components, rule types and alerts. How it works? — by combining elasticsearch Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match. This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts. Several Rule types with common monitoring are included with ElastAlert: Match when there are X events in Y time — Frequency Match when the rate of events increase/decrease — Spike Match when there are less than X events in Y time — flatline Match when there a certain field matches a blacklist/whitelist — blacklist/whitelist Match on any event matching a given a filter — any Match when a field has two different values within some time — Change — Jira ,email, Google Chat,the Hive,Telegram,Slack… Support In Built are In Addition to this basic usage elastalert supports : Alert links to Kibana Dashboard Aggregate Counts for arbitrary fields Combine alerts into periodic reports Separate Alerts by a unique field Intercept/enhance match data. : Installation Clone Repo : git clone or pip install elastalert https://github.com/Yelp/elastalert.git See Requirements in Cloned Repo and install using pip. Elasticsearch Python3 python3 setup.py install Yes! we have installed ElastAlert. In the next part, we'll talk about configuring elasticsearch to elastalert and configuring built in type queries. In case of queries please feel free to comment if any issues!. Thanks for reading Read Part II Here

Stories, News & Mentions about THANX

Thanx’s Stories on HackerNoon