Real Links

reallinks

172 Stories To Learn About Ruby

Today I would like to introduce to you the blockchain project Hydrogen. The thing that caught my attention about this project was an article on TD Bank a couple of weeks ago. This article wasn’t actually about Hydrogen, it was about TD Bank buying over another financial institution. But in that article, there was one sentence that said “The bank is partnering with U.S. firm Hydrogen on these products”. That got my attention. TD bank is the world’s 13th largest bank with an asset value of over 1.2 Trillion dollars. That’s 4x the entire crypto market asset worth. If this piece of news was true, then Hydrogen would have itself a partnership that was worth more than the partnership of EOS, ADA, LTC and TRX combined. It would be as big as Stellar and IBM’s partnership and we all know what that announcement did for Stellar’s token price last year when that was announced. The news of that one partnership literally pushed Stellar into the top 10 cryptocurrencies. I honestly believe the actual news was a leak. It was not meant to be announced because neither Hydrogen nor TD Bank had made such an announcement. I believe the person writing the article leaked a news that he shouldn’t have. Now sometimes, after a news is leaked, e.g. a partnership announced beforehand or a coin being listed on an exchange beforehand etc.. the result can be devastating and the whole deal could even be called off, simply because the news was leaked unprofessionally. So in this case, I wanted to see what would happen and have been keeping a close eye on the situation. Quite recently, on their Telegram group, the team pinned a message that said “Other upcoming events: TD Bank PR announcement by end of July, FINDI Launch — we already have a $20+ Bil company signed up. Big PR possible!” So I believe the TD Bank partnership is legit news and they are possibly getting other big partnerships on board as well. In other words, this project is probably going to get a lot of momentum in the very near future. 25% of all Blockchain projects currently revolve around the financial industry. Hydrogen describes itself as “the FinTech Blockchain”. It’s a rather generic description that makes it hard to know how this project stands out from other financial Blockchain projects. But Hydrogen is actually a very unique project in the way it approaches the Fintech scene. Most financial based Blockchain e.g. Ripple, Stellar aim to improve the financial transaction itself. They aim to introduce a cheaper and faster way to do cross-border payments etc… But the difficulty they face is firstly scalability because the Blockchain is primary made for security not speed and secondly they face difficulty of mass adoption because using their technology demands that the financial company e.g. the bank has to make huge adjustments to their current practice to use Blockchain technology. Hydrogen though is a second layer solution to financial transactions that adds an additional layer of super security and open ledger to all the transactions, after the bank has done it’s thing. In other’s words, it doesn’t interfere at all with how they bank currently runs, it just adds a second layer of security. It’s non-intrusive, easy to adopt, and that’s why banks and other major financial institutions will find it easy to use. It’s providing the benefits of the Blockchain decentralised open ledger, with zero disruption to existing current practices. To understand Hydrogen, you need to understand 2 aspects of the project, the Technology and the APIs. The technology of Hydrogen has 5 core features called Raindrop, Snowflake, Ice, Tide and Mist. These are the 5 technologies they are bringing to provide the advantages of Blockchain technology to financial institutions. Raindrop is a protocol that enables single of multi-factor authentication process. This authentication process helps to prevent system breaches and data compromises. This feature is already up and running. It’s a rather complicated feature that comprises of both a server side validation and a client side validation. Think of it as validating an account before doing a transaction. So for example, if I wanted to send a large sum of money to your crypto wallet. What I might do is send a micro-transaction, a small sum to your wallet, initially to confirm the transaction has gone through before sending the larger sum. In a similar but much more complicated way, raindrop does it, but as the name raindrop suggests, the protocol validates very specific transactions that are unpredictable, like the raindrops in a storm. And rather than sending the user and amount directly and having it sent back, Hydro defines a transaction and the user must execute it from a known wallet. This can only be done by having direct access to the wallet in question. Currently the safest way most people know to do financial transactions, even us using crypto exchanges, is the 2 factor authentication. But as you can see, Raindrop as an authentication feature, has so many additional features, google authentication can’t even compare with it. I believe security of accounts is paramount, and people are willing to spend to protect their account and their assets. We’ve seen 2FA take off over the last decade, and I think we will see Blockchain authentication solutions like Raindrop take off over the next decade. The next core feature is Snowflake. This is basically an identity management system. So just as other projects such as Ontology, TheKey or SelfKey uses the immutable untamperable nature of the Blockchain to create identity for individual and institutions, Snowflake does the same to create unique identities for each participant. Unlike ID cards or passports which can be faked, or digital IDs which can be hacked and tampered, Blockchain identities are decentralised, open ledger, meaning they can never be hacked or tampered with, a Blockchain identity is even more accurate and secure than a passport. So just as every Snowflake in real life is unique and different in pattern from another, this Snowflake technology creates and uses unique digital fingerprints for users. Both Raindrop and Snowflakes are working products and each have their own whitepaper which is accessible from the Hydrogen website. The other three technologies Ice, Tide and Mist are not released yet. But Ice will be a service to stamp and record contracts on the Blockchain, Tide will extend payment solutions to the unbanked and Mist aims to integrate AI into this Blockchain project to utilise machine learn and network optimisation. And that’s the technology of Hydrogen in a nutshell, an impressive technology with a wide array of features that is very practical to the financial industry. The way that Hydrogen makes this technology usable by the banks and financial institutions, is by creating what is known as the Hydrogen Atom. The Atom is basically a platform on which API modules are built. APIs are not applications. They are applications for applications. So your mobile phone will have various applications. One of those applications might be Ubereats. Ubereats will have different features e.g. payment service, restaurant menu you can choose the food from, GPS to track the delivery etc.. each of these features, are known as an API. Hydrogen doesn’t ask banks to discard their existing software and learn a new program. Instead, they create APIs, to integrate into existing software. So again very minimal disruption and a very low barrier to adoption. The APIs they are creating include: Nucleus APIs — These are APIs that provides the core digital infrastructure to the Fintech Application. They include services such as authentication, storing and encrypting client information, management of client accounts and more… Proton APIs which provides financial engineering to the Fintech Application e.g. Risk scoring, goals tracking, simulations and more Electron APIs which provide functionality to grow the Fintech Application e.g. Support, Marketing, Billings etc.. So it’s basically a very holistic platform that covers all the essential needs of a Fintech company. The entire project is also designed to have a lot of token use, which is great for token demand and price. Raindrop will use Hydro tokens to pay for authentication attempts, Snowflake will require Hydro Tokens by companies that use the identity data, Ice will use Hydro tokens to pay for the stamping of contracts, Tide and Mist we will have to wait and see as those features are not due until 2019, but they are planned to integrate Token use as well. The Hydro technology is run by smart contracts, and the easiest way for companies to access these smart contracts is by using the APIs. But should they not wish to install or use the APIs, they can still use the smart contracts, by “staking” Hydro tokens, and what this does is it locks the tokens into an escrow contract that fuels the smart contracts. So it’s a good project, with good technology and good token use. This is the team behind the project. One thing I don’t like about their website is there is not resume or links to the resumes for any of their team members. Almost every other website gives a little spill or link to linkedin for the staff. But this profile page only has a few faces and names. You can find their resumes by searching them up individually, but it’s pretty troublesome. The Co-founders are twin brothers, Michael and Matthew Kane. They previously founded a company called Hedgeable in 2009, which has since won Finovate best of show award in 2015 and 2017 as well as UK great tech awards top Fintech 2015. So it’s good to know they have previous success and experience in the Fintech space. Hedgeable their previous projects has also attracted Venture Capital funding, so I’m hoping Hydrogen will at some point attrack similar VC attention. For now, their website does not list any advisors, partners or investors. Which is a bit of a downer for me personally, because 25% of all Blockchain projects are focused in the financial industry. That’s about 500 projects. With dominant big projects like Ripple and Stella, it’s going to take more than just a good technology for a project to stand out from the rest. Good advisors and investors do more than give advice and money to the project, they are prominent, successful individuals in the industry who have a personal stake to see the project succeed and many times we’ve seen connections and in-roads to big partnerships happen because of these personal relationships. So I think these are important aspects of any project to consider and it would good of Hydro to update these details when they can. Of course the upcoming partnership with TD Banks, itself is huge news and that’s an outstanding partnership I expect to have a big impact on this project and it’s standing in the crypto market. The other thing the project doesn’t have is a clear roadmap for the whole project. This is the roadmap for their Technology, and it’s simple and easy to read. But a project is more than this, it would include plans for marketing milestones, getting the APIs out etc… So whilst we can appreciate the technology of the project is expected to be completed in late 2019, I think the website is lacking a proper roadmap to help provide accountability and expectation to the community. The only other consideration I have for this project is regarding scalability. As we mentioned before, Hydrogen is a second layer security solution to financial transactions. It doesn’t need to have super high throughput, e.g. Ripple, Stellar etc… but it still needs to have a decent throughput if it is to be used by multiple big financial institutions. The way I think of it is like a cashier and book keeper. Ripple and Stellar and the other projects that focus on doing the actual transactions are like cashiers who need to have high speed. The book keeper is the person who processes the record of the transaction after it is done. It doesn’t need to be super fast, but it still needs to be fairly fast in order to keep up with the volume of work. Hydrogen is built on the Ethereum Blockchain which I think is a terrible Blockchain for Dapps at the moment. It’s entire scalability capability is dependent or limited by how scalable Ethereum is. The best they can offer is that Ethereum has proposed to implement Plasma or Raiden. But the thing is Ethereum has been looking at sharding solutions for years, and nothing has been practically achieved yet. It is still very very slow. Look at what happened over the past few weeks, with one exchange Fcoin flooding the network, the entire Ethereum network came to a crawl and gas wars erupted. Price of doing a transaction was ridiculously high and timings for transactions were as slow as 3 hours. Ethereum has not gotten better compared to cryptokitties 8 months ago. It is still very very slow. I also think alot of the practical scaling solutions that I do see workable on Ehtereum are the second layer solutions e.g. Loom, OMG etc.. but these will benefit new projects being built, not so much project that are already built on Ethereum. Just because a new project is able to implement Plasma, doesn’t mean Ethereum can also use Plasma. The difference is that Ethereum has a thousand Dapps built on it and running, to implement any scaling solution to Ethereum is like trying to upgrade a train engine whilst it is running at full speed. It’s incredibly hard. Im not saying it’s impossible, Im just saying it’s very hard. So when projects say in their white paper Vitalik has proposed Plasma or Raiden Network as a scaling solution, I don’t buy it. That’s not enough to put my mind at ease. Il believe when I see it actually working and Ethereum actually having cheap and fast transactions. Until then, any project that is restricted by Ethereum in terms of scalability is not out of the woods. The other thing they mention is at this time, raindrop will put very minimal strain on the Ethereum framework. That’s fair enough. But the key phases are “at this time.” And “minimal strain”. With a mega partnership coming on board and the other features expected to roll out by end of 2019, at some point this project will be a significant volume on Ethereum. Furthermore, it doesn’t matter if Raindrop doesn’t put much strain on the Ethereum network. What matters is that Raindrops depends on the Ethereum network to function. Which means, even though they put very little strain on the Ethereum network, all you need is one joker like Fcoin to flood and clog the system up for everyone using the system, whether it’s Hydro or you and me doing a simple Ethereum transaction to suffer. So my concern is not with this project and it’s technology. I think the technology is great. My concern is with Ethereum and how bad the network is at the moment, and the fact that Hydro is working on the network. They are a working product and need to function well now, not in the future when Ethereum can finally scale, but now when Ethereum has not yet scaled, so personally, I think they would be better off moving to an Ethereum compatible platform. That’s just my very unprofessional and layman’s way of looking at things. Conclusion All in all, I think Hydro is a project with alot of potential in the Fintech space and worth paying attention to, definitely worth knowing about in the crypto space. None of this is financial advice, just my personal thoughts on this project. Always do your own homework and make your own decisions. Drop a comment below, let me know what your thoughts on Hydro is, and if you agree or disagree with what my thoughts. I’m not always right, and I would love to hear your thoughts on this one. Thank you so much for reading this. If you liked this, give it that clap and subscribe, we have some other great projects coming up for review, so make sure you follow us for that. It’s Friday, I hope you guys have a great start to the weekend, enjoy yourselves and catch you guys again very soon!

Hydrogen (HYDRO) Review: Seamless integration of Blockchain into Fintech

Hard links and symbolic links have been available since time immemorial, and we use them all the time without even thinking about it. In machine learning projects they can help us, when setting up new experiments, to rearrange data files quickly and efficiently in machine learning projects. However, with traditional links, we run the risk of polluting the data files with erroneous edits. In this blog post we’ll go over the details of using links, some cool new stuff in modern file systems (reflinks), and an example of how DVC (Data Version Control, ) leverages this. https://dvc.org/ As I am studying machine learning, I’m wishing for a tool that would let us inspect ML projects the way we do regular software engineering projects. That is, to retrieve the state of the project at any given time, create branches or tags (with Git) based on an earlier state of a project, handle collaboration with colleagues, and so on. What makes ML projects different is the tremendous amount of data, thousands of images, audio, or video files, the trained models, and how difficult it is to manage those files with regular tools like Git. In my earlier articles I went over why , and why Git-LFS is not a solution for machine learning projects, as well as some . Git by itself is insufficient principles that seem to be useful for tools to manage ML projects DVC has proved to be very good at managing ML project datasets and workflow. It works hand-in-hand with Git, and can show you the state of the datasets corresponding to any Git commit. Simply by checking out a commit, DVC can rearrange the data files to exactly match what was present at the time of that commit. The speed is rather magical, considering that potentially many gigabytes of data are being rearranged nearly instantaneously. So I was wondering: How does DVC pull off this trick? The trick to rearranging gigabytes of training data Turns out DVC’s secret to rearranging data and model files as quickly as Git is to link files rather than copy them. Git, of course, copies files into place when it checks out a commit, but Git typically deals with relatively small text files, as opposed to the large binary blobs used in ML projects. Linking a file, like DVC does, is incredibly fast, making it possible to rearrange any amount of files in the blink of an eye, while avoiding copying, thus saving disk space. Using file linking techniques is nothing new to the field, actually. Some data science teams use symlinks to save space and avoid copying large datasets. But symlinks are not the only sort of link which can be used. We will start with a strategy of copying files into place, then using hard links and symbolic links, then ending up with a new type of link, reflinks, which implements in the file system. We will use DVC as an example of how tools can use different linking strategies. Copy On Write capabilities Test setup Since we’ll be testing different linking strategies, we need a sample workspace. The workspace was setup on my laptop, a MacBook Pro where the main drive is formatted with the file system. Further tests were done on Linux on a drive formatted with . APFS XFS The data used is two “stub-articles” dumps of the Wikipedia website retrieved from two different days. Each is about 38 GB of XML, giving us enough data to be similar to an ML project. We then set up a Git/DVC workspace where one can switch between these two files by checking out different Git commits. $ ls -hl wikidatawiki-20190401-stub-articles.xml
-rw-r--r--  1 david  staff    35G Jul 20 21:35 wikidatawiki-20190401-stub-articles.xml
$ time cp wikidatawiki-20190401-stub-articles.xml wikidatawiki-stub-articles.xml

real    14m16.918s
... As a base-line measure we’ll note that copying these files into the workspace took about 15 minutes apiece. Obviously an ML researcher would not have a pleasant life if it took 15 minutes to switch between commits in the repository. Instead, we’ll be exploring another technique DVC and some other tools utilize - linking. There are two types of links all modern OSs support: Hard Link, Symbolic Link. A new type of link, the Reflink (copy-on-write), is starting to be available in newer releases of Mac OS X and Linux (for which one needs the desired filesystem driver). We’ll use each of them in turn and see how well they work. DVC discusses corresponding to the three link types (the 4th strategy is to just copy files) in its documentation. The strategy used depends on the file system capabilities, and whether the "dvc config cache.type" command has changed the configuration. the four strategies DVC defaults to using reflinks and, if not available, to fall back to file copying. It avoids using symlinks and hardlinks because of the risk of accidental cache or repository corruption. We’ll see all this in the coming sections. Versioned datasets using File Copying The basic (or naive) strategy of copying files into place when checking out a Git tag is the equivalent of these commands: $ rm data/wikidatawiki-stub-articles.xml
$ cp .dvc/cache/ / c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml 40 58 This will run on any filesystem, but it will take a long time to copy the file, and it will consume twice the disk space. What’s with the strange file name? It’s the filename within the DVC cache, the hex digits being the MD5 checksum. Files in the DVC cache are indexed by this checksum, allowing there to be multiple versions of the same file. The contains more details about the DVC cache implementation. DVC documentation In practice this is how it works in DVC: To set up the workspace we created two Git tags, one for each file we downloaded. The ( ) should give you an idea of what’s involved with setting up the workspace. To test different file copying/linking modes we first change the DVC configuration then check out the given Git commit. Running "dvc checkout" causes the corresponding data file to be inserted into the directory. DVC example on versioned datasets or this alternate tutorial Oh boy, that sure took a long time. The Git portion of this is very fast, but DVC took a very long time. This is as expected, since we told DVC to perform a file copy, and we already knew it took about 16 minutes or so to use the cp command to copy the file. As for disk space, obviously there are now two copies of the data file. There is the copy in the DVC cache directory, and the other that was copied into the workspace. Source: https://www.xkcd.com/981/ Versioned datasets using Hard Links and Symlinks Clearly copying files around to handle dataset versioning is slow and an inefficient use of disk space. An option that has existed since time immemorial in Unix-like environments are both hard links and symbolic links. While Windows historically did not support file links, the " " can do both styles of links as well. mklink Hard links are a byproduct of the Unix model for file systems. What we think of as the filename is really just an entry in a directory file. The directory file entry contains the file name, and the “inode number” which is simply an index into the inode table. Inode table entries are data structures containing file attributes, and a pointer to the actual data. A hard link is simply two directory entries with the same inode number. In effect, it is the exact same file appearing at two locations in the filesystem. Hard links can only be made within a given mounted volume. A symbolic link is a special file where the attributes contains a pathname specifying the target of the link. Because it contains a pathname, symbolic links can point to any file in the filesystem, even across mounted volumes or across network file systems. The equivalent commands in this case are: $ rm data/wikidatawiki-stub-articles.xml
$ ln .dvc/cache/40/58c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml This is for a hard link. For a symbolic link use " ". ln -s Then to perform the hard link scenario: The symbolic link scenario is the same, but setting to . The timing is similar for both cases. cache.type symlink Two seconds (or less) is sure a lot faster than the 16 minutes or so it took to copy the files. It happens so fast we use the word “instantaneous”. File links are that much faster than copying files around. This is a big win. As for disk space consumption, consider this: $ ls -l data/
total 8
lrwxr-xr-x  1 david  staff   70 Jul 21 18:43 wikidatawiki-stub-articles.xml -> /Users/david/dvc/linktest/.dvc/cache/2c/82d0130fb32a17d58e2b5a884cd3ce The link takes up a negligible amount of disk space. But there is a wrinkle to consider. Ok, looks great, right? Fast, no extra space consumed … But, let’s think about what would happen if you were to edit data/wikidatawiki-stub-articles.xml in the workspace. Because that file is a link to a file in the DVC cache, the file in the cache would be changed, polluting the cache. You’ll need to take extra measures, and learn how to avoid that problem. The has instructions on avoiding the problem when using DVC. It means always remembering to use a specific process for editing a data file that, while not a deal breaker, is less than convenient. The better option though is to use reflinks. DVC documentation Versioned Datasets using Reflinks Hard links and symbolic links have been in the Unix/Linux ecosystem for a long time. I first used symbolic links in 1984 on 4.2BSD, and hard links date back even further. Both hard links and symbolic links can be used to do what DVC does, namely quickly rearranging data files in a working directory. But surely in the last 35+ years there has been an advancement or two in file systems? Indeed there has, and the Mac OS X “ ” and Linux “ ” features are examples. clonefile reflink Copy On Write links, a.k.a. reflinks, offer a solution to quickly linking a file into the workspace while avoiding any risk of polluting the cache. The hard link and symbolic link approaches are big wins because of their speed, but doing so runs the risk of polluting the cache. With reflinks, the copy-on-write behavior means that if someone were to modify the data file the copy in the cache would not be polluted. That means we’d have the same performance advantage as traditional links, with the added advantage of data safety. Maybe, like me, you don’t know what a reflink is. This technique means to duplicate a file on the disk such that the “copy” is a “clone” similar to a hard link. Unlike a hard link where two directory entries refer to the same inode entry, with reflinks there are two inode entries, and it is the data blocks that are shared. It happens as quickly as a hard link, but there is an important difference. Any write to the cloned file causes new data blocks to be allocated to hold that data. The cloned file appears changed, and the original file is unmodified. The clone is perfectly suitable for the case of duplicating a dataset, allowing modifications to the dataset without polluting the original dataset. Like with hard links, reflinks only work within a given mounted volume. Reflinks are easily available on Mac OS X, and with a little work is available on Linux. This feature is supported only on certain file systems: Linux: BTRFS, XFS, OCFS2 Mac OS X: APFS APFS is supported out of the box on macOS, and Apple strongly suggest we use it. For Linux, XFS is the easiest to set up as shown in this tutorial. For APFS the equivalent commands are: $ rm data/wikidatawiki-stub-articles.xml
$ cp -c .dvc/cache/40/58c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml With the option, the macOS cp command uses system call. The function sets up a reflink clone of the named file. On Linux the cp command uses the option instead. -c clonefile(2) clonefile --reflink Then to run the test: The performance is, as expected, similar to the hard links and symbolic links strategies. What we learn is that reflinks are about as fast as hard links and symlinks, and disk space consumption is again negligible. The cool stuff about this link is even though the files are connected you can edit the file without modifying the file in the cache. The changed data are copied under the hood. On Linux the same scenario runs with similar performance. Conclusion We’ve learned something about how to efficiently manage a large dataset, like is typical in machine learning projects. If we need to revisit any development stage in such projects, we’ll wanta system for efficiently rearranging large datasets to match each stage. We’ve seen it is possible to keep a list of files that were present at any Git commit. With that list we can link or copy those files into the working directory. That is exactly how DVC manages data files in a project. Using links, rather than file copying, lets us quickly and efficiently switch between revisions of the project. Reflinks are an interesting new feature for file systems, and they are perfect for this scenario. Reflinks are as fast to create as traditional hard links and symbolic links, letting us quickly duplicate a file, or a whole directory structure, while consuming negligible extra space. And, since reflinks keeps modifications in the linked file, they give us many more possibilities than traditional links. In this article we examined using reflinks in machine learning projects, but they are used in other sorts of applications. For example, some database systems utilizing them to manage data on disk more efficiently. Now that you’ve learned about reflinks, how will you go about using them?

Reflinks vs symlinks vs hard links, and  how they can help machine learning projects

Stream Processing - Windows

Introduction This is a companion guide to (the "Part 1 guide"). I wrote the Part 1 guide on how to finetune Meta AI's model on Chilean Spanish. It is assumed that you have completed that guide and have generated your own finetuned XLS-R model. This guide will explain the steps to run on your finetuned XLS-R model via a . Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition wav2vec2 XLS-R ("XLS-R") inference Kaggle Notebook Prerequisites and Before You Get Started To complete the guide, you will need to have: A finetuned XLS-R model for the Spanish language. An existing . Kaggle account Intermediate knowledge of Python. Intermediate knowledge of working with Kaggle Notebooks. Intermediate knowledge of ML concepts. Basic knowledge of ASR concepts. Building the Inference Notebook Step 1 - Setting Up Your Kaggle Environment Step 1.1 - Creating a New Kaggle Notebook Log in to Kaggle. Create a new Kaggle Notebook. The name of the notebook can be changed as desired. This guide uses the notebook name . spanish-asr-inference Step 1.2 - Adding the Test Datasets This guide uses the as its source for test data. Like the , the Peruvian speakers dataset also consists of two sub-datasets: 2,918 recordings of male Peruvian speakers and 2,529 recordings of female Peruvian speakers. Peruvian Spanish Speech Data Set Chilean Spanish Speech Data Set This dataset has been uploaded to Kaggle as 2 distinct datasets: Recordings of Peruvian male speakers Recordings of Peruvian female speakers Add both of these datasets to your Kaggle Notebook by clicking on . Add Input Step 1.3 - Adding the Finetuned Model You should have saved your finetuned model in of the guide as a . Step 4 Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition Kaggle Model Add your finetuned model to your Kaggle Notebook by clicking on . Add Input Step 2 - Building the Inference Notebook The following 16 sub-steps build each of the inference notebook's 16 cells in order. You will note that many of the same utility methods from the Part 1 guide are used here. Step 2.1 - CELL 1: Installing Packages The first cell of the inference notebook installs dependencies. Set the first cell to: ### CELL 1: Install Packages ###
!pip install --upgrade torchaudio
!pip install jiwer Step 2.2 - CELL 2: Importing Python Packages The second cell imports required Python packages. Set the second cell to: ### CELL 2: Import Python packages ###
import re
import math
import random
import pandas as pd
import torchaudio
from datasets import load_metric
from transformers import pipeline Step 2.3 - CELL 3: Loading WER Metric The third cell imports the HuggingFace WER evaluation metric. Set the third cell to: ### CELL 3: Load WER metric ###
wer_metric = load_metric("wer") WER will be used to measure the performance of the finetuned model on test data. Step 2.4 - CELL 4: Setting Constants The fourth cell sets constants that will be used throughout the notebook. Set the fourth cell to: ### CELL 4: Constants ###
# Testing data
TEST_DATA_PATH_MALE = "/kaggle/input/google-spanish-speakers-peru-male/"
TEST_DATA_PATH_FEMALE = "/kaggle/input/google-spanish-speakers-peru-female/"
EXT = ".wav"
NUM_LOAD_FROM_EACH_SET = 3

# Special characters
SPECIAL_CHARS = r"[\d\,\-\;\!\¡\?\¿\।\‘\’\"\–\'\:\/\.\“\”\৷\…\‚\॥\\]"

# Sampling rates
ORIG_SAMPLING_RATE = 48000
TGT_SAMPLING_RATE = 16000 Step 2.5 - CELL 5: Utility Methods for Reading Index Files, Cleaning Text, and Creating Vocabulary The fifth cell defines utility methods for reading the dataset index files, as well as for cleaning transcription text and generating a random set of samples from test data. Set the fifth cell to: ### CELL 5: Utility methods for reading index files, cleaning text, random indices generator ###
def read_index_file_data(path: str, filename: str):
    data = []
    with open(path + filename, "r", encoding = "utf8") as f:
        lines = f.readlines()
        for line in lines:
            file_and_text = line.split("\t")
            data.append([path + file_and_text[0] + EXT, file_and_text[1].replace("\n", "")])
    return data

def clean_text(text: str) -> str:
    cleaned_text = re.sub(SPECIAL_CHARS, "", text)
    cleaned_text = cleaned_text.lower()
    return cleaned_text

def get_random_samples(dataset: list, num: int) -> list:
    used = []
    samples = []
    for i in range(num):
        a = -1
        while a == -1 or a in used:
            a = math.floor(len(dataset) * random.random())
        samples.append(dataset[a])
        used.append(a)
    return samples The method reads a dataset index file and produces a list of lists with audio filename and transcription data, e.g.: read_index_file_data line_index.tsv [
    ["/kaggle/input/google-spanish-speakers-chile-male/clm_08421_01719502739", "Es un viaje de negocios solamente voy por una noche"]
    ...
] The method is used to strip each text transcription of the characters specified by the regular expression assigned to in . These characters, inclusive of punctuation, can be eliminated as they don't provide any semantic value when training the model to learn mappings between audio features and text transcriptions. clean_text SPECIAL_CHARS Step 2.4 The method returns a set of random test samples with the quantity set by the constant in . get_random_samples NUM_LOAD_FROM_EACH_SET Step 2.4 Step 2.6 - CELL 6: Utility Methods for Loading and Resampling Audio Data The sixth cell defines utility methods using to load and resample audio data. Set the sixth cell to: torchaudio ### CELL 7: Utility methods for loading and resampling audio data ###
def read_audio_data(file):
    speech_array, sampling_rate = torchaudio.load(file, normalize = True)
    return speech_array, sampling_rate

def resample(waveform):
    transform = torchaudio.transforms.Resample(ORIG_SAMPLING_RATE, TGT_SAMPLING_RATE)
    waveform = transform(waveform)
    return waveform[0] The method loads a specified audio file and returns a multi-dimensional matrix of the audio data along with the sampling rate of the audio. All the audio files in the training data have a sampling rate of Hz. This "original" sampling rate is captured by the constant in . read_audio_data torch.Tensor 48000 ORIG_SAMPLING_RATE Step 2.4 The method is used to downsample audio data from a sampling rate of to the target sampling rate of . resample 48000 16000 Step 2.7 - CELL 7: Reading Test Data The seventh cell reads the test data index files for the recordings of male speakers and the recordings of female speakers using the method defined in . Set the seventh cell to: read_index_file_data Step 2.5 ### CELL 7: Read test data ###
test_data_male = read_index_file_data(TEST_DATA_PATH_MALE, "line_index.tsv")
test_data_female = read_index_file_data(TEST_DATA_PATH_FEMALE, "line_index.tsv") Step 2.8 - CELL 8: Generating Lists of Random Test Samples The eighth cell generates sets of random test samples using the method defined in . Set the eighth cell to: get_random_samples Step 2.5 ### CELL 8: Generate lists of random test samples ###
random_test_samples_male = get_random_samples(test_data_male, NUM_LOAD_FROM_EACH_SET)
random_test_samples_female = get_random_samples(test_data_female, NUM_LOAD_FROM_EACH_SET) Step 2.9 - CELL 9: Combining Test Data The ninth cell combines the male test samples and female test samples into a single list. Set the ninth cell to: ### CELL 9: Combine test data ###
all_test_samples = random_test_samples_male + random_test_samples_female Step 2.10 - CELL 10: Cleaning Transcription Test The tenth cell iterates over each test data sample and cleans the associated transcription text using the method defined in . Set the tenth cell to: clean_text Step 2.5 ### CELL 10: Clean text transcriptions ###
for index in range(len(all_test_samples)):
    all_test_samples[index][1] = clean_text(all_test_samples[index][1]) Step 2.11 - CELL 11: Loading Audio Data The eleventh cell loads each audio file specified in the list. Set the eleventh cell to: all_test_samples ### CELL 11: Load audio data ###
all_test_data = []

for index in range(len(all_test_samples)):
    speech_array, sampling_rate = read_audio_data(all_test_samples[index][0])
    all_test_data.append({
        "raw": speech_array,
        "sampling_rate": sampling_rate,
        "target_text": all_test_samples[index][1]
    }) Audio data is returned as a and stored in as a list of dictionaries. Each dictionary contains the audio data for a particular sample, the sampling rate, and the text transcription of the audio. torch.Tensor all_test_data Step 2.12 - CELL 12: Resampling Audio Data The twelfth cell resamples audio data to the target sampling rate of . Set the twelfth cell to: 16000 ### CELL 12: Resample audio data and cast to NumPy arrays ###
all_test_data = [{"raw": resample(sample["raw"]).numpy(), "sampling_rate": TGT_SAMPLING_RATE, "target_text": sample["target_text"]} for sample in all_test_data] Step 2.13 - CELL 13: Initializing Instance of Automatic Speech Recognition Pipeline The thirteenth cell initializes an instance of the HuggingFace library class. Set the thirteenth cell to: transformer pipeline ### CELL 13: Initialize instance of Automatic Speech Recognition Pipeline ###
transcriber = pipeline("automatic-speech-recognition", model = "YOUR_FINETUNED_MODEL_PATH") The parameter must be set to the path to finetuned model added to the Kaggle Notebook in , e.g.: model your Step 1.3 transcriber = pipeline("automatic-speech-recognition", model = "/kaggle/input/xls-r-300m-chilean-spanish/transformers/hardy-pine/1") Step 2.14 - CELL 14: Generating Predictions The fourteenth cell calls the initialized in the previous step on the test data to generate text predictions. Set the fourteenth cell to: transcriber ### CELL 14: Generate transcriptions ###
transcriptions = transcriber(all_test_data) Step 2.15 - CELL 15: Calculating WER Metrics The fifteenth cell calculates WER scores for each prediction as well as an overall WER score for all predictions. Set the fifteenth cell to: ### CELL 15: Calculate WER metrics ###
predictions = [transcription["text"] for transcription in transcriptions]
references = [transcription["target_text"][0] for transcription in transcriptions]

wers = []
for p in range(len(predictions)):
    wer = wer_metric.compute(predictions = [predictions[p]], references = [references[p]])
    wers.append(wer)

zipped = list(zip(predictions, references, wers))
df = pd.DataFrame(zipped, columns=["Prediction", "Reference", "WER"])
wer = wer_metric.compute(predictions = predictions, references = references) Step 2.16 - CELL 16: Printing WER Metrics The sixteenth and final cell simply prints the WER calculations from the previous step. Set the sixteenth cell to: ### CELL 16: Output WER metrics ###
pd.set_option("display.max_colwidth", None)
print(f"Overall WER: {wer}")
print(df) WER Analysis Since the notebook generates predictions on random samples of test data, the output will vary each time the notebook is run. The following output was generated on a run of the notebook with set to for a total of 6 test samples: NUM_LOAD_FROM_EACH_SET 3 Overall WER: 0.013888888888888888
                                                                              Prediction  \
0                                     quiero que me reserves el mejor asiento del teatro   
1                                         el llano en llamas es un clásico de juan rulfo   
2         el cuadro de los alcatraces es una de las pinturas más famosas de diego rivera   
3                           hay tres cafés que están abiertos hasta las once de la noche   
4  quiero que me recomiendes una dieta pero donde uno pueda comer algo no puras verduras   
5        cuántos albergues se abrieron después del terremoto del diecinueve de setiembre   

                                                                               Reference  \
0                                     quiero que me reserves el mejor asiento del teatro   
1                                         el llano en llamas es un clásico de juan rulfo   
2         el cuadro de los alcatraces es una de las pinturas más famosas de diego rivera   
3                           hay tres cafés que están abiertos hasta las once de la noche   
4  quiero que me recomiendes una dieta pero donde uno pueda comer algo no puras verduras   
5       cuántos albergues se abrieron después del terremoto del diecinueve de septiembre   

        WER  
0  0.000000  
1  0.000000  
2  0.000000  
3  0.000000  
4  0.000000  
5  0.090909 As can be seen, the model did an excellent job! It only made one error with the sixth sample (index ), mis-spelling the word as . Of course, running the notebook again with different test samples and, more importantly, a larger number of test samples, will produce different and more informative results. Nonetheless, this limited data suggests the model can perform well on different dialects of Spanish - i.e. it was trained on Chilean Spanish, but appears to perform well on Peruvian Spanish. 5 septiembre setiembre Conclusion If you are just learning how to work with wav2vec2 models, I hope that the guide and this guide were useful for you. As mentioned, the finetuned model generated by the Part 1 guide is not quite state-of-the-art, but should still prove useful for many applications. Happy building! Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition

Working with wav2vec2 Part 2 - Running Inference on Finetuned ASR Models

wav2vec2 পার্ট 2-এর সাথে কাজ করা - Finetuned ASR মডেলগুলিতে অনুমান চালানো

Arbeiten mit wav2vec2 Teil 2 - Ausführen von Inferenzen auf fein abgestimmten ASR-Modellen

Trabajar con wav2vec2 Parte 2: Ejecución de inferencia en modelos ASR ajustados

Travailler avec wav2vec2 Partie 2 - Exécution d'inférence sur des modèles ASR affinés

wav2vec2 के साथ कार्य करना भाग 2 - फ़ाइनट्यून्ड ASR मॉडल पर इन्फ़रेंस चलाना

wav2vec2 の操作パート 2 - 微調整された ASR モデルでの推論の実行

wav2vec2 작업 2부 - 미세 조정된 ASR 모델에서 추론 실행

Trabalhando com wav2vec2 Parte 2 - Executando inferência em modelos ASR ajustados

Работа с wav2vec2. Часть 2. Выполнение вывода на точно настроенных моделях ASR

wav2vec2 ile Çalışmak Bölüm 2 - İnce Ayarlı ASR Modellerinde Çıkarım Çalıştırmak

Làm việc với wav2vec2 Phần 2 - Chạy suy luận trên các mô hình ASR đã được tinh chỉnh

使用 wav2vec2 第 2 部分 - 在经过微调的 ASR 模型上运行推理

Working With Wav2vec2 Part 1: Finetuning XLS-R for Automatic Speech Recognition

Wav2vec2 এর সাথে কাজ করা পার্ট 1: স্বয়ংক্রিয় বক্তৃতা স্বীকৃতির জন্য XLS-R ফাইনটিউনিং

Arbeiten mit Wav2vec2 Teil 1: Feinabstimmung von XLS-R für die automatische Spracherkennung

Trabajar con Wav2vec2 Parte 1: Ajuste de XLS-R para el reconocimiento automático de voz

Travailler avec Wav2vec2 Partie 1 : Affiner XLS-R pour la reconnaissance vocale automatique

Wav2vec2 के साथ कार्य करना भाग 1: स्वचालित वाक् पहचान के लिए XLS-R को परिष्कृत करना

Wav2vec2 の操作パート 1: 自動音声認識のための XLS-R の微調整

Wav2vec2 작업 1부: 자동 음성 인식을 위한 XLS-R 미세 조정

Trabalhando com Wav2vec2 Parte 1: Ajuste fino do XLS-R para reconhecimento automático de fala

Работа с Wav2vec2. Часть 1. Точная настройка XLS-R для автоматического распознавания речи

Wav2vec2 ile Çalışmak Bölüm 1: Otomatik Konuşma Tanıma için XLS-R'ye İnce Ayar Yapmak

Làm việc với Wav2vec2 Phần 1: Tinh chỉnh XLS-R để nhận dạng giọng nói tự động

使用 Wav2vec2 第 1 部分：微调 XLS-R 以实现自动语音识别

If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment.
Document what needs to be checked: main user journeys and definitions of success.
Gather on a call or in a room, use the product, discuss, and exchange feedback.
Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. Python interpreter I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000”

From Routine to Rewarding: Integrating ChatGPT Into Product Testing Workflow

In the realm of system debugging, particularly on Linux platforms, strace stands out as a powerful and indispensable tool. Its simplicity and efficacy make it the go-to solution for diagnosing and understanding system-level operations, especially when working with servers and containers. In this blog post, we'll delve into the nuances of strace, from its history and technical functioning to practical applications and advanced features. Whether you're a seasoned developer or just starting out, this exploration will enhance your diagnostic toolkit and provide deeper insights into the workings of Linux systems. https://www.youtube.com/watch?v=bgi7PJXtEzc&embedable=true As a side note, if you like the content of this and the other posts in this series check out my his subject. If you have friends that are learning to code I'd appreciate a reference to my get back to Java after a while check out my Debugging book that covers t Java Basics book. If you want to . Java 8 to 21 book Understanding strace and its Origins A Look Back: strace and dtrace The journey of strace begins with its predecessor, dtrace, . However, dtrace's availability is limited, particularly on Linux systems where most server and container debugging takes place. This is where strace comes into the picture, offering a simpler yet effective alternative. which we covered last time Originating from Sun Microsystems strace, like dtrace, traces its roots back to Sun Microsystems, emerging in the 90s (a decade before dtrace). This isn't surprising given the impressive array of technologies that originated from Sun. However, strace differentiates itself by its straightforwardness in both usage and capabilities. Unlike DTrace, which demands deep operating system support and thus remained absent as an official feature in common Linux distributions, strace thrives in the Linux environment. Its simplicity and ease of implementation make it a popular choice for Linux users, offering a distinct approach to system diagnostics. Technical Functioning of strace The Role of ptrace in strace The cornerstone of strace's functionality is the ptrace kernel feature. ptrace, pre-existing in Linux, spares users from the need to add additional kernel code or modules, a requirement often associated with DTrace. This fundamental difference not only simplifies the use of strace but also broadens its accessibility. Comparing with DTrace While DTrace offers a more in-depth analysis through deeper kernel support, strace operates on a more surface level. This simplicity, however, does not undermine its effectiveness. strace works essentially by logging every kernel call made by a process, providing verbose but incredibly detailed insights into the system's operation. This method allows users to trace the inner workings of a process, understanding each interaction with the kernel. Practical Usage and Advantages Ease of Use and Accessibility One of the most appealing aspects of strace is its user-friendly nature. It doesn't require special privileges or complex setup procedures. This ease of use is particularly beneficial for developers and system administrators who need to quickly diagnose and address issues in a Linux environment. Unlike DTrace, strace is readily available and doesn’t demand advanced configurations or permissions. Favored in Linux Environments strace's popularity in Linux circles is not only due to its accessibility but also its practicality. Being able to run without special privileges makes it a go-to tool for diagnosing various system-related issues. However, it's important to note that strace should be used cautiously in production environments. Its extensive logging can create a significant performance overhead, potentially impacting the efficiency of a live system. This is why strace is generally recommended for use in development or isolated testing environments rather than in production. strace in Action: A Closer Look at System Calls Basic Usage and Output Analysis Using strace is straightforward: you simply pass the command line to it. strace java -classpath . PrimeMain This simplicity belies its power, as the output offers a wealth of information. Each line in the strace output corresponds to a system call made by the process as you can see below: execve("/home/ec2-user/jdk1.8.0_45/bin/java", ["java", "-classpath.", "PrimeMain"], 0x7fffd689ec20 /* 23 vars */) = 0
brk(NULL)                               = 0xb85000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0294272000
readlink("/proc/self/exe", "/home/ec2-user/jdk1.8.0_45/bin/j"..., 4096) = 35
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64", 0x7fff37af09a0) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls", 0x7fff37af09a0) = -1 ENOENT (No such file or directory) By analyzing these calls, users can gain insights into the intricate operations of their applications. For instance, if a Java process attempts to load a library and fails, strace can reveal the underlying system call and its exit code, providing clues about potential issues like missing files or directories. E.g. in this line: COPY COPY open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) Java tries to load the library from the directory using a system call open to load the file. The exit code of the system call is which means that the file isn't there. Under normal circumstances, we should get back a file descriptor value from this API (positive non-zero integer). Looking in the directory, it seems the directory is missing. I'm guessing that this is because of a missing (Java Cryptography Extensions) installation. This is probably OK but might have been interesting in some cases. pthread tls -1, tls JCE Interpreting System Calls for Debugging The output of strace, while verbose, is a goldmine for troubleshooting. For example, a negative exit code in a system call indicates an error, such as a missing file, which could be crucial for diagnosing issues in an application. This level of detail, although overwhelming at times, is invaluable for understanding the interactions between your application and the Linux system. Advanced Features and Tips Filtering System Calls for Efficiency A common challenge with strace is managing its voluminous output. Fortunately, strace offers options to filter system calls, significantly enhancing its usability. By using the argument, you can instruct strace to log only specific types of system calls, such as or e.g.: -e open connect strace -e open java -classpath . PrimeMain This selective logging not only makes the output more manageable but also allows for focused troubleshooting, speeding up the debugging process. Exploring a Variety of System Calls strace's utility extends beyond just tracking file access or network interactions. It can be used to monitor a range of system calls, offering insights into various aspects of application behavior. By understanding and utilizing different system calls, users can gain a comprehensive view of their application's interaction with the operating system, leading to more effective debugging and optimization. strace and Java: A Special Case strace with the JVM While strace predates Java and operates at a low level with no specific awareness of the Java Virtual Machine (JVM), it remains highly effective for debugging Java applications. The JVM, like most platforms, relies on system calls for its operations, which strace can monitor and report. However, certain aspects of the JVM's behavior may be less visible to strace due to its unique approach to problem-solving. Allocations and Threading in Java For instance, Java's memory management differs significantly from standard system tools. While typical applications use malloc, which directly maps to kernel allocation logic, Java manages its own memory. This approach, aimed at efficiency and streamlined garbage collection, means that some memory allocation activities are obscured from strace's view. Similarly, Java threading is currently well-represented in strace output, but this is changing with Java 21 and Project Loom. Java 21 added support for Virtual Threads which are only partially visible to the operating system hence 1,000 threads can seem like 16 threads. These changes could affect the clarity of strace outputs in complex, heavily threaded Java applications. Final Word strace stands out as an exceptionally versatile and powerful tool in the Linux debugging arsenal. Its ability to provide detailed insights into system calls makes it invaluable for diagnosing and understanding the inner workings of applications. Despite its simplicity, strace is capable of handling complex debugging scenarios, especially when used with its advanced filtering options. For developers and system administrators working in Linux environments, strace is more than just a diagnostic tool; it's a lens through which the intricate interactions between applications and the operating system can be viewed and understood. As technologies evolve, tools like strace adapt, continuing to offer relevant and critical insights into system behaviors. Whether you are troubleshooting a stubborn issue or simply curious about how your applications interact with the Linux kernel, strace is a tool that you will likely find yourself returning to time and again.

Revisiting Strace: Simple is Beautiful

306 Stories To Learn About Artificial Intelligence

AI for Knowledge Management: Iterating on RAG with the QE-RAG Architecture

জ্ঞান ব্যবস্থাপনার জন্য AI: QE-RAG আর্কিটেকচারের সাথে RAG-তে পুনরাবৃত্তি করা

KI für Wissensmanagement: Iteration auf RAG mit der QE-RAG-Architektur

IA para la gestión del conocimiento: iteración en RAG con la arquitectura QE-RAG

IA pour la gestion des connaissances : itérer sur RAG avec l'architecture QE-RAG

ज्ञान प्रबंधन के लिए एआई: क्यूई-आरएजी आर्किटेक्चर के साथ आरएजी पर पुनरावृत्ति

ナレッジ管理のための AI: QE-RAG アーキテクチャによる RAG の反復

지식 관리를 위한 AI: QE-RAG 아키텍처를 사용한 RAG 반복

IA para Gestão do Conhecimento: Iterando em RAG com a Arquitetura QE-RAG

ИИ для управления знаниями: итерация RAG с архитектурой QE-RAG

Bilgi Yönetimi için Yapay Zeka: QE-RAG Mimarisi ile RAG'ı Yineleme

AI cho Quản lý Tri thức: Lặp lại trên RAG với Kiến trúc QE-RAG

AI 知识管理：使用 QE-RAG 架构迭代 RAG

Stories, News & Mentions about REALLINKS

Reallinks’s Stories on HackerNoon