We provide technology backed solutions to companies that turn referrals into a primary hiring source and allow them to manage and grow their employees in a highly effective way

Real Links

reallinks

172 Stories To Learn About Ruby

Today I would like to introduce to you the blockchain project Hydrogen. The thing that caught my attention about this project was an article on TD Bank a couple of weeks ago. This article wasn’t actually about Hydrogen, it was about TD Bank buying over another financial institution. But in that article, there was one sentence that said “The bank is partnering with U.S. firm Hydrogen on these products”. That got my attention. TD bank is the world’s 13th largest bank with an asset value of over 1.2 Trillion dollars. That’s 4x the entire crypto market asset worth. If this piece of news was true, then Hydrogen would have itself a partnership that was worth more than the partnership of EOS, ADA, LTC and TRX combined. It would be as big as Stellar and IBM’s partnership and we all know what that announcement did for Stellar’s token price last year when that was announced. The news of that one partnership literally pushed Stellar into the top 10 cryptocurrencies. I honestly believe the actual news was a leak. It was not meant to be announced because neither Hydrogen nor TD Bank had made such an announcement. I believe the person writing the article leaked a news that he shouldn’t have. Now sometimes, after a news is leaked, e.g. a partnership announced beforehand or a coin being listed on an exchange beforehand etc.. the result can be devastating and the whole deal could even be called off, simply because the news was leaked unprofessionally. So in this case, I wanted to see what would happen and have been keeping a close eye on the situation. Quite recently, on their Telegram group, the team pinned a message that said “Other upcoming events: TD Bank PR announcement by end of July, FINDI Launch — we already have a $20+ Bil company signed up. Big PR possible!” So I believe the TD Bank partnership is legit news and they are possibly getting other big partnerships on board as well. In other words, this project is probably going to get a lot of momentum in the very near future. 25% of all Blockchain projects currently revolve around the financial industry. Hydrogen describes itself as “the FinTech Blockchain”. It’s a rather generic description that makes it hard to know how this project stands out from other financial Blockchain projects. But Hydrogen is actually a very unique project in the way it approaches the Fintech scene. Most financial based Blockchain e.g. Ripple, Stellar aim to improve the financial transaction itself. They aim to introduce a cheaper and faster way to do cross-border payments etc… But the difficulty they face is firstly scalability because the Blockchain is primary made for security not speed and secondly they face difficulty of mass adoption because using their technology demands that the financial company e.g. the bank has to make huge adjustments to their current practice to use Blockchain technology. Hydrogen though is a second layer solution to financial transactions that adds an additional layer of super security and open ledger to all the transactions, after the bank has done it’s thing. In other’s words, it doesn’t interfere at all with how they bank currently runs, it just adds a second layer of security. It’s non-intrusive, easy to adopt, and that’s why banks and other major financial institutions will find it easy to use. It’s providing the benefits of the Blockchain decentralised open ledger, with zero disruption to existing current practices. To understand Hydrogen, you need to understand 2 aspects of the project, the Technology and the APIs. The technology of Hydrogen has 5 core features called Raindrop, Snowflake, Ice, Tide and Mist. These are the 5 technologies they are bringing to provide the advantages of Blockchain technology to financial institutions. Raindrop is a protocol that enables single of multi-factor authentication process. This authentication process helps to prevent system breaches and data compromises. This feature is already up and running. It’s a rather complicated feature that comprises of both a server side validation and a client side validation. Think of it as validating an account before doing a transaction. So for example, if I wanted to send a large sum of money to your crypto wallet. What I might do is send a micro-transaction, a small sum to your wallet, initially to confirm the transaction has gone through before sending the larger sum. In a similar but much more complicated way, raindrop does it, but as the name raindrop suggests, the protocol validates very specific transactions that are unpredictable, like the raindrops in a storm. And rather than sending the user and amount directly and having it sent back, Hydro defines a transaction and the user must execute it from a known wallet. This can only be done by having direct access to the wallet in question. Currently the safest way most people know to do financial transactions, even us using crypto exchanges, is the 2 factor authentication. But as you can see, Raindrop as an authentication feature, has so many additional features, google authentication can’t even compare with it. I believe security of accounts is paramount, and people are willing to spend to protect their account and their assets. We’ve seen 2FA take off over the last decade, and I think we will see Blockchain authentication solutions like Raindrop take off over the next decade. The next core feature is Snowflake. This is basically an identity management system. So just as other projects such as Ontology, TheKey or SelfKey uses the immutable untamperable nature of the Blockchain to create identity for individual and institutions, Snowflake does the same to create unique identities for each participant. Unlike ID cards or passports which can be faked, or digital IDs which can be hacked and tampered, Blockchain identities are decentralised, open ledger, meaning they can never be hacked or tampered with, a Blockchain identity is even more accurate and secure than a passport. So just as every Snowflake in real life is unique and different in pattern from another, this Snowflake technology creates and uses unique digital fingerprints for users. Both Raindrop and Snowflakes are working products and each have their own whitepaper which is accessible from the Hydrogen website. The other three technologies Ice, Tide and Mist are not released yet. But Ice will be a service to stamp and record contracts on the Blockchain, Tide will extend payment solutions to the unbanked and Mist aims to integrate AI into this Blockchain project to utilise machine learn and network optimisation. And that’s the technology of Hydrogen in a nutshell, an impressive technology with a wide array of features that is very practical to the financial industry. The way that Hydrogen makes this technology usable by the banks and financial institutions, is by creating what is known as the Hydrogen Atom. The Atom is basically a platform on which API modules are built. APIs are not applications. They are applications for applications. So your mobile phone will have various applications. One of those applications might be Ubereats. Ubereats will have different features e.g. payment service, restaurant menu you can choose the food from, GPS to track the delivery etc.. each of these features, are known as an API. Hydrogen doesn’t ask banks to discard their existing software and learn a new program. Instead, they create APIs, to integrate into existing software. So again very minimal disruption and a very low barrier to adoption. The APIs they are creating include: Nucleus APIs — These are APIs that provides the core digital infrastructure to the Fintech Application. They include services such as authentication, storing and encrypting client information, management of client accounts and more… Proton APIs which provides financial engineering to the Fintech Application e.g. Risk scoring, goals tracking, simulations and more Electron APIs which provide functionality to grow the Fintech Application e.g. Support, Marketing, Billings etc.. So it’s basically a very holistic platform that covers all the essential needs of a Fintech company. The entire project is also designed to have a lot of token use, which is great for token demand and price. Raindrop will use Hydro tokens to pay for authentication attempts, Snowflake will require Hydro Tokens by companies that use the identity data, Ice will use Hydro tokens to pay for the stamping of contracts, Tide and Mist we will have to wait and see as those features are not due until 2019, but they are planned to integrate Token use as well. The Hydro technology is run by smart contracts, and the easiest way for companies to access these smart contracts is by using the APIs. But should they not wish to install or use the APIs, they can still use the smart contracts, by “staking” Hydro tokens, and what this does is it locks the tokens into an escrow contract that fuels the smart contracts. So it’s a good project, with good technology and good token use. This is the team behind the project. One thing I don’t like about their website is there is not resume or links to the resumes for any of their team members. Almost every other website gives a little spill or link to linkedin for the staff. But this profile page only has a few faces and names. You can find their resumes by searching them up individually, but it’s pretty troublesome. The Co-founders are twin brothers, Michael and Matthew Kane. They previously founded a company called Hedgeable in 2009, which has since won Finovate best of show award in 2015 and 2017 as well as UK great tech awards top Fintech 2015. So it’s good to know they have previous success and experience in the Fintech space. Hedgeable their previous projects has also attracted Venture Capital funding, so I’m hoping Hydrogen will at some point attrack similar VC attention. For now, their website does not list any advisors, partners or investors. Which is a bit of a downer for me personally, because 25% of all Blockchain projects are focused in the financial industry. That’s about 500 projects. With dominant big projects like Ripple and Stella, it’s going to take more than just a good technology for a project to stand out from the rest. Good advisors and investors do more than give advice and money to the project, they are prominent, successful individuals in the industry who have a personal stake to see the project succeed and many times we’ve seen connections and in-roads to big partnerships happen because of these personal relationships. So I think these are important aspects of any project to consider and it would good of Hydro to update these details when they can. Of course the upcoming partnership with TD Banks, itself is huge news and that’s an outstanding partnership I expect to have a big impact on this project and it’s standing in the crypto market. The other thing the project doesn’t have is a clear roadmap for the whole project. This is the roadmap for their Technology, and it’s simple and easy to read. But a project is more than this, it would include plans for marketing milestones, getting the APIs out etc… So whilst we can appreciate the technology of the project is expected to be completed in late 2019, I think the website is lacking a proper roadmap to help provide accountability and expectation to the community. The only other consideration I have for this project is regarding scalability. As we mentioned before, Hydrogen is a second layer security solution to financial transactions. It doesn’t need to have super high throughput, e.g. Ripple, Stellar etc… but it still needs to have a decent throughput if it is to be used by multiple big financial institutions. The way I think of it is like a cashier and book keeper. Ripple and Stellar and the other projects that focus on doing the actual transactions are like cashiers who need to have high speed. The book keeper is the person who processes the record of the transaction after it is done. It doesn’t need to be super fast, but it still needs to be fairly fast in order to keep up with the volume of work. Hydrogen is built on the Ethereum Blockchain which I think is a terrible Blockchain for Dapps at the moment. It’s entire scalability capability is dependent or limited by how scalable Ethereum is. The best they can offer is that Ethereum has proposed to implement Plasma or Raiden. But the thing is Ethereum has been looking at sharding solutions for years, and nothing has been practically achieved yet. It is still very very slow. Look at what happened over the past few weeks, with one exchange Fcoin flooding the network, the entire Ethereum network came to a crawl and gas wars erupted. Price of doing a transaction was ridiculously high and timings for transactions were as slow as 3 hours. Ethereum has not gotten better compared to cryptokitties 8 months ago. It is still very very slow. I also think alot of the practical scaling solutions that I do see workable on Ehtereum are the second layer solutions e.g. Loom, OMG etc.. but these will benefit new projects being built, not so much project that are already built on Ethereum. Just because a new project is able to implement Plasma, doesn’t mean Ethereum can also use Plasma. The difference is that Ethereum has a thousand Dapps built on it and running, to implement any scaling solution to Ethereum is like trying to upgrade a train engine whilst it is running at full speed. It’s incredibly hard. Im not saying it’s impossible, Im just saying it’s very hard. So when projects say in their white paper Vitalik has proposed Plasma or Raiden Network as a scaling solution, I don’t buy it. That’s not enough to put my mind at ease. Il believe when I see it actually working and Ethereum actually having cheap and fast transactions. Until then, any project that is restricted by Ethereum in terms of scalability is not out of the woods. The other thing they mention is at this time, raindrop will put very minimal strain on the Ethereum framework. That’s fair enough. But the key phases are “at this time.” And “minimal strain”. With a mega partnership coming on board and the other features expected to roll out by end of 2019, at some point this project will be a significant volume on Ethereum. Furthermore, it doesn’t matter if Raindrop doesn’t put much strain on the Ethereum network. What matters is that Raindrops depends on the Ethereum network to function. Which means, even though they put very little strain on the Ethereum network, all you need is one joker like Fcoin to flood and clog the system up for everyone using the system, whether it’s Hydro or you and me doing a simple Ethereum transaction to suffer. So my concern is not with this project and it’s technology. I think the technology is great. My concern is with Ethereum and how bad the network is at the moment, and the fact that Hydro is working on the network. They are a working product and need to function well now, not in the future when Ethereum can finally scale, but now when Ethereum has not yet scaled, so personally, I think they would be better off moving to an Ethereum compatible platform. That’s just my very unprofessional and layman’s way of looking at things. Conclusion All in all, I think Hydro is a project with alot of potential in the Fintech space and worth paying attention to, definitely worth knowing about in the crypto space. None of this is financial advice, just my personal thoughts on this project. Always do your own homework and make your own decisions. Drop a comment below, let me know what your thoughts on Hydro is, and if you agree or disagree with what my thoughts. I’m not always right, and I would love to hear your thoughts on this one. Thank you so much for reading this. If you liked this, give it that clap and subscribe, we have some other great projects coming up for review, so make sure you follow us for that. It’s Friday, I hope you guys have a great start to the weekend, enjoy yourselves and catch you guys again very soon!

Hydrogen (HYDRO) Review: Seamless integration of Blockchain into Fintech

Hard links and symbolic links have been available since time immemorial, and we use them all the time without even thinking about it. In machine learning projects they can help us, when setting up new experiments, to rearrange data files quickly and efficiently in machine learning projects. However, with traditional links, we run the risk of polluting the data files with erroneous edits. In this blog post we’ll go over the details of using links, some cool new stuff in modern file systems (reflinks), and an example of how DVC (Data Version Control, ) leverages this. https://dvc.org/ As I am studying machine learning, I’m wishing for a tool that would let us inspect ML projects the way we do regular software engineering projects. That is, to retrieve the state of the project at any given time, create branches or tags (with Git) based on an earlier state of a project, handle collaboration with colleagues, and so on. What makes ML projects different is the tremendous amount of data, thousands of images, audio, or video files, the trained models, and how difficult it is to manage those files with regular tools like Git. In my earlier articles I went over why , and why Git-LFS is not a solution for machine learning projects, as well as some . Git by itself is insufficient principles that seem to be useful for tools to manage ML projects DVC has proved to be very good at managing ML project datasets and workflow. It works hand-in-hand with Git, and can show you the state of the datasets corresponding to any Git commit. Simply by checking out a commit, DVC can rearrange the data files to exactly match what was present at the time of that commit. The speed is rather magical, considering that potentially many gigabytes of data are being rearranged nearly instantaneously. So I was wondering: How does DVC pull off this trick? The trick to rearranging gigabytes of training data Turns out DVC’s secret to rearranging data and model files as quickly as Git is to link files rather than copy them. Git, of course, copies files into place when it checks out a commit, but Git typically deals with relatively small text files, as opposed to the large binary blobs used in ML projects. Linking a file, like DVC does, is incredibly fast, making it possible to rearrange any amount of files in the blink of an eye, while avoiding copying, thus saving disk space. Using file linking techniques is nothing new to the field, actually. Some data science teams use symlinks to save space and avoid copying large datasets. But symlinks are not the only sort of link which can be used. We will start with a strategy of copying files into place, then using hard links and symbolic links, then ending up with a new type of link, reflinks, which implements in the file system. We will use DVC as an example of how tools can use different linking strategies. Copy On Write capabilities Test setup Since we’ll be testing different linking strategies, we need a sample workspace. The workspace was setup on my laptop, a MacBook Pro where the main drive is formatted with the file system. Further tests were done on Linux on a drive formatted with . APFS XFS The data used is two “stub-articles” dumps of the Wikipedia website retrieved from two different days. Each is about 38 GB of XML, giving us enough data to be similar to an ML project. We then set up a Git/DVC workspace where one can switch between these two files by checking out different Git commits. $ ls -hl wikidatawiki-20190401-stub-articles.xml
-rw-r--r--  1 david  staff    35G Jul 20 21:35 wikidatawiki-20190401-stub-articles.xml
$ time cp wikidatawiki-20190401-stub-articles.xml wikidatawiki-stub-articles.xml

real    14m16.918s
... As a base-line measure we’ll note that copying these files into the workspace took about 15 minutes apiece. Obviously an ML researcher would not have a pleasant life if it took 15 minutes to switch between commits in the repository. Instead, we’ll be exploring another technique DVC and some other tools utilize - linking. There are two types of links all modern OSs support: Hard Link, Symbolic Link. A new type of link, the Reflink (copy-on-write), is starting to be available in newer releases of Mac OS X and Linux (for which one needs the desired filesystem driver). We’ll use each of them in turn and see how well they work. DVC discusses corresponding to the three link types (the 4th strategy is to just copy files) in its documentation. The strategy used depends on the file system capabilities, and whether the "dvc config cache.type" command has changed the configuration. the four strategies DVC defaults to using reflinks and, if not available, to fall back to file copying. It avoids using symlinks and hardlinks because of the risk of accidental cache or repository corruption. We’ll see all this in the coming sections. Versioned datasets using File Copying The basic (or naive) strategy of copying files into place when checking out a Git tag is the equivalent of these commands: $ rm data/wikidatawiki-stub-articles.xml
$ cp .dvc/cache/ / c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml 40 58 This will run on any filesystem, but it will take a long time to copy the file, and it will consume twice the disk space. What’s with the strange file name? It’s the filename within the DVC cache, the hex digits being the MD5 checksum. Files in the DVC cache are indexed by this checksum, allowing there to be multiple versions of the same file. The contains more details about the DVC cache implementation. DVC documentation In practice this is how it works in DVC: To set up the workspace we created two Git tags, one for each file we downloaded. The ( ) should give you an idea of what’s involved with setting up the workspace. To test different file copying/linking modes we first change the DVC configuration then check out the given Git commit. Running "dvc checkout" causes the corresponding data file to be inserted into the directory. DVC example on versioned datasets or this alternate tutorial Oh boy, that sure took a long time. The Git portion of this is very fast, but DVC took a very long time. This is as expected, since we told DVC to perform a file copy, and we already knew it took about 16 minutes or so to use the cp command to copy the file. As for disk space, obviously there are now two copies of the data file. There is the copy in the DVC cache directory, and the other that was copied into the workspace. Source: https://www.xkcd.com/981/ Versioned datasets using Hard Links and Symlinks Clearly copying files around to handle dataset versioning is slow and an inefficient use of disk space. An option that has existed since time immemorial in Unix-like environments are both hard links and symbolic links. While Windows historically did not support file links, the " " can do both styles of links as well. mklink Hard links are a byproduct of the Unix model for file systems. What we think of as the filename is really just an entry in a directory file. The directory file entry contains the file name, and the “inode number” which is simply an index into the inode table. Inode table entries are data structures containing file attributes, and a pointer to the actual data. A hard link is simply two directory entries with the same inode number. In effect, it is the exact same file appearing at two locations in the filesystem. Hard links can only be made within a given mounted volume. A symbolic link is a special file where the attributes contains a pathname specifying the target of the link. Because it contains a pathname, symbolic links can point to any file in the filesystem, even across mounted volumes or across network file systems. The equivalent commands in this case are: $ rm data/wikidatawiki-stub-articles.xml
$ ln .dvc/cache/40/58c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml This is for a hard link. For a symbolic link use " ". ln -s Then to perform the hard link scenario: The symbolic link scenario is the same, but setting to . The timing is similar for both cases. cache.type symlink Two seconds (or less) is sure a lot faster than the 16 minutes or so it took to copy the files. It happens so fast we use the word “instantaneous”. File links are that much faster than copying files around. This is a big win. As for disk space consumption, consider this: $ ls -l data/
total 8
lrwxr-xr-x  1 david  staff   70 Jul 21 18:43 wikidatawiki-stub-articles.xml -> /Users/david/dvc/linktest/.dvc/cache/2c/82d0130fb32a17d58e2b5a884cd3ce The link takes up a negligible amount of disk space. But there is a wrinkle to consider. Ok, looks great, right? Fast, no extra space consumed … But, let’s think about what would happen if you were to edit data/wikidatawiki-stub-articles.xml in the workspace. Because that file is a link to a file in the DVC cache, the file in the cache would be changed, polluting the cache. You’ll need to take extra measures, and learn how to avoid that problem. The has instructions on avoiding the problem when using DVC. It means always remembering to use a specific process for editing a data file that, while not a deal breaker, is less than convenient. The better option though is to use reflinks. DVC documentation Versioned Datasets using Reflinks Hard links and symbolic links have been in the Unix/Linux ecosystem for a long time. I first used symbolic links in 1984 on 4.2BSD, and hard links date back even further. Both hard links and symbolic links can be used to do what DVC does, namely quickly rearranging data files in a working directory. But surely in the last 35+ years there has been an advancement or two in file systems? Indeed there has, and the Mac OS X “ ” and Linux “ ” features are examples. clonefile reflink Copy On Write links, a.k.a. reflinks, offer a solution to quickly linking a file into the workspace while avoiding any risk of polluting the cache. The hard link and symbolic link approaches are big wins because of their speed, but doing so runs the risk of polluting the cache. With reflinks, the copy-on-write behavior means that if someone were to modify the data file the copy in the cache would not be polluted. That means we’d have the same performance advantage as traditional links, with the added advantage of data safety. Maybe, like me, you don’t know what a reflink is. This technique means to duplicate a file on the disk such that the “copy” is a “clone” similar to a hard link. Unlike a hard link where two directory entries refer to the same inode entry, with reflinks there are two inode entries, and it is the data blocks that are shared. It happens as quickly as a hard link, but there is an important difference. Any write to the cloned file causes new data blocks to be allocated to hold that data. The cloned file appears changed, and the original file is unmodified. The clone is perfectly suitable for the case of duplicating a dataset, allowing modifications to the dataset without polluting the original dataset. Like with hard links, reflinks only work within a given mounted volume. Reflinks are easily available on Mac OS X, and with a little work is available on Linux. This feature is supported only on certain file systems: Linux: BTRFS, XFS, OCFS2 Mac OS X: APFS APFS is supported out of the box on macOS, and Apple strongly suggest we use it. For Linux, XFS is the easiest to set up as shown in this tutorial. For APFS the equivalent commands are: $ rm data/wikidatawiki-stub-articles.xml
$ cp -c .dvc/cache/40/58c95964df74395df6e9e8e1aa6056 data/wikidatawiki-stub-articles.xml With the option, the macOS cp command uses system call. The function sets up a reflink clone of the named file. On Linux the cp command uses the option instead. -c clonefile(2) clonefile --reflink Then to run the test: The performance is, as expected, similar to the hard links and symbolic links strategies. What we learn is that reflinks are about as fast as hard links and symlinks, and disk space consumption is again negligible. The cool stuff about this link is even though the files are connected you can edit the file without modifying the file in the cache. The changed data are copied under the hood. On Linux the same scenario runs with similar performance. Conclusion We’ve learned something about how to efficiently manage a large dataset, like is typical in machine learning projects. If we need to revisit any development stage in such projects, we’ll wanta system for efficiently rearranging large datasets to match each stage. We’ve seen it is possible to keep a list of files that were present at any Git commit. With that list we can link or copy those files into the working directory. That is exactly how DVC manages data files in a project. Using links, rather than file copying, lets us quickly and efficiently switch between revisions of the project. Reflinks are an interesting new feature for file systems, and they are perfect for this scenario. Reflinks are as fast to create as traditional hard links and symbolic links, letting us quickly duplicate a file, or a whole directory structure, while consuming negligible extra space. And, since reflinks keeps modifications in the linked file, they give us many more possibilities than traditional links. In this article we examined using reflinks in machine learning projects, but they are used in other sorts of applications. For example, some database systems utilizing them to manage data on disk more efficiently. Now that you’ve learned about reflinks, how will you go about using them?

Reflinks vs symlinks vs hard links, and  how they can help machine learning projects

Introduction This is a companion guide to (the "Part 1 guide"). I wrote the Part 1 guide on how to finetune Meta AI's model on Chilean Spanish. It is assumed that you have completed that guide and have generated your own finetuned XLS-R model. This guide will explain the steps to run on your finetuned XLS-R model via a . Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition wav2vec2 XLS-R ("XLS-R") inference Kaggle Notebook Prerequisites and Before You Get Started To complete the guide, you will need to have: A finetuned XLS-R model for the Spanish language. An existing . Kaggle account Intermediate knowledge of Python. Intermediate knowledge of working with Kaggle Notebooks. Intermediate knowledge of ML concepts. Basic knowledge of ASR concepts. Building the Inference Notebook Step 1 - Setting Up Your Kaggle Environment Step 1.1 - Creating a New Kaggle Notebook Log in to Kaggle. Create a new Kaggle Notebook. The name of the notebook can be changed as desired. This guide uses the notebook name . spanish-asr-inference Step 1.2 - Adding the Test Datasets This guide uses the as its source for test data. Like the , the Peruvian speakers dataset also consists of two sub-datasets: 2,918 recordings of male Peruvian speakers and 2,529 recordings of female Peruvian speakers. Peruvian Spanish Speech Data Set Chilean Spanish Speech Data Set This dataset has been uploaded to Kaggle as 2 distinct datasets: Recordings of Peruvian male speakers Recordings of Peruvian female speakers Add both of these datasets to your Kaggle Notebook by clicking on . Add Input Step 1.3 - Adding the Finetuned Model You should have saved your finetuned model in of the guide as a . Step 4 Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition Kaggle Model Add your finetuned model to your Kaggle Notebook by clicking on . Add Input Step 2 - Building the Inference Notebook The following 16 sub-steps build each of the inference notebook's 16 cells in order. You will note that many of the same utility methods from the Part 1 guide are used here. Step 2.1 - CELL 1: Installing Packages The first cell of the inference notebook installs dependencies. Set the first cell to: ### CELL 1: Install Packages ###
!pip install --upgrade torchaudio
!pip install jiwer Step 2.2 - CELL 2: Importing Python Packages The second cell imports required Python packages. Set the second cell to: ### CELL 2: Import Python packages ###
import re
import math
import random
import pandas as pd
import torchaudio
from datasets import load_metric
from transformers import pipeline Step 2.3 - CELL 3: Loading WER Metric The third cell imports the HuggingFace WER evaluation metric. Set the third cell to: ### CELL 3: Load WER metric ###
wer_metric = load_metric("wer") WER will be used to measure the performance of the finetuned model on test data. Step 2.4 - CELL 4: Setting Constants The fourth cell sets constants that will be used throughout the notebook. Set the fourth cell to: ### CELL 4: Constants ###
# Testing data
TEST_DATA_PATH_MALE = "/kaggle/input/google-spanish-speakers-peru-male/"
TEST_DATA_PATH_FEMALE = "/kaggle/input/google-spanish-speakers-peru-female/"
EXT = ".wav"
NUM_LOAD_FROM_EACH_SET = 3

# Special characters
SPECIAL_CHARS = r"[\d\,\-\;\!\¡\?\¿\।\‘\’\"\–\'\:\/\.\“\”\৷\…\‚\॥\\]"

# Sampling rates
ORIG_SAMPLING_RATE = 48000
TGT_SAMPLING_RATE = 16000 Step 2.5 - CELL 5: Utility Methods for Reading Index Files, Cleaning Text, and Creating Vocabulary The fifth cell defines utility methods for reading the dataset index files, as well as for cleaning transcription text and generating a random set of samples from test data. Set the fifth cell to: ### CELL 5: Utility methods for reading index files, cleaning text, random indices generator ###
def read_index_file_data(path: str, filename: str):
    data = []
    with open(path + filename, "r", encoding = "utf8") as f:
        lines = f.readlines()
        for line in lines:
            file_and_text = line.split("\t")
            data.append([path + file_and_text[0] + EXT, file_and_text[1].replace("\n", "")])
    return data

def clean_text(text: str) -> str:
    cleaned_text = re.sub(SPECIAL_CHARS, "", text)
    cleaned_text = cleaned_text.lower()
    return cleaned_text

def get_random_samples(dataset: list, num: int) -> list:
    used = []
    samples = []
    for i in range(num):
        a = -1
        while a == -1 or a in used:
            a = math.floor(len(dataset) * random.random())
        samples.append(dataset[a])
        used.append(a)
    return samples The method reads a dataset index file and produces a list of lists with audio filename and transcription data, e.g.: read_index_file_data line_index.tsv [
    ["/kaggle/input/google-spanish-speakers-chile-male/clm_08421_01719502739", "Es un viaje de negocios solamente voy por una noche"]
    ...
] The method is used to strip each text transcription of the characters specified by the regular expression assigned to in . These characters, inclusive of punctuation, can be eliminated as they don't provide any semantic value when training the model to learn mappings between audio features and text transcriptions. clean_text SPECIAL_CHARS Step 2.4 The method returns a set of random test samples with the quantity set by the constant in . get_random_samples NUM_LOAD_FROM_EACH_SET Step 2.4 Step 2.6 - CELL 6: Utility Methods for Loading and Resampling Audio Data The sixth cell defines utility methods using to load and resample audio data. Set the sixth cell to: torchaudio ### CELL 7: Utility methods for loading and resampling audio data ###
def read_audio_data(file):
    speech_array, sampling_rate = torchaudio.load(file, normalize = True)
    return speech_array, sampling_rate

def resample(waveform):
    transform = torchaudio.transforms.Resample(ORIG_SAMPLING_RATE, TGT_SAMPLING_RATE)
    waveform = transform(waveform)
    return waveform[0] The method loads a specified audio file and returns a multi-dimensional matrix of the audio data along with the sampling rate of the audio. All the audio files in the training data have a sampling rate of Hz. This "original" sampling rate is captured by the constant in . read_audio_data torch.Tensor 48000 ORIG_SAMPLING_RATE Step 2.4 The method is used to downsample audio data from a sampling rate of to the target sampling rate of . resample 48000 16000 Step 2.7 - CELL 7: Reading Test Data The seventh cell reads the test data index files for the recordings of male speakers and the recordings of female speakers using the method defined in . Set the seventh cell to: read_index_file_data Step 2.5 ### CELL 7: Read test data ###
test_data_male = read_index_file_data(TEST_DATA_PATH_MALE, "line_index.tsv")
test_data_female = read_index_file_data(TEST_DATA_PATH_FEMALE, "line_index.tsv") Step 2.8 - CELL 8: Generating Lists of Random Test Samples The eighth cell generates sets of random test samples using the method defined in . Set the eighth cell to: get_random_samples Step 2.5 ### CELL 8: Generate lists of random test samples ###
random_test_samples_male = get_random_samples(test_data_male, NUM_LOAD_FROM_EACH_SET)
random_test_samples_female = get_random_samples(test_data_female, NUM_LOAD_FROM_EACH_SET) Step 2.9 - CELL 9: Combining Test Data The ninth cell combines the male test samples and female test samples into a single list. Set the ninth cell to: ### CELL 9: Combine test data ###
all_test_samples = random_test_samples_male + random_test_samples_female Step 2.10 - CELL 10: Cleaning Transcription Test The tenth cell iterates over each test data sample and cleans the associated transcription text using the method defined in . Set the tenth cell to: clean_text Step 2.5 ### CELL 10: Clean text transcriptions ###
for index in range(len(all_test_samples)):
    all_test_samples[index][1] = clean_text(all_test_samples[index][1]) Step 2.11 - CELL 11: Loading Audio Data The eleventh cell loads each audio file specified in the list. Set the eleventh cell to: all_test_samples ### CELL 11: Load audio data ###
all_test_data = []

for index in range(len(all_test_samples)):
    speech_array, sampling_rate = read_audio_data(all_test_samples[index][0])
    all_test_data.append({
        "raw": speech_array,
        "sampling_rate": sampling_rate,
        "target_text": all_test_samples[index][1]
    }) Audio data is returned as a and stored in as a list of dictionaries. Each dictionary contains the audio data for a particular sample, the sampling rate, and the text transcription of the audio. torch.Tensor all_test_data Step 2.12 - CELL 12: Resampling Audio Data The twelfth cell resamples audio data to the target sampling rate of . Set the twelfth cell to: 16000 ### CELL 12: Resample audio data and cast to NumPy arrays ###
all_test_data = [{"raw": resample(sample["raw"]).numpy(), "sampling_rate": TGT_SAMPLING_RATE, "target_text": sample["target_text"]} for sample in all_test_data] Step 2.13 - CELL 13: Initializing Instance of Automatic Speech Recognition Pipeline The thirteenth cell initializes an instance of the HuggingFace library class. Set the thirteenth cell to: transformer pipeline ### CELL 13: Initialize instance of Automatic Speech Recognition Pipeline ###
transcriber = pipeline("automatic-speech-recognition", model = "YOUR_FINETUNED_MODEL_PATH") The parameter must be set to the path to finetuned model added to the Kaggle Notebook in , e.g.: model your Step 1.3 transcriber = pipeline("automatic-speech-recognition", model = "/kaggle/input/xls-r-300m-chilean-spanish/transformers/hardy-pine/1") Step 2.14 - CELL 14: Generating Predictions The fourteenth cell calls the initialized in the previous step on the test data to generate text predictions. Set the fourteenth cell to: transcriber ### CELL 14: Generate transcriptions ###
transcriptions = transcriber(all_test_data) Step 2.15 - CELL 15: Calculating WER Metrics The fifteenth cell calculates WER scores for each prediction as well as an overall WER score for all predictions. Set the fifteenth cell to: ### CELL 15: Calculate WER metrics ###
predictions = [transcription["text"] for transcription in transcriptions]
references = [transcription["target_text"][0] for transcription in transcriptions]

wers = []
for p in range(len(predictions)):
    wer = wer_metric.compute(predictions = [predictions[p]], references = [references[p]])
    wers.append(wer)

zipped = list(zip(predictions, references, wers))
df = pd.DataFrame(zipped, columns=["Prediction", "Reference", "WER"])
wer = wer_metric.compute(predictions = predictions, references = references) Step 2.16 - CELL 16: Printing WER Metrics The sixteenth and final cell simply prints the WER calculations from the previous step. Set the sixteenth cell to: ### CELL 16: Output WER metrics ###
pd.set_option("display.max_colwidth", None)
print(f"Overall WER: {wer}")
print(df) WER Analysis Since the notebook generates predictions on random samples of test data, the output will vary each time the notebook is run. The following output was generated on a run of the notebook with set to for a total of 6 test samples: NUM_LOAD_FROM_EACH_SET 3 Overall WER: 0.013888888888888888
                                                                              Prediction  \
0                                     quiero que me reserves el mejor asiento del teatro   
1                                         el llano en llamas es un clásico de juan rulfo   
2         el cuadro de los alcatraces es una de las pinturas más famosas de diego rivera   
3                           hay tres cafés que están abiertos hasta las once de la noche   
4  quiero que me recomiendes una dieta pero donde uno pueda comer algo no puras verduras   
5        cuántos albergues se abrieron después del terremoto del diecinueve de setiembre   

                                                                               Reference  \
0                                     quiero que me reserves el mejor asiento del teatro   
1                                         el llano en llamas es un clásico de juan rulfo   
2         el cuadro de los alcatraces es una de las pinturas más famosas de diego rivera   
3                           hay tres cafés que están abiertos hasta las once de la noche   
4  quiero que me recomiendes una dieta pero donde uno pueda comer algo no puras verduras   
5       cuántos albergues se abrieron después del terremoto del diecinueve de septiembre   

        WER  
0  0.000000  
1  0.000000  
2  0.000000  
3  0.000000  
4  0.000000  
5  0.090909 As can be seen, the model did an excellent job! It only made one error with the sixth sample (index ), mis-spelling the word as . Of course, running the notebook again with different test samples and, more importantly, a larger number of test samples, will produce different and more informative results. Nonetheless, this limited data suggests the model can perform well on different dialects of Spanish - i.e. it was trained on Chilean Spanish, but appears to perform well on Peruvian Spanish. 5 septiembre setiembre Conclusion If you are just learning how to work with wav2vec2 models, I hope that the guide and this guide were useful for you. As mentioned, the finetuned model generated by the Part 1 guide is not quite state-of-the-art, but should still prove useful for many applications. Happy building! Working With wav2vec2 Part 1 - Finetuning XLS-R for Automatic Speech Recognition

Working with wav2vec2 Part 2 - Running Inference on Finetuned ASR Models

wav2vec2 পার্ট 2-এর সাথে কাজ করা - Finetuned ASR মডেলগুলিতে অনুমান চালানো

Arbeiten mit wav2vec2 Teil 2 - Ausführen von Inferenzen auf fein abgestimmten ASR-Modellen

Trabajar con wav2vec2 Parte 2: Ejecución de inferencia en modelos ASR ajustados

Travailler avec wav2vec2 Partie 2 - Exécution d'inférence sur des modèles ASR affinés

wav2vec2 के साथ कार्य करना भाग 2 - फ़ाइनट्यून्ड ASR मॉडल पर इन्फ़रेंस चलाना

wav2vec2 の操作パート 2 - 微調整された ASR モデルでの推論の実行

wav2vec2 작업 2부 - 미세 조정된 ASR 모델에서 추론 실행

Trabalhando com wav2vec2 Parte 2 - Executando inferência em modelos ASR ajustados

Работа с wav2vec2. Часть 2. Выполнение вывода на точно настроенных моделях ASR

wav2vec2 ile Çalışmak Bölüm 2 - İnce Ayarlı ASR Modellerinde Çıkarım Çalıştırmak

Làm việc với wav2vec2 Phần 2 - Chạy suy luận trên các mô hình ASR đã được tinh chỉnh

使用 wav2vec2 第 2 部分 - 在经过微调的 ASR 模型上运行推理

Working With Wav2vec2 Part 1: Finetuning XLS-R for Automatic Speech Recognition

Wav2vec2 এর সাথে কাজ করা পার্ট 1: স্বয়ংক্রিয় বক্তৃতা স্বীকৃতির জন্য XLS-R ফাইনটিউনিং

Arbeiten mit Wav2vec2 Teil 1: Feinabstimmung von XLS-R für die automatische Spracherkennung

Trabajar con Wav2vec2 Parte 1: Ajuste de XLS-R para el reconocimiento automático de voz

Travailler avec Wav2vec2 Partie 1 : Affiner XLS-R pour la reconnaissance vocale automatique

Wav2vec2 के साथ कार्य करना भाग 1: स्वचालित वाक् पहचान के लिए XLS-R को परिष्कृत करना

Wav2vec2 の操作パート 1: 自動音声認識のための XLS-R の微調整

Wav2vec2 작업 1부: 자동 음성 인식을 위한 XLS-R 미세 조정

Trabalhando com Wav2vec2 Parte 1: Ajuste fino do XLS-R para reconhecimento automático de fala

Работа с Wav2vec2. Часть 1. Точная настройка XLS-R для автоматического распознавания речи

Wav2vec2 ile Çalışmak Bölüm 1: Otomatik Konuşma Tanıma için XLS-R'ye İnce Ayar Yapmak

Làm việc với Wav2vec2 Phần 1: Tinh chỉnh XLS-R để nhận dạng giọng nói tự động

使用 Wav2vec2 第 1 部分：微调 XLS-R 以实现自动语音识别

AI-Powered Localization Using an Open-source Project as a Case Study

If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment.
Document what needs to be checked: main user journeys and definitions of success.
Gather on a call or in a room, use the product, discuss, and exchange feedback.
Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. Python interpreter I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000”

From Routine to Rewarding: Integrating ChatGPT Into Product Testing Workflow

In the realm of system debugging, particularly on Linux platforms, strace stands out as a powerful and indispensable tool. Its simplicity and efficacy make it the go-to solution for diagnosing and understanding system-level operations, especially when working with servers and containers. In this blog post, we'll delve into the nuances of strace, from its history and technical functioning to practical applications and advanced features. Whether you're a seasoned developer or just starting out, this exploration will enhance your diagnostic toolkit and provide deeper insights into the workings of Linux systems. https://www.youtube.com/watch?v=bgi7PJXtEzc&embedable=true As a side note, if you like the content of this and the other posts in this series check out my his subject. If you have friends that are learning to code I'd appreciate a reference to my get back to Java after a while check out my Debugging book that covers t Java Basics book. If you want to . Java 8 to 21 book Understanding strace and its Origins A Look Back: strace and dtrace The journey of strace begins with its predecessor, dtrace, . However, dtrace's availability is limited, particularly on Linux systems where most server and container debugging takes place. This is where strace comes into the picture, offering a simpler yet effective alternative. which we covered last time Originating from Sun Microsystems strace, like dtrace, traces its roots back to Sun Microsystems, emerging in the 90s (a decade before dtrace). This isn't surprising given the impressive array of technologies that originated from Sun. However, strace differentiates itself by its straightforwardness in both usage and capabilities. Unlike DTrace, which demands deep operating system support and thus remained absent as an official feature in common Linux distributions, strace thrives in the Linux environment. Its simplicity and ease of implementation make it a popular choice for Linux users, offering a distinct approach to system diagnostics. Technical Functioning of strace The Role of ptrace in strace The cornerstone of strace's functionality is the ptrace kernel feature. ptrace, pre-existing in Linux, spares users from the need to add additional kernel code or modules, a requirement often associated with DTrace. This fundamental difference not only simplifies the use of strace but also broadens its accessibility. Comparing with DTrace While DTrace offers a more in-depth analysis through deeper kernel support, strace operates on a more surface level. This simplicity, however, does not undermine its effectiveness. strace works essentially by logging every kernel call made by a process, providing verbose but incredibly detailed insights into the system's operation. This method allows users to trace the inner workings of a process, understanding each interaction with the kernel. Practical Usage and Advantages Ease of Use and Accessibility One of the most appealing aspects of strace is its user-friendly nature. It doesn't require special privileges or complex setup procedures. This ease of use is particularly beneficial for developers and system administrators who need to quickly diagnose and address issues in a Linux environment. Unlike DTrace, strace is readily available and doesn’t demand advanced configurations or permissions. Favored in Linux Environments strace's popularity in Linux circles is not only due to its accessibility but also its practicality. Being able to run without special privileges makes it a go-to tool for diagnosing various system-related issues. However, it's important to note that strace should be used cautiously in production environments. Its extensive logging can create a significant performance overhead, potentially impacting the efficiency of a live system. This is why strace is generally recommended for use in development or isolated testing environments rather than in production. strace in Action: A Closer Look at System Calls Basic Usage and Output Analysis Using strace is straightforward: you simply pass the command line to it. strace java -classpath . PrimeMain This simplicity belies its power, as the output offers a wealth of information. Each line in the strace output corresponds to a system call made by the process as you can see below: execve("/home/ec2-user/jdk1.8.0_45/bin/java", ["java", "-classpath.", "PrimeMain"], 0x7fffd689ec20 /* 23 vars */) = 0
brk(NULL)                               = 0xb85000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0294272000
readlink("/proc/self/exe", "/home/ec2-user/jdk1.8.0_45/bin/j"..., 4096) = 35
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64", 0x7fff37af09a0) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls", 0x7fff37af09a0) = -1 ENOENT (No such file or directory) By analyzing these calls, users can gain insights into the intricate operations of their applications. For instance, if a Java process attempts to load a library and fails, strace can reveal the underlying system call and its exit code, providing clues about potential issues like missing files or directories. E.g. in this line: COPY COPY open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) Java tries to load the library from the directory using a system call open to load the file. The exit code of the system call is which means that the file isn't there. Under normal circumstances, we should get back a file descriptor value from this API (positive non-zero integer). Looking in the directory, it seems the directory is missing. I'm guessing that this is because of a missing (Java Cryptography Extensions) installation. This is probably OK but might have been interesting in some cases. pthread tls -1, tls JCE Interpreting System Calls for Debugging The output of strace, while verbose, is a goldmine for troubleshooting. For example, a negative exit code in a system call indicates an error, such as a missing file, which could be crucial for diagnosing issues in an application. This level of detail, although overwhelming at times, is invaluable for understanding the interactions between your application and the Linux system. Advanced Features and Tips Filtering System Calls for Efficiency A common challenge with strace is managing its voluminous output. Fortunately, strace offers options to filter system calls, significantly enhancing its usability. By using the argument, you can instruct strace to log only specific types of system calls, such as or e.g.: -e open connect strace -e open java -classpath . PrimeMain This selective logging not only makes the output more manageable but also allows for focused troubleshooting, speeding up the debugging process. Exploring a Variety of System Calls strace's utility extends beyond just tracking file access or network interactions. It can be used to monitor a range of system calls, offering insights into various aspects of application behavior. By understanding and utilizing different system calls, users can gain a comprehensive view of their application's interaction with the operating system, leading to more effective debugging and optimization. strace and Java: A Special Case strace with the JVM While strace predates Java and operates at a low level with no specific awareness of the Java Virtual Machine (JVM), it remains highly effective for debugging Java applications. The JVM, like most platforms, relies on system calls for its operations, which strace can monitor and report. However, certain aspects of the JVM's behavior may be less visible to strace due to its unique approach to problem-solving. Allocations and Threading in Java For instance, Java's memory management differs significantly from standard system tools. While typical applications use malloc, which directly maps to kernel allocation logic, Java manages its own memory. This approach, aimed at efficiency and streamlined garbage collection, means that some memory allocation activities are obscured from strace's view. Similarly, Java threading is currently well-represented in strace output, but this is changing with Java 21 and Project Loom. Java 21 added support for Virtual Threads which are only partially visible to the operating system hence 1,000 threads can seem like 16 threads. These changes could affect the clarity of strace outputs in complex, heavily threaded Java applications. Final Word strace stands out as an exceptionally versatile and powerful tool in the Linux debugging arsenal. Its ability to provide detailed insights into system calls makes it invaluable for diagnosing and understanding the inner workings of applications. Despite its simplicity, strace is capable of handling complex debugging scenarios, especially when used with its advanced filtering options. For developers and system administrators working in Linux environments, strace is more than just a diagnostic tool; it's a lens through which the intricate interactions between applications and the operating system can be viewed and understood. As technologies evolve, tools like strace adapt, continuing to offer relevant and critical insights into system behaviors. Whether you are troubleshooting a stubborn issue or simply curious about how your applications interact with the Linux kernel, strace is a tool that you will likely find yourself returning to time and again.

Revisiting Strace: Simple is Beautiful

306 Stories To Learn About Artificial Intelligence

AI for Knowledge Management: Iterating on RAG with the QE-RAG Architecture

জ্ঞান ব্যবস্থাপনার জন্য AI: QE-RAG আর্কিটেকচারের সাথে RAG-তে পুনরাবৃত্তি করা

KI für Wissensmanagement: Iteration auf RAG mit der QE-RAG-Architektur

IA para la gestión del conocimiento: iteración en RAG con la arquitectura QE-RAG

IA pour la gestion des connaissances : itérer sur RAG avec l'architecture QE-RAG

ज्ञान प्रबंधन के लिए एआई: क्यूई-आरएजी आर्किटेक्चर के साथ आरएजी पर पुनरावृत्ति

ナレッジ管理のための AI: QE-RAG アーキテクチャによる RAG の反復

지식 관리를 위한 AI: QE-RAG 아키텍처를 사용한 RAG 반복

IA para Gestão do Conhecimento: Iterando em RAG com a Arquitetura QE-RAG

ИИ для управления знаниями: итерация RAG с архитектурой QE-RAG

Bilgi Yönetimi için Yapay Zeka: QE-RAG Mimarisi ile RAG'ı Yineleme

AI cho Quản lý Tri thức: Lặp lại trên RAG với Kiến trúc QE-RAG

AI 知识管理：使用 QE-RAG 架构迭代 RAG

With the growth of LLM models like ChatGPT, there has been a rush by firms to commercialize language-based deep learning applications. Companies like Duolingo and Blinkist are building educational chat applications, firms like Cocounsel are building document analysis models, and some, like MedGPT, are even building specialist models that can do things like medical diagnosis. In a previous article, I wrote about how someone can use ChatGPT and prompt engineering to build a power document analyzer. In order to support more powerful and domain-specific LLM applications, technology providers have made many cloud solutions available. OpenAI, the company behind ChatGPT, for example, has made a simple but powerful fine-tuning API available to users, allowing users to build their own language models based on GPT3 technology. Google, not to be outdone, made their bison-text model, widely considered a capable competitor to GPT 3 and GPT 3.5, available for fine-tuning through the Google Cloud platform. In a previous article, I wrote about how to use the fine-tuning API to create a domain expert LLM. As powerful as these services can be, a company considering a serious investment in LLM technology will want to learn to train their own models from open-source technologies. : Compared to using these vendor-provided endpoints, training your own model gives the following advantages You gain the flexibility of choosing and changing your deployment infrastructure. This can lead to cost savings, closer integration, and perhaps most importantly in medical and financial applications, more privacy. You get more control over the underlying technology, giving you a choice of open-source models to use. Different open source models are built with different use cases in mind, and you can choose the best tool for the job. Your applications become more future-proof. By using open source technology, you can set your own pace of development. You can use the state-of-the-art technology, and you won’t have to worry about vendor deprecations and service outages. In this article, we will take a popular and capable open-source LLM model, train it on our own data similar to what we did in a previous article, and validate the results. While the example we’re tackling is non-commercial and based on public information, the techniques can be easily cross-applied to commercial endeavors. in the “Expert LLM Model” section where we define the problem we’ll be solving in this article. We will delve into specific suggestions on what commercial applications can be built using this technique Underlying Technologies Flan-T5 For today’s experiment, we will be on Flan-T5 Large, which is a large language model released by Google. While this is not the technology that underlies Bard, this model is widely considered to be competitive with GPT-based technologies. What’s impressive about the Flan T5 models, however, is that they achieve satisfactory results using far fewer parameters than GPT based models. Even the XL version of the model, for example, only has 3 billion parameters, compared to GPT3, which has 175 billion. relying As a result of this compactness, it is relatively cheap to train and store these models on cloud computing assets. Additionally, the Flan-T5 family of models are released with the Apache license, which allows for commercial use, reducing potential license headaches that accompany some of the other open source LLMs. Facebooks’ LLaMa, for example, is still available only for research and non-commercial purposes. To write this article, I experimented with a few different classes of tasks to test the effectiveness of the technology. Generally, Flan-T5, especially the XL variant, seems to have great natural language understanding capabilities similar to some of the GPT models on the market. However, the model falls short somewhat when drawing abstract connections, and has some trouble generating long outputs. Therefore, one should take care to select the right model for the right task. Replicate Replicate is a Platform-as-a-Service company that people to rent GPUs for training and running large AI models at an affordable price. Their suite of AI model management tools allows users to focus on working with data instead of managing server resources. allows In order to write this article, I tried several AI training PaaS offerings including AWS SageMaker, Google Colab, and PaperSpace Gradient. Replicate was by far the easiest platform to get started with, and offered very competitive pricing relative to the other services mentioned. Python Python is the lingua franca of data engineering. The extensive ecosystem allows programmers to quickly ingest, analyze, and process data. Most major AI training platforms have first-class support for Python, which makes our job much easier. Because of Replicate’s excellent integration, we will be writing all of our code today in Python. Expert LLM Model Playwright Classifier Because the Flan-T5 family of models is much better at understanding text than generating text, we want to choose a task that is heavy on input but light on output. Natural language classifiers are a perfect use-case for this type of scenario, so today we will be building a playwright identifier. Specifically, we will be giving the model passages from either William Shakespeare or Anton Chekhov, and see if we can teach the model to identify the playwright based on the writing style and word choice. Of course, because this is a public tutorial, we are intentionally choosing a model with public and easily accessible data. . Here are some examples where natural language classifiers may be useful: However, this can easily be adapted to a commercial context Sorting customer reviews and complaints into various categories such as shipping problems, product quality, customer service, etc. Performing sentiment analysis on a sales call transcript to see if the prospect had a change in their mood during the call. Analyzing a large number of earnings calls transcripts to determine if CEOs are generally bullish or bearish. Building the Training Data To create the training data, we can download some Anton Chekhov and William Shakespeare plays from Project Gutenberg. To set up the data ingestion, we can run the following Python script. import requests
import openai
import replicate
import os
import pandas as pd
import random

texts = {
    'chekhov': 'https://www.gutenberg.org/files/7986/7986-0.txt',
    'chekhov_2': 'https://www.gutenberg.org/cache/epub/1755/pg1755.txt',
    'shakespeare_midsummer': 'https://www.gutenberg.org/cache/epub/1514/pg1514.txt',
    'shakespeare_romeo_juliet': 'https://www.gutenberg.org/cache/epub/1112/pg1112.txt',
    'shakespeare_macbeth': 'https://www.gutenberg.org/cache/epub/2264/pg2264.txt',
    'shakespeare_hamlet': 'https://www.gutenberg.org/cache/epub/2265/pg2265.txt',
} Now we create the training data folder and download the texts: if not os.path.exists('training_text'):
    os.mkdir('training_text')
for name, url in texts.items():
    print(name)
    res = requests.get(url)
    with open(os.path.join('training_text', '%s.txt' % name), 'w') as fp_write:
        fp_write.write(res.text) You should see some outputs like this to show you it succeeded: chekhov
chekhov_2
shakespeare_midsummer
shakespeare_romeo_juliet
shakespeare_macbeth
Shakespeare_hamlet You can also check the training_text folder to see that the files were properly downloaded. Now we want to read these files back into memory and split them into a list of lines. While we’re at it, we’ll count the number of lines in each file. lines_by_file = {}
for fn in os.listdir('training_text'):
    if not fn.endswith('.txt'):
        continue
    with open(os.path.join('training_text', fn)) as fp_file:
        lines_by_file[fn.split('.')[0]] = '\n'.join(fp_file.readlines())
        print(fn, len(lines_by_file[fn.split('.')[0]])) You should see output like the below: shakespeare_midsummer.txt 120198
shakespeare_romeo_juliet.txt 179726
shakespeare_macbeth.txt 140022
shakespeare_hamlet.txt 204169
chekhov.txt 419063
chekhov_2.txt 148324 Now comes the fun part. We want to split the lines into real training data. To do so, we first remove the first and last 1000 lines, which is taken up by introduction, header and footer content. Then, we will grab the remaining text 50 lines at a time. We will then turn the 50 lines into a prompt-and-completion pair. train_data = []
for k in lines_by_file:
    is_chekhov = 'chekhov' in k
    useful_lines = lines_by_file[k].split('\n')[1000:-1000]
    
    prompt_fmt = "Which playwright wrote the following passage? \n ==== \n %s \n ====" 
    for i in range(0, len(useful_lines), 50):
        training_set = useful_lines[i: i+50]
        train_data.append({
            'prompt': prompt_fmt % '\n'.join(training_set),
            'completion': 'Anton Chekhov' if is_chekhov else 'William Shakespeare'
        }) So now we have clearly defined the problem - given 50 lines of text from a play, determine if the playwright is Anton Chekov or William Shakespeare. We’re not done just yet. We need to write the data into jsonl (JSON Lines) format for training, and we also want to reserve a few samples for testing purposes. Run the following code like so: df = pd.DataFrame(train_data)
df_chekhov = df[df['completion'] == 'Anton Chekhov']
df_shakespeare = df[df['completion'] == 'William Shakespeare']
chekhov_test_indices = random.sample(df_chekhov.index.tolist(), 15)
shakespeare_test_indices = random.sample(df_shakespeare.index.tolist(), 15)
df_chekhov_test = df_chekhov.loc[chekhov_test_indices]
df_shakespeare_test = df_shakespeare.loc[shakespeare_test_indices]
df_chekhov_train = df_chekhov.loc[[i for i in df_chekhov.index if i not in chekhov_test_indices]]
df_shakespeare_train = df_shakespeare.loc[[i for i in df_shakespeare.index if i not in shakespeare_test_indices]]

pd.concat([df_chekhov_train, df_shakespeare_train]).to_json('chekhov_shakespeare_train.jsonl', orient='records', lines=True)
pd.concat([df_chekhov_test, df_shakespeare_test]).to_json('chekhov_shakespeare_test.jsonl', orient='records', lines=True) Of course, if you want to use the entire corpus for training, you can simply run . pd.DataFrame(train_data).to_json(‘output.jsonl’, orient=’records’, lines=True) Training with Replicate We need to do two things before we can invoke the training - first, we need to upload the training data onto somewhere accessible by replicate. One very easy way to do this would be to upload the file to a google cloud bucket, make the bucket and the file public, and supply the url in the format of . https://storage.googleapis.com/<bucket_name>/<file_name> Next, we need to create a destination. To do this, simply log into Replicate (which you can do through Github OAuth), and create a new model. Once the model is created and named, you will be able to push your model to this space. Once everything is set up, you can kick off the training like so: training = replicate.trainings.create(
  version="[flant5-large location]",
  input={
    "train_data": "[Data Location]",
  },
  destination="[Destination]"
)

print(training) You should see some output that tells you the training is starting. Wait a few minutes and check back with the training by running the following code: training.reload()
print(training) You can also monitor training progress on the Replicate website. Once the training is done, you can reload the training object to get the output name and proceed to the next step. Be warned that there are time periods when GPU resources are scarce, and you may get a “training failed” error. If this happens to you, just wait a few hours and try again. There is a GPU shortage, and PaaS providers are not immune! Testing the Model All right! Now that we have our fine-tuned model, we need to test it. Remember that we reserved 15 Chekhov and Shakespeare passages for testing. We can use them here like so: for _, row in df_chekhov_test.iterrows():
    output = replicate.run(
      training.output["version"],
      input={"prompt": row['prompt']}
    )
    for s in output:
        print(s, end="", flush=True)
    print('') After a short start-up period, you should see the output being printed to the console. The model should be extremely accurate and return “Anton Chekhov” each time. Let’s try this with Shakespeare: for _, row in df_shakespeare_test.iterrows():
    output = replicate.run(
      training.output["version"],
      input={"prompt": row['prompt']}
    )
    for s in output:
        print(s, end="", flush=True)
    print('') Similar to the Chekhov example, you should see that the model is able to identify Shakespeare every time. For good measure, let’s see if the base model is able to identify Shakespeare or Chekhov: for _, row in df_shakespeare_test.iterrows():
    output = replicate.run(
      "[base flant5-large location]",
      input={"prompt": row['prompt']}
    )
    for s in output:
        print(s, end="", flush=True)
    print('')

for _, row in df_chekhov_test.iterrows():
    output = replicate.run(
      "[base flant5-large location]",
      input={"prompt": row['prompt']}
    )
    for s in output:
        print(s, end="", flush=True)
    print('') You should see that the base model is to reliably identify the playwright for the same passages. This shows that our fine tuning reliably gave the model new information, and we have built ourselves a natural language playwright classifier! unable Conclusion In today’s article, we trained a simple natural language classifier based on Flan-T5, a large language model provided by Google. Because of its compact size and permissive license, Flan-T5 can be trained and deployed on private infrastructure, which sets it apart from many of the other popular models on the market, such as ChatGPT. While the example today was based on public data and was decidedly non-commercial, this proof of concept can be easily adapted to many other commercial applications as outlined above. If you have an idea with LLMs that you like to see turned into reality, feel free to start a conversation by visiting my or page. Also, feel free to read my previous LLM articles including one about , and API. GitHub LinkedIn Building a Document Analyzer using ChatGPT Creating a Domain Expert LLM using OpenAI’s Fine Tuning Happy hacking!

Train Your Own ChatGPT-like LLM with FlanT5 and Replicate

FlanT5 এবং প্রতিলিপি দিয়ে আপনার নিজের ChatGPT-এর মতো LLM-কে প্রশিক্ষণ দিন

Trainieren Sie Ihr eigenes ChatGPT-ähnliches LLM mit FlanT5 und Replicate

Entrene su propio LLM similar a ChatGPT con FlanT5 y Replicate

Entraînez votre propre LLM de type ChatGPT avec FlanT5 et Replicate

FlanT5 और रेप्लिकेट के साथ अपनी खुद की चैटजीपीटी-जैसी एलएलएम को प्रशिक्षित करें

FlanT5 を使用して独自の ChatGPT のような LLM をトレーニングし、複製する

FlanT5 및 복제를 사용하여 나만의 ChatGPT 유사 LLM 교육

Treine seu próprio LLM semelhante ao ChatGPT com FlanT5 e Replicate

FlanT5 ile Kendi ChatGPT benzeri LLM'nizi Eğitin ve Replikalayın

Huấn luyện LLM giống GPT trò chuyện của riêng bạn với FlanT5 và nhân rộng

使用 FlanT5 和 Replicate 训练您自己的类似 ChatGPT 的 LLM

78 Stories To Learn About Operating Systems

Manual and AI-Powered Test Runtime Approaches You Can Start Using Today Software development is getting progressively more complex as time passes, making software testing equally essential to releasing a stable and reliable product. But we know bloated test suites are infamous bottlenecks to faster releases. Dev teams need to be able to speed up their tests to keep progress moving, but it isn’t that simple. Measuring the runtime of your test suites can significantly impact your development cycle, enabling fast feedback loops, effective resource utilization, and better maintainability for your tests. With all that in mind, you can see how important measuring test runtime can be. Traditional Approaches for Measuring Test Runtime There are several ways of measuring test runtime, enough to suit almost anyone’s needs. That doesn’t always mean they’re the best approach, however. Let’s talk about some of the most common methods you may have heard of: - The most basic way possible, it’s not entirely unheard of for timing these tests with traditional clocks and stopwatches. All you need is a way to track the passage of time (like the stopwatch function on your phone) and some good timing. However, it’s nowhere near the most accurate, as it still relies on human interaction. Manual timing - Plenty of CI tools include features for measuring test runtime. Tools like TravisCI and Jenkins can record the start and end time of the jobs provided, giving you a glimpse of how they perform and if it changes. Integrations with CI tools - Another more “manual” way is to simply set up logs in your tests as they run. No matter what language you use, there will always be a print() function equivalent somewhere. You can also set your tests to log these results for easier parsing, but either choice will make writing tests longer. Analyzing logs and using timestamps Analyzing Logs Example in Python When your test framework generates logs, you can parse them to extract the necessary information such as test start time and test completion time. import re

def parse_logs(log_file):
    with open(log_file, 'r') as f:
        logs = f.readlines()

    test_start_time = None
    test_end_time = None

    for line in logs:
        if 'Test started' in line:
            test_start_time = extract_timestamp(line)
        elif 'Test completed' in line:
            test_end_time = extract_timestamp(line)

    if test_start_time and test_end_time:
        test_runtime = test_end_time - test_start_time
        print(f"Test runtime: {test_runtime}")

def extract_timestamp(log_line):
    # Regular expression to extract timestamp (assuming format: [YYYY-MM-DD HH:MM:SS])
    timestamp_pattern = r'\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\]'
    match = re.search(timestamp_pattern, log_line)
    if match:
        timestamp = match.group(1)
        return convert_to_datetime(timestamp)
    return None

def convert_to_datetime(timestamp):
    from datetime import datetime
    datetime_format = "%Y-%m-%d %H:%M:%S"
    return datetime.strptime(timestamp, datetime_format)

# Usage
parse_logs('test_logs.txt') In this example, the parse_logs function reads the log file line by line and searches for specific markers ('Test started' and 'Test completed') to extract the corresponding timestamps. It uses regular expressions to extract the timestamp and converts it to a DateTime object for further calculations. Timestamps Example With Python Another approach is to record timestamps directly in your test code. In the below example, the run_test function records the start time using time.time() before executing the test logic. After the test completes, it calculates the runtime by subtracting the start time from the end time, which is also obtained using time.time(). The result is printed as the test runtime in seconds. import time

def run_test():
    start_time = time.time()

    # Test logic goes here
    # ...

    end_time = time.time()
    test_runtime = end_time - start_time
    print(f"Test runtime: {test_runtime} seconds")

# Usage
run_test() Regardless of your approach, these methods tend to have some pitfalls. It’s difficult to get wholly accurate results from these methods, whether from the chance of human error or external factors. Plus, many of these can’t scale up with your testing. That means you need to find a more accurate way to measure. Why Should I Be Measuring Test Runtime Anyway? We’ve gone on and on about the how, but what about the why? Your tests are critical, but the testing pipeline can be a delivery bottleneck. By tracking how your tests run, you can see the full picture of their performance, string together test suite intelligence, and take steps to streamline your testing further, including three common areas: Parallelization Optimization Measuring test suite runtime allows you to spot what tests have the lengthiest runtime. This transparency enables you to which gives you faster results and a shorter feedback loop. optimize testing further, when possible, by parallel testing Resource Allocation Accurately measuring your test runtime can help you plan your testing phases more efficiently. You’ll be able to allocate time and to your lengthier tests and avoid bottlenecks later down the pipeline. compute power Test Prioritization Identifying your slowest tests also allows you to take a step back and That way, you can have your faster tests run first, giving you crucial insights into the build before your lengthier tests get to work. prioritize the most critical tests and what can be saved for later. Continuous Improvement Test runtimes are critical in CI/CD pipelines, where fast feedback is essential. By measuring and optimizing your test runtimes, you can , enabling faster deployment. reduce the time required for the CI/CD process This, in turn, improves the overall agility of the development process and allows for more frequent iterations. Automate Tracking Test Runtime and Critical KPIs With  Launchable It’s pretty clear that measuring test runtime can be a huge factor in your overall testing process. And with Launchable, you can start measuring immediately and make your tests more efficient. How Does Launchable Measure Test Runtime? Launchable automates test runtime tracking by integrating with all your favorite CI/CD tools, including TravisCI, Jenkins, and GitHub Actions. That means you can easily slide Launchable into your existing pipeline, allowing our ML model to analyze your tests. And once we’re in, we can seamlessly measure test runtime across multiple builds, . giving you critical insights into your tests beyond just runtime Deeper Testing Insights Beyond Runtime for Complete Test Suite Health Empower your team to quantify the impact of changes in your test suites beyond test runtime. for deeper test suite transparency for data-driven QA. Launchable gives test suite health metrics for QA teams, sucking up time and effort. Launchable identifies flaky tests based on their impact, allowing you to address and run them more reliably. Get insights from our daily Flakiness report and fix your tests with confidence. Prioritize impactful flaky tests: Flaky tests can be a huge headache Measure the time it takes for your test suite to run across multiple sessions. Our tracking feature highlights tests that exceed expected durations, helping you identify and resolve any performance issues. Track test session duration: Monitor how often your test suites are executed and combine it with session duration. Ensure tests are run at the right intervals. Plus, saves time and resources by running the most relevant tests at the optimal times. Optimize test session frequency: Predictive Test Selection Pinpoint tests that fail frequently and investigate potential issues with the tests or the current build. Gain valuable insights into the stability of your testing process and make informed decisions to improve overall quality. Identify test session failure ratio: Utilizing your test infrastructure efficiently is key to minimizing idle time and maximizing your resources. Easily identify any bottlenecks hindering performance by measuring test runtime with Launchable. Get all the information you need to test suite patterns and trends within your tests, allowing you to make informed, data-driven decisions to optimize your tests. And by doing so, you can . streamline your overall testing process

Measuring Test Runtime: How to Optimize Your Test Suite Performance

Medición del tiempo de ejecución de las pruebas: cómo optimizar el rendimiento de su conjunto de pruebas

Mesurer la durée d'exécution des tests : comment optimiser les performances de votre suite de tests

टेस्ट रनटाइम को मापना: अपने टेस्ट सूट के प्रदर्शन को कैसे अनुकूलित करें

テスト実行時間の測定: テスト スイートのパフォーマンスを最適化する方法

Medindo o tempo de execução do teste: como otimizar o desempenho do seu conjunto de testes

Đo lường thời gian chạy thử nghiệm: Cách tối ưu hóa hiệu suất bộ thử nghiệm của bạn

测量测试运行时间：如何优化测试套件性能

257 Stories To Learn About Linux

194 Stories To Learn About Git

There are a lot of exciting and valuable Computer Vision tasks. For instance, Let’s say we have a project where we want to search for lost people in the forest using camera-fitted drones and Computer Vision. Or maybe we need to find something small and we have a high-quality camera. In cases like these, we can get high-resolution images in our dataset. However, most CV models will lower image resolution because it helps with speed (both on train and inference) and usually, you don't need very high resolutions to detect an object. In our case, however, we really need full resolution, because we would be using camera-fitted drones for our search. Meaning that even people would appear as tiny objects in our images. How do we go about a task like this? Dataset Everything starts with a dataset. I've tried to find an open dataset and ended up using . This is not a perfect dataset for searching for lost people, but we will use it as an example. TinyPerson I've converted the dataset from format to , as I was going to use a model from the YOLO family. Furthermore, I've filtered the dataset a little bit, but nothing really important. To convert the dataset, I used and customized it a tiny bit, so it would work in my case. My final dataset consists of 1495 images. I've split it into train/val, where 15% was left for validation. COCO YOLO format this repo A few notes on this Dataset. Images are hard, there can be a lot of people on it, and they can be really far away. It's not rare when not everyone is labeled, which is not good. Anyway, here is an ‘ok’ example from the dataset: It’s important to keep in mind that there are not a lot of high-res images in this dataset, which is unfortunate. The median size is 1920x1080. Baseline As a baseline model, I took pre-trained , which uses 1280x1280 images and is a pretty beefy model. I've also tried with the same image size and similar model size, but I got worse results on this dataset, so I settled on YOLOv5. YOLOv5l6 YOLOv7-W6 Having the correct dataset, it's easy to train Yolov5 with commands like this one: python train.py --data dataset/dataset.yaml --weights yolov5m6.pt --img 1280 --batch 15 --epochs 80 I got mAP50 around for my baseline. You that on this dataset that's an ok mAP. 0.477 can see So we still lose some data in most cases in our baseline, as we resize every image to 1280x1280. When our objects are that small, we don't really want to do that. It would be better to preserve full resolution. But, models are optimized for their training image size, so we need to use 1280x1280 or 640x640 in the case of the YOLO family. Patch-based approach This is another approach that could help us. What if we take every image and cut it with a fixed-size sliding window and use that crop as training data? Let's take a smaller model with an input size of 640x640 and cut our dataset to that size. Here is how I see it: We'll use an image with a size of 1920x1080 as an example. We can divide it into 6 images (patches), 640x640. We can't slice it exactly, we will have intersections and that's fine: Blue boxes are first-row patches, and green ones are second. We don't have an intersection between columns here, just because 1920 is divisible by 640, but we do have an intersection between the first and second rows. With this intersection, we lower the chance that our object will get cut on the edge of the patch, although it is already small because our objects are small. And don't forget that you need to carry out the same process with your labels. Anyway, we can use this approach for any image, no matter the size (if it is bigger than our patch size). This allows us: Not to lose image resolution To use optimal input size for the model, no matter what image sizes we originally have This approach is also good if you don't have enough memory on your GPU. You can find yourself in that situation with 3D really often. is a good lib for that tasks. But in our case, I decided to use a simple custom patcher, specifically written for YOLO models. TorchIO Here is my utils.py: from typing import List

def xywh_to_xyxy(
    lines: List[str],
    img_height: int,
    img_width: int) -> List[List[int]]:

    '''
    This function gets list with YOLO labels in a format:
    label, x-center, y-center, bbox width, bbox height
    coordinates are in relative scale (0-1).
    Returns list of lists with xyxy format and absolute scale.
    '''
    labels = []
    for _, cur_line in enumerate(lines):
        cur_line = cur_line.split(' ')
        cur_line[-1] = cur_line[-1].split('\n')[0]

        # convert from relative to absolute scale (0-1 to real pixel numbers)
        x, y, w, h = list(map(float, cur_line[1:]))
        x = int(x * img_width)
        y = int(y * img_height)
        w = int(w * img_width)
        h = int(h * img_height)

        # convert to xyxy
        left, top, right, bottom = x - w // 2, y - h // 2, x + w // 2, y + h // 2
        labels.append([int(cur_line[0]), left, top, right, bottom])

    return labels


def xyxy_to_xywh(
    label: List[int],
    img_width: int,
    img_height: int) -> List[float]:

    '''
    This function gets list with label and coordinates in a format:
    label, x1, y1, x2, y2
    coordinates are in absolute scale.
    Returns list with xywh format and relative scale
    '''

    x1, y1, x2, y2 = list(map(float, label[1:]))
    w = x2 - x1
    h = y2 - y1

    x_cen = round((x1 + w / 2) / img_width, 6)
    y_cen = round((y1 + h / 2) / img_height, 6)
    w = round(w / img_width, 6)
    h = round(h / img_height, 6)

    return [label[0], x_cen, y_cen, w, h] And here is the patcher itself: class Patcher:
    def __init__(self, path_to_save: Union[Path, str], base_path: Union[Path, str]) -> None:
        self.path_to_save = path_to_save
        self.create_folders()
        self.base_path = base_path

    def create_folders(self) -> None:
        self.path_to_save.mkdir(parents=True, exist_ok=True)
        (self.path_to_save / "images").mkdir(exist_ok=True)
        (self.path_to_save / "labels").mkdir(exist_ok=True)

    def patch_sampler(
        self,
        img: np.ndarray,
        fname: str,
        patch_width: int = 640,
        patch_height: int = 640,
    ) -> None:
        # Get image size and stop if it's smaller than patch size
        img_height, img_width, _ = img.shape
        if img_height < patch_height or img_width < patch_width:
            # copy image and label to patched folder
            io.imsave(self.path_to_save / "images" / f"{fname}.jpg", img)
            label_path = (self.base_path / "labels" / fname).with_suffix(".txt")
            copy(label_path, self.path_to_save / "labels")
            return

        # Get number of horisontal and vertical patches
        horis_ptch_n = int(np.ceil(img_width / patch_width))
        vertic_ptch_n = int(np.ceil(img_height / patch_height))
        y_start = 0

        ##### Prepare labels
        label_path = (self.base_path / "labels" / fname).with_suffix(".txt")
        with open(label_path) as f:
            lines = f.readlines()

        all_labels = xywh_to_xyxy(lines, *img.shape[:2])
        #####

        # Run and create every crop
        for v in range(vertic_ptch_n):
            x_start = 0

            for h in range(horis_ptch_n):
                idx = v * horis_ptch_n + h

                x_end = x_start + patch_width
                y_end = y_start + patch_height

                # Get the crop
                cropped = img[y_start:y_end, x_start:x_end]

                ##### Get labels patched
                cur_labels = []
                for label in all_labels:
                    cur_label = label.copy()

                    # Check if label is insde the crop
                    if (
                        label[1] > x_start
                        and label[2] > y_start
                        and label[3] < x_end
                        and label[4] < y_end
                    ):
                        # Change scale from original to crop
                        cur_label[1] -= x_start
                        cur_label[2] -= y_start
                        cur_label[3] -= x_start
                        cur_label[4] -= y_start

                        label_yolo = xyxy_to_xywh(cur_label, patch_width, patch_height)
                        cur_labels.append(label_yolo)

                # Save the label file to the disk
                if len(cur_labels):
                    with open(self.path_to_save / "labels" / f"{fname}_{idx}.txt", "a") as f:
                        f.write("\n".join("{} {} {} {} {}".format(*tup) for tup in cur_labels))
                        f.write("\n")
                #####

                # Save the crop to disk
                io.imsave(self.path_to_save / "images" / f"{fname}_{idx}.jpg", cropped)

                # Get horisontal shift for the next crop
                if img_width % patch_width == 0:
                    x_start += patch_width
                else:
                    x_start += int(
                        patch_width
                        - (patch_width - img_width % patch_width) / (img_width // patch_width)
                    )

            # Get vertical shift for the next crop
            y_start += int(
                patch_height
                - (patch_height - img_height % patch_height) / (img_height // patch_height)
            )


def main():
    '''
    base path structure:

    -> dataset
    ---> train
    -----> images (folder with images)
    -----> labels (folder with labels)
    ---> valid
    -----> images (folder with images)
    -----> labels (folder with labels)
    '''

    base_path = Path("")

    # path were you want to save patched dataset
    path_to_save = Path("")

    for split in ["train", "valid"]:
        images_folder_path = base_path / split / "images"

        patcher = Patcher(path_to_save / split, base_path / split)

        for image_path in images_folder_path.glob("*"):
            if image_path.name.startswith("."):
                continue
            image = io.imread(image_path)
            fname = image_path.stem

            patcher.patch_sampler(image, fname)


if __name__ == "__main__":
    main() Training YOLOv5 is still easy: python train.py --data dataset/dataset.yaml --weights yolov5m.pt --img 640 --batch 40 --epochs 80 mAP increased a little bit - to : 0.499 So we have increased our mAP by , which is not much, and our new solution is going to be a little bit slower (both on inference and while training). But this technique is going to perform better, the higher the resolution of the images you have. And we can see a difference even with our image size. 4% Conclusion The best use case for this technique is when you don't mind losing some speed to gain accuracy, and you have high-res images with small objects. In other similar cases, you can also try this solution after you have a baseline. Don't forget to split the dataset first and sample patches second, so you don't create a data leakage.

Small Object Detection in Computer Vision: The Patch-Based Approach

Usually, I found myself reading more than 100 tweets per day, so I thought that using all that 'reading power' to read a book using tweets could be a good idea. During the holiday I made the first version of this crazy idea, and I called it , a mix between Twitter and Book (yes, I know it's a bad name 😄). ‘25 de Mayo’ Took In this post I will explain how I made Took using Python and Tweepy, how I hosted it using Heroku (and later on a Raspberry Pi), and if it was actually useful to read a book on Twitter using this bot. Are you looking for the code 👀? You can check the Took GitHub repository . here Coding a Twitter bot In this section, I will explain how Took works, and the changes it got since the first version. In the beginning the idea was simple: Take the book I want to tweet, put all the book’s sentences into a list, verify that those sentences have less than 280 characters, and tweet one sentence every 30 minutes. This simple initial logic is still the core of Took, only some minor changes have been done. To start explaining in deep how the bot works, we need to go to the file. This file is in charge of the pre-processing of the book, this means that inside the file there are utility functions that will help us with the job of dividing the book into tweetable sentences. book-preprocessing.py The main function here is : First, it will convert the book into a list of sentences using the function , that uses the tokenize function of the nlkt library to do the job in a simple and readable way. tweetify_book txt_to_list_of_sentences # Converts a .txt to a list of sentences to tweet
def txt_to_list_of_sentences(txt_name):
    with open(txt_name, 'r') as file:
        data = file.read().replace('\n', ' ')
    return nltk.tokenize.sent_tokenize(data) After generates that list of sentences the function moves on and verify if the current sentence is available to tweet, this means that it verifies if the current sentence is 280 characters or fewer. If the length is more than 280 characters, the function will be called and it will re-tokenize that sentence to make it available to tweet. txt_to_list_of_sentences tweetify_book make_sentence_available_for_tweet # Takes a sentence and replaces , for . to tokenize it and be able to tweet it.
def make_sentence_available_for_tweet(sentence):
    sentence = sentence.replace(",", ".")
    return nltk.tokenize.sent_tokenize(sentence) Let’s see an example: Original Text: Adell was just drunk enough to try, just sober enough to be able to phrase the necessary symbols and operations into a question which, in words, might have corresponded to this: Will mankind one day without the net expenditure of energy be able to restore the sun to its full youthfulness even after it had died of old age? Or maybe it could be put more simply like this: How can the net amount of entropy of the universe be massively decreased? Multivac fell dead and silent. The function will do nothing with this text because there are not ‘.’ involved to tokenize, so it will put the full-length text of 476 characters into the list of sentences. When the verification time comes, the function will be called and it will convert the previous text into this: txt_to_list_of_sentences make_sentence_available_for_tweet [“Adell was just drunk enough to try”, ”just sober enough to be able to phrase the necessary symbols and operations into a question which”, ”in words”, ”might have corresponded to this: Will mankind one day without the net expenditure of energy be able to restore the sun to its full youthfulness even after it had died of old age?”, ”Or maybe it could be put more simply like this: How can the net amount of entropy of the universe be massively decreased?”, ”Multivac fell dead and silent.“ ] Now the resulting sentences will be ready to tweet since all of them have 280 characters or fewer. The magic here is in this line inside the function : make_sentence_available_for_tweet sentence = sentence.replace(",", ".") That converts all ‘,’ in ‘.’, letting us tokenize that sentence again using nlkt. In Python tokenization basically refers to splitting up a larger body of text into smaller lines, words or even creating words for a non-English language. - Source: Tutorials Point Covering how nlkt works is out of the scope of this article, if you want to read more you can go to . nltk.org After tokenizing all the sentences of the book, the function will create a .txt file where each line is a tweet ready to be sent, you can see an example . tweetify_book here Now let's focus on the file, where all the magic happens. main.py If you take a look at the , you will see that the first lines are just boring constants used to save the Twitter API keys. I will not be covering how you can use Twitter API, so let’s skip this part of the code and go straight to the end of the file, specifically where we open our already tokenized book. file Inside the while loop at the end of is where everything happens, let's take a look: main.py # Tweets the book, one line every TIME_DELAY seconds
with open('tweetify-books/tweetify-' + BOOK_TO_TWEET, 'r') as file:
    tweets = file.readlines()
    while current_index <= len(tweets):
        original_tweet_id = get_last_tweet_id()
        client.create_tweet(text = tweets[current_index], in_reply_to_tweet_id = original_tweet_id)
        print(tweets[current_index])
        current_index= current_index + 1
        update_index(current_index)
        time.sleep(TIME_DELAY) Basically, the loop will run while we have tweets left to send. Inside the loop, the program will get the last tweet id using the function and save it in a variable called , why? Because we need to connect the tweets we are sending to make a thread on Twitter where the book will be available to read. get_last_tweet_id original_tweet_id # Returns last tweet id, to create a thread
def get_last_tweet_id():
    # Get the last tweet (the one created upside)
    tweets = client.get_users_tweets(id=USER_ID, tweet_fields=['context_annotations','created_at','geo'], user_auth=True)

    return tweets.data[0].id After that, the current tweet will be sent using tweepy, and the variable will be incremented in one. Let's make a little interruption here and talk about why the program should save the current index. current_index The function that is called next, , is in charge of saving the current index of the most recent tweet into a local txt file. What is this for? We need to save the position of the last send tweet because in case the program stop running we need to resume the bot from the last sentence it tweet, and not from the beginning. update_index After tweeting the bot waits 30 minutes before repeating the process again. In fact, that is the logic behind Took: a simple and fun Twitter bot. Remember that if you want to see the complete code you can check the Took GitHub repository . here Hosting a Twitter bot In this section, I will be briefly discussing how I hosted Took, and the different options I have tried. When I released the first version of Took I was using . At the time I thought it will be great for that job and I choose it because I used to use it in older projects. Heroku Heroku was the home of Took for some days, but after some time I decided to use my own Raspberry Pi to do the job. Hosting the bot using a Raspberry was a pretty straightforward job. I just clone the repository into the device, installed the necessary dependencies, and voilà! It was hosted perfectly and running 24/7. The Raspberry Pi and Heroku are just two options inside the vast sea of hosting services you have to host a bot. Feel free to do your own research and find the best option for your work. Final Thoughts about Took Let’s talk about Took utility and what I have learned developing it. Took’s development begin on the 25th of May 2022, and some days later, on the 27th, I tweeted about Took’s existence in my main account. That tweet caught some attention and made Took’s account reach 80 followers that day. After some testing Took started tweeting the first book on June 15th, and finished it on August 6th. It is important to say that the bot was inactive for a lot of days due to testing and improving stuff, and that is why there is a time gap in some tweets. “The last question” by Isaac Asimov was the book that Took tweeted. The book has a length of 398 tweets. , and I never intended to violate any copyright. For future books, I have been thinking about using books from exclusively. Please note that Took has educational purposes only Project Gutenberg Did Took meet his objective? Yes. The book was completely tweeted in an only thread. Is a Twitter thread a good way to read a full-length book? Let’s be honest, no. It is a bit difficult to only read one sentence at a time, and also bookmarking what was the last sentence you read is something difficult to do (you can use Twitter likes for this purpose, but it is not so practical). Also ~10 users unfollowed the bot because it was annoying to see a new tweet from Took on their timeline every 30 minutes. It was a good project to learn how to use the Twitter API and make something fun. Absolutely yes, and a lot of people from Reddit and Twitter loved the idea. I encourage you to try to do something similar! Is Took going to keep tweeting other books? For the moment Took will be paused, feel free to fork the code and start your own bot if you want to see something similar running! Also Published here

Took - A Twitter Bot that Tweets Books.

Often when debugging, we need to step outside of the comforting embrace of the IDE to reproduce or track an issue. In this series, I’d like to go over some tools you might find useful for these cases. I’ll try to limit myself to tools that are 100% debugging tools and not those that are useful for development testing. E.g. tools like or are remarkably useful. You can/should use them while debugging. But you probably used them while building and testing the feature. So you would already be familiar with them and should have some sense of what they do. I want to focus on tools that you would mostly reach for when debugging. In that sense, tools like etc. also make no sense here. curl jq SDKMan I also want to avoid tools like database tools. They are remarkably useful when debugging, but again, you’d probably use them during development, too. They’re also very vendor specific, so it’s a vast subject that’s too broad to cover here. The tools I'll cover in this series encompass the following categories: System monitoring tools - as is the case with strace and dtrace, which we'll discuss today Network monitors - also fits in the above bracket but warrants a category of its own VM/Runtime monitoring - e.g. tools that let us inspect the JVM, etc. Profilers and memory monitors In this first post I'd like to discuss two heavyweight champions: dtrace and strace. If you're a Java developer or a Windows user, there's a good chance you never heard of these tools. You might have inadvertently used one of them since so many tools were built on top of them, but that's probably not the case. Both tools will let you debug anything without the source code. You could detect problems and gain a level of understanding you never imagined. Dtrace Back in 2004, I first heard about DTrace while working at Sun Microsystems. It became all the rage in the hallways as it was an innovation that Sun Microsystems was promoting. Dtrace was ported later into MacOS X (it originated on Solaris). Today there are ports on Windows and Linux as well. Dtrace is a powerful low-level dynamic tracing framework. But that's just another superlative and if you never used such a tool and don't have a background in systems programming, you might be feeling a bit confused: what the hell does it do? It lets you "see" everything. Want to know which files a process has opened? OK Want to know who invokes a kernel API and get a stack trace to the caller? OK Want to know why a process dies? OK Want to know how much CPU Time is spent on an operation? OK You might think that dtrace is one of those tools that will destroy your CPU completely... But here's the killer feature: it's fast enough to run in production with minimal to no performance impact! It was revolutionary when it was launched almost two decades ago and it's still the case to this day! Running Dtrace Before we begin, a word of warning. Save your data! This tool can easily crash your machine and enabling it requires disabling important security facilities on MacOS. This is a risky "low level" system service and should be treated as such... On a Mac dtrace conflicts with "System Integrity Protection" which is a security feature that blocks some interactions between processes (among other things). Under normal circumstances, this would be great. But if you want to run dtrace this would be a problem. The solution is booting to recovery mode on Intel Macs; this means holding the keys on boot. On an ARM mac just long-press the power button. Command-R Then, in a recovery mode terminal, issue the command: . csrutil disable Then upon reboot, dtrace should work as expected. Basic Usage As mentioned before, dtrace is a very powerful tool. There are whole books written about it. It has its own programming language based on C syntax that you can use to build elaborate logic. E.g. the following command will log some information from the given callbacks: sudo dtrace -qn 'syscall::write:entry, syscall::sendto:entry /pid == $target/ { printf("(%d) %s %s", pid, probefunc, copyinstr(arg1)); }' -p [PID] The code snippet passed to the dtrace command listens to the sendto callback on the target process ID. Then prints out the information to the console, e.g.: (pid) text If this seems like a bit too much and too hard to get started with... You're 100% right. It's a powerful tool when you need it. But for most of our day-to-day usage, it's just too powerful. What we want is to know a bit of basic stuff and this is just too much! Simple Usage As luck would have it, we have a simple solution: man -k dtrace This prints out a list of tools that's worth reading just to get a sense of how extensive this thing is. Here are a few interesting lines of output from that command: bitesize.d(1m)           - analyse disk I/O size by process. Uses DTrace
dapptrace(1m)            - trace user and library function usage. Uses DTrace
errinfo(1m)              - print errno for syscall fails. Uses DTrace
iotop(1m)                - display top disk I/O events by process. Uses DTrace
plockstat(1)             - front-end to DTrace to print statistics about POSIX mutexes and read/write locks It's worth your time going over this list to realize what you can truly do here. Examples You're facing elevated disk write issues that are causing the performance of your application to degrade... But is it your app at fault or some other app? Just run: sudo rwbypid.d It will print out the reads/writes to the disk: PID CMD                       DIR    COUNT
  2957 wordexp-helper              W        1
  2959 wc                          W        1
  2961 grep                        W        1

... snipped for clarity ...

   637 firefox                     R     6937
   637 firefox                     W    15325
   343 sentineld                   W   100287 The security software is really holding performance down... You can also use to get more specific results on the number of bytes written/distribution. bitesize.d That's pretty high level though. What if you want specifics: file name, process name, etc? sudo iosnoop -a Prints out output that includes pretty much everything you would need: STRTIME              DEVICE  MAJ MIN   UID   PID D      BLOCK     SIZE                     PATHNAME ARGS
2022 Jun 30 12:16:56 ??        1  17   501  1111 W  150777072     4096 ??/idb/3166453069wcaw.sqlite-wal firefox\0
2022 Jun 30 12:16:56 ??        1  17   501   661 W  150777175   487424  ??/index-dir/the-real-index Slack Helper\0
2022 Jun 30 12:16:57 ??        1  17   499   342 W  150777294     4096 ??/persistent/.dat.nosync0156.ztvXap sentineld\0 I can see the process id and how many bytes it wrote to the specific file! Let's say your program spans processes and you want to see what's going on. E.g. I run a source code build in a server I built: sudo errinfo Lets me detect the error returned from system calls to and the command that originally triggered that: EXEC          SYSCALL  ERR  DESC
    WindowServer workq_kernreturn   -2 
    WindowServer workq_kernreturn   -2 
   SentinelAgent workq_kernreturn   -2 
   SentinelAgent workq_kernreturn   -2 
          Signal           Helper    0 
          Google           Chrome    0 
           Brave          Browser    0 
          Google           Chrome    0 These are just the tip of the iceberg. I suggest checking out this old from Oracle or . Disclaimer: I didn't read the book... dtrace tutorial the book strace Amusingly enough, the strace tool also originated at Sun Microsystems in this case in the 90s. This isn't surprising though as the list of technologies originating from Sun Microsystems is absolutely mind-numbing. Strace is much simpler than dtrace in both usage and capabilities. For better and for worse. Since dtrace requires deep OS support, it never became an official feature of common Linux distributions and as a result, on Linux people use strace instead of dtrace. They aren't exactly interchangeable, though. Strace is enabled thanks to the kernel feature known as ptrace. Since ptrace is already in Linux, we don't need to add additional kernel code or modules. Typically, dtrace requires deeper kernel support, to get around licensing issues on Linux it's in a separate loadable module but that still presents some challenges. Working with strace is akin to printing a log entry every time we make a kernel call. This creates very verbose logging for every command you execute. As a result, you can follow what's going on under the hood of a running process. really Running Strace Nowadays, strace is commonly used in Linux, it's my favorite system diagnostic tool on that platform. It's remarkably convenient to work with since we can run it with no special privileges. Notice that, unlike dtrace, you should keep strace away from production environments (unless the code is segregated). It carries a significant performance overhead and would bring a production system down. The most basic usage of strace is just passing the command line to it: strace java -classpath. PrimeMain The output of strace for this is pretty long, lets go over a few of the lines: execve("/home/ec2-user/jdk1.8.0_45/bin/java", ["java", "-classpath.", "PrimeMain"], 0x7fffd689ec20 /* 23 vars */) = 0
brk(NULL)                               = 0xb85000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0294272000
readlink("/proc/self/exe", "/home/ec2-user/jdk1.8.0_45/bin/j"..., 4096) = 35
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64", 0x7fff37af09a0) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls", 0x7fff37af09a0) = -1 ENOENT (No such file or directory) Every one of these lines is a Linux system call. We can google each one of them to get a sense of what's going on. Here's a simple example: open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) Java tries to load the library from the directory using a system open call to load the file. The exit code of the system call is , which means that the file isn't there. Under normal circumstances, we should get back a file descriptor value from this API. Looking in the directory, it seems the directory is missing. I'm guessing that this is because of a missing JCE installation. This is probably OK but might have been interesting in some cases. pthread tls -1 tls Obviously, the amount of output is overwhelming sometimes. We usually just want to see things like "which file was opened" and "what's going on with our network calls". We can easily accomplish that by only looking at specific system calls using the argument. -e strace -e open java -classpath . PrimeMain Will only show the open system calls: open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/libjli.so", O_RDONLY|O_CLOEXEC) = 3
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) There are many system calls you can learn and use to track many behaviors such as this: connect, write, etc. This is only the tip of the iceberg of what you can do using strace. Julia Evans wrote some of the most . If you want to learn more about it, there's probably no better place (also check out her other stuff... Amazing resources!). exhaustive and entertaining posts on strace Strace and Java As you saw before, strace works great with the JVM. Since strace predates Java and is a very low-level tool, it has no awareness of the JVM. The JVM works like most other platforms and invokes system calls which you can use to debug its behavior. However, because of its unique approach to some problems, some aspects might not be as visible with strace. A good example is allocations. System tools use malloc, which maps to kernel allocation logic, but Java takes a different route. It manages its own memory for efficiency and easier garbage collection logic. As a result, some aspects of memory allocation will be hidden from strace output. This can be a blessing in disguise, as the output can be overwhelming sometimes. At the time of this writing, threading works well with strace. But this might not be the case moving forward as might change the one-to-one mapping between Java threads and system threads. This might make strace output harder to pinpoint in heavily threaded applications. project Loom Finally There's an alphabet soup of "*trace" utilities in various forms that keep borrowing ideas from one another. It's a significant challenge to keep up with all of that noise. There are just too many great tools to cover, I would like to discuss btrace in a future installment though. It's very similar to dtrace but also very JVM specific so probably worth a different post. The tools I discussed today take different approaches to solve similar problems: how do we understand what a binary application "really" does? Security researchers and hackers use these tools to understand your program. They don't need the code and don't need a disassembly to see what you're actually doing. You can also use these tools to understand the impact of your actions. We often invoke an API and let things end there. But the devil is in the details and those details can carry a heavy toll. As a Java developer, I rarely think about signal delivery, process management or such low-level staples. But I do spend time looking at these things since they ultimately impact the stability and performance of my application. Also Published Here

Using External Debugging Tools dtrace and strace

Stories, News & Mentions about REALLINKS

Reallinks’s Stories on HackerNoon