This article is excerpted from chapters 6 and 9 of my Manning book, Linux in Action . Besides the book, you can also work through Linux in Motion — a hybrid course made up of more than two hours of video and around 40% of the text of Linux in Action. You know that the passwords chosen by the people you support are probably not strong enough to protect your infrastructure against a serious attack. And even the few exceptions to the rule are probably being reused on multiple servers and accounts. You beg and nag, but it’s a losing battle. But all is not entirely lost. The problem of keeping track ofsufficiently complex passwords can be largely solved by using a goodpassword vault like KeePass2 or LastPass. And the problem of overusingpasswords can be at least blunted by implementing a single sign-onsolution like Kerberos. Ok. Not Kerberos, but . like Kerberos Still, dumb mistakes are always going to happen. So what’s bound to happen to the one or two users who care enough to dream up good, strong passwords for each server they access? Every now and then they’ll forget a password, of course. That won’t be a problem if there’s another admin with sudo power who can log into the server and run passwd to create a new password for the user. $ sudo  passwd username[sudo] password for yourname:Enter new UNIX password:Retype new UNIX password:passwd: password updated successfully But if your unlucky and forgetful user was the only admin with an account on that machine, you’ve got trouble. Except that you don’t. — the grandfather of all Linux virtualization — is going to save your day. chroot Here’s one way that it might work. Use a live-boot drive to power up the server that’s got you locked out. Steps for creating a Linux live boot USB Then open a terminal and run lsblk to determine the designation of your root partition on the server’s hard disk, and mount the root partition to a temporary directory. # mkdir /run/mountdir/# mount /dev/sdb1 /run/mountdir/ Then you whisper the magic words and you’re in: # chroot /run/mountdir/root@ubuntu:/# That’s all it takes. At this point, you’re free to run commands as though you were working on a running version of the physical hard drive. Use passwd to give your admin a new password to replace the lost one and, after typing exit to shut down your chroot session, reboot the machine (without the live-boot USB). Everything should now be fine. To encrypt or not to encrypt Encrypting the data on your storage drives using tools like ecryptfs or dm-crypt makes it a great deal less likely that your data will be compromised. But on the other hand, many rescue and recovery operations like the above chroot trick simply won’t work on an encrypted volume. Striking a balance between security and accessibility isn’t an exactscience. Many admins, for instance, will leave local servers and desktop workstations unencrypted — because they’re at least protected by locked office doors — but insist that mobile devices be encrypted. Recovering a locked VM You can apply the magic of chroot to clean up all kinds messes. Locked out of a local server (or LXC container)? Feel free to open a chroot shell to disable or even reconfigure your firewall. Getting that done on a physical machine should be straightforward by now. But here’s how it would work on an LXC container. First of all, stop the container and then run chroot against the rootfs directorythat’s within the directory hierarchy used by your LXC container ( var/lib/lxc/your-container-name/ ). The command prompt you’ll get will allow you to execute commands as if the container was actually running. Now disable ufw or, if you prefer, run the necessary commands to fix the problem and then exit the chroot shell. When you start the container back up again, you should now have SSH access. # lxc-stop -n your-container-name# chroot /var/lib/lxc/your-container-name/rootfs/# ufw disable# exit# lxc-start -d -n your-container-name This article is excerpted from my Manning “Linux in Action” book . There’s lots more fun where this came from , including a hybrid course called Linux in Motion that’s made up of more than two hours of video and around 40% of the text of Linux in Action. Who knows…you might also enjoy my recently published Learn Amazon Web Services in a Month of Lunches .

Amazon

SysAdmin books, courses, and other good stuff

See what other trouble David's been up to

Buy My Books

Take My Courses

Read My Stories

Chroot: the magical healing powers of the original Linux virtualization tool

Flow

Osmosis

Twitter

The Uprooting of Mind: Can AI Save Us?

This article was excerpted from chapter 9 of my Manning book, Linux in Action . Besides the book, you can also work through Linux in Motion — a hybrid course made up of more than two hours of video and around 40% of the text of Linux in Action. Building a LAMP server and getting it all nicely configured with reliable data handling, a domain, and a TLS certificate is only half the battle. You’ll also need to make sure your infrastructure is protected from the internet’s many frightening threats. In this article, I’ll explore website security through the proper use of system groups, process isolation, and regular audits of your system resources. It’s not the whole story (my covers additional tools like installing TLS certificates and working with SELinux), but it’s a great start. Linux in Action book System groups and the principle of least privilege The developers you support have (finally) come to realize that they need to restrict access to the data and configuration files living on the application server while still allowing access to various dev and IT teams. public The first part of the solution is . A group is a system object — much the same as a user — except that no one will ever log in to the system as a group. The power of groups is in how they, like users, can be “assigned” to files or directories, allowing any group members to share the group powers. This is illustrated in the figure. groups Developers who are members of the Developers group can be given access to a particular directory, as opposed to those individuals who are not part of the group Try this yourself: use a text editor to create a new file. Add some “Hello world” text so you’ll be able to easily tell when you can successfully access it. Now edit its permissions using chmod 770 so that the file’s owner and group have full rights over the file, but others can’t even read it. $ nano datafile.txt$ chmod 770 datafile.txt If your system doesn’t already have an extra user besides your account, createone using either adduser — the Debian/Ubuntu way — or useradd if you’re onCentOS. useradd will also work on Ubuntu. The useradd command (as opposed to the Debian adduser) requires you togenerate a user password separately: # useradd otheruser# passwd otheruserEnter new UNIX password:Retype new UNIX password:passwd: password updated successfully Use to switch to your new user. Once you enter the user’s password, all the commands you execute will be run as that user. You’ll be working with only that user’s authority: no more and no less. If you try reading the datafile.txt file (using cat ), you’ll have no luck since, as you remember, others were denied read permission. When you’re done, type exit to leave the new user shell and return to your original shell. su $ su otheruserPassword:$ cat /home/ubuntu/datafile.txtcat: /home/ubuntu/datafile.txt: Permission denied$ exit All this is expected and easy to understand. And, as you’ve seen, not being able to read the file belonging to a different reader can sometimes be a problem. Let’s see what we can do about it by associating the file with a group and then properly configuring the file’s permissions. Create a new group you can use to manage your application data and then edit the properties of your data file using chown . The ubuntu:app-data-group argument leaves the file ownership in the hands of the ubuntu user, but changes its group to your new app-data-group . # groupadd app-data-group# chown ubuntu:app-data-group datafile.txt Run ls with “long” output against the file to view its new permissions and status. Note that, as expected, ubuntu is the file’s owner and app-data-group is its group. $ ls -l | grep datafile.txt-rwxrwx — — 1 ubuntu app-data-group 6 Aug 9 22:43 datafile.txt You can use usermod to add your user to the app-data-group group and then,once again, su to switch to a shell deploying the other user’s account. This time, even though the file’s permissions lock others out — and you’re definitely acting as an “other” user right now — you should be able to read it thanks to your group membership. # usermod -aG app-data-group otheruser$ su otheruser$ cat datafile.txtHello World Use su to switch between user accounts. These happened to be the contents of my datafile.txt file. This kind of organization is the correct and effective way to deal with many of the complicated permissions issues that will arise on a multi-user system. In fact, not only is it used to give individual users the access they need, but many system processes couldn’t do their jobs without special group memberships. Take a quick look through the /etc/group file and note how many system processes have their own groups. A partial listing of the contents of the /etc/group file: $ cat /etc/grouproot❌0:daemon❌1:bin❌2:sys❌3:adm❌4:syslogtty❌5:disk❌6:lp❌7:mail❌8:news❌9:uucp❌10:man❌12:proxy❌13:[…] Isolating processes within containers Worried that the multiple services you’ve got running on a single server will, should one service be breached, all be at risk? One way to limit the damage that careless or malicious users can cause is by isolating system resources and processes. This way, even if someone might want to expand their reach beyond a set limit, they won’t have physical access. The old approach to the problem was provisioning a separate physical machine for each service. But virtualization can make it a lot easier -and more affordable - to build a “siloed” infrastructure. this architecture is often referred to as and would have you launch multiple containers with one, perhaps, running only a database, another Apache, and a third containing media files that might be embedded in your web pages. Besides the many performance and efficiency benefits associated with microservices architectures, this can greatly reduce each individual component’s risk exposure. microservices By “containers” I don’t necessarily mean those of the LXC persuasion.These days, for this kind of deployment, Docker containers are far morepopular. If you’re interested in learning more about Docker, check out that touch on the topic. my Pluralsight courses Scanning for dangerous User ID values While any admin user will be able to temporarily assume root authority using sudo, only root is actually root . As you’ve seen already, it isn’t safe to perform regular functions as root. But it can happen — whether by innocent accident or malicious tampering — that a regular user can effectively get admin rights full-time. The good news is that it’s easy to spot imposters: their user and/or group IDnumbers will, like root, be zero (0). Take a look at the passwd file in /etc/. This file contains a record for each regular and system user account that currently exists. The first field contains the account name (root and ubuntu in this case) and the second field might contain an x in place of a password (which, if it exists, will appear encrypted in the /etc/shadow file). But the next two fields contain the user and group IDs. In the case of ubuntu in this example, both IDs are 1000 . And, as you can see, root has zeroes. $ cat /etc/passwdroot❌0:0:root:/root:/bin/bash[…]ubuntu❌1000:1000::/home/ubuntu:/bin/bash If you ever see a regular user with a user or group ID of 0, however, then you know there’s something nasty going on and you should get to work fixing it. The quick and easy way to spot a problem is to run this awk command against the passwd file, which will print out any line whose third field contains only a 0. In this case, to my great relief, the only result was root . You can run it a second time substituting $4 for $3 to pick up the group ID field. $ awk -F: ‘($3 == “0”) {print}’ /etc/passwdroot❌0:0:root:/root:/bin/bash Auditing system resources The more things you’ve got running, the greater the odds of something breaking. So it makes sense that you’ll want to keep track of what’s running. This will apply to network ports (if they’re “open” then, by definition, there must be a way in), services (if they’re active, then people can run them), and installed software (if it’s installed, it can be executed). For audits to be useful you’ll have to remember to run them once in a while. Since you just know you’re going to forget, you’ll be much better off incorporating your auditing tools into a script that not only executes regularly but, ideally, also parses the results to make them more readable. Here, however, I’ll focus on introducing you to three key audit tools to help you scan for open ports, active services, and unnecessary software packages. Getting it automated will be your job. Scanning for open ports A port is considered “open” if there’s some process running on the host that’s listening on that port for requests. Keeping an eye on your open ports can keep you plugged into what’s really going on with your server. You already know that a regular web server is probably going to have HTTP (80) and SSH (22) open, so it shouldn’t come as a surprise to come across those. But you’ll really want to focus on other unexpected results. netstat will display open ports along with a wealth of information about how they’re being used. In this example run against a fairly typical multi-purpose server, -n tells netstat to include the numeric ports and addresses. -l includes only listening sockets, and -p adds the process ID of the listening program. Naturally, if you see something, do something. # netstat -nplActive Internet connections (only servers)Proto Local Address Foreign Address State PID/Program nametcp 127.0.0.1:3306 0.0.0.0:* LISTEN 403/mysqldtcp 0.0.0.0:139 0.0.0.0:* LISTEN 270/smbdtcp 0.0.0.0:22 0.0.0.0:* LISTEN 333/sshdtcp 0.0.0.0:445 0.0.0.0:* LISTEN 270/smbdtcp6 :::80 :::* LISTEN 417/apache2[…] In recent years, has begun to replace netstat for many uses. Just in case you find yourself at a party one day and someone asks you about , this example (which lists all established SSH connections) should give you enough information to save you from deep embarrassment: ss ss $ ss -o state established ‘( dport = :ssh or sport = :ssh )’Netid Recv-Q Send-Q Local Address:Port Peer Address:Porttcp 0 0 10.0.3.1:39874 10.0.3.96:sshtimer:(keepalive,18min,0) Scanning for active services Getting a quick snapshot of the systemd-managed services currently enabled on your machine can help you spot activity that doesn’t belong. systemctl can list all existing services, which can then be narrowed down to only those results whose descriptions include enabled. This will return only active services. # systemctl list-unit-files — type=service — state=enabledautovt@.service                       enabledbind9.service                         enabledcron.service                          enableddbus-org.freedesktop.thermald.service enableddocker.service                        enabledgetty@.service                        enabledhaveged.service                       enabledmysql.service                         enablednetworking.service                    enabledresolvconf.service                    enabledrsyslog.service                       enabledssh.service                           enabledsshd.service                          enabledsyslog.service                        enabledsystemd-timesyncd.service             enabledthermald.service                      enabledunattended-upgrades.service           enabledureadahead.service                    enabled If you do find something that shouldn’t be there, you can use systemctl to bothstop the service and make sure it doesn’t start up again with the next boot. # systemctl stop haveged# systemctl disable haveged There’s actually nothing dark and sinister about the service I’mstopping in this example: it’s a very small tool I often install to generaterandom background system activity when I’m creating encryption keys. haveged Searching for installed software Could someone or something have installed software on your system without you knowing? Well, how would you know if you don’t look? yum list installed or, on Debian/Ubuntu, dpkg — list will give you the whole briefing, while remove <packagename> should delete any packages that don’t belong. # yum list installed# yum remove packageName Here’s how it goes on Ubuntu: # dpkg --list# apt-get remove packageName It’s also a good idea to be aware of changes to your system configuration files -which is something I cover in chapter 11. This article is excerpted from my Manning “Linux in Action” book . There’s lots more fun where this came from , including a hybrid course called Linux in Motion that’s made up of more than two hours of video and around 40% of the text of Linux in Action.. Who knows…you might also enjoy my recently published Learn Amazon Web Services in a Month of Lunches .

Apache

Securing your Linux web server

Ethereum

Litecoin

Bitcoin Cash

Chain

BUNCH

Coinbase

Facebook

Google

Netflix

Target

YouTube

Crypto Newbie Gains: Your Quick Start Guide to Getting Comfortable with Cryptocurrencies and…

As cryptocurrencies have exploded in value, so too have the . Even more exciting is a recent uptick in . Spreadsheet models have risen in popularity as a tool for evaluating and predicting trends. attempts to understand them quantitative analysis The effort and analytic rigor behind these models is phenomenal. However, there are strong reasons why most top funds do use this as part of their evaluation methodology. Often, there is no objective measure — what am I trying to predict, price in 1 week? 1 year? 10 years? And the feature set is often not very predictive (are we really so sure that the number of Telegram users is a strong indicator of future price?). In this space, it’s all too easy to fall into the trap of analysis: building complex models that are not all that predictive once the assumptions and objectives are stated and the is tested. not cargo cult model To illustrate my point, I’ll walk through an example on an existing dataset and finish with my own take on token sale evaluation. All that’s required to follow along is a bit of basic statistics. I’ll post the code at the end of the article. Crunching the numbers For the sake of example, let’s use the following which has the benefit of being very detailed and complete (a godsend for anyone playing around with data). It tracks quite a few interesting metrics that intuitively seem very predictive: codebase activity (Github contributors/commits), community activity (Reddit/Telegram members), exchange listings, and public presence (tweets, news mentions). Most people I talk to use some combination of these as a major part of their decision process. spreadsheet We can write some quick code to read the CSV and dump it into a useful format: a grid of numbers across the feature space (our input matrix) and a list of numbers that we want to “predict” (more specifically, check for correlations). Before we do any number-crunching, there’s an important observation to make. There are 51 rows and 21 columns: a ton of data for a lowly human to process. But, to draw useful conclusions, you typically need much more data than this. Models that I’ve run in the past have been trained on at least tens of thousands of samples. , the go-to database for training computer vision algorithms, has over 14 million images! ImageNet The main issue with our dataset is something known as dimensionality.¹ In a spreadsheet, this can be be seen as the number of rows in the spreadsheet versus the number of columns. What we’re looking for is something that looks long and thin — a ton of rows but just a few columns. The dataset famously has over 1.4 million rows but only three columns (user, movie, and rating). Netflix Challenge The more complex the model (the more columns we have), the more samples we need for the data to generalize (the more rows we need to add so we can still trust our results). Without going into , the rule of thumb I’ve internalized is that . So, the next time you see someone making a model out of a square spreadsheet (dimensionality about the same as the sample size), you can be a bit more skeptical of the conclusions they draw. more detail you need at least ten times the rows as you have columns Back to our example. 51 rows of data is fine, but to keep things simple (and statistically significant), let’s start with checking correlations between pairs of data (a bivariate regression). Looking at the data The following section has some numbers and graphs. If that’s not really your thing, feel free to skip to the next section, “Ok, so what model should I use?” So we run the regression and it looks something like this: Total # Commits vs Market Cap R^2: 0.138249Total # Contributors vs Market Cap R^2: 0.130249One Month # Commits vs Market Cap R^2: 0.002144One Month # Contributors vs Market Cap R^2: 0.091301Telegram Members in Top Group vs Market Cap R^2: 0.159053Reddit Members vs Market Cap R^2: 0.806415Exchanges Listed vs Market Cap R^2: 0.298320# of Top 5 Exchanges vs Market Cap R^2: 0.150254# of Hashtag Tweets (30 days) vs Market Cap R^2: 0.368655# of News Mentions (30 days) vs Market Cap R^2: 0.771270Twitter Followers vs Market Cap R^2: 0.443522 For those of you unsure how to interpret this, the numbers to the right are the of the best fit line. As long as our data satisfy certain properties, we can be sure that the line is our of a correlation. coefficients of determination best possible guess Let’s take some examples. “Total # Commits” and “Market Cap” have an R² of 0.138249. That ends up looking something like this: Total # Commits vs Market Cap Probably not too useful. We’re looking for something that looks a little more like this: The closer R² is to 1, the stronger the correlation. Fortunately, our results have some strong contenders: “Reddit Members vs Market Cap” at 0.81 and “# of News Mentions (30 days) vs Market Cap” at 0.77. Let’s take a look at one of those graphs, “Reddit Members vs Market Cap”: Reddit Members vs Market Cap Oh no — even though the R² is pretty high, the graph doesn’t look like the one we’re expecting. That’s because the data suffer from something known as — where the points kind of fan out as the numbers get larger. This is something we see often in venture capital data because the sizes of companies grow by orders of magnitude, and with larger companies come larger variances. Fortunately, there’s a very easy fix: normalize the data on a logarithmic scale. Now, the plot looks much more correlative: heteroskedasticity Reddit Members vs Market Cap on a log-log plot Total # Commits vs Market Cap R^2: 0.083247Total # Contributors vs Market Cap R^2: 0.094291One Month # Commits vs Market Cap R^2: 0.091181One Month # Contributors vs Market Cap R^2: 0.151921Telegram Members in Top Group vs Market Cap R^2: 0.130186Reddit Members vs Market Cap R^2: 0.357688Exchanges Listed vs Market Cap R^2: 0.303370# of Top 5 Exchanges vs Market Cap R^2: 0.395840# of Hashtag Tweets (30 days) vs Market Cap R^2: 0.426500# of News Mentions (30 days) vs Market Cap R^2: 0.484609Twitter Followers vs Market Cap R^2: 0.353295 The correlations took a hit: the R² of “Reddit Members vs Market Cap” is down from 0.81 to 0.36. But these values aren’t bad. At the risk of offending the readers, a more precise evaluation of how “not bad” these values are is to perform a on the coefficients of the regressors. Without going into too much detail, the idea is to compare your model against a model that assumes no relationship (just a horizontal line). If there is enough evidence to show that adding “Reddit Members” significantly improves the model, we can, with confidence, say there is some relationship. Bayesian hypothesis test The formal measure of this is the derived from the , which represents the likelihood our data would’ve been generated purely by random chance — that is, if there were no actual relationship between “Reddit Members” and “Market Cap”. Put another way, if the p-value is very small, there’s almost no chance that the data we’re looking at could’ve been randomly generated. There be some relationship. p-value F-statistic must Most academic studies look for p-values less than either 0.05 or 0.01. In fact, the p-values we get hover around numbers very close to zero. It seems that the popularity of a might be nontrivially related to its market cap after all (imagine that)! Though it seems that developer activity is a bit less useful. cryptocurrency Total # Commits vs Market Cap p-value: 0.074861Total # Contributors vs Market Cap p-value: 0.057240One Month # Commits vs Market Cap p-value: 0.061724One Month # Contributors vs Market Cap p-value: 0.014175Telegram Members in Top Group vs Market Cap p-value: 0.009292Reddit Members vs Market Cap p-value: 0.000004Exchanges Listed vs Market Cap p-value: 0.000028# of Top 5 Exchanges vs Market Cap p-value: 0.000001# of Hashtag Tweets (30 days) vs Market Cap p-value: 0.000000# of News Mentions (30 days) vs Market Cap p-value: 0.000000Twitter Followers vs Market Cap p-value: 0.000007 But think about this for a second. All these p-values mean is that you can perhaps eyeball a general trend of dots moving up and to the right in the plots. How does this inform which cryptocurrencies you should buy? Do you buy the currencies at the top right — the ones with the most Reddit members and highest market caps? Do you buy all currencies above the trend line because their market caps are proportionally higher per additional Reddit member? We’ve talked ad nauseam about correlations — the relationships between variables. But, it turns out that prediction is an entirely different problem. This is where the question of comes in: does being in the news lead to a higher market cap, or does a higher market cap lead to more news coverage (or both)? correlation vs. causation We can do a pretty quick sanity check. Instead of comparing these factors to market cap, we can instead compare them to the in market cap since the spreadsheet was made. When running the code over the new data (current as of 2/13/2018), the plots no longer look so meaningful. change Reddit Members vs Change in Market Cap on a log-log plot Removing the outliers makes the randomness even more apparent. Reddit Members vs Change in Market Cap on a log-log plot, sans outliers Now, the implied correlation is ! More Reddit users apparently leads to decreases in market cap. We can blame this more on fitting noisy data rather than any negative relationship, as evidenced by the final set of p-values. negative Total # Commits vs Market Cap Change p-value: 0.839070Total # Contributors vs Market Cap Change p-value: 0.709478One Month # Commits vs Market Cap Change p-value: 0.227949One Month # Contributors vs Market Cap Change p-value: 0.615143Telegram Members in Top Group vs Market Cap Change p-value: 0.019781Reddit Members vs Market Cap Change p-value: 0.810512Exchanges Listed vs Market Cap Change p-value: 0.311267# of Top 5 Exchanges vs Market Cap Change p-value: 0.512172# of Hashtag Tweets (30 days) vs Market Cap Change p-value: 0.930046# of News Mentions (30 days) vs Market Cap Change p-value: 0.698512Twitter Followers vs Market Cap Change p-value: 0.821432 Most of these values are well outside our tolerance of statistical significance and are more or less entirely random. Interestingly enough, for the one feature that might be related, “Telegram Members”, the correlation is negative, which implies more Telegram members relates to a decrease in price. But, I wouldn’t read so much into that. The only conclusion I’d draw is that the data we have do not predict what we expect. Ok, so what model should I use? So what does this mean? Can we definitively say the number of Reddit or Telegram members has no predictive power? Unfortunately not. There’s nothing ruling out better models or correlations that come out of larger and more comprehensive datasets. For example, try using “change in Telegram members” as a regressor, using a more expressive model, or framing the problem as a .² The point I’m making is that, with the current dataset, it’s hard to imagine a truly predictive model without some change. classification problem The larger point is this: if you construct a model, start with a formalization of your objective and assumptions. Make sure it’s testable and, more importantly, verifiable/falsifiable. If it works — fantastic, you now have an edge!³ If not, back to the drawing board. Often, when predicting future prices of an asset, you’re probably looking for . There is an devoted to proving that these might exist devoted to disproving it. In public equities markets, it’s commonly believed that prices follow a random walk or, more specifically, a ( ). That is, the expected change in stock price is not influenced by the previous prices. This is the source of the commonly held belief that the stock market is . According to some of my quant fund friends, there is reason to believe that this might not hold true just yet in cryptocurrency markets. Perhaps there are predictive signals of future price movements (past price, social media activity, etc.), but the results are yet unpublished. At best, this requires more analysis.⁴ autocorrelations entire body of economic work as well as another body of work martingale or not? impossible to predict So what did we learn? A few things: The data that we might believe are meaningful are probably less than we think. It’s easy to fool ourselves into thinking we have a good strategy or model. Often times, we look for patterns that aren’t really there. This can be avoided with good ol’ data analysis. Quantitative analysis is hard to get right. Correlations are an important first step. But there is a much larger body of work around what we’re really trying to do: predict future price. As promised, here’s a link to the code. Feel free to play around with it on your own, and let me know if you find any bugs: https://github.com/opentoken-inc/crypto-dd Update as of 3/24: Here are a few footnotes to clarify some technical points. Footnotes When considering how well a model will generalize, model complexity relative to the number of samples is the right thing to look at. Input dimensionality probably isn’t, but we use it in this example because it is easier to visualize the shape of a spreadsheet than parameters in a model. Although input dimensionality and model complexity aren’t always the same, in this case they are: we’re considering linear models with the same number of parameters as inputs. In practice, you might use dimensionality reduction methods such as to pick a more meaningful subset of features. This is often an early step in quantitative models for price prediction over much larger datasets. PCA More specifically, you have a backtested model that performs well on historical datasets. However, it’s well known that a backtested model doesn’t guarantee good live performance. But, at the very least, you have confidence that your model have some predictive power. might Autocorrelations may not be directly used for price prediction, but a lack of autocorrelations may correspond to a difficulty in finding predictive signals.

The Graph

Quant

Building A Good Cryptocurrency Model Is Harder Than You Think

In , we introduced the . We learned about the type and the basic combinators like and . We saw how we could combine those together with applicative functions like and to parse strings into data structures. This week, we’ll put these pieces together in an actual parser for our Gherkin syntax. To follow along with the code examples, check out on the Github repository. last week’s article Applicative parsing library RE sym string many <*> Parser.hs Starting next week, we’ll explore some other parsing libraries, starting with . For a little more information about those and many other libraries, download our It summarizes many libraries on topics from databases to Web APIs. Attoparsec Production Checklist! If you’ve never written Haskell at all, get started! Download our free . Beginners Checklist! Value Parser In keeping with our approach from the last article, we’re going to start with smaller elements of our syntax. Then we can use these to build larger ones with ease. To that end, let’s build a parser for our type, the most basic data structure in our syntax. Let’s recall what that looks like: Value data Value =ValueNull |ValueBool Bool |ValueString String |ValueNumber Scientific Since we have different constructors, we’ll make a parser for each one. Then we can combine them with alternative syntax: valueParser :: RE Char ValuevalueParser =  nullParser <|>  boolParser <|>  numberParser <|>  stringParser Now our parsers for the null values and boolean values are easy. For each of them, we’ll give a few different options about what strings we can use to represent those elements. Then, as with the larger parser, we’ll combine them with . <|> nullParser :: RE Char ValuenullParser =  (string “null” <|>  string “NULL” <|>  string “Null”) *> pure ValueNull

boolParser :: RE Char ValueboolParser =  trueParser *> pure (ValueBool True) <|>   falseParser *> pure (ValueBool False)  where    trueParser = string “True” <|> string “true” <|> string “TRUE”    falseParser = string “False” <|> string “false” <|> string “FALSE” Notice in both these cases we discard the actual string with and then return our constructor. We have to wrap the desired result with . *> pure Number and String Values Numbers and strings are a little more complicated since we can’t rely on hard-coded formats. In the case of numbers, we’ll account for integers, decimals, and negative numbers. We’ll ignore scientific notation for now. An integer is simple to parse, since we’ll have many characters that are all numbers. We use instead of to enforce that there is at least one: some many numberParser :: RE Char ValuenumberPaser = …  where    integerParser = some (psym isNumber) A decimal parser will read some numbers, then a decimal point, and then more numbers. We’ll insist there is at least one number after the decimal point. numberParser :: RE Char ValuenumberPaser = …  where    integerParser = some (psym isNumber)    decimalParser =       many (psym isNumber) <*> sym ‘.’ <*> some (psym isNumber) Finally, for negative numbers, we’ll read a negative symbol and then one of the other parsers: numberParser :: RE Char ValuenumberPaser = …  where    integerParser = some (psym isNumber)    decimalParser =       many (psym isNumber) <*> sym ‘.’ <*> some (psym isNumber)    negativeParser = sym ‘-’ <*> (decimalParser <|> integerParser) However, we can’t combine these parsers as is! Right now, they all return different results! The integer parser returns a single string. The decimal parser returns two strings and the decimal character, and so on. In general, we’ll want to combine each parser’s results into a single string and then pass them to the function. This requires mapping a couple functions over our last two parsers: read numberParser :: RE Char ValuenumberPaser = …  where    integerParser = some (psym isNumber)    decimalParser = combineDecimal <$>       many (psym isNumber) <*> sym ‘.’ <*> some (psym isNumber)    negativeParser = (:) <$>       sym ‘-’ <*> (decimalParser <|> integerParser)

    combineDecimal :: String -> Char -> String -> String    combineDecimal base point decimal = base ++ (point : decimal) Now all our number parsers return strings, so we can safely combine them. We’ll map the constructor over the value we read from the string. ValueNumber numberParser :: RE Char ValuenumberPaser = (ValueNumber . read) <$>  (negativeParser <|> decimalParser <|> integerParser)  where    ... Note that order matters! If we put the integer parser first, we’ll be in trouble! If we encounter a decimal, the integer parser will greedily succeed and parse everything before the decimal point. We’ll either lose all the information after the decimal, or worse, have a parse failure. The last thing we need to do is read a string. We need to read everything in the example cell until we hit a vertical bar, but then ignore any whitespace. Luckily, we have the right combinator for this, and we’ve even written a function already! trim stringParser :: RE Char ValuestringParser = (ValueString . trim) <$> readUntilBar And now our will work as expected! valueParser Building an Example Table Now that we can parse individual values, let’s figure out how to parse the full example table. We can use our individual value parser to parse a whole line of values! The first step is to read the vertical bar at the start of the line. exampleLineParser :: RE Char [Value]exampleLineParser = sym ‘|’ *> ... Next, we’ll build a parser for each cell. It will read the whitespace, then the value, and then read up through the next bar. exampleLineParser :: RE Char [Value]exampleLineParser = sym ‘|’ *> ...  where    cellParser =       many isNonNewlineSpace *> valueParser <* readThroughBar

isNonNewlineSpace :: RE Char CharisNonNewlineSpace = psym (\c -> isSpace c && c /= ‘\n’) Now we read of these and finish by reading the newline: many exampleLineParser :: RE Char [Value]exampleLineParser =   sym ‘|’ *> many cellParser <* readThroughEndOfLine  where    cellParser =       many isNonNewlineSpace *> valueParser <* readThroughBar Now, we need a similar parser that reads the title column of our examples. This will have the same structure as the value cells, only it will read normal alphabetic strings instead of values. exampleColumnTitleLineParser :: RE Char [String]exampleColumnTitleLineParser = sym ‘|’ *> many cellParser <* readThroughEndOfLine  where    cellParser =       many isNonNewlineSpace *> many (psym isAlpha) <* readThroughBar Now we can start building the full example parser. We’ll want to read the string, the column titles, and then the value lines. exampleTableParser :: RE Char ExampleTableexampleTableParser =  (string “Examples:” *> readThroughEndOfLine) *>  exampleColumnTitleLineParser <*>  many exampleLineParser We’re not quite done yet. We’ll need to apply a function over these results that will produce the final . And the trick is that we want to map up the example keys with their values. We can accomplish this with a simple function. It will return zip the keys over each value list using : ExampleTable map exampleTableParser :: RE Char ExampleTableexampleTableParser = buildExampleTable <$>  (string “Examples:” *> readThroughEndOfLine) *>  exampleColumnTitleLineParser <*>  many exampleLineParser  where    buildExampleTable :: [String] -> [[Value]] -> ExampleTable    buildExampleTable keys valueLists = ExampleTable keys (map (zip keys) valueLists) Statements Now we that we can parse the examples for a given scenario, we need to parse the Gherkin statements. To start with, let’s make a generic parser that takes the keyword as an argument. Then our full parser will try each of the different statement keywords: parseStatementLine :: String -> RE Char StatementparseStatementLine signal = …

parseStatement :: RE Char StatementparseStatement =  parseStatementLine “Given” <|>  parseStatementLine “When” <|>  parseStatementLine “Then” <|>  parseStatementLine “And” Now we’ll get the signal word out of the way and parse the statement line itself. parseStatementLine :: String -> RE Char StatementparseStatementLine signal = string signal *> sym ' ' *> ... Parsing the statement is tricky. We want to parse the keys inside brackets and separate them as keys. But we also want them as part of the statement’s string. To that end, we’ll make two helper parsers. First, will parse everything in a string up through a bracket (or a newline). nonBrackets nonBrackets :: RE Char StringnonBrackets = many (psym (\c -> c /= ‘\n’ && c /= ‘<’)) We’ll also want a parser that parses the brackets and returns the keyword inside: insideBrackets :: RE Char StringinsideBrackets = sym ‘<’ *> many (psym (/= ‘>’)) <* sym ‘>’ Now to read a statement, we start with non-brackets, and alternate with keys in brackets. Let’s observe that we start and end with non-brackets, since they can be empty. Thus we can represent a line a list of non-bracket/bracket pairs, followed by a last non-bracket part. To make a pair, we combine the parser results in a tuple using the constructor enabled by : (,) TupleSections parseStatementLine :: String -> RE Char StatementparseStatementLine signal = string signal *> sym ‘ ‘ *>  many ((,) <$> nonBrackets <*> insideBrackets) <*> nonBrackets From here, we need a recursive function that will build up our final statement string and the list of keys. We do this with . buildStatement parseStatementLine :: String -> RE Char StatementparseStatementLine signal = string signal *> sym ‘ ‘ *>  (buildStatement <$>     many ((,) <$> nonBrackets <*> insideBrackets) <*> nonBrackets)  where    buildStatement ::       [(String, String)] -> String -> (String, [String])    buildStatement [] last = (last, [])    buildStatement ((str, key) : rest) rem =      let (str', keys) = buildStatement rest rem      in (str <> "<" <> key <> ">" <> str', key : keys) The last thing we need is a final helper that will take the result of and turn it into a . We’ll call this , and then we’re done! buildStatement Statement finalizeStatement parseStatementLine :: String -> RE Char StatementparseStatementLine signal = string signal *> sym ‘ ‘ *>  (finalizeStatement . buildStatement <$>     many ((,) <$> nonBrackets <*> insideBrackets) <*> nonBrackets)  where    buildStatement ::       [(String, String)] -> String -> (String, [String])    buildStatement [] last = (last, [])    buildStatement ((str, key) : rest) rem =      let (str', keys) = buildStatement rest rem      in (str <> "<" <> key <> ">" <> str', key : keys)

    finalizeStatement :: (String, [String]) -> Statement    finalizeStatement (regex, variables) = Statement regex variables Scenarios Now that we have all our pieces in place, it’s quite easy to write the parser for scenario! First we get the title by reading the keyword and then the rest of the line: scenarioParser :: RE Char ScenarioscenarioParser = string “Scenario: “ *> readThroughEndOfLine ... After that, we read many statements, and then the example table. Since the example table might not exist, we’ll provide an alternative that is a pure, empty table. We can wrap everything together by mapping the constructor over it. Scenario scenarioParser :: RE Char ScenarioscenarioParser = Scenario <$>  (string “Scenario: “ *> readThroughEndOfLine) <*>  many (statementParser <* sym ‘\n’) <*>  (exampleTableParser <|> pure (ExampleTable [] [])) We can also make a “Background” parser that is very similar. All that changes is that we read the string “Background” instead of a title. Since we’ll hard-code the title as “Background”, we can include it with the constructor and map it over the parser. backgroundParser :: RE Char ScenariobackgroundParser = Scenario “Background” <$>  (string “Background:” *> readThroughEndOfLine) *> many (statementParser <* sym ‘\n’) <*>  (exampleTableParser <|> pure (ExampleTable [] [])) Finally the Feature We’re almost done! All we have left is to write the itself! As with scenarios, we’ll start with the keyword and a title line: featureParser featureParser :: RE Char FeaturefeatureParser = Feature <$>  (string “Feature: “ *> readThroughEndOfLine) <*>  ... Now we’ll use the combinator to parse the if it exists, but return if it doesn’t. Then we’ll wrap up with parsing many scenarios! optional Background Nothing featureParser :: RE Char FeaturefeatureParser = Feature <$>  (string “Feature: “ *> readThroughEndOfLine) <*>  (optional backgroundParser) <*>  (many scenarioParser) Note that here we’re ignoring the “description” of a feature we proposed as part of our original syntax. Since there are no keywords for that, it turns out to be painful to deal with it using applicative parsing. When we look at monadic approaches starting next week, we’ll see it isn’t as hard there. Conclusion This wraps up our exploration of applicative parsing. We can see how well suited Haskell is for parsing. The functional nature of the language means it’s easy to start with small building blocks like our first parsers. Then we can gradually combine them to make something larger. It can be a little tricky to wrap our heads around all the different operators and combinators. But once you understand the ways in which these let us combine our parsers, they make a lot of sense and are easy to use. To further your knowledge of useful Haskell libraries, download our free ! It will tell you about libraries for many tasks, from databases to machine learning! Production Checklist If you’ve never written a line of Haskell before, never fear! Download our to learn more! Beginners Checklist

Different

Applicative Parsing II: Putting the Pieces Together

Anyone getting toe-deep into the world of crypto-currencies understands there is a scalability crisis there. 2017 has made it evident to users of the platform resulting in High Commissions when trading bitcoin High gas prices in the Ethereum platform The scalability issue, if inherent to the platform and concept, can be deadly to its future. Most current altcoins that brag about low commissions only have them because they are not yet of scale, but face the same inherent obstacles bitcoin is facing and thus their bragging is laughable. I will examine the areas where scaling issues appear in blockchain, and the solutions that were suggested to tackle them. — The Blockchain as suggested by Satoshi Nakamoto has two limitations that prevent it from scaling — Constant Block size and setting an average difficulty of arriving at a valid block to a constant time. The combination of the two at the current settings yields ~3 transactions per second (tps), much less than achieved by centralized alternatives such as Paypal or Visa. What’s worse, lowering the average time for block calculation would result in many forks and network splitting, so it is not a viable solution. Removing the block size limit would enable DOS attacks as very large invalid blocks could be sent to nodes and congest their network traffic. Thus protocol changes of some kind are required to tackle these issues, as the limitations in the original setting seem to be important. Throughput The reason the tps problem is so important, beside constricting the viability of cryptocurrencies as full blown replacements to existing online payment solutions at scale, is its effect on commissions — commissions rise when many people wish to transact — high demand, but the supply is limited due to architecture reasons. The same goes for Ethereum storage prices — as there is only as much data that can be written to the blockchain per second, costs are very high. — In bitcoin a transaction is never fully validated. As the blockchain is defined as the longest chain of valid blocks, an extremely strong miner may appear at any point to traverse the history of the entire existing blockchain by creating a fork from a previous block that will become longer than the existing one. This is very hard to do though, and so in bitcoin the reasonable wait time is 6 blocks, after which it is believed no one will be able to pull such a trick. With block validation time averaging at 10 minutes, this means a merchant receiving funds will have to wait circa 1 hour to know the payment he received is trustworthy enough. The alternative in centralized payments is a few seconds and one can think of a bundle of use cases where this feature is critical. Validation latency — To prevent and , blockchain uses Proof of work (PoW). Letting alone the environmental effects of it, this has a peculiar effect on the coin which I will thus analyze. Of course, in some way overall costs of energy miners take on themselves have to percolate to the whole coin users. The way it actually happens is that when coin price are rising due to high demand, the equilibrium of how many new miners will take advantage of this to generate more coins is skewed. Thus prices are rising more than they would under normal circumstances where many more miners would join in the game, making the currency unstable as a means of exchange. In other words, assuming rational miners, the equation holds Security Sybil attacks Double spending attacks C + P * S >= M With — C — Total of commissions paid to date P — Current Price of Bitcoin S — Current Supply of Bitcoins M — Total mining costs to date This means one of the following undesired consequences - Unstoppable coin deflation, leading it to be unable to serve as a reliable medium of exchange. Commission prices spiking I am planning a future blog post concerning coin price stability, where I will discuss this specific point more. My Point of view is that of the three, throughput is the worst and most concerning issue. Validation times are important for many use-cases but not all, but currently the main part of lags in bitcoin exchanges occur because of the throughput issue, so the throughput issue creates a latency issue, which does not happen vice-versa. The security scaling issue, though it’s the one dealt with the most and is the most hyped for some reason, is the least concerning one. I don’t see it as having very consequential effects on the future of the technology or its current state, and it has many suggested alternatives. I will not discuss it here. I will now go through the existing proposed solutions. - Bitcoin-ng Instead of the hash-riddle being solved and submitted together with the validated block, it is solved beforehand in a special block, and then allows the solver to issue as many transactions as ‘mini-blocks’ until another solver appears on the chain. Thus, bitcoin-ng brings two new ideas to the table - There is no block size limit. The miner can add as many transactions as he can process. Even if some throttling is imposed, effective block size will be much higher than today’s blockchain. The miner can use the time he dedicates for Proof of Work to propagate the blocks through the network. This can save much time and unnecessary forks. Analyzing these two ideas, I would say the first seems like a good feature but it is dependent on the second, as otherwise as mentioned before it would be equivalent to forking bitcoin to have no block size limit, which will result in problems described . here The Second idea is undoubtedly beneficial but introduces new risks that are not addressed in the Bitcoin-Ng white paper. Basically, the knowledge of who validates transactions propagates to the network when the validator is still actively validating transactions, rather than hand in hand with the final set of transactions he’s validated. This is valuable information, that may cause behaviors such as - — adversaries will send invalid transactions to the node, in quantities that are enough to hold it from processing the valid transactions. This is hard to duplicate as an attack on all nodes simultaneously, but much easier achieved when an isolated node become the single point of failure for the entire network. To perform this attack, an adversary will need to traverse a blockchain address to a mining node, but this may be achieved given the node would send a huge bulk of ‘implicating’ micro-blocks, in a way that would expose him to all first-tier nodes. DOS — It will be much easier to implement the following scenario — the chosen validator will publish a log of pending transactions, and ask commissions to include them in his micro-blocks. The Scenario is say on black friday, Amazon wishes to cause DOS to small merchants, so that all deals would flow through Amazon. They would then offer commissions to the validator not to process these deals — in an interactive bidding way that is impossible to propagate through a network with innumerable amount of possible future validators. Bribes Iota Iota is probably the most innovative protocol in the way they go out of the original blockchain conventions and way of thinking. In a few sentences, instead of a chain of blocks there is a DAG — Directed Acyclic Graph of transactions, where anyone can add his own transactions using PoW, pointing to two previous transactions. Double spending is tackled by a very strong herd-mentality initiative for the platform users. You want your transaction to be linking to the main branch of the DAG, and so if it is very well connected at some point — causing it to be approved by the merchant — it will be very hard to make a double spending transaction more well connected than it in the future. This is the basic intuition that you can read more about in and the . iota white paper tangle equilibrium paper Still, it seems that: The scenario described in the bitcoin-ng discussion where an entity prevents other entities from using the blockchain is even easier to achieve in Iota, using a variation of Aviv Zohar’s splitting attack. This will not require bribing any Single point of failure but rather using a small amount of computational capability to keep the transaction from ever acquiring enough validity. In general their white paper’s response to the splitting attack is multiple heuristics that do not amount to anything verifiable, some requiring deus ex-machinas such as quote “Another effective method for defending against a splitting attack would be for a sufficiently powerful entity to instantaneously publish a large number of transactions on one branch, thus rapidly changing the power balance and making it difficult for the attacker to deal with this change” This informal attitude is quite unique with iota and it’s highly disturbing as their protocol is so different from the others and many other flaws seem possible in it. Everything is presumably verified with their own conducted undisclosed simulations, not with proofs or thorough logical discussion. This is enough for me to not consider them a real future scaling solution at the moment, though I believe they introduced very interesting probabilistic elements into the cryptocurrency building blocks. SPECTRE I recommend first watching Aviv Zohar’s video lecture explaining the SPECTRE architecture . here In a few sentences, SPECTRE uses a DAG where each user submits his blocks using PoW pointing to all visible tips of the graph. Double spending is tackled by having a pairwise order between the blocks, so that if block A > block B, a node will accept contradicting transactions according to A. It is highly likely that if block A > block B at some point, it will stay this way due to the way this ordering is calculated — SPECTRE voting mechanism. In fact, there is a great similarity between Spectre voting mechanism and Iota’s random walk equilibrium — They also even both suffer from splitting attacks, though in Spectre it is mitigated. if it could be proven that they are essentially the same, it could be a huge thing for both Iota and Spectre — dealing with the voting complexity of Spectre and the lack of formal proof in Iota. In the voting mechanism, to determine block A ? block B, all blocks in the DAG vote on who they ‘prefer’. The more a block is linked to and from, the more votes it will get. This means future blocks will only amplify well-linked blocks and so double spending is highly unlikely, once a transaction was accepted. The thing though about this voting, is it needs to be done for each small block, on every pair of two other small blocks. It seems as the DAG grows larger and transactions become more frequent, this could be in the trillions of computations per second and much more, just to keep verifying everything — let alone generating PoW for submitting anything to the DAG. Say you have 100 transactions per second rate, these new blocks will need to vote on the ordering of ~N² blocks of the DAG, which at this transaction rate can amount in a year to ~3B blocks. Without going into implementation details and optimizations, on its face it looks a challenge today’s strongest super computer can’t deal with. Aviv Zohar claims it’s not a big deal and the guys at DagLabs who are working on implementing the protocol obviously think it’s not a major flow that will hold them back. I did not see a concise discussion of this matter. I would say though there seems to be a lot of possible paths for optimizing a naive implementation of voting, so it might be solvable, but for me it’s the biggest cloud hanging over SPECTRE at the moment. Bitcoin cash Raising the block size gradually over time, as seems to be the solution advocated by bitcoin cash will work, but it will depend heavily on the integrity of the community for years and years to come. This will need to balance between two different factors — the demand for blockchain transactions, and the computational and network capability of modern day computers — so DOS will not be a threat. While this will work, it is not a leap forward in scaling, such that will enable Ethereum’s vision of Dapps, Iota’s vision of IOT microtransactions, and it will even might face future constrictions in raising block size for native bitcoin transactions, if at some point demand will rise much more than what current viable choices of block sizes enable. Also worth noting & To Summarize Iota and SPECTRE also solve the latency issue, bringing it to a few seconds. As for implementation, bitcoin-ng is currently running in beta under the waves-ng platform. Iota is one of the dominant altcoins, though it is not clear how much the network follows the white paper versus how many hacks and centrally controlled elements are still incorporated into it. You could say it’s still in beta with a ~4.8B$ market cap as of this blog post publication. The insane world of Cryptocurrencies. SPECTRE is being implemented by Jerusalem based DagLabs and is expected to be released to beta by 2018Q4. Bitcoin cash is running as the second most dominant fork of bitcoin — the first being bitcoin of course. Only time will tell what comes out from crypto currencies scaling. It doesn’t seem to me like someone pulled the chestnuts out of the fire here and cryptocurrency scaling is now just a matter of waiting for some protocol to be implemented. I would say my favorite is SPECTRE, if indeed the voting will prove itself to be a non-issue. Otherwise, it may just be that the most simplistic attitude, of ‘scaling as it comes’ advocated by the bitcoin-cash guys could go for the win. Iota and bitcoin-ng are very innovative but I’m not sure how they will survive the test of time. To Be Briefly Mentioned — is a good idea, but helps mainly small scale applications. Ethereum sharding — Very recently published by the authors of SPECTRE, I was not able to refer to it yet but I will add the link for further reading. Phantom — Main advantage it brings to the table is around reducing latency. I’m yet to figure out whether it’s a heuristic solution that compromises security or a viable error-free solution. Ripple — Though advocated as a huge thing for Bitcoin, It’s not a real protocol solution to the issues, rather a mitigation that raises the question why not just use third party services and be done with it. Lightning networks I want to thank Michael Mirkin for discussing these issues together in a great process that I believe will produce many results in the future. He’s not to be held responsible though for any mistakes or wild claims I have made here. Further reading: https://techcrunch.com/2018/02/08/whats-needed-to-unlock-the-real-power-of-blockchain-and-distributed-apps/ https://hackernoon.com/blockchains-dont-scale-not-today-at-least-but-there-s-hope-2cb43946551a https://ethereum.stackexchange.com/questions/872/what-is-the-cost-to-store-1kb-10kb-100kb-worth-of-data-into-the-ethereum-block/896 https://medium.com/loom-network/scalability-tradeoffs-why-the-ethereum-killer-hasnt-arrived-yet-8f60a88e46c0 Bitcoin-ng white paper — https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eyal.pdf Phantom https://eprint.iacr.org/2018/104.pdf

IOTA

Waves

Super

Techcrunch

Visa

State of Cryptocurrencies Scaling

February 2018. : this post explores how to send requests to a node in order to create raw transactions. The goal is to understand and see what is going on in the background when using a high level library such as or . Goal JSON-RPC Geth web3py web3js Also I am not interested in . If anything goes wrong it will . This post is indented to be educational only. For production, consider using . handling errors and exceptions fail loudly web3py : we will deploy and interact (calling functions and reading variables) with a smart contract on a private network using only HTTP requests. Transactions are signed and only then sent to a geth node for processing. Summary public offline For this guide I am using a private network. If you would like to create such a network I invite you to read my . Otherwise using (previously called TestRPC) or any Ethereum is totally fine. Therefore I am not gonna cover anything about setting up a network here as the focus is using to send HTTP requests to a Geth node. Proof-of-Authority previous post Ganache network python If you just wanna see the full code, jump at the last Chapter. Prerequisite Having access to a Ethereum network over IPC or RPC (might it be public, private, or a simulator like Ganache). Having python 3 installed. Personally I like the . Anaconda distribution Install the latest version of . web3py 1. Sending a simple request to Geth Let’s warm up by sending a very simple request to Geth. Like asking the network id. The first step is to . The method we need is called and is described . read the doc net_version here My Geth node URL and Port are: . If you’re using Ganache with default value, the URL is probably . http://localhost:8501 http://localhost:7545 I am using the to make my HTTP requests. Enough talking. Requests python library import requests # create persistent HTTP connectionsession = requests.Session() # as defined in method = 'net_version'params = []payload= {"jsonrpc":"2.0","method":method,"params":params,"id":1} https://github.com/ethereum/wiki/wiki/JSON-RPC#net_version headers = {'Content-type': 'application/json'} response = session.post(' , json=payload, headers=headers)print('raw json response: {}'.format(response.json()))print('network id: {}'.format(response.json()['result'])) http://localhost:8501' prints: raw json response: {'id': 1, 'jsonrpc': '2.0', 'result': '1515'}network id: 1515 Goob job ! From there we are ready to deploy and transact with a contract. Better build on good foundations. 1515 is the network id of my private Blockchain as defined in the genesis file. Everything looks great. With Ganache you should get 5777 for the network id. But before being able to sign and send a transaction, we need an address, a private key and some ether. 2. Creating a private-public key pair and getting some ether The web3py (release 4) library will help us create a key pair. import web3 w3 = web3.Web3() myAccount = w3.eth.account.create('put some extra entropy here')myAddress = myAccount.addressmyPrivateKey = myAccount.privateKey print('my address is     : {}'.format(myAccount.address))print('my private key is : {}'.format(myAccount.privateKey.hex())) in my case, I get: my address is    : 0xF464A67CA59606f0fFE159092FF2F474d69FD675my private key is: 0x94cb9f766ef067eb229da85213439cf4cbbcd0dc97ede9479be5ee4b7a93b96f Please ! I do it here because it is a local private development network that I am destroying and reboot a few times a day. I am not using this key pair on any public network. NEVER SHARE YOUR PRIVATE KEY Now to get some ether to this address there are multiple ways: 1. A very simple way is to add this address in your file and to start a new network. From my , here is my genesis file that includes the address (remove ) we have just created. genesis.json previous guide 0x {"config": {"chainId": 1515,"homesteadBlock": 1,"eip150Block": 2,"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000","eip155Block": 3,"eip158Block": 3,"byzantiumBlock": 4,"clique": {"period": 5,"epoch": 30000}},"nonce": "0x0","timestamp": "0x5a722c92","extraData": "0x000000000000000000000000000000000000000000000000000000000000000008a58f09194e403d02a1928a7bf78646cfc260b087366ef81db496edd0ea2055ca605e8686eec1e60000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","gasLimit": "0x8000000","difficulty": "0x1","mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000","coinbase": "0x0000000000000000000000000000000000000000","alloc": {"08a58f09194e403d02a1928a7bf78646cfc260b0": {"balance": "0x200000000000000000000000000000000000000000000000000000000000000"},"87366ef81db496edd0ea2055ca605e8686eec1e6": {"balance": "0x200000000000000000000000000000000000000000000000000000000000000"},"F464A67CA59606f0fFE159092FF2F474d69FD675": {"balance": "0x200000000000000000000000000000000000000000000000000000000000000"}},"number": "0x0","gasUsed": "0x0","parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"} 2. If you have a mining node or ganache, open a and create a transaction by hand Geth Javascript Console $ geth attach ipc:'http://localhost:8501' // 7545 for ganacheWelcome to the Geth JavaScript console! instance: Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9coinbase: 0x87366ef81db496edd0ea2055ca605e8686eec1e6at block: 1585 (Wed, 14 Feb 2018 11:46:04 CET)modules: eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0 > eth.sendTransaction({'from':eth.coinbase, 'to':'0xF464A67CA59606f0fFE159092FF2F474d69FD675', 'value':1000000000000000000000})"0xdbc86acbe3644ac2cdb68132bbeecda40733c10f07ca16d87a2e5001e50eec4c"> exit here I am sending 1000 ether from to my address . (1 followed by 18 zeros). The . 0x87366... 0xF464A... 1 ether is 1e18 wei unit of the [value](https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_sendtransaction) field is wei 3. On public testnets, use a faucet. 3 Deploy and Transact with a Smart Contract Great, now that we have an address with some ether (to pay for the gas cost) we can create our transaction offline, sign it, and send it to a node with a raw JSON-RPC HTTP request. We are gonna use the that takes as inputs the signed of a . send_rawTransaction method parameters transaction The python code is looking for the json file containing the contract abi and bytecode that truffle creates when compiling a smart contract. Before testing the python code, create a truffle workspace and compile the dummy contract . AdditionContract.sol $ truffle init// add the smart contract in contracts/$ truffle compile then update the python code with the URL of your geth node, and the path to the truffle workspace and your genesis file (don’t forget to replace my userName with yours in the path). Everything else is in the code and should be self-explanatory. I kept everything dead simple to ease modification and experimentation. Have fun :)

Ethereum: create raw JSON-RPC requests with Python for deploying and transacting with a smart…

I hope you enjoyed your bear market baptism. Image courtesy of ElevenNews Whenever I teach someone how to get involved in cryptocurrency investing, I know that whenever the market suffers I’m going to get frantic calls from that person asking where all of their money went. It happens every time, without fail. I’m not complaining, I think more people in this market should go out of their way to help new investors — it’s the only way this market will grow: which is in our best interest as current investors. What exactly has caused the recent correction in Bitcoin is hard to pinpoint. There are several new developments regarding cryptocurrency regulations that may be affecting the price. Whatever the catalyst is, this is a normal & healthy market correction, that if anything, should bring you a sigh of relief. Don’t Panic. Bubbles don’t have healthy market corrections. Bitcoin (and the majority of other cryptocurrencies) just finished a historic bull market cycle. No financial market in history has ever had as profitable of a bull run as cryptocurrency as a whole has had in the past few months. If the price of Bitcoin were to continue rising at the rate it was — I’d get out as fast I could. That’s unsustainable growth. That’s how bubbles burst. I’m going to drastically oversimplify this for the sake of keeping it under 100 pages — but the recent drop in Bitcoins price can be attributed to the following factors. 1. NEGATIVE NEWS It all starts with a little panic. Do you notice a theme here? The Cryptocurrency market is extremely emotional. To my knowledge, there have been three major negative catalysts that have largely aided in the downward spiral Bitcoin seems to be in. Confusion Surrounding Indian Regulations In my opinion, people have given this news a lot more weight than it should have. India is simply going through the process that every country will have to when their population starts to adopt cryptocurrency more and more. Much like the U.S., India is trying to determine how they want to regulate this market — but due to some statements from their Prime Minister regarding Bitcoin never being a part of their payment systems, people have started to panic. [1] Chinese Crackdown There are also rumors of another Chinese crackdown on cryptocurrency exchanges and ICO’s. The Financial News released an article stating that the Peoples Bank of China would no longer be providing services for accounts tied to cryptocurrency exchanges. Keep in mind that China has tried to ban cryptocurrency over 3 times. It always bounces back. [2] South Korean Regulations While not as big in mainstream media — South Korea has also been increasing regulations surrounding cryptocurrency. Credit Card Crypto Purchases Now Classified as “Cash Advance” The U.S. recently made it so that if someone purchases cryptocurrency with a credit card, it is considered a “cash advance”. Which is not very favorable to those taking the advance to say the least. [3] 2. INVESTORS TAKING PROFITS/CLAIMING LOSSES BEFORE TAX SEASON Going back to my previous statement about how we just came out of a historic bull market, people are taking their profits. Bitcoin went from around $5,000 to $20,000 in an extremely short period of time. While people who bought in closer to $20,000 are panicking at the moment, investors who got in only a couple short months ago are still in the green. We’ve seen a huge amount of profit taking — as well as investors claiming losses. Claiming your capital loss offsets your capital gains tax and allows you to pay less to Uncle Sam. 3. PANIC This is the biggest contributing factor to the recent Bitcoin crash. Cryptocurrency is unique in the sense that it is the only financial market that is made up primarily of new investors, or investors that haven’t been trained in finance. Which means, this is the most emotional market in existence. I’ve found that Elliot Wave theory is the best tool in my Technical Analysis arsenal for predicting price action in Crypto. The underlying concept behind Elliot Wave is that markets move in these “psychological impulse waves” which continuously go up and down as old investors claim profits\losses and new investors buy in. I could go on and on for hours about Elliot Wave, but I’ll save that for another day! we’re in an extremely emotional market. So when a catalyst occurs such as negative news about regulations in India, we see far more drastic price movement than we would see in a seasoned and developed investment vehicle like a traditional stock or security. Long story short; Many people who trade cryptocurrencies, have never traded before. When you combine negative news, with the fact that and institutional investors/high net worth individuals are claiming their losses to offset capital gains taxation — you create mass hysteria for the not so seasoned investors that make up the vast majority of the cryptocurrency market. tax season is coming One thing I want to reiterate: this is completely normal. It would be extremely worrisome if Bitcoin didn’t retrace its price to allow new investors to come in, and let the last wave of investors who have seen enough percentage gain to be happy make their exit. I like to think of Bitcoin like a rubber-band, the lower it corrects, the higher it tends to catapult. Some would call me naive for thinking like this, I’m young enough that I’ve never lived through a multi-year bear market — but all I know is: Buying the dip in Bitcoin has been a winning strategy since 2009. By the way, not to sound like a crazy tin-foil hat wearing kind of guy, but it’s a pretty crazy coincidence that Bitcoin dropped in price to almost exactly where most large investment groups and financial institutions said that the risk/reward ratio would be favorable enough to enter long positions. Footnotes [1] Over $60 billion wiped off value of cryptocurrencies as bitcoin drops to lowest level since Nov 16 [2] Ban Complete: China Blocks Foreign Crypto Exchanges To Counter ‘Financial Risks’ [3] Coinbase warns that banks now process credit-card crypto purchases as ‘cash advances’

Catapult

3 Reasons Bitcoin Crashed & Why You Should Stop Panicking

If you have been involved in trading either stock, fiat currencies or crypto currencies then you may have come across the term ‘buy wall’ or ‘sell wall’. Throwing it in a sentence certainly helps to make you look smart, but what does it actually mean? And how can this knowledge be utilized by people like you and me — the simple trader? In order to answer these questions, we’ll need to take a step back and look at what buy/sell orders are first. When you want to purchase some coins it isn’t as simple as setting up an account on an exchange and buying the coins off the organization. The exchange is just a facility in which two users can meet to swap assets. You want to buy their coins with another coin of value (normally ETH or BTC), they want to sell their coins and receive BTC or ETH in exchange. When you agree on a price the transaction is made and each involved user’s wallets adjust reflecting the completion of the transaction. Now this sounds very complicated and tedious. And it would be if you had to do all the work. But you don’t. All you need to do is put in a request, selecting the coin, the amount and the price per coin. This is called a buy order. The exact same occurs for those that want to sell, but this is called a sell order. Normally these will be set as list until cancelled — which means it will stay as an unfilled order until either it gets filled or you cancel it. Now you can imagine, thousands of people putting buy and sell orders on the exchange, creating a myriad of orders. The exchange then matches any up that overlap automatically. The rest are listed publicly until filled or cancelled. Figure 1. On the inner right side there are unfilled sell orders (above — red) and buy orders (below — green) listed and the far right lists all the completed orders chronologically — newest first. Since all orders that are overlapping are automatically fulfilled, it makes it easy to set and forget until your order gets filled. Sometimes, for a large buy (or sell) order, it might take multiple smaller orders to fill it up, this might mean you will see your order partially filled, then a little more, until it gets filled completely. Now, inherently there will always be a gap between the highest buy order and the lowest sell order. This gap is essentially where the current coins price will sit. Normally it will be the last completed order price. This is how the market can drive up or down the coin’s price. Now that we understand how the buy and sell orders work, we can look at what a buy wall/sell wall is. Figure 2. On the right side you can see the combination of the buy orders in the green pile, compared to the sell orders in the red pile. Figure 2 shows the buy/sell depth. This means there are more buy orders compared to sell orders as the green pile is larger than the red pile. When there is a large buy order then the green pile will increase vertically instead of in small increments. It looks similar to a large step — or… a wall! What this means is that if there is a massive buy order of 10,000 coins at 0.003 ETH, when the majority of sell orders are only 10’s or 100’s of coins then it will be very difficult to fulfill this order. If this order cannot be fulfilled then it means no orders below this wall price will be fulfilled, essentially stopping the price from going below the wall. The same thing happens when there is a sell wall. Now, when a buy wall is put in place, everyone who has an order below this knows their order will be unlikely to be filled AND anyone looking to buy quickly also knows they must put their buy order at a higher price than the wall, ensuring it will get filled first. This will then drive the price up as the people can then start putting sell orders in at higher prices too due to the circumstantial demand. A sell wall will have the opposite effect, lowering the price. A little bit of background on these walls. Some of these walls are legitimate, and it just means that a lot of people or one person with a lot of money believes that at that particular price it is the right decision to purchase, but sometimes these walls are attempts at price manipulation. Someone who holds a large percentage of the market share of a coin is called a whale — they are the big fish in the sea, they cause a splash. So when a whale has enough funds to put a large buy order in they can single-handedly create a buy wall. They would do this to try and manipulate the market, creating fake hype. I say fake, as they never mean to buy this many coins at this high price. But they might already own some coins they purchased at a lower price earlier. So by putting this ‘fake’ buy order in they hope to drive the price up, then allowing them to sell their currently owned coins for a nice profit. This large dump of coins will then make the market realize the fake hype and bring the price crashing back down. This allows the whale to purchase more coins with the profits they have just made. Sometimes things don’t always go as planned. If the buy wall doesn’t manipulate the price to go higher, then the whale will cancel the massive order if the price gets too low where their order might start to get filled. This can have the opposite affect which could cause the price to drop as people believe this massive order has been filled and everyone wants to sell before the price drops further. This could even have been the whale’s plan all along in order to buy some cheap coins. Essentially what I’m trying to say is that a fake buy wall might be put in place to either push price up or pull the price down. This is why it is something that you cannot use alone to guide your trade choices. Taking this advice, there is one thing to watch for with these. If you place an order just below a buy order or just above a sell order might mean you miss out on some profit. The reason for this is that if the buy wall is successful in pushing the price up/holding it steady, then your order will never get fulfilled thus you could have got involved by just pushing your buy order price up only a few points so that it is either equal to or slightly above the wall. On the other end though, if the buy wall is (successful) in pushing the price down when it is removed then the price will likely move down a number of points. Therefore your order will be filled, but it would also have been filled if you had lowered the buy order’s price down further. These walls might seem quite solid and unwavering, even confronting at times. But if you see a buy/sell wall make sure you keep an eye on it as they can be cancelled and reset within seconds. In fact you will often see them moving up and down in response to the current price. For example, let’s say I want to push the price up from $1.00 to $1.20 as opposed to holding it steady or dropping it. I will set my buy wall at $0.99 which will force others to put their orders above this price, say spread from $1.00 to $1.05. I then reset my buy wall at $1.04, again forces others to put their orders above this price, spreading from $1.05 to $1.10. I continue again until the price reaches $1.20 and then put in a number of medium sell orders so as to not disrupt the high price, but allowing me to sell my coins. Then put larger sell orders and pull my buy order off completely when I’m ready for price to drop again. I then can start the process again. Even if you do basic day trading and these walls are one of the few things you look for to help with your order placement, then you will be doing much better than if you ignore them. Adding this to a few other basic trading principles and you will be sure to start making some profits as this volatile coin movement is a trader’s paradise. Joshua (Mappo) has been investing and trading in fiat currencies since 2013. He has recently moved into the crypto world spreading his portfolio over long term coin investments, ICOs and day trading.

Guide to Crypto Trading Part 2: Day Trading — The Great Buy Wall

The last month was not easy for cryptocurrency lovers and traders, everything was going down and red was the color. For anyone except for that not only didn’t suffered so much but actually gained a lot more confidence and bullishness. Achainers If we take a look at the chart of ACT of the last month what you can see is a first pump at the beggining of January caused by the , then the first attempt of the market to go up and then the consolidation of the price aournd 4000 this week. ABTC fork satoshi If you look closely at the circled part, that is refering to the last two weeks, you can see how the price instead of going down quickly, like in the last two pumps, it went down veeeery slowly, and those were the weeks while almost every cryptocurrency was going down and people were panicking on social media and press. This is to reiterate the goodness and the solidity of the project and of the who support this project. Achainers Below you can take a look at the comparison of the price market between Achain and a few of its competitors: Ethereum, Neo and EOS. How we can see Ethereum tried to go from 10 million satoshi to almost 12 million satoshi to then stabilize around 10 million satoshi. NEO instead went down from 1,5 million satoshi to 1,25 million satoshi but with very big ups and downs, it must been very adrenalic for NEO holders during those weeks. EOS lost 30,000 satoshi but in very calm way, similary to Achain. The future of Achain seems bright, ACT overcomed very well this general negative period and, above all, must be taken into account that Achain is at the beginning of its path, which will be completed at the end of 2018 with the creation of the Cosmos interconnected network. Achain roadmap In order to arrive to that goal they need to do other forks and this will bring a lot more people in. Also a lot of other projects built upon Achain or other their partners (which continue to increase) will start their activities so Achain will gain a lot of visibility in the near future, take a look for yourself: , , BitcoinOS, … Kcash Acute Angle Cloud PundiX

Cosmos

Achain: price analysis during the last weeks of drama

Finally, and I know this is a long shot, there may be a chapter or two in here that inspires some economists to change the way they teach the subject, by relating at least some economic ideas to their practical uses or consequences in business settings. — , Trillion Dollar Economists: How Economists and their Ideas have Transformed Business_,_ p. 5 Robert Litan Most economists undertake research that generates nothing of value. The papers that they publish contribute neither to economic understanding nor to social improvement. But, as Robert Litan’s book illustrates, some economists have made major contributions to our well-being by developing and applying concepts in practical settings. Their ideas have proven valuable in the real world of business and regulation. They have influenced airline seat discounts, online dating service algorithms, Google’s ad system, Amazon’s fetish with conducting experiments, Uber’s pricing methods, the FCC’s approach to spectrum regulation, and more. I believe that more economists would have keener insights and contribute greater social value if they were required to spend time as interns in businesses or at regulatory agencies. Just as we believe that educators and medical professionals benefit from internships, so too would economists. Economists out of touch I am not arguing for internships because I believe that applied work is better than theoretical work. The main reason that I believe economists need internships is that much of economic theory has lost touch with reality. Classical economics was developed to describe businesses that transform inputs into outputs. Today, mental-cultural factors are clearly important. tangible tangible Many goods that provide utility to consumers, such as Google Maps, are non-rival and not-easily excludable. According to classical economic theory, this makes them public goods. Yet they are provided by private firms. So we need to find a way to talk about . public goods in a market economy Many determinants of the income of firms and workers are intangible. These include knowledge, strategic choices, teamwork, business relationships, reputations, credentials, and more Beliefs held by individuals affect their choices of occupation and consumption, business decisions, and asset selection. The process by which beliefs spread among groups affects the process by which innovations diffuse throughout the economy and the evolution of regional economic differences. 2. Before the Industrial Revolution developed momentum in the mid-1800s, economic change took place slowly. Now, the nature of economic activity changes every few decades. The economy has become more complex and specialized. The economic concept of “labor” treats workers as interchangeable with one another. This may have been approximately true in 1800, but it is completely inappropriate today. Farming and manufacturing production work, which accounted for three-fourths of the American labor force one hundred years ago, account for less than one tenth of all workers today. Many people work in occupations that did not exist even forty years ago. The textbook picture of the entrepreneur, as someone who makes decisions about allocating capital and labor and choosing the level of output, is now hopelessly out of date. Business strategy today is highly complex and multi-faceted. The process of creative destruction operates much faster than it did throughout most of human history. Today, people have had to put much more time and effort into adapting to economic and technological change. Naive and anachronistic Because economists do not spend time in the real world of business or regulation, they approach their discipline with concepts that are naive and anachronistic. Economists continue to describe entrepreneurs as solving problems of resource allocation, when in fact business executives worry internally about talent development and culture. They worry externally about strategy in a complex environment. Economists continue to describe the distribution of economic rewards using marginal productivity theory. But value is created jointly by teams and value is captured by strategic interactions among firms. In most of the economy, it is impossible to isolate and measure any one person’s marginal product. Economists have a stunted picture of the financial system, one which ignores the vast majority of financial instruments and strategies that have been developed since 1950. For example, repurchase agreements have been at the core of real-world securities markets and monetary policy implementation for more than forty years, but most economists did not know what a “repo” was until the financial crisis of 2008, when explained its role. Gary Gorton and Andrew Metrick Economists’ main measure of economic performance, GDP, was designed for a world where value resided in forms of output that could be counted. Today, consumer wants are satisfied by goods and services that are not quantifiable or even tangible. Health care, education, and finance make up an increasing share of spending and employment, but we do not know how to measure output in those sectors. Also, for services like Google Maps or Facebook, we have no idea how much value they provide, because consumers are not charged. Ph.D programs in economics need to get out of the 19th century and into the 21st. I think that requiring an internship in the world of business or regulation would help.

Uber

Require Internships for Economists

Blockchain startups are using nitial oin fferings to raise capital and to gain community support. Community members or investors who want to support the project send Ether/ETH to a given smart contracts address and in return receive custom tokens instantly, or sometime later after the event ends. I C O ICO Let’s take a step back: Just like any other investment most community members do some homework before investing. A number of website sites, bloggers and Vloggers are doing a brilliant job sharing their honest analysis before ICO events. I strongly believe one of the last few analysis steps before investing should be the analysis of the ICO and Token generation Smart Contracts. This will truly give the community confidence in the promises made by project team. is a computer protocol intended to digitally facilitate, verify, or enforce the negotiation or performance of a . allow the performance of credible transactions without third parties.” [ ] “A smart contract contract Smart contracts 1 As mentioned in my previous post about s, I would really like to encourage ICO management teams to release these contracts with source code at-least few hours before the ICO event starts — better to use ENS instead of contract address. Best Practices for ICO From the community point of view in this era of scammers, this will give us enough time to double check that the smart contract is genuine and it is doing what it is supposed to do. I would like to share few tips on how to read the basic information from the smart contracts. . To make it useful for wider audience i will try not to jump into code — at least not in part 1 of the series :). You don’t need to be a coder to do basic smart contract analysis Lets do it! Token Contract Analysis We can easily read some basic information about ERC20 based tokens. “The Ethereum token standard ( ) is used for smart contracts. Developed in 2015, defines a common list of rules that an Ethereum token has to implement. Giving developers the ability to program how new tokens will function within the Ethereum ecosystem” [ ] ERC20 Ethereum ERC20 2 So let’s take BEE token as an example: Open .io website, search and select BEE in top right search bar etherscan or Click this link BEE Token Contract For new tokens you might not be able to search the tokens by symbol but you can search by token address given by ICO management team. Have a quick look at the basic information provided under “Read Smart Contract Tab” Snapshot of BEE token contract via etherscan Compare it with information provided by ICO team. You must have noticed number “ & train of zeros” :). That’s because value supports 18 decimals places. 1 BEE = 1 x 10 to the power 18. 2 . > totalSupply -> 5 You can also check the “Token Holders” tab which will list address of token holders. You might be able to check how many token are already transferred during pre-sales or private sale — assuming token were transferred to them before ICO. You should be able to see this sort of information for most of ERC20 based tokens. ICO Contract Analysis As far as i know there is no standard for ICO smart contracts. ICO management teams members are free to design and choose what short of information they would like to expose via providing Contract pplication inary nterface — ABI. A B I Now lets have a look at our example and analyse the information available BEE ICO Smart Contract Snapshot of BEE ICO contract “Read Contract Tab” Select “Read Contract Information” tab and there you Go!. Most of the information shared is self descriptive. Token address given at number number “17.” — same token address we analysed in previous section. Hard cap at “14.” and Minimum Contribution at number “11.” — this is in ETH. Remember . 1 ETH = 10 to power 18 wei This means minimum contribution was set to 0.1 ETH. Nicely done ICO contracts also let users query for more information using “Query” feature of etherscan. I will let you explore that part :). High five! Now you know how to explore a smart contract and do some basic analysis.

Instantly

Smart Contract basic analysis

I have watched and re-watched Seinfeld over and over for the last 6–7 years now and it remains, till date, one of my most favorite shows. For a show that claims to be about nothing, it has certainly managed to fill up nine seasons worth of content, all with seemingly no plot. The show, if you’re unfamiliar with it, traces the day-to-day lives of (mainly) 4 quite unremarkable people — Jerry Seinfeld, George Costanza, Cosmo Kramer and Elaine Benes. These 4, along with plenty of other side characters and guest stars engage in what seems to be a series of meaningless conversations and activities, which somehow all come together towards the end of the duration of each episode. Yesterday while animatedly watching one of the episodes in which Frank Costanza talks about the a festival he had created, “Festivus”, I found myself thinking how much I adored Frank Costanza as a character. But he hadn’t featured in many episodes right? Neither had Jackie Chiles, the versatile lawyer, and yet he’d had the same effect as had Frank. Which episodes were those again? Damn. You’d thinking having watched this show so many times, such information would be “tattooed in my brain”. But admittedly, it was not. And there Had to be a more complicated, yet fun way to get this information than looking it up on Wikipedia. And thus, the weekend data-dive game was on. The measure used in this analysis is the number of times a person has a line to speak per episode. So it follows that the first time a person (say Jackie Chiles) had a line throughout the show would be the first time he/she was featured. Other results that followed were how important a character was to the show or an episode, or in how many episodes was say, Elaine or Kramer, not present. Also, we know the central 4 characters, but how do the 3 characters apart from Jerry rank against each other? So, this is how I went about it. Scraping and parsing the data. I obtained the list of episodes and scraped every episode’s script off of seinfeldscripts.com <Hat-tip> using Requests at a consistent rate so as to not overwhelm them, even if it was just around 200 automated API calls. This website also has the listing of all episodes with the episode numbers from 1–180 alongside it.[1] The data scraped was neither very manageable nor consistent in the form obtained. So a pattern was slightly troublesome to derive, but finally I found a way to just get the dialogues and more importantly, the person who delivered them using Beautiful Soup (bs4). soup = BeautifulSoup(req.content, ‘html.parser’)paragraphs = soup.find_all('p') ... if (j.isupper() and len(j)>1 and ‘:’ in j):if j in chars_dict:chars_dict[j]=chars_dict[j]+1chars_dict[‘TOTAL’]=chars_dict[‘TOTAL’]+1else:chars_dict[j]=1chars_dict[‘TOTAL’]=chars_dict[‘TOTAL’]+1 In case it was a well written script, the pattern would be something like - KRAMER: “Hoochiemama!” And the above mentioned code snippet would handle. But in many of the files you had entries like George: “Vandelay Industries! Say Vandelay Industries” And the only indication here that a dialogue has begun is the colon (“:”). So this was the other case to handle. Storing and filtering the data Looping through the entire set of scripts resulted in a dictionary of dictionaries, where the key was an episode number and the value was a dictionary of the cast for that episode with their corresponding counts of dialogues. Next I consider the characters I’m primarily interested in, since for this usecase, I don’t really care about some people like or Stan the Caddy. Sue Ellen Mischke characters =[“JERRY”,”GEORGE”,”ELAINE”,”SUSAN”,”KRAMER”,”JACKIE”,”FRANK”,”RAVA”,”NEWMAN”,”TOTAL”] After translating the keys of the dictionary obtained in Step 3 to uniformly be in upper case, I store the occurrence values per episode for the characters mentioned above along with the total number of dialogues per episode (“TOTAL”) in a dataframe since Pandas is a very good tool to manipulate data.The occurrence values were then normalized to be in percentages of the TOTAL. Despite all the attempts to obtain complete data, there were some files i.e. episode scripts that were able to be only partially processed or were unable to be processed because there was no way to distinguish beyond a shadow of the doubt between the character names and the dialogues themselves. This was obviously because the only source considered was Seinfeldscripts.com. A future experiment will probably consider other sources to fill in the missing data. To this end, I’ve filtered out entries that had a TOTAL value < 100 df = df[df.TOTAL > 100].sort_values(by=”episode_num”) The finale episodes have also been omitted from this analysis. Results: The main result was a set of graphs based on the following criteria I chose -a) % of dialogues per episode of Elaine vs Kramerb) % of dialogues per episode of Elaine vs Georgec) % of dialogues per episode of George vs Kramerd) Episodes in which Frank Costanza featured e) Episodes in which Susan featuredf) Episodes in which Jackie Chiles featuredg) Episodes in which Newman featured Other miscellaneous graphs were also computed. The X-axis for all graphs shows the episode number (albeit unclear) and the Y-axis tracks the % occupied per episode. All characters Understandably, Jerry has the largest number of dialogues in every episode. a) : Elaine vs George ELAINE BENES vs GEORGE COSTANZA The trend shows that George (~20–30%) always has more screen space than Elaine(~5–25%) . b) Elaine vs Kramer : ELAINE BENES vs COSMO KRAMER While they seem to be uniformly matched for the most part, it looks like Elaine almost always has more to say than Kramer, atleast for the first 60% of the total number of episodes. Towards the end however, Kramer seems to be having more to say. c) George vs Kramer: GEORGE COSTANZA vs COSMO KRAMER No surprises here. George dominates with ~20–30% of the conversations while Kramer averages out at close to 15% of the conversations. So, we see that in terms of main character importance, Jerry > George > (Elaine = Kramer). Moving on, d) Frank Costanza: FRANK COSTANZA Frank Costanza first appears in episode 62 — “The Handicap Spot” and finally in episode 166 — “The Strike” (the legendary Festivus episode) barring the finale episodes. e) Susan Ross SUSAN ROSS Susan first appears in episode 43 — “The Pitch” and finally in episode 134 — “The Invitations”, when… you know.. f) Jackie Chiles JACKIE CHILES Jackie Chiles plays a major role in only 3 episodes Episodes 113, 122 and 143, again barring the Finale. I finally have my answer :D g) Newman We see that Newman plays a key role in more episodes than any of the other side characters, going upto 14% in episode 132 — The Bottle Deposits — part 2 where he runs a bottle scam with Kramer (Aah nostalgia). This experiment was a lot of fun. I hope to fill in the blanks in the next ones and also work out stats and relationships for the other characters. SERENITY NOW!!!! [1] : http://www.seinfeldscripts.com/seinfeld-scripts.html Code https://github.com/abhiii5459/DataViz/blob/master/Seinfeld/Seinfeld_Analysis.ipynb

Alongside

Studying the show about nothing

Image courtesy of on Pixabay Geralt I’m a software engineer currently building chatbots. I’ve published more than twenty custom skills on the Amazon Alexa platform, and last year published my first Facebook Messenger chatbot named . Chuck I measure success by how much users are educated or entertained by the bot. The best KPI’s for this are retention, and depth of the dialog. Here is my latest insight in how this applies to applications written for Facebook Messenger. The NLU and orchestration layer is written using AWS Lex, as well as AWS Lambda functions written in NodeJS. What is the Ideal Chatbot? The ideal chatbot is what we saw with the HAL 9000 in A Space Odyssey. Just a blinking cursor or microphone that we can ask any question. No constraints, just answers. In applying this to software tools around Facebook Messenger, here is what that architecture looks like. Natural Language Understanding (NLU) models are good at deciphering user requests, and few scale issues. What limits applications is how many features can be written to support the thousands of different requests users make. Once those features proliferate, the challenge becomes discovery of the features — what the bot “knows”. A blinking cursor has limitless options, but a cold conversation starter. Users will have expectations based on the branding of the bot, but little else. Traditional visual applications handle feature discovery by providing menus and widgets on a device screen that provides options. An effective chatbot needs an approach for this as well. A good approach is to provide context with each response to the user, providing hints at what is possible. Building discovery into the responses encourages dialog, and actively engages the user. How to Improve a Facebook Messenger Chatbot Back in , I began adding buttons to my Facebook Messenger bot in an effort to improve the usability. This was in response to a lack of growth in usage. My bot was growing in features, as it ingested more data and algorithms to manipulate it. Without users knowing of it’s newfound capabilities, it was stuck. December I tried different ways of advertising the features on the Facebook Page, but none were effective. For example, I saw trivial gains through posting video demonstrations of the bot demonstrating new features. Similar results were achieved when writing articles in the news feed. Adding buttons into the bot responses did provide the lift I was looking for. Designing Buttons into User Interaction Buttons augment responses by providing clear direction to the user on what’s possible. Let’s start by the initial response the bot has when the user initiates a conversation with “Hello” or an introduction. Scripting clear language in the response is important, including providing a sample phrase (i.e. “How many calories in a Chicken Sandwich”.) Adding buttons takes this a step further, and proves far more effective. All a user needs to do is click one, and another response is coming their way. It also saves the user time for typing the potential requests. Mobile devices are tricky to navigate spelling and users tend to abbreviate sentances. Buttons simplify the user experience. Results of Adding Buttons Now for seeing the progress. I use the analytics tools within Facebook to track detailed usage. It is essential for measuring feedback for changes made within the app. Here’s an example dashboard that is provided if you haven’t used before. The first success metric is around depth of the dialog. I want users to be educated with my bot. This can be quantified by tracking how many times a given user replies back and forth with the bot before exiting. Here is the chart tracking the amazing growth after I added buttons. More than double the activity per user. Growth of session length after buttons added in early January. The next success metric is around retention. If a user believes they received useful information, they will be back. The easiest view on this is around weekly retention, and daily statistics shows similar success. Retention of users after buttons added in early January. The first week retention numbers are almost double since I added buttons. Less successful has been the second week retention which is slightly down. Sample size for week 2+ isn’t significant with these cohorts, but will be a trend to monitor in the next month as this grows. Limitations with Buttons The biggest limitation is the amount of screen space that buttons take up. The chat dialog becomes overwhelmed with the space being consumed, and with Messenger, there is no font size to shrink this. History of the session gets lost from immediate viewing, but the user can scroll back. Messenger renders up to three buttons at once, so there is the potential that users limit their thinking to only what they can click. Here is an actual dialog talking about pizza. These buttons can constrain users thinking, and skew results towards what is presented. For example, the bot knows about Domino’s, but is not picked at the same frequency as those rendered as buttons. I attempt to phrase the label above the buttons to highlight this — “Click button below .” or type response Conclusion Buttons are an essential part of designing a . It introduces some visual constraints, but worth a doubling of retention and transaction depth. chatbot For more information on the underlying , here is the source code for the bot, including design documentation. technology _caloriecounter - AWS Lex based chatbot that calculates calories based on different fast food restaurants. This was an…_github.com terrenjpeterson/caloriecounter

Discovery

How to Double Retention and Dialog Length for your Chatbot

Using AWS Lambda functions for micro jobs with no startup delay

If you’ve been following me for a while, you will know that I am a huge fan of the concept of the new paradigm. However, when it comes to managing a large number of functions all behind an gateway, it starts to become a bit of a burden. Serverless API Thankfully, new tools such as have proved to be incredibly useful when it comes to managing a large number of functions together. Serverless In this article we are going to have a look at some of the amazing advantages that Serverless provides when dealing with a vast empire of different lambda functions across multiple environments! Key Advantages Serverless allows you to develop and deploy a wide number of changes to multiple AWS Lambda functions with ease. This is a huge win. By allowing us to define everything about our Lambda functions within a file, it reduces the time for us to deploy changes down to less than a minute. yml When it comes to developing anything, shortening the feedback loop is vital. You want to minimize the number of times where a simple change will take 5–10 minutes to deploy and check. The quick feedback loop we experience using a tool like serverless truly makes this architecture style a very viable option when it comes to building highly scalable production applications with minimal fuss. Cross Platform One of the key concerns of many Product Owners is that they are becoming to reliant on the services of a single cloud service provider. This is otherwise known as “vendor lock-in” and is a very real issue. You don’t want to develop a multi-million dollar solution that relies purely on a single cloud provider, only to find out that in 6 months time, the particular service you are relying upon will be deprecated. Serverless addresses this command by remaining agnostic to what underlying cloud service provider you use. You aren’t locked into AWS and you can design your serverless applications to be deployable across multiple cloud service providers. Installation Installing the Serverless tool requires npm. It can be installed by calling: npm install -g serverless If you’ve already setup your aws credentials file on your machine and have a user set up with the correct privileges, you should be good to go! Structuring Your Functions As I’m currently writing an imgur-like website built purely on AWS, I’ll be basing this article on what I’ve done so far for that particular project! Say I wanted to develop a series of different endpoints for my new project. I could define all of these individual functions within a distinct python file within an directory. imgur/ .serverless/imgur/- list.py- upload.py- ...serverless.yml Within my file within the root directory I could then do something like so: serverless.yml So within the above config, I’ve essentially defined 2 distinct API endpoints that are powered using functions that I’ve defined within my directory. imgur/ Once I’ve made my changes to my function, in order to deploy these changes, I simply have to call: serverless deploy And the rest is handled for me. This. is. amazing! An example of the serverless deploy command As you can see from the above screenshot, the cli handles the creation of a CloudFormation file, it packages and uploads the artifacts and then also pushes them to S3 and deploys them. Finally, once everything is done, it prints out the “Service Information”. Within this information, you should see a list of all the endpoints deployed and what verbs are needed to hit them. This entire process took about 20 seconds for the 2 endpoints I have defined. HTTP Managing Resource Access More often than not, you’ll want your lambda functions to be able to talk to other AWS resources, serverless handles this exceptionally well. Within the confines of your file, you are able to specify to a very granular level, the different access levels of your functions. serverless.yml Dev, Test & Production When it comes to segregating your development environment from your test and production environments, serverless allows you to specify what stage you want to deploy to through either the file or through command-line arguments. yml This means that I can build a full deployment pipeline around this within the likes of Jenkins that will automatically deploy tests to test, perform integration tests and then push straight to production, with no added fuss. This is ideal. Conclusion Hopefully, you found this article useful! Having just recently started working with Serverless, I can already see massive potential with the framework and I very much look forward to doing more development with it in the future. I’m currently trying to build my YouTube channel and hit the new 1k subscriber mark to remain a partner, if you want to support me then please feel free to head over there and subscribe: https://www.youtube.com/tutorialedge I’d be really eager to hear the thoughts of anyone who has been using Serverless in production for a while now! Let me know either in the comments section below or through my twitter: Elliot Forbes

Managing Your Lambda Empire with Serverless

In this post, I’ll show how to containerize an existing project using Docker. I’ve picked a random project from GitHub that had an saying Dockerize to contribute and use as an example here. open issue Why in the world do you want to Dockerize an existing Django web application? There are plenty of reasons, but if you don’t have one just do it for fun! I decided to use docker because one of my applications was getting hard to install. Lots of system requirements, multiple databases, celery, and rabbitmq. So every time a new developer joined the team or had to work from a new computer, the system installation took a long time. Difficult installations lead to time losses and time losses lead to laziness and laziness leads to bad habits and it goes on and on… For instance, one can decide to use SQLite instead of Postgres and not see truncate problems on a table until it hits the Test server. If you don’t know what docker is, just picture it as a huge virtualenv that instead of containing just some python packages have Containers for isolating everything from the OS to your app, databases, workers, etc. Getting Things Done Ok, talk is cheap. Show me some code, dude. First of all, Docker. I did it using Ubuntu and Mac OS without any problem, but on Windows Home, I couldn’t have it working. install To tell Docker how to run your application as a container you’ll have to create a Dockerfile FROM python:3.6ENV PYTHONUNBUFFERED 1RUN mkdir /webappsWORKDIR /webapps# Installing OS DependenciesRUN apt-get update && apt-get upgrade -y && apt-get install -y \libsqlite3-devRUN pip install -U pip setuptoolsCOPY requirements.txt /webapps/COPY requirements-opt.txt /webapps/RUN pip install -r /webapps/requirements.txtRUN pip install -r /webapps/requirements-opt.txtADD . /webapps/# Django serviceEXPOSE 8000 So, let’s go line by line: Docker Images FROM python:3.6 Here we’re using an Image from . e.q. One pre-formated container that helps build on top of that. In this case, Python 3.6 is an Ubuntu container that already has Python3.6 installed on it. docker hub Environment Variables You can create all sort of Environment Variables using Env. ENV PYTHONUNBUFFERED 1  # Here we can create all Environment variables for our container For instance, if you use them for storing your Django’s Secret Key, you could put it here: ENV DJANGO_SECRET_KEY abcde0s&&$uyc)hf_3rv@!a95nasd22e-dxt^9k^7!f+$jxkk+$k- And in your code use it like this: import osSECRET_KEY = os.environ['DJANGO_SECRET_KEY'] Run Commands Docker Run Commands are kinda obvious. You’re running a command “inside” your container. I’m quoting because docker creates something as sub-containers, so it doesn’t have to run the same command again in case of rebuilding a container. inside RUN mkdir /webappsWORKDIR /webapps # Installing OS DependenciesRUN apt-get update && apt-get upgrade -y && apt-get install -y \libsqlite3-dev RUN pip install -U pip setuptools COPY requirements.txt /webapps/COPY requirements-opt.txt /webapps/ RUN pip install -r /webapps/requirements.txtRUN pip install -r /webapps/requirements-opt.txt ADD . /webapps/ In this case, We are creating the directory that will hold our files . webapps/ Workdir is also kind of self-evident. It just docker to run the commands in the indicated directory. After that, I am including one OS dependency. When we’re just using requirements.txt usually we are not including any OS requirement for the project and believe me, for large projects you’ll have lots and lots of OS requirements. COPY and ADD Copy and ADD are the same. Both a file from your computer (the Host) into the container (The Guest OS). In my example, I’m just coping requirements to pip install them. But Add can COPY from a URL. copy EXPOSE Expose Instruction is for forwarding a port from Guest to the Host. # Django serviceEXPOSE 8000 Ok, so now what? How can we add more containers and make them work together? What if a need a Postgresql inside a container too? Don’t worry, here we go. Docker-Compose is a tool for running multiple Docker containers. It’s a yml, you just need to create a docker-compose.yml on your project folder. Docker-compose version: '3.3' services: Postgres db:image: postgresenvironment:- POSTGRES_USER=postgres- POSTGRES_PASSWORD=postgres- POSTGRES_DB=postgres web:build: .command: ["./run_web.sh"]volumes:- .:/webappsports:- "8000:8000"links:- dbdepends_on:- db In this case, I’m using an from Docker Hub. Image of Postgres Now, let’s change the settings.py to use Postgres as Database. DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2','NAME': 'postgres','USER': 'postgres','PASSWORD': 'postgres','HOST': 'db','PORT': '5432',}} We’re almost, done. Let me talk a little about the docker compose file. VOLUMES Remember ? vagrant Once upon a time, there was Vagrant and it was a form to run a project inside a Virtual Machine but easily configuring it forwarding ports, provisioning requirements and, sharing volumes. Your machine (Host) could share a volume with your Virtual Machine (Guest). In docker, it’s exactly the same. When you’re writing a file on a shared volume this file is being written on your container as well. volumes: .:/webapps In this case, the current directory (.) is being shared as webapps on the container. LINKS links: db You can refer to another container that belongs to your compose using its name. Since we created a container for our Postgres we can link it to our container. You can see that in our settings.py file I’ve used ‘ ’ as host. db web db DEPENDS_ON In order to your application work, your database has to be before web container, otherwise, it will raise an exception. ready for use depends_on: db Command Command is the default command that your container will run right after it is up. For our example, I’ve created a run_web.sh, that will run the migrations, collect the static files and start the development server. #!/usr/bin/env bash cd django-boards/python manage.py migratepython manage.py collectstatic --noinputpython manage.py runserver 0.0.0.0:8000 One can argue that run the migrate at this point, automatically, every time the container is up is . You can run it directly on the web machine. You can access your container (just like the good’ol ) : not a good practice I agree. vagrant ssh docker-compose run web bash if you’d like you can run it without accessing the container itself, just change the last argument from the previous command. docker-compose run web python manage.py migrate The same for other commands docker-compose run web python manage.py testdocker-compose run web python manage.py shell Running Docker With our Dockerfile, docker-compose.yml and run_web.sh set in place, just run it all together: docker-compose up You can see this project here . on my GitHub Originally published at Fernando Alves .

Dockerizing Django for Development

New programming languages are born every day. Dart, Go, Kotlin, Elixir and more! Beating the learning curve can be difficult, but there's some tips that will help you to get through. There are some steps you want to take to accelerate the learning process and increase your fluency in the programming language or framework you are learning. Photo: Sai Kiran Anagani/Unsplash Master the language. Then learn frameworks. Some people tend to learn frameworks, like Ruby on Rails, without getting to know the language, in this case, Ruby. That's a huge problem, specially when talking about Rails, that plays with Ruby as a DSL (domain-specific language). Solve coding challenges. There are a lot of websites like , and that can help you to reach fluency. You'll learn more about the syntax, libraries and logic. That's a great way to prepare yourself to an interview too. LeetCode HackerRank Project Euler Use StackOverflow and Code Review as much as you want. When you start solving coding challenges, you'll find problems. To solve them, you can use , a website that needs no introduction. StackOverflow The ace in the hole is on , a StackExchange community, the same network from StackOverflow. The whole point about Code Review is to . Take a look at the and join this awesome community! Code Review get a review of your code from experts in the language you are learning beginners section Recently I asked for a code review there, click here if you want to see it. Read a lot of code. is essential to beat the learning curve. There you'll find in the language you are learning. If you are confident to contribute, that's even better. GitHub production code Install a linting tool. Linters are code-analyzer tools that flags bugs, idiomatic offenses, stylistic errors and more. A linting tool will flag the error for you to fix, and that's when you learn. For Ruby, I use RuboCop as a linter. Code idiomatically. When I was learning Python, I transferred my C# knowledge to Python. That’s a great way to learn a new programming language, but it’s not enough to reach fluency, because you can’t learn the idiomatic way of programming in that language. To illustrate, take a look at this Python code (credits to ): Oleksandr Kaleniuk def matrix_of_floats(matrix_of_anything):n = len(matrix_of_anything)n_i = len(matrix_of_anything[0])new_matrix_of_floats = []for i in xrange(0, n):row = []for j in xrange(0, n_i):row.append(float(matrix_of_anything[i][j]))new_matrix_of_floats.append(row)return new_matrix_of_floats versus the Pythonic way (idiomatic Python): def matrix_of_floats(matrix_of_anything):return [[float(a_ij) for a_ij in a_i]for a_i in matrix_of_anything] To learn how to code idiomatically, you have to read a lot of code written by the pros. Build something! This one is cool. The main purpose of learning a programming language is to build something with it. When I was learning Ruby, I used Rails framework to build a blog, a todo app, a Wikipedia clone and a Rotten Tomatoes like website. Building something will evolve every aspect of your learning. Don't give up. Beating learning curves can be really hard. You need time and focus to get there. Don't give up because you may be closer than you think. If you are stuck, ask for help. There's a lot of resources out there. Jedi tip: while(true) => study() I know a programmer studies C# for more than 20 years. He still discovering new things about C# and .NET every day. This guy is , the . Antonio Maniero most influent member of StackOverflow in Portuguese The fact is: we'll never learn everything about a language, and that's what will keep us learning more and more every day. You can be good at something, become a reference in it, but there will always be more to be learnt. May the force be with you! Leave a comment telling what else helps you to learn a new programming language. Thank you for reading! Don’t forget to follow me on , and . Medium Instagram LinkedIn

Instagram

2022 - HackerNoon Contributor of the Year - Career Development

2022 - HackerNoon Contributor of the Year - Learning

2022 - HackerNoon Contributor of the Year - Productivity

Coding Guru of 2022

Mastodon

How to Learn a New Programming Language or Framework

As a new graduate, I recently went through the recruiting process for Associate Product Manager positions at technology companies in the Bay Area. I’m hoping my experience can help those looking to transition into these roles, whether directly from college or otherwise. In this post, I’ll discuss how I approach most product questions. In the next, I’ll apply this framework to an actual product interview question. While there are numerous approaches that work, I’ll present the British Council’s Double Diamond as an industry standard and then discuss how I’ve refined it to meet my own needs. Design courtesy of the British Design Council ( ) https://www.designcouncil.org.uk The Double Diamond is a structured design approach to tackle challenges in four phases: — insight into the problem (diverging) Discover/Research — the area to focus upon (converging) Define/Synthesis — the potential solutions (diverging) Develop/Ideation — solutions that work (converging) Deliver/Implementation In the diverging phases, an open mind and a breadth of ideas is necessary. In the converging phases, ruthless prioritization and measurable specificity is required. This expansion and contraction can be repeatedly followed to make sure that the right problem and the best solution are being identified at every step of the way. Using this as a guide, I’ve outlined in more specificity the steps I take when addressing feature or improvement questions in product interviews. development Take a broader look at the strengths and weaknesses of the product and the larger company, and the opportunities and threats that exist in the space. This sets us up on the right track for categorizing and classifying action later on. SWOT Analysis Identify the user that you are trying to optimize the experience for. While playing to a general audience may seem intuitive, Gartner’s Adoption Curve indicates that an optimized use case for a niche audience will lead to the early adoption that is necessary to achieve mainstream growth later on. Target Personas Outline what the current experience is like for the persona that you have selected. What are their uses cases? What is the current flow from start to finish for the use cases that you identify? Current Experience For each of the steps in the current experience, identify the bottlenecks and areas that could use improvement. A good place to start is with Don Norman’s seven stages of action. Pain Points courtesy of Don Norman, Design of Everyday Things Pain points can arise at any step across these two bridges around how to execute an intended action or why the result was different was the intended result. User testing or data can also surface bottlenecks or drop-off points in a flow or process. Also establish a goal: what should a solution accomplish for the specific problem? This is a point of discussion for many designers. Some find inspirational starting points in the work of their competitors. Others advocate a design approach that looks inward for core-competency driven development and builds on a specific company’s differentiation factors. I feel that while you should always design for your strengths, it is important to keep your eyes open to how similar problems are being solved in your competitive space. Competitors Use the constraints, strengths, problem statement, and user goals to guide the ideation process. For each pain point, brainstorm design, business, and technology solutions. Ideation Compare the solutions and analyze the tradeoffs of each one. Evaluate based on appropriate axes — a couple important ones include technical effort, cost, cannibalization of existing products, market size, impact, and appeal. How well will this solution address the goal you’ve outlined? Select the best option. Opportunity Assessment An oft-overlooked part of the design process is evaluating how well your design is doing in practice. Establish metrics launching or testing to know what success looks like. Metrics before Think carefully about what minimum functionality needs to exist to validate your design or hypothesis. How can the launch be structured (where, when, how, MVP) to most quickly validate your hypothesis? Launch fast and start looking at data immediately. Prototype + Launch You might have selected the wrong problem or the wrong solution. You might have made an incorrect assumption about your target audience. Look through the data and conduct user interviews to uncover these mistakes. Iterate and relaunch. Iterate Next, I’ll use this framework to approach the following question: How can Uber improve its rider experience?

My Product Interview Process

Python 3.7 is in Beta! It’s time to get testing… is it any faster? Here are the major speed boosts you’ll get with Python 3.7 versus 3.6 : some of the topics in this article are quite detailed and beyond the level I normally blog at. If you don’t know some of the terms or meanings, just download and explore the examples — keep reading and playing! Warning Time for some go faster stripes.. 1. Calling methods faster (maybe) The title of is “Speedup method calls 1.2x”, which is a bit misleading.. this change There are many ways to change CPython, either by modifying the of the Opcode, or by adding new . Adding new Opcodes requires a lot of discussion and testing and this change introduces new Opcodes. Opcodes are selected by the process in CPython. Once your code is converted into an Abstract-Syntax-Tree, the compiler explores each branch and converts it to Opcodes. The execution of your code goes through the Opcodes in a massive switch statement inside a loop and calls different C-functions for each opcode. execution Opcodes compilation For reference, has 3 Opcodes for calling functions. All of these were either added or modified in Python 3.6. Python 3.6 For calling positional argument-only functions: , CALL_FUNCTION For calling positional and keyword functions: , CALL_FUNCTION_KW For calling variable positional or keyword functions: CALL_FUNCTION_EX Python 3.7 adds 2 new Opcodes, for when the compiler sees it uses these new Opcodes. [LOAD_METHOD](https://github.com/python/cpython/blob/master/Python/ceval.c#L3021-L3105) and [CALL_METHOD](https://github.com/python/cpython/blob/master/Python/ceval.c#L3021-L3105) x.method(...) As an example calling 3 functions with different signatures: Running this on Python 3.6 and Python 3.7 we can see no change in the resulting code, or performance. Another example with methods (ie those belonging to an instance of a class), bound The results of this show: The new opcode replaces loading bound methods as attributes and just calling them as normal functions. Remember, and are faster than for instance methods. LOAD_METHOD LOAD_METHOD CALL_METHOD CALL_FUNCTION Bound methods with keyword-arguments are the same as in Python 3.6, you won’t get any performance change. Bound methods with are now faster no arguments replaces , which is essentially getting the BoundMethod instance on the object instance. the logic in but better optimised when the method hasn’t been overridden and it has positional arguments. LOAD_METHOD LOAD_ATTR [LOAD_METHOD](https://github.com/python/cpython/blob/master/Objects/object.c#L1074-L1155) is a copy of LOAD_ATTR Coming out of this, you might have some questions So if I put my functions into a class, will that make them faster? No, because this speed boost is to remove object-related slow-downs What is it about Keyword and Variable Arguments that they get special treatment? Keyword arguments require special treatment in the execution loop because there is no equivalent in C (which CPython is written in), some extra code has to compile 2 tuples to pass to the method. Variable arguments, whether positional or keyword also require special treatment. So many caveats, I like keyword-arguments, will I see any difference? This change should encourage you in class-design to follow the DRY (don’t repeat yourself) principle and add private methods that reduce duplication of logic across multiple public methods. Prior to 3.7, the performance hit would have been a strong consideration and copying+pasting code was an accepted practice where speed was required. In future, we might see more . scenarios undergo similar treatment 2. str.find() is faster for some characters Some unicode characters have an unfortunate issue when scanning a string for occurrences using , seeing up to str.find(x) 25x slow down. $ ./python -m perf timeit -s 's = "一丁丂七丄丅丆万丈三上下丌不与丏丐丑丒专且丕世丗丘丙业丛东丝丞丟丠両丢丣两严並丧丨丩个丫丬中丮丯丰丱串丳临丵丶丷丸丹为主丼丽举丿乀乁乂乃乄久乆乇么义乊之乌乍乐乑乒乓乔乕乖乗乘乙乚乛乜九乞也习乡乢乣乤乥书乧乨乩乪乫乬乭乮乯买乱乲乳乴乵乶乷乸乹乺乻乼乽乾乿亀亁亂亃亄亅了亇予争 亊事二亍于亏亐云互亓五井亖亗亘亙亚些亜亝亞亟亠亡亢亣交亥亦产亨亩亪享京亭亮亯亰亱亲亳亴亵亶亷亸亹人亻亼亽亾亿什仁仂仃仄仅仆仇仈仉今介仌仍从仏仐仑仒仓仔仕他仗付仙仚仛仜 仝仞仟仠仡仢代令以仦仧仨仩仪仫们仭仮仯仰仱仲仳仴仵件价仸仹仺任仼份仾仿"*100' -- 's.find("乎")' Unpatched:  Median +- std dev: 761 us +- 108 usPatched:    Median +- std dev: 117 us +- 9 us In Python 3.7, the expected Unicode code-point size is no longer hard-coded and the methods are optimised long (mostly unusual) characters. These are still slower, but now 3x slower than ASCII characters instead of 25x! 3. os.fwalk is 2x faster The function in the module (only in Python 3) is a directory-tree generator. fwalk os It behaves exactly like walk(), except that it yields a 4-tuple response (dirpath, dirnames, filenames, dirfd) The the implementation to use the method instead of , which is Operating-System optimised and much faster. change was to modify scandir listdir 4. Regular expressions are faster* In the regular-expression module ( ) there is a method which compiles a regular-expression string and an optional set of flags. These flags can be RegEx flags, passed to the RegEx library. re compile A change was made in Python 3.6 which slowed down this call when flags were passed which were integers. Python 3.7 “fixes” the slowdown but is still not as fast as Python 3.5 * Faster than 3.6 5. Regular expressions are faster for case-insensitive matching Per change change log Matching and searching case-insensitive regular expressions is much slower than matching and searching case-sensitive regular expressions. Case-insensitivity requires converting every character in input string to lower case and disables some optimizations. But there are only 2669 cased characters (52 in ASCII mode). For all other characters in the pattern we can use case-sensitive matching. The speed improvement is significant, if you’re matching ASCII characters you can see up to a 20x improvements in matching time since it’s now doing a lookup instead of running over each character. lower() Still stuck on Python 2? Check out my new course on Pluralsight for moving from Python 2 to 3.

Keep

Checkout my Pluralsight courses!

5 speed improvements in Python 3.7

Something that few know about me is that I have an alter ego, maybe I’m not a batman, but close. In a world parallel to and , I call myself blackcerv. programming entrepreneurship Before having been a developer or even having an idea that I was interested in technology, my greatest passion was music. Since I was twelve years old I spent every afternoon in my room with my guitar, pedals and laptop making weird sounds. Those were good times. Every day I could not wait to get out of school to get home and start over, it was not about being famous, it was about creating landscapes with sound, of bringing those kinesthetic feelings to reality. translate into frequency what I felt. That was all. Time has continued to go by and the restlessness is still there within me, maybe I will soon take enough time to produce new weird sounds and make new creations with twisted circuits. And what about you? Do you have any passion that moves you to create great things? If so, let me know.

HackerNoon Tech Stories Archive February 11th, 2018

@dclintonDavid Clinton

@jacquesrlegaultJacques Legault, C.Psych.

@dclintonDavid Clinton

@AlykhanGulamaliAlykhan Gulamali

@dychenDaniel Chen

@james_32022James Bowen

@yotamyachmoorgafniYotam Gafni

@ethervolutionSalanfe

@rezaReza Jafery

@rezaReza Jafery

@MappoJoshua

@rusnac14Dan Rusnac

@arnoldklingArnold Kling

@waqas.64M Waqas Raza

@sinewaverAbhiram

@Terren_in_VATerren Peterson

@kavehmzKaveh Mousavi Zamani

@elliot_fElliot Forbes

@ffreitasalvesFernando Freitas Alves

@vinibVini Brasil

@revati.kRevati Kapshikar

@anthony.p.shawAnthony Shaw

@luisacervLuisAcerv

@johnpcutlerJohn Cutler

@dclinton
David Clinton

@jacquesrlegault
Jacques Legault, C.Psych.

@dclinton
David Clinton

@AlykhanGulamali
Alykhan Gulamali

@dychen
Daniel Chen

@james_32022
James Bowen

@yotamyachmoorgafni
Yotam Gafni

@ethervolution
Salanfe

@reza
Reza Jafery

@reza
Reza Jafery

@Mappo
Joshua

@rusnac14
Dan Rusnac

@arnoldkling
Arnold Kling

@waqas.64
M Waqas Raza

@sinewaver
Abhiram

@Terren_in_VA
Terren Peterson

@kavehmz
Kaveh Mousavi Zamani

@elliot_f
Elliot Forbes

@ffreitasalves
Fernando Freitas Alves

@vinib
Vini Brasil

@revati.k
Revati Kapshikar

@anthony.p.shaw
Anthony Shaw

@luisacerv
LuisAcerv

@johnpcutler
John Cutler