As always, the fields of deep learning and natural language processing are as busy as ever. Despite many industries being hindered by the quarantine restrictions in many countries, the machine learning industry continues to move forward. It seems almost every week, new models are being released, and new startups are showing off AI-powered technologies that will help build a better world. In this article, we will briefly go over some of the biggest recent news in NLP and deep learning, as well as some must-read guides, feature articles, tools, resources, and datasets you may want to check out. Machine Learning News 1. How Deep Learning Can Keep You Safe with Real-Time Crime Alerts From Nikunj Aggarwal, the Machine Learning Lead at Citizen, this article gives us a great example of how deep learning is being used to create life-changing (or life-saving) technologies. Citizen is an emergency and safety alert app that warns people of incidents and crimes that have taken place in their area in real-time. Image from Citizen The company used a speech-to-text engine and a convolutional neural network to analyze first responder radio frequencies. In doing so, the company was able to scale their app to multiple cities in the United States. This technology could mark a huge change in the police and first responder infrastructure in years to come. 2. The Release of Open AI API The release of GPT-3 by Open AI was likely the biggest news in the field of NLP this year. However, what many people may have missed is the release of Open AI’s API. The purpose of the API is to give people access to future models developed by the company, including GPT-3. This is big news, as it marks a shift for the company’s normal practices of open-sourcing their models (as they did with GPT-2). In the article, the company explains why they decided to release a commercial product, why they went away from open-source this time around, and how they will control potential misuse of their API. 3. IBM will no longer offer, develop, or research facial recognition technology In , the CEO of IBM publicly stated that the company would be halting development and service offerings of general-purpose facial recognition technology. a letter to congress This was a huge step for the company and a big message to the data science community as a whole. IBM’s move to prioritize ethics and safety might have encouraged other large tech companies (including ) to do the same. Microsoft 4. Introducing the Model Card Toolkit for Easier Model Transparency Reporting With the creation of larger and possibly more complicated deep learning models, it becomes increasingly difficult to explain their intended use cases and other information to users downstream. To help solve this problem, researchers at Google have developed the “Model Card Toolkit” to help make model transparency reports easier to create. 5. You Don’t Need College Anymore, Says Google Do you need a Ph.D. to work in data science? Well, Google’s new certification program may change the game. On July 14th, 2020, Google announced their new professional certification programs in the fields of UX design, project management, and data analysis. Whether or not a Google Certificate in data analysis will be enough to land you a job at a data science team is yet to be determined. However, a certification from the largest tech company in the world may end up being worth more than a 4-year degree. 6. The Court of Justice invalidates Decision 2016/1250 on the adequacy of the protection provided by the EU-US Data Protection Shield In July of 2020, a big decision was made by the Court of Justice of the European Union that may greatly affect data transfer between Europe and the United States. Essentially, the decision was made to invalidate “Decision 2016/1250”, which fostered in a data transfer agreement titled the “EU-U.S. Privacy Shield”. Instead, those whose data are transferred to a country not in the EU, must be afforded “a level of protection essentially equivalent to that guaranteed within the EU by the GDPR.” This means that if a company like Tik Tok wants to transfer data from users in the EU to be processed on servers in the United States, authorities have the responsibility to prohibit this data transfer if they deem that data privacy and security measures in the United States don’t comply with GDPR standards. If you want to learn more about this, is a much easier read than the actual legal document. TechCrunch’s article Machine Learning Guides & Feature Articles 7. Deep Learning Algorithms — The Complete Guide From Sergios Karagiannakos, the founder of AI Summer, this article serves as a meaty guide to deep learning. It introduces many topics, from the different kinds of neural networks to deep learning baselines in NLP and computer vision. 8. OpenAI's GPT-3 Language Model: A Technical Overview As mentioned previously, Open AI’s launch of GPT-3 was likely the biggest news in NLP so far this year. For those of you that don’t know, GPT-3 is a text-generating neural network that has 175 billion parameters, which is incredibly larger than the previous model, GPT-2 (1.5 billion parameters). This guide serves as a great overview of the model, with key takeaways and explanations about the model and data used to train it. 9. Philosophers On GPT-3 (updated with replies by GPT-3) From Daily Nous, this is an interesting thought piece where 9 philosophers take a deep dive into Open AI’s GPT-3. These thought leaders explore the possible ethical and moral issues, as well as the lingering questions brought forth by the technology. 10. End to End Multiclass Image Classification Using Pytorch and Transfer Learning From Rahul Agarwal, a data scientist at WalmartLabs, this guide is a step-by-step tutorial on creating a multiclass image classification model. Furthermore, Agarwal explains what transfer learning is and how to use it to improve your own image classification models. 11. The 100-Year History of Self-Driving Cars This feature piece from OneZero talks about the long history of autonomous vehicles, from the first manual auto-pilot maneuvers on ships to the self-driving cars we see from the likes of Tesla and Google today. Machine Learning Tools & Resources 12. 5 Fantastic Natural Language Processing Books Written by KDnuggets Editor Matthew Mayo, this useful guide introduces five books on NLP from his personal library. Unlike other book lists you may find online, Matthew has personally read all of these books and vouches for their quality. Please note that these books are not free, so they require a bit of investment on your part. 13. Tools to Spot Deepfakes and AI-Generated Text With the rampant spread of misinformation on social media, I was very concerned when I saw this spread reach my own inner circles. As it has become easier and easier to create deepfakes and generate fake articles using AI, I wanted to help combat the malicious use of these technologies. This article introduces a few simple methods and browser plugins that may help you detect both deepfakes and AI-generated text. 14. 30 Largest TensorFlow Datasets for Machine Learning This listicle is a simple curation of the largest datasets in the TensorFlow library that may prove useful in improving your deep learning models. It introduces the largest audio, video, image, and text datasets on the platform and some of their intended use cases. 15. Machine Learning Developer Hourly Rate Calculator From Toptal, this handy tool can help you determine the average hourly rate for data scientists based on your location, programming languages, and skills. You can use this calculator to compare the average salary for your position in your own country and other countries to help you evaluate your career and plan your next steps. We hope that these NLP and deep learning articles and guides helped you catch up with some of the big things happening in machine learning this year. For more reading, please take a look at the top stories below. Previously published on: https://www.kdnuggets.com/2020/08/must-read-nlp-deep-learning-articles.html

Google

Microsoft

Techcrunch

Tesla

The 🚀 Booster Pack 🚀 is a Utility NFT special collection for users of the [NFTs2Me](https://www.nfts2me.org/?pm=50) platform that offers a number of unique benefits and enhancements for the holders of this NFT. There are three tiers of Booster Packs. All of them unlock special features as well as early access to new features.

NFTs2Me Starter Booster Pack

The NFTs2Me Item

Check Out My Mobile Suit Gundam Watch Order Guide

Read My Stories

Follow my blog about Japan

Book a call

VP of Editorial

Check out my review of the latest Shinkai blockbuster!

If Humans Could Talk

Why Targeting Single-Word Keywords Doesn't Work for Evergreen Content

How to Connect Oculus Quest Bluetooth Headphones: No Latency

15 Must-read Machine Learning Articles for Data Scientists

Making the transition to a work from home arrangement has been a heavy lift for a lot of organizations. However, due to various risk factors and regulations, making the sudden shift to working from home has been more complicated for some sectors than others. Industries like finance and healthcare, as well as those working for the government, face tighter restrictions on remote work. This is because the risks to these sectors are deemed to be higher due to and security considerations. privacy In many cases, it is against the rules for certain jobs to be performed remotely out of concern for security. Under normal circumstances, it would make perfect sense to forbid the employees of large financial institutions from making sensitive transactions over insecure home networks. But in the time of COVID-19, many of these regulations have been weakened, , in order to allow work to continue on while keeping workers safely at home. if temporarily At the same time that regulators and organizations are attempting to find workarounds to accommodate the need to work away from the office, security threats are mounting as hackers look to take advantage of the situation. In hopes of helping organizations in sensitive sectors better understand their risks, we examined each one’s threat models and provided a couple of suggestions on how to mitigate them. Defining Security Concepts — The CIA Triad When we talk about cybersecurity, it's worth taking a moment to define our terms. More than just a buzzword, cybersecurity describes the effort to protect information. Yes, there are examples of cyber crossing into kinetic like we saw in Stuxnet, power stations in Ukraine, and a lot of machines that became expensive paperweights after the NotPetya attacks. But for most organizations, the target is the data on their systems that is either itself valuable or can be used to access something of value. In practice, this can be personally identifiable information like a Social Security number for use in fraud, a company’s intellectual property, sensitive government information, voting information, credit card numbers, or even the ability to access the data itself. Thinking about these examples laid out above, we can break information security into three categories: confidentiality, integrity, and availability. The CIA triad, as it is often known, asks us whether the information in our systems is still secret, trustworthy and, well, available if we need to access it. If any of these three conditions have been compromised, then we may be in trouble. Let’s look first at the example of healthcare to understand how the CIA concept impacts our sensitive organization types in practice. Healthcare Confidentiality is extremely important when it comes to healthcare. Whether it is communications with a doctor, medical records, or other information nobody else has a right to know about, people rightly take the privacy of their medical information seriously. Beyond the expectation that health records remain private, they contain a lot of personal information that can be used for identity theft and fraud. They have addresses, birth dates, family details, and plenty of other tidbits that can be sold to fraudsters looking to apply for credit cards or loans under someone else’s name. Recognizing the need to secure these kinds of data and doctor/patient confidentiality, the government has issued regulations for healthcare providers and services. These include the well known Health Insurance Portability and Accountability Act ( ) and the more recent Health Information Technology for Economic and Clinical Health Act (HITECH). HIPAA Looking at HIPAA, its Security Rule lays out the standards for dealing with electronic protected health information (e-PHI). It that covered entities must: states Identify and protect against reasonably anticipated threats to the security or integrity of the information; Protect against reasonably anticipated, impermissible uses or disclosures; and Ensure compliance by their workforce. Ensure the confidentiality, integrity, and availability of all e-PHI they create, receive, maintain or transmit; On a good day, many organizations have trouble staying compliant with HIPAA. The regulations require that they take reasonable measures to keep their systems secure and employees in line with best practices. This is easier said than done on outdated systems with IT teams that are stretched thin, and a workforce that is often far from hardened against attacks by hackers. Keeping data secure during the COVID-19 outbreak has become a bigger challenge as more medical services moved from the in-person appointment to the digital. Telehealth services, where a patient communicates with their doctor electronically, usually over a video chat app on their phone or transmits data to them from a device, have been crucial in helping the public continue to access important care. While there are a certified as HIPAA-compliant, the Department of Health and Human Services (HHS) has temporarily allowed for the use of additional services such as FaceTime, Zoom, and even Facebook Messenger to conduct these visits. This is good for patients who need to speak with their doctor without taking additional risks, but there are also risks if healthcare providers fail to take the necessary security precautions. number of platforms The first concern is that not all applications utilize end-to-end (e2e) encryption. Essentially, this is where the data being sent from one device to another can only be read by the person it is being sent to since only they have the keys to decrypt the messages. This prevents the data from being intercepted by a “man-in-the-middle” attack. Zoom took a lot of heat for initially claiming that it was using e2e before admitting that they were not. Features like their call-in numbers for those not using the app mean that the calls cannot be encrypted. The second issue stems from the security of endpoints like mobile phones and computers. Implementing updates as they become available is crucial for preventing the exploitation of software vulnerabilities. Misconfigurations on communication apps like Zoom can open the door to eavesdropping and put patient privacy at risk. While working remotely is not the cause of these security concerns, it puts a lot of stress on a system that already struggles to get it right day-to-day. Ensuring devices are up to date is not easy. Many healthcare providers will choose to go with the telehealth option that is most usable for their staff and patients, not necessarily the one that is most secure. These are significant challenges to overcome. Unfortunately, this is not the only sector to face significant issues from the remote work situation. Financial Institutions There’s an old joke about why bank robbers rob banks. Because it’s where the money is. Whereas an old-fashioned stickup is less of an issue for these financial organizations when most of their transactions are performed digitally, there are still plenty of risks they must mitigate. Organizations that handle financial information and transactions have long been aware of the need for security. Unlike the case of healthcare providers, security is generally well-funded. But financial institutions still face very real risks to all three of our CIA triad criteria. Our trust in these institutions depends on their ability to keep our accounts and transactions private (confidential), accurate (integrity), and of course accessible (availability). Any threat to these factors and the system could find itself in serious trouble. Now, in the current work-from-home moment, the financial industry faces challenges in maintaining security and adhering as closly as possible to regulations aimed at guarding against abuse from insider threats and external attackers. However, faced with the balancing act of keeping services running for customers vs security controls, the Financial Industry Regulatory Authority (FINRA) has issued for the pandemic. The regulator has already made about relaxing rules for how Wall Street firms are required to supervise their employees involved in trading from remote locations. special guidance noises One they are currently allowing is that documents normally needed to be transferred by hard copy are now permitted to be sent by email. This is good news for limiting employees to the risk of exposure. At the same time, it puts additional challenges on securing devices and communications. significant change When working in the office, employees at financial institutions use employer-provided IT networks and computers. But what happens when employees work from home on unsecured home networks? Is their VPN properly configured? Are they using devices supplied by their employer or a personal computer that hasn't been updated in years? Then there are the more human challenges. Hackers are taking advantage of remote work to launch phishing campaigns aimed at tricking workers into handing over credentials. One concern is that hackers might pretend to be from the support team and ask an employee for access to their account. Under normal circumstances, it would be easy enough to walk down to double-check on a questionable request in person. However in the remote experience, this becomes a harder nut to crack. Government The next concern is government work. Local, state, or federal, all levels of government are further strained by our current work from home arrangement. While every department has its own specific requirements, the National Institute for Standards and Technology (NIST) has issued the standard for government compliance. The Department of Homeland Security also has a say when it comes to data security and the Federal Information Security Management Act of 2002 ( ) provides another foundational layer of cyber-protocol to be followed. cybersecurity framework FISMA Similar to healthcare, public-facing and typically under-resourced, government agencies start at a significant cybersecurity disadvantage. While certain departments may have expectedly higher standards for data security given their assumed risk level (the NSA on taking your work home with you), others like the Office of Personnel Management have been the precisely because of their lax security. frowns target of high-profile attacks One of the more significant challenges for government departments is that, even as a significant number have been using VPNs and employee monitoring software, there has never been this volume of workers transferring to remote work all at once. The are many. Using unsecured internet connections, the lack of vetted/updated devices, and phishing attempts all threaten institutional cybersecurity. working remotely for years potential pitfalls Additionally, as the number of workers needing to be independently secured rises and IT teams pulling together solutions with a mixture of popsicle sticks and chewing gum, adversaries rightly vire this reshuffling of policies as an for hacking. opportunity Government organizations are targeted for many reasons. On one hand, state actors like China are launching against researchers working on COVID-19 or in an effort to identify . On the other hand, cities and states are facing an uptick in the number of from cybercriminals out to make a quick buck. massive intrusions intelligence assets ransomware attacks Given the range of threats facing the government, healthcare, and financial sectors during this mass migration to remote work, how can their organizations work to improve their chances of making it through with minimal cyber scrapes and bruises? Three Tips for Cybersecurity Threat Mitigation During COVID-19 There is no shortage of excellent advice available online for those looking to make their organization a little bit safer during the COVID-19 outbreak. I always recommend looking at the provided by the Electronic Frontier Foundation (EFF) for becoming better educated about how to protect yourself. resources But before you go on a deep dive of cybersecurity wisdom seeking, here are a few tips to help you and your team avoid the most pressing threats out there today. 1. Think Before You Click Ransomware is one of the biggest concerns for organizations in all sectors today. These attacks lock users out of their systems, leaving them at the mercy of hackers to let them back in at a price. Along with cities, hospitals have found themselves to be particularly vulnerable to these attacks, since being locked out of systems means lives are at risk. Considering the risk, many have been quick to pay out to regain access. hundreds of thousands of dollars As organizations have become smarter about backing up their files, hackers have also evolved. Now many pose the double threat of not only locking organization outs of their machines or networks but also threatening to publicly dump data if not paid, compromising data security compliance as well. In most cases, attackers start with a phishing email enticing an employee to open a boobytrapped document or click on a link. Once they gain a foothold on a device, the send a malware payload to infect their target. Since many organizations are public-facing, avoiding clicking on links is easier said than done. Sure, you can look out for telltale signs like poor spelling or other mistakes, but many hackers have gotten better at their craft or simply buy high-quality phishing emails off black markets. Educating your team to spot suspicious emails is the first line of defense. If an email looks suspicious, then avoid opening it or any docs or links contained within. It's always better to send a suspicious communication to security for inspection than risk harming the organization. As a backup though, we recommend that your system admins and macros in Office products. These are two of the most common ways malware infects systems. They are also features the majority of users do not need, so it is far better to simply avoid having them open as avenues of attack. disable Powershell 2. Verify with a Second Channel Sticking with phishing, one of the most common threats facing organizations is business email compromise ( ). While there are many forms of this attack, all involve using social engineering to trick an employee into sending them money. Most often attackers pretend to be an executive at the target's company or a vendor sending an invoice. Typically, the goal is to fool an insider into making a deposit or purchase that will materially benefit the hacker. BEC Defending against these kinds of tricks, unfortunately, puts the onus on the user. It's advisable to always check that an email or communication really comes from the right address, and not someone creating a fraudulent address. If there's ever any doubt, the best thing to do is ask. Remote workers make this more difficult since there are many more opportunities for hackers to pose as someone from within an organization. But even if you cannot just pop down the hall to the CFO’s office, you can at least pick up the phone. Never ask for confirmation on the same channel that you suspect might be compromised. 3. Update, Patch, Repeat One of the most important steps for improving an organization's chance of success against attackers is to regularly update software. This is understandably annoying for IT teams an workers. It can be time-consuming and impact the performance of essential software. But updates often explicitly call out the vulnerabilities they are released to addressed, tipping their weak hand to cybercriminals. In recent years, some of the most notorious hacks have been carried out not by using zero-day exploits, but by leveraging known vulnerabilities on unpatched systems. Think WannaCry and its use of the EternalBlue exploit that the NSA had found and developed. Microsoft had issued patches well before the attack was launched, but many organizations like the UK’s National Healthcare System (NHS) were still running old versions of Windows that were not protected. Staying Secure in Uncertain Times Even as some states have begun to plot their course towards a post-pandemic future, many aspects of how we work are likely to remain in flux. The only certainty is uncertainty as we learn more and adjust to the new normal. Whether your organization returns to work full-time or on a hybrid model, it's best to stick to proven practices for staying secure. While complex threats will continue to emerge, organizations can take a significant step in fending off attacks by covering the basics. This article was originally published on IT Security Central and reprinted with permission.

Facebook

Target

2022 - HackerNoon Contributor of the Year - Cyber Security Awareness

2022 - HackerNoon Contributor of the Year - Cyber Threats

2022 - HackerNoon Contributor of the Year - Cybercrime

2022 - HackerNoon Contributor of the Year - Data Security

Check out new Teramind Academy 

Nominated for 2022 - HackerNoon Contributor of the Year - Cyber Threats

Nominated for 2022 - HackerNoon Contributor of the Year - Cybercrime

Nominated for 2022 - HackerNoon Contributor of the Year - Cyber Security Awareness

Nominated for 2022 - HackerNoon Contributor of the Year - Data Security

The Sectors Most at Risk from Remote Work and How to Secure Them

( ) is an encrypted version of the protocol. It uses or to encrypt all communication between a client and a server. This secure connection allows clients to safely exchange sensitive data with a server, such as when performing banking activities or online shopping. HTTPS HyperText Transfer Protocol Secure HTTP SSL TLS View Previous Terms: Block cipher mode of operation Certificate authority Challenge-response authentication Cipher Cipher suite Ciphertext CORS CORS-safelisted request header CORS-safelisted response header Cross-site scripting Cryptanalysis Cryptographic hash function Cryptography CSP CSRF Decryption Digital certificate DTLS (Datagram Transport Layer Security) Encryption Forbidden header name Forbidden response header name Hash HMAC HPKP HSTS Key MitM OWASP Preflight request Public-key cryptography Reporting directive Robots.txt Same-origin policy Session Hijacking SQL Injection Symmetric-key cryptography TOFU Transport Layer Security (TLS) Credits Source: https://developer.mozilla.org/en-US/docs/Glossary/HTTPS Published under license Open CC Attribution ShareAlike 3.0

Mozilla

Glossary of Security Terms: HTTPS

A key is a piece of information used by a for and/or . Encrypted messages should remain secure even if everything about the , except for the key, is public knowledge. cipher encryption decryption cryptosystem In , the same key is used for both encryption and decryption. In , there exists a pair of related keys known as the and . The public key is freely available, whereas the private key is kept secret. The public key is able to encrypt messages that only the corresponding private key is able to decrypt, and vice versa. symmetric-key cryptography public-key cryptography public key private key View Previous Terms: Block cipher mode of operation Certificate authority Challenge-response authentication Cipher Cipher suite Ciphertext CORS CORS-safelisted request header CORS-safelisted response header Cross-site scripting Cryptanalysis Cryptographic hash function Cryptography CSP CSRF Decryption Digital certificate DTLS (Datagram Transport Layer Security) Encryption Forbidden header name Forbidden response header name Hash HMAC HPKP HSTS HTTPS MitM OWASP Preflight request Public-key cryptography Reporting directive Robots.txt Same-origin policy Session Hijacking SQL Injection Symmetric-key cryptography TOFU Transport Layer Security (TLS) Credits Source: https://developer.mozilla.org/en-US/docs/Glossary/Key Published under license Open CC Attribution ShareAlike 3.0

Glossary of Security Terms: Key

Proof-of-Stake (PoS) has been rising as one of the sexiest techniques for replacing Proof-of-Work (PoW) currently used in mainstream blockchain technologies such as Ethereum and Bitcoin. The objectives of PoS-based blockchains such as EOS.IO and XinFin Hybrid Blockchain are not only to eliminate the PoW-related electricity consumption but also to provide a scalable solution to tackle the transaction processing performance problem in Ethereum and Bitcoin. The latter can only process around 10–15 transactions per second, which is not comparable to Visa and MasterCard. In this post, we analyse in depth and compare in details the similarities and differences between XinFin Hybrid Blockchainand EOS.IO. This post is the continuum in the series of technical posts aiming for comparing XinFin Hybrid Blockchain and some notable PoS-based blockchains, including EOS.IO, Casper FFG, Cardano, and Tendermint. The CAP theorem and the blockchain’s trilemma The classic mathematically proven CAP theorem says that a distributed system cannot together guarantee all three requirements: consistency (C), availability (A), and partition tolerance (P). Consistency means that the results of reading some data on any node of a distributed system are either the most recent result or an error. Availability requires that every request to read receives responses without guarantee that the received results are the most recent ones. Partition tolerance is inherent in the distributed system meaning that “the system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes”. Since partition tolerance is the nature of distributed systems, many system designs have to choose either consistency or availability while sacrificing the other requirement. Blockchains, which run in a distributed environment, also cannot break out of this loop. Everyone wants to get the most recent result right after a read request to any node (availability), but that would lose the system consistency since there is block propagation latency that makes the data in different nodes be not the same. Even though the CAP theorem is not obviously visible to users of blockchains, blockchains have their own trilemma where three requirements, namely decentralization, security and performance/scalability, are difficult to guarantee a convergence into the same blockchain. Bitcoin and Ethereum are good at decentralization and security but offer a very poor performance. Security seems the obviously visible and sensible requirement required by users. Losing money or double-spending should not happen. Many blockchains are trying to balance the trade-off between decentralization and performance, by reducing the number of block producers in the network. A lower number of producers means quicker network propagation, thus quicker consensus between block producers. EOS and XinFin Hybrid Blockchain are both trying to improve their performance, while the differences between them lie in their consensus details and the decentralisability they sacrifice for the performance gain. EOS at a first glance EOS is, however, still a relatively new project compared with Bitcoin and Ethereum. EOS inherits the Delegated Proof-of-Stake (DPoS) invented by Dan Larimer from its application-specific ancestor blockchains, namely Bitshares and Steemit, and generalizes multiple features from these ancestors. Similar to other blockchains, EOS presents some security features such that the system can be fault tolerant if no more than 1/3 total of nodes are attackers. On the other hand, in the impossible triangle of blockchain where decentralization, security, and scalability cannot be done all together, EOS sacrifices decentralization for scalability. Therefore, only 21 block producers can create blocks by being elected, but block time can be decreased down to 500ms. Consensus “Consensus is the heart and the soul of any decentralized blockchain system”. It is the main actor who drives the different requirements of the system, including decentralization, security, and performance. EOS.IO and Bitshares both use a new PoS-based consensus algorithm called Delegated Proof-of-Stake (DPoS). In this latter, instead of miners, as in Bitcoin and Ethereum, who participate into a race of hashing operations, block producers in EOS, namely witnesses or block producers, are elected through a voting system. The witnesses are stakeholders who are the most voted by other stakeholders: the stakeholders delegate the block production ability to the witnesses. In both EOS and XinFin Hybrid Blockchain the ability to produce blocks is the responsibility of the nodes, either witnesses in EOS or masternodes in XinFin Hybrid Blockchain. This responsibility is dispatched equally to these block producers so that every block producer can and must create blocks. The set of the block producers is fixed within a period, called epoch, but can be changed dynamically for every epoch. There is a pre-determined order of block producers, each of which produces blocks. EOS employs what we called single validation to validate blocks while XinFin Hybrid Blockchain has designed a double validation technique. Then, what is the point of doing double validation? To understand that, let me show the differences between these two validation techniques as follows In single validation, each block is created by one block producer and asynchronously validated by other block producers. The next block producer in the pre-determined order then validates that block and builds another block on top of it. In double validation, another layer of block validation is added in the way that a created block must be validated by another randomly selected block producer, namely block verifier, before the next block producer lengthens the blockchain from that block. What this means is that, if, in EOS, there is no order of which block producers should sign off the block after its creation. XinFin Hybrid Blockchain with its double validation technique requires that the second masternode who signs off on the created block must be the block verifier, that is randomly selected. If this block verifier detects the abnormality in the block, it rejects the block and all other masternodes will soon reject it as well. An epoch is measured by the maximum number of blocks that can be created within that period. In EOS, an epoch is equivalent to 126 blocks (each block producer creates 6 blocks in each epoch), while this epoch is 900 blocks in XinFin Hybrid Blockchain. At the end of each epoch, block producers are rewarded within a reward block. In EOS, a block producer is rewarded for the successfully validated produced blocks that it created for that epoch, while the reward of each successful block in XinFin Hybrid Blockchain is shared equally between masternodes who validate and sign the block to finalize it. XinFin Hybrid Blockchain requires 3/4 of the masternodes to sign a block in order to finalize it. Once 3/4 of the signatures of the masternodes are available for a block, the latter is finalized meaning that it will never be modified later by any means. EOS aims to provide zero fee transactions but it is not zero in reality. This idea of ‘zero user-fee’ is disingenuous because it suggests that economic incentives are no longer required to maintain the network. EOS does indeed have fees, but they are hidden in the form of inflation. Security Nothing-at-stake problem If there’s a fork in the chain, the optimal strategy for any validator is to validate on every chain, so that the validator gets their reward regardless of the outcome of the fork. Attackers can create multiple forks and block producers will sign every block in every fork in order to get incentives regardless of which chain will win. An attacker then takes advantage of this security hole by making a transaction for paying something on one chain and another transaction sending the same money to his controlled account, thus double-spending occurs. EOS and XinFin Hybrid Blockchain both choose to down-vote the block producers who sign multiple chains, instead of having slashing punishment as in Casper. Moreover, having the order of block producers to create blocks also provides a means of defense to this problem, though not all cases. The network might converge to the chain since it is longer than the other chain that missed a slot of block time. But, if the another chain misses a block time caused by either its block producer being on the other chain or an attacker subjectively missing a block time, thus resulting in two chains with equal length, therefore having nothing-at-stake, which chain should go for all block producers? XinFin Hybrid Blockchain  provides more protection on this problem by having double validation. The chain can continue running when the block verifier signs On the other hand, for the other chain to continue, it needs both block creator and block verifier on that chain separated from the  chain, which is unlikely to happen. Long-range attack Both EOS and XinFin Hybrid Blockchain can defend against long-range attack by block finality and the order of block producers who can create blocks at a specific point in time. Therefore, there is no chance for one malicious block producer to create a longer valid chain because other block producers will refuse it. Censorship attack “If blockchain platforms do not provide censorship resistance — i.e. they rely on a set number of trusted actors to produce and validate blocks — then we have simply returned to legacy database systems, albeit at the expense of the efficiency that legacy systems like Amazon Web Services provide”. Both EOS and XinFin Hybrid Blockchain are trying to reinforce the censorship resistance through the system, where nodes have to stake a significant amount of tokens in order to become block producers and then can be rewarded for their honest behavior of work, therefore, they are financially disincentivized from acting maliciously. Censorship attacks can happen if 2/3 in EOS or 3/4 in XinFin Hybrid Blockchain of block producers are controlled, thus requiring an enormous amount of money staked, turning the system into a plutocracy, where no one wants to use it, therefore reduce the token price, which is not wanted by the attackers. Compared to EOS, XinFin Hybrid Blockchain censorship resistance is seemingly stronger since (1) XinFin Hybrid Blockchain has 108 masternodes with 192 backup node compared to 21 block producers of EOS, and (2) stakes of XinFin Hybrid Bolockchain masternodes are locked in smart contracts that will only unlock after one month. If any masternode misbehaves, it will be out and its stake cannot be used in any way for a long month. Performance It is clear that EOS has better performance in terms of block time (500ms) than what XinFin Hybrid Blockchain targets (2s) since only 21 masternodes would reduce some issues related to network propagation and synchronisation. EOS also has better transaction confirmation latency which is approximately less than 1 second (0.5 second of block time + 0.25 second of finality + some time for sending the transaction) as what their team expected. In my opinion, with a block time of 0.5 second, EOS is trying to put a lot of expectations on the network bandwidth and propagation. A block producer must verify the transactions which are included in the previous block, create another block with another set of verified transactions, and quickly send the created block to the next block producer in the order so that this next block producer does not consider it as a missing block. On the other hand, XinFin Hybrid Blockchain tries to be more secure by providing the double validation technique, which neatly requires some additional latency to get the block verifier reached and signed on the block so that other masternodes can accept it. More on performance, EOS is itself designed as an operating system-like construct that can schedule transactions and what are called “actions”. Independent actions can be scheduled to execute in parallel. XinFin Hybrid Blockchain, however, has a different vision on improving its performance. In EOS, smart contracts are written in C++, instead of EVM smart contract languages such as Solidity used in Xinfin Hybrid Blockchain since the latter aims for providing an EVM-compatible platform, thus puts the focus on the portability of EVM smart contracts currently running on Ethereum. XinFin Hybrid blockchain Team is currently researching on two orthogonal directions for increasing its performance. One is state sharding and the other one is EVM parallelization. State sharding divides the network into groups of masternodes, each of which helps maintain a portion of the blockchain, called shards. The assignment of masternodes to specific shards is based on a uniform distribution and publicly verifiable decentralized randomization in order to secure the network since one node cannot determine itself to assign to a specific shard to avoid shard attacks where all attacking nodes will actively join the same shard. Each shard will then process a set of non-overlapping transactions in parallel with other shards, thus increase the whole throughput of the network. More on sharding can be found in our sharding proposal. EVM parallelization scaling aims to improve the performance of smart contract processing by making multiple EVM instances run on multiple machines connected to a masternode. In other words, EVM parallelization is trying to schedule smart contracts and outsource their executions to other machines in order to increase the performance of each masternode. Roadmap EOS started the development in summer 2017 and had its minimal viable test network in the fall of 2017. In June 2018, the EOS mainnet has been launched after receiving enough votes (more than 15% of total EOS token) for mainnet launch. Only one thread is enabled for each block producer in the current mainnet. The parallelization mechanism of EOS is still under development and optimization and will be probably released in the fall of 2018. Founded later than EOS, XinFin Hybrid Blockchain started its development in summer 2018. XinFin Hybrid Blockchain had released its testnet 1in December 2018. XinFin Hybrid Blockchain mainnet with its newly designed XinFin Delegated Proof of stake (XDPoS) Masternodes consensus Have been officially launched on 1st june 2019. In my opinion, both EOS and XinFin Hybrid Blockchain are hardworking to bring the best products to their community, and even founded later, XinFin Hybrid Blockchain Team is catching up EOS and other Ethereum-based initiatives Ecosystem Many DApps have been developed and are about to run on the EOS blockchain. Readers are kindly invited to check for the list of these DApps here. For these DApps and the strong community of EOS after its mainnet launch, I put EOS over XinFin Hybrid Blockchain. But, remember that, if EOS has already developed a strong community, even with its C++-based smart contract programming language, XinFin Hybrid Blockchain also has a lot of potential in building a strong ecosystem because it is EVM-compatible. Therefore, any DApp running on the current Ethereum blockchain can be ported to run on XinFin Hybrid Blockchain without any struggle. The conclusion If the DPoS-based Lisk project has been criticized for the formation of Mafia coalition where a set of block producers control the network and the transaction processing, EOS and XinFin Hybrid Blockchain are trying to avoid that by providing almost zero-fee transactions and censorship-resistant consensus mechanism (double validation in XinFin Hybrid Blockchain. Both EOS and XinFin Hybrid Blockchain sacrifice the decentralization for the transaction processing performance. If EOS only allows for 21 block producers (less than 108 block producers of XinFin Network) to be able to produce 1 block every 0.5s, XinFin seems to balance the trade-off between decentralization and performance, by allowing 108 masternodes to create blocks but increase the block time to 2 second. This two second block time also allows XinFin to improve the security of the blockchain by providing the double validation mechanism. EOS has its parallelisation architecture and mechanism for processing potentially millions of transactions per second. XinFin, on the other hand, is working on both sharding and EVM parallelization targeting to raise the XinFin performance to 2000+ Transactions per second. EOS has accomplished its mainnet, which is one step over XinFin, and also has a strong ecosystem with many potential DApps. XinFin, on the other hand, is focusing on the development of its EVM-compatible blockchain, enabling thousands of DApps currently running on the Ethereum blockchain. I won’t say which one is the blockchain winner for the moment (even though we all know EOS has a better community and a strong ecosystem) since it is unfair for XinFin because it was founded quite late than EOS but it is moving up towards very quickly. Thus, I will let the time answer and the users decide on it. About XinFin Hybrid Blockchain XinFin Hybrid Blockchain is an open-source enterprise-ready Hybrid Blockchain for Global Trade and Finance. It combines the power of public and private Blockchains with interoperable smart contracts. XinFin is fully EVM compatible.

Cardano

Chain

Amazon

Dapp

fees

Lisk

Mastercard

Visa

XinFin Hybrid Blockchain vs EOS.IO: The Battle of XDPoS vs DPoS 

First of all, the main question is: What is Artificial Intelligence in the gaming world? has been in the gaming industry since the . But the technology was so different now from then. Artificial intelligence beginning of video games Artificial intelligence works with the NPC's (non-playable characters), and is used to determine the behavior of the non-playable characters by generating the response or actions from the NPC's. Artificial intelligence mainly works on interaction, so it does not include the firing, slashing, or killing an enemy. What are NPC AIs? A Non-Playable Character is like the protagonist's friend or family member, or it can be anybody in the game. In most games today, the NPC AIs don't really use machine learning technology, because they don't have to. If there are 4 dialogue options to choose from and 1 NPC response to each option, this can be achieved with traditional computer programming. There is no need for machine learning here. In terms of traditional NPCs, we use the term AI because these characters are supposed to be made in a way that they appear real and alive. The main work of an NPC is to help the player to advance in the storyline or to say something about the tasks of the game or different places the player has to go or how to defeat a certain enemy. They are controlled by artificial intelligence, as with the help of artificial intelligence an NPC adapts the behavior of the player and generates human-like responses which help in the progression of the mission or storyline. For example, RDR2 (Red Dead Redemption 2) is one of the great games which has shown us the true power of artificial intelligence. The whole storyline has beautifully shown us the importance of NPCs in a game and how they adapt from the player's different decisions and the storyline changes from those decisions. In short words, artificial intelligence determines the player's behaviors and emulates the behavior of the NPC's to make the storyline meaningful. Now on to the main question. 4 Ways Artificial Intelligence Can Transform the Gaming Industry 1. Smarter games Artificial intelligence has done wonders over the past 10 years. AI has made the games so much smarter than they used to before. The choices you made in the game determine how the story turns out, which was less common 10-15 years ago. Think back to games like Prince of Persia, Devil May Cry, and many more RPG games. Remember, as a player, you did not have many choices. There is a rigid storyline that one has to follow, and nowadays games like Horizon Zero Dawn, RDR2 have a lot of choices, and you can define your own storyline. Though scripting is still used for NPC responses, the changes in responses like which response will be perfect for the situation are determined by Artificial intelligence, and the same is the case with the behavior of the NPC's. The goal is to . Nowadays, you can even choose anything in a game from buying a house to dating a girl. New algorithms and techniques are tested every day to make games even smarter. make the game more real lifelike 2. Improving Developer Skills As artificial intelligence technology increases, the game developers are getting more and more chances to show their creativity to the world. As the technology keeps improving the options and the opportunities are infinite, it is also an opportunity for the developers to hone their skills and give the best game possible. People are adapting to AI games more than anything because they give them more options and more sense of control in the game. As developers have a lot of good options to build the game, it is also important for the developers to have a sound knowledge of the latest AI algorithms and technologies. New technologies like pathfinding, function-based movements, and a lot more. 3. Realistic games Things like and virtual reality has really shown that AI has incredible ability to improve our gaming experiences. It has been just amazing how much AI has inclined its technology in the past 10 years. augmented reality Physics-based simulations and 3D visualization techniques are also a big revolution in the gaming universe; all of these advanced technologies have made gaming for the people very realistic. With the help of Virtual Reality(VR), the player can feel like he/she is in the game and living the game in spite of playing it. The more options AI creates, the more realistic games can be made. 4. Changing Technology & Changing Experience All of us want new things to experience, as the old saying goes new is always better. AI technology is always changing and keeps on finding how to go one step forward. As we told you above, about the new AI technology we also want to tell you about the changing experience as we have seen in many games like NBA2k and NFL and many more. When it's the end of the game and scores are tied in the NBA2k the whole crowd gets on a different vibe the chanting is different the intensity is different it almost feels like you have the ball in the final seconds of the game are about to shoot the basketball, you can really feel the intensity of the game while playing on your couch. Nowadays, everyone is looking to drift slightly away from real life and just go into the game and feel the reality there. Artificial intelligence makes sure to give you every detail through high-end graphics and visual appearance but also with its interactive gameplay. Now the question arises: Can Artificial Intelligence somehow become bad for gaming? If you are a , then sometimes doing too much can also make your game bad, and nobody would want to play it. game developer Like if you are using AI for your game make sure you are putting every attention to detail because your game might not be enjoyable if the AI you use is terrible, like the Daikatana it has one of the worst AI in the history of games. AI is machine learning technology, and it is sometimes its not such a great idea to put AI in some games because usually the AI is used to beat the player and its not fun like the game Nioh it's too difficult, and after a while, It just doesn't feel good to play it the goal of AI is to challenge the player but not so much that it stopped being fun. Sometimes it's frustrating when the game has too many choices as a gamer, I myself like to play every bit of a game, and it's just frustrating sometimes when there are a lot of options, and a player can pick only one. The gaming industry is one of the most improved, advanced, and grown industries in the past 2 decades, the industry worth over a hundred billion dollars. I remember how games used to look like in the early '90s; they were all glitchy, they lagged, and they used to have limited choices and a rigid storyline. But the years have past and now look at the games we are all playing, it's all player-friendly, and the graphics are just unbelievable and so realistic, and it is all because of the improving technology of Artificial intelligence.

Ball

Contact @ Facebook

How Artificial intelligence is Transforming the Gaming Industry

I have spent the last few weeks understanding the impact of a great revolution in the world of Artificial Intelligence and NLP on the customer experience. Not from a purely technical point of view, but trying to estimate the competitive advantage that this new approach can generate. We are facing yet another disruptive innovation, and it can bring significant advantages, let's try to find out which ones. It all started with the paper " " that has put the NLP world in turmoil. It was immediately understood that something new appeared in the world of artificial intelligence. This was immediately followed by the release of and the associated . Skipping a few milestones, OpenAI presented his new model , a model that, according to those who have tried it, offers incredible performance. Unfortunately, GPT-3 will not be an open-source model as it is GPT-2. Attention Is All You Need BERT paper GPT-3 But what can these models really do, and why are they hailed as the new NLP frontier? I will try to explain it to you in simple terms the competitive advantage they can provide to those who can skillfully employ them to support business processes. First of all, we are talking about pre-trained models. They are models trained by millions and millions of data. Models designed to act in a completely unsupervised way, generalizing the context as much as possible. It is, of course, possible to fine-tune them with a minimum effort, to focus on the specific context. In this case, we are talking about , something like training a human brain to get the knowledge to pass the maturity exam, and then later make them study a very precise field. Fine-tune the brain to understand and generalize in that field. I realize it's a bit of a long-shot similarity, but you get the idea. transfer learning In business terms, it means leaving something very expensive and cumbersome: the creation of training datasets to train models. Whoever, in these last years, has clashed with this reality, knows what I'm talking about: hours and hours of people manually classifying datasets, such as those for sentiment analysis. Read some tests, and classify them in positive, neutral, and negative. Quite a nuisance, in terms of cost and time. I have faced my study of these last weeks with a library in particular: . This library is a small revolution within the great transformer revolution. huggingface.co Think that today to work with these kinds of models you have to use two reference frameworks: and/or . In the second case, you can use a slightly higher-level interface, , sacrificing flexibility. Well, huggingface.co is like the for Unix distributions. It is the Mac OS of transformers. An extremely simple interface to deal with transformers. PyTorch TensorFlow Keras X Window System But let's see what a huggingface pipeline can do today, out-of-the-box, from a business point of view: , returning the sentiment of a text, for example, feedback from a customer. Sentiment Analysis No need for previous training. , give the beginning of the sentence to the template, and it will complete it. For example to comment on financial or operational results. Text Generation No need for previous training. (NER) by exactly identifying the entities described in a text, an extremely important capability for all chatbots that we want to automatically dialogue with clients. Name Entity Recognition No need for previous training. : in this case you provide the model with a context, a long text like a section of a user manual, and a series of questions. The model will automatically answer the questions. Question answering No need for previous training. : given a text with masked words, fill the blanks. This is a fundamental ability of transformers to "understand" the word in its context, and not in sequence. Filling Masked Text : generate a summary of a long text. Summarisation : translate a text in another language. Translation : return a tensor representation of the text. Feature extraction But what really surprised me in the last few days from the point of view of using these models in NLP is what is called zero-shot classification. Practically pre-trained models with millions and millions of data, which need no further training to classify new texts. Just give them the corpus you want to classify, and the candidate labels you want to classify. The model, without any further training, will be able to classify with an accuracy that in my tests proved to be very accurate. ( No need for previous training ) We are certainly faced with something extremely innovative that can completely change the analysis of unstructured data and the competitive advantage that this can create. The transformers will definitely help to better understand and provide services to your customers. It can be a chatbot, a Q&A service, a framework to analyze sentiment, a topic detection data visualization, etc. Transformers will help us to humanize better interactions between customers and machines. They will definitely help to provide The Phygital Journey faster and easier for your clients. In my small world, this is the next AI step I will bring to to manage and improve the performance of our Customer Experience platform, with something truly revolutionary, helping our clients to gain competitive advantage through Customer Experience Management. sandsiv+ For the geeks only: https://colab.research.google.com/drive/1jocViLorbwWIkTXKwxCOV9HLTaDDgCaw?usp=sharing#scrollTo=xWiovVJG9ei_ Previously published at https://www.linkedin.com/pulse/impact-ai-transformers-customer-experience-federico-cesconi

Geeks

The Impact of AI Transformers on the Customer Experience

From vehicle counting and smart parking systems to Autonomous Driving Assistant Systems, the demand for detecting cars, buses, and motorbikes is increasing and soon will be as common of an application as face detection. And of course, they need to run real-time to be usable in most real-world applications, because who will rely on an Autonomous Driving Assistant Systems if it cannot detect cars in front of us while driving. In this post, I will show you how you can implement your own car detector using pre-trained models that are available for download: MobileNet SSD and Xailient Car Detector. Before diving deep into the implementation, let’s gets a bit familiar and know about these models. But feel free to skip to the code and results if you wish. MobileNet SSD MobileNet is a light-weight deep neural network architecture designed for mobiles and embedded vision applications. In many real-world applications such as a self-driving car, the recognition tasks need to be carried out in a timely fashion on a computationally limited device. To fulfil this requirement, MobileNet was developed in 2017. The core layers of MobileNet is built on depth-wise separable filters. The first layer, which is a full convolution, is an exception. To learn further about MobileNet, please refer to . Around the same time (2016), SSD: Single Shot detector was also developed by Google Research team to cater the need for models that can run real-time on embedded devices without a significant trade-off in accuracy. the paper Single Shot object detection or SSD takes one single shot to detect multiple objects within the image. The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes. It’s composed of two parts: Extract feature maps, and Apply convolution filter to detect objects SSD is designed to be independent of the base network, and so it can run on top of any base networks such as VGG, YOLO, MobileNet. In the original paper, Wei Liu and team used VGG-16 network as the base to extract feature maps. To learn further about SSD, please refer to . To further tackle the practical limitations of running high resource and power-consuming neural networks on low-end devices in real-time applications, MobileNet was integrated into the SSD framework. So, when MobileNet is used as the base network in the SSD, it became the paper MobileNet SSD. The MobileNet SSD method was first trained on the COCO dataset and was then fine-tuned on PASCAL VOC reaching 72.7% mAP (mean average precision). We’ll use a MobileNet pre-trained downloaded from that was trained in Caffe-SSD framework. Download the pre-trained MobileNet SSD model and prototxt from . MobileNetSSD_deploy.caffemodel MobileSSD for Real-time Car Detection Step 1: Download pre-trained MobileNetSSD Caffe model and prototxt. https://github.com/chuanqi305/MobileNet-SSD/ here MobileNetSSD_deploy.prototxt Step 2: Implement Code to use MobileNet SSD time cv2 cv numpy np math print( )

prototxt_path = model_path = CLASSES = [ , , , , , , , , , , , , , , , , , , , , ]

net = cv.dnn.readNetFromCaffe(prototxt_path, model_path) rgb = cv.cvtColor(next_frame, cv.COLOR_BGR2RGB)
    (H, W) = next_frame.shape[: ] blob = cv.dnn.blobFromImage(next_frame, size=( , ), ddepth=cv.CV_8U)
    net.setInput(blob, scalefactor= / , mean=[ , , ])
    detections = net.forward() i np.arange( , detections.shape[ ]): confidence = detections[ , , i, ] confidence > : idx = int(detections[ , , i, ]) CLASSES[idx] != : box = detections[ , , i, : ] * np.array([W, H, W, H])
            (startX, startY, endX, endY) = box.astype( )
            
            cv.rectangle(next_frame, (startX, startY), (endX, endY), ( , , ), ) next_frame cap = cv.VideoCapture(filename) frame_width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) fps = size = (int(frame_width),int(frame_height))
    fourcc = cv.VideoWriter_fourcc( , , , )
    out = cv.VideoWriter()
    success = out.open( , fourcc, fps, size, )

    frame_count = t1 = time.time() :
        ret, next_frame = cap.read() ret == : frame_count += next_frame = process_frame_MobileNetSSD(next_frame) out.write(next_frame)
        
        key = cv.waitKey( ) key == : t2 = time.time() fps = str( float(frame_count / float(t2 - t1))) + print( )
    print( .format(frame_count))
    print( .format(float(t2 - t1)))
    print( .format(fps))

    cap.release()
    cv.destroyAllWindows()
    out.release() import import as import as import # load our serialized model from disk "Load MobileNetSSD model" "MobileNetSSD_deploy.prototxt" "MobileNetSSD_deploy.caffemodel" # initialize the list of class labels MobileNet SSD was trained to detect "background" "aeroplane" "bicycle" "bird" "boat" "bottle" "bus" "car" "cat" "chair" "cow" "diningtable" "dog" "horse" "motorbike" "person" "pottedplant" "sheep" "sofa" "train" "tvmonitor" : def process_frame_MobileNetSSD (next_frame) 2 # convert the frame to a blob and pass the blob through the # network and obtain the detections 300 300 1.0 127.5 127.5 127.5 127.5 # loop over the detections for in 0 2 # extract the confidence (i.e., probability) associated # with the prediction 0 0 2 # filter out weak detections by ensuring the `confidence` # is greater than the minimum confidence if 0.7 # extract the index of the class label from the # detections list 0 0 1 # if the class label is not a car, ignore it if "car" continue # compute the (x, y)-coordinates of the bounding box # for the object 0 0 3 7 "int" 0 255 0 3 return : def VehicheDetection_UsingMobileNetSSD (filename) # Write output file # Define the codec and create VideoWriter object 20 'm' 'p' '4' 'v' 'output_mobilenetssd.mov' True 0 # start timer while True # Reads the next video frame into memory if False break 1 # write frame 50 if 27 # Hit ESC key to stop break # end timer # calculate FPS ' FPS' "/MobileNetSSD Car Detector" "Frames processed: {}" "Elapsed time: {:.2f}" "FPS: {}" I ran the above code on two different devices: (Parts of this code is inspired from PyImageSearch blog.) Experiments: On my dev machine, which is Lenovo Yoga 920 with Ubuntu18.04 operating system. On low-cost, resource-constrained device, which is Raspberry Pi 3B+ with Raspbian Buster operating system. Results: On my dev machine, Lenovo Yoga, with MobileNet SSD, I got an inference speed of 23.3 FPS and when I ran RaspberryPi 3B+, the inference speed was 0.9 FPS, using all 4 cores.Pretty dramatic. This experiment shows that if you have a powerful device to run the MobileNetSSD, it performs well and will serve the real-time requirement.  But if your application is targeted to be deployed on a computationally limited IoT/embedded device such as the Raspberry Pi, this does not seem to be a good fit for a real-time application. Xailient Xailient model uses selective attention approach to perform detection. It is inspired by the working mechanism of the human eye.Xailient models are optimized to run on low power devices that are memory and resource-constrained. Now let’s see how Xailient Pre-trained Car detector performs. We’ll use a Xailient’s pre-trained car detector model downloaded from Xailient Car Detector for Real-time Car Detection Step-1: Download pre-trained Car Detector model. console.xailient.com. Step 2: Implement Code to use Xailient Car detector mode time cv2 cv numpy np math xailient dnn print( )
THRESHOLD = detectum = dnn.Detector() _, bboxes = detectum.process_frame(next_frame, THRESHOLD) i bboxes:
        cv.rectangle(next_frame, (i[ ], i[ ]), (i[ ], i[ ]), ( , , ), ) next_frame cap = cv.VideoCapture(filename) frame_width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) fps = size = (int(frame_width),int(frame_height))
    fourcc = cv.VideoWriter_fourcc( , , , )
    out = cv.VideoWriter()
    success = out.open( , fourcc, fps, size, )

    frame_count = t1 = time.time() :
        ret, next_frame = cap.read() ret == : frame_count += next_frame = process_frame_xailient(next_frame) out.write(next_frame)

        key = cv.waitKey( ) key == : t2 = time.time() fps = str( float(frame_count / float(t2 - t1))) + print( )
    print( .format(frame_count))
    print( .format(float(t2 - t1)))
    print( .format(fps))

    cap.release()
    cv.destroyAllWindows()
    out.release() import import as import as import from import # initialize Xailient model "Initialize Xailient model" 0.6 # Value between 0 and 1 for confidence score : def process_frame_xailient (next_frame) # Extract bbox coords # Loop through list (if empty this will be skipped) and overlay green bboxes # Format of bboxes is: xmin, ymin (top left), xmax, ymax (bottom right) for in 0 1 2 3 0 255 0 3 return : def VehicheDetection_UsingXailient (filename) # Write output file # Define the codec and create VideoWriter object 20 'm' 'p' '4' 'v' 'output_xailient.mov' True 0 # start timer while True # Reads the next video frame into memory if False break 1 # write frame 50 if 27 # Hit ESC key to stop break # end timer # calculate FPS ' FPS' "/nXailient Car Detector" "Frames processed: {}" "Elapsed time: {:.2f}" "FPS: {}" I ran the above code the same two sets of devices: Experiments: On my dev machine, which is Lenovo Yoga 920 with Ubuntu18.04 operating system. On low-cost, resource constrained device, which is Raspberry Pi 3B+ with Raspbian Buster operating system. Results: On dev machine, there is a slight improvement on inference speed when using Xailient Car Detector even when only 1 core is used. On Raspberry Pi, however, Xailient processes 8x more frames per second with a single core. Summarizing the results of both models: The video I used for this experiment was downloaded from In this post, we looked the need for real-time detection models, briefly introduced MobileNet, SSD, MobileNetSSD and Xailient, all of which were developed to solve the same challenge: to run detection models on low-powered, resource-constrained IoT/embedded devices with a right balance of speed and accuracy. We used pre-trained MobileNetSSD and Xailient car detector models and performed experiments on two separate devices: dev machine and a low-cost IoT device. Results show a slight improvement in speed of Xailient Car detector over MobileNetSSD, in the dev machine and a significant improvement in the low-cost IoT device, even when only 1 core was used. If you want to extend your car detection application to car tracking and speed estimation, a very good blog by PyImageSearch. Pexels.com Originally published in xailient.com/blog here’s References Howard, Andrew G., et al. "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861 (2017). Liu, Wei, et al. "Ssd: Single shot multibox detector." European conference on computer vision. Springer, Cham, 2016. https://www.pyimagesearch.com/2019/12/02/opencv-vehicle-detection-tracking-and-speed-estimation/ https://honingds.com/blog/ssd-single-shot-object-detection-mobilenet-opencv/ https://github.com/chuanqi305/MobileNet-SSD https://mc.ai/object-detection-with-ssd-and-mobilenet/ https://machinethink.net/blog/mobilenet-v2/#:~:text=SSD%20is%20designed%20to%20be,detection%20portion%20of%20the%20network. Previously published at https://www.xailient.com/post/real-time-vehicle-detection-with-mobilenet-ssd-and-xailient

Timely

Building Real-Time Vehicle Detection System

A distributed architecture brings in several challenges when it comes to operability and monitoring. Here, one may be dealing with tens if not hundreds of microservices, each of which may or may not have been built by the same team. With a large-scale system, it is very important to have eyes on key system metrics, application health and enough data to be able to track down issues quickly and mitigate them. Being in a CLOUD environment such as AWS, Google Cloud, Azure further accentuate the problem and makes it difficult to detect, troubleshoot and isolate an issue due to the dynamic nature of the infrastructure (scale up / down, temporal machines, dynamic IPs etc.). : KEY PILLARS OF OBSERVABILITY - Application metrics, Host/System metrics, Network metrics etc. Metrics - Application and supporting infrastructure logs Logs - Track down a flow of a request through the distributed systemIn this article I will be focusing on the Logs (Application generated only) & Traces aspect of the observability. Traces LOGS With the distributed architecture it has become a standard practice to use centralized logging systems to be able to gather logs and visualize them. Logs generated by different microservice will be automatically shipped to a central location for storage and analysis. Some of the popular centralized logging solutions for application logs are : Splunk Datadog Logstash Fluentd Logs provide useful information about events that have occurred within your application. These could be INFO level logs or error logs with detailed stack traces around exceptions. LOG CORRELATION ACROSS MICROSERVICES A centralized logging system in a large distributed application may be ingesting Gigabytes of data per hour, if not more. Given that a request may be travelling through multiple microservices, one way to query all logs related to a request spanning several microservices is to attribute each request with some kind of unique identifier. In most cases this could be a associated with the request or some sort of generated at the first point of ingress into the application. These identifiers will be attached to each and every log message and are propagated downstream from one microservice to another via request headers (in case the identifier is not part of the request being made downstream). Now one can easily use the or to query the logging system to find all logs associated with the request across multiple services !!! userId unique UUID requestId userId Fig 1: Centralized Logging Below are few examples of how to tag your logs with required information in Java using Request Filters. Fig2 : Log4J2 Config & Sample Log Fig 3: Request Filters with UUID or UserId TRACING Tracing is a method which allows one to profile and monitor applications while they are running. Traces provide useful information such as: Pathway of a request through the distributed system Latency of a request at every hop/call (as in from one service to another) Below is a sample trace for a request interacting with two microservices (ad-bidding-service and ad-integrator service). Fig 4: Tracing In the above example the data was captured and visualized via tool. There are several other ways to capture traces which I will highlight in the upcoming section. DataDog COMPONENTS OF A TRACE A trace is composed of a tree like structure with parent trace and child spans. A request trace has scope over several services and is further broken down into smaller chunks by operations/functionality known as spans. For example, a call from one microservice to another encompasses a span. Within a microservice there may be multiple spans( depending on how many layers of classes/function or dependent microservices are called to serve the request). Tracing works by generating a unique id per request at the point of ingress and propagating that to downstream systems as in request headers. This allows the different trace information emitted from several services to be tied together at a central location for analysis and visualization. trace-context LOGS & TRACE CORRELATION Now that we are able to search logs by or some other unique identifier (such as ) and are able to trace the performance/behavior of a single request. Would it not be great if we are able to tie these together and be able to correlate logs and traces for a single request !! userId generated UUID Having this correlation between logs and request allows one to: Correlate performance metrics directly with logs Be able to make ad-hoc request to the system for troubleshooting issues Enables synthetic transactions to be made against the system at different points in time and be able to compare current traces with historic ones and collect system logs related to these requests automatically IMPLEMENTING DISTRIBUTED TRACING VIA LOG-TRACE CORRELATION APPROACH #1: INSTRUMENTING WITH 3RD PARTY TOOLS SUCH AS DATADOG Reference: DataDog APM In this approach we are instrumenting the services in the distributed systems with DataDog APMs (application performance monitors). Datadog does 100% tracing of requests and also is able to collect logs emitted by your applications. Datadog essentially takes care of all your centralized logging and trace information collection. Datadog generates unique traceIds and propagates the same across all the instrumented downstream microservices automatically. The only thing we need to do is attribute the DD traceId with the logs and we are able to achieve log-trace correlation. Fig 6: Application Instrumentation Using DataDog Fig 7: Log-Trace Correlation as seen in DataDog APPROACH #2: ZIPKINS, CLOUD-SLEUTH WITH SPRING BOOT APPLICATION Reference: , Zipkins Cloud Sleuth Advantages: Fully integrated into SPRING boot Simple and easy to use Traces can be visualized using Zipkins UI Supports OpenTracing standards via external libraries Supports log correlation via Log4j2 MDC contexts Disadvantages: Does not have a solution to automatically collect LOGs related to trace. We would have to ship logs to ElastiSearch ourself and search using the cloud-sleuth generated trace ids (as header X-B3-TraceId ) Design: Fig 8: Zipkins, Cloud Sleuth and Spring Boot Application APPROACH #3: AMAZON XRAY Reference: Amazon XRAY Advantages: Supports all AWS resources natively. Good if your distributed services deployed and running in AWS AWS Load balancers auto-generate a REQUEST ID per incoming request. Frees up the APPLICATION to generate one. (Reference: ) https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html Allows tracing all the way from API gateway to load balancer, to service and other dependent AWS resources Allows log correlation with help of logs in CLOUDWATCH logs Disadvantages: Cloudwatch logs can get very expensive with large log volumes APPROACH #4: JAGER Reference: Jager Advantages: Supports opentracing by default Has libraries that work with Spring Supports a Jager Agent which can be installed as a side car to propagate traces and logs Disadvantages: More complicated from infrastructure maintenance perspective and setup CONCLUSION Logs and traces are definitely helpful by themselves. But when tied together via log-trace correlation they become a powerful tool to expedite troubleshooting of issue in PRODUCTION environment and at the same time giving operations and development time insight into the health, performance and behavior of the distributed systems. As described about there are several ways to implement the solution. Pick your poison :-)

Flow

Trace

Check out my Blog

Introduction To Distributed Tracing Pattern 

'Have you developed or are you in the process of creating an API Server that will be used on a production or cloud environment? In this 4th instalment of my series, I show you how to test the availability of your API Server, so that you can understand how many requests per second it can handle whilst performing heavy duty tasks. Node JS Performance Optimizations This is a very important measure, especially for production environments, because the last thing you want is to have incoming requests queued as a result of your API Server peaking and not freeing up resources often enough. A Quick Note While I’ll be referencing in this article, most of the theory and principles mentioned can apply to any platform and environment. NodeJS Watch The Video On YouTube Which Benchmark Testing Tool To Use The tool I’ll be using for my tests is called . It’s written entirely in and is very similar to , , , , etc. This is good news because you are not forced to use AutoCannon to follow along with this article. If your benchmark testing tool can perform load tests against HTTP requests and can determine the average requests per second, you’re good to go. AutoCannon NodeJS Apache Benchmark Artillery K6 Wrk That being said, should you wish to make use of AutoCannon, it’s as easy as installing it globally as an NPM Module: npm i -g autocannon How To Test The Availability Of Your API Server Firstly, there is an online that you can reference should you wish to run these examples on your local environment. All you will need is NodeJS installed. Source Code Repo The snippet of code below gets you pretty much 99% to where you want to be, with the exception of setting up your package.json, setting the NODE_ENV to Production and PORT to 6000 (reference sample code in provided Source Code Repo and embedded video should you struggle). Add the following to an file. app.js ( ).config() Express = ( ) App = Express() HTTP = ( ) BCrypt = ( ) App.get( , (req, res) => {
  res.send( )
})

App.get( , (req, res) => { hash = BCrypt.hash( , )
  res.send(hash)
}) port = process.env.PORT server = HTTP.createServer(App)

server.listen(port, () => { .log( , port)
}) 'use strict' require 'dotenv' const require 'express' const const require 'http' const require 'bcryptjs' // Router Setup '/pulse' '' '/stress' async const await 'this is a long password' 8 // Server Setup const const console 'NodeJS Performance Optimizations listening on: ' This is a very simple Express Server that exposes 2 Routes: /pulse /stress The endpoint represents a very lightweight API that contains no business logic and returns an empty string as a response. There should be no reason for any delays when processing this endpoint. /pulse The endpoint, on the other hand, uses to salt and hash a password. This is quite a heavy process and because it’s completely written in JavaScript, it will land up blocking the Event Loop quite badly. /stress BcryptJS For our first test, we’re going to use AutoCannon to run a load test against the endpoint, to see how many requests per second the API Server can handle when it’s running idle. The process is as follows: /pulse 1. Start up the Express Server node app 2. Run the AutoCannon test autocannon http://127.0.0.1:6000/pulse 3. This is a simple load test that runs 10 concurrent connections for 10 seconds After the test run, you should receive a report that includes an average requests per second. Depending on the speed of your machine, it should vary between 15 000 and 25 000 requests: Now that we have our baseline measurement, let’s see what happens when the API Server is performing heavy duty tasks: 1. Make sure the Express server is running node app 2. Open 2 Terminal windows for the tests to be performed 3. In window 1, run AutoCannon against the endpoint for a duration of seconds /stress 30 autocannon -d 30 http://127.0.0.1:6000/stress 4. In window 2, while the test is running, run AutoCannon against the endpoint, same as before /stress /pulse autocannon http://127.0.0.1:6000/pulse Make sure the test runs to completion whilst the test is running /pulse /stress After this test run, you should see a significant drop in the requests per second for the test. /pulse As you can imagine, this is a frightening result…one that needs to be handled sooner rather than later. What Does This Mean In The Real World While this example won’t make too much sense in the real world, it forms a template for the kind of tests you should be running on your environment. You need to identify when your API Server is running at peak, and then hit it with load testing against lightweight APIs that belong to your server. You need to determine if they can be processed without much delay, or if they get blocked because your code might not be managing the Event Loop well enough. How do I fix the problem? Well, I have good news: As mentioned at the start, I am busy with a series on “ “. Having already used examples from content I’ve published and content to come, I managed to increase the requests per second for the test from to over . Node JS Performance Optimizations /pulse 117 4 000 What you want to do is to my YouTube Channel, as I publish everything there first and almost on a weekly basis. There are already 4 videos for this series, an important one which is “ “. subscribe Bleeding Code Managing The Event Loop Phases I hope this article proved valuable. Stay tuned for more to come Cheers Previously published at http://bleedingcode.com/how-to-test-availability-of-your-api-server/

Apache

YouTube

Join me on my BleedingCode YouTube Channel

How To Monitor Your API Server

Having a humble beginning of starting as a language that was intended to handle browser validations to a full-blown programming language that powers a lot of desktop applications as well, JavaScript has traveled a long journey. It’s important to understand this journey of JavaScript to work with the trickiest parts of the language. Let’s start from where it all began. Why Was A Language Like JavaScript Needed? Many years ago, back in 1995, there was a browser called Netscape Navigator (sounds like a story already;). The internet connectivity was rather very slow as people used to use dial-up connections using telephone lines and modems. So when you were supposed to fill a form on a website, which was usually coded using a language like Perl, the validation of the input fields of any form required a full round trip to the server. This was not only a painful exercise as the internet speed was very slow but also resulted in the loss of data if the form was not filled correctly. Netscape Navigator sought to solve this problem by finding out a way to validate the input fields on the client side itself to save that complete round to the server just to validate the input fields. This is how JavaScript was born, merely as a scripting language to validate forms on the client-side. Why Is It Important To Know The History Of JavaScript JavaScript was born as a scripting language to validate input and it has traveled its journey of becoming a full-blown programming language that can be used to build anything from mobile apps, web applications to even desktop applications. Due to the way it has evolved, I would say that its a kind of a programming language, which might take a few hours to get started with but years to master. So when you start learning JavaScript knowing its history would help you better understand its nature, strengths, and limitations. It’s easier to understand why some of the inherited problems exist and why some things behave in a certain way once you know how JavaScript evolved as a language. The Birth Of JavaScript And The Birth Of A Conflict Brendan Eich (then a developer at Netscape Navigator) was assigned a task of developing this new scripting language which was then named Mocha. This was intended to be released as a part of Netscape Navigator 2. Mocha was later renamed to LiveScript before the release of Netscape Navigator 2. Netscape Communications then entered into an alliance with Sun Microsystems which was then really pushing hard for promoting Java. Just before the launch LiveScript was renamed as JavaScript to bank upon the traction that Java was getting from the press and portray JavaScript as a natural extension of Java. Since Java was just too complicated and big for handling server-side tasks LiveScript was seen as a natural extension to Java for handling the client-side tasks. This was supposed to be a companion to Java which would take care of the heavy server-side tasks and LiveScript would make client-side validations easy. LiveScript inherited a lot of its syntax and inherited nature from Java due to these obvious reasons. Sun Microsystems helped to complete and release LiveScript with Netscape Navigator 2 and just before the launch it was renamed as JavaScript to bank upon the traction that Java was getting from the press and portray JavaScript as a natural extension of Java. The Evolution Of The Conflict And Browser Incompatibility JavaScript 1.0 was a huge success and internet adoption was increasing tremendously. So much so that JavaScript 1.1 was released along with the Netscape Navigator 3. By now Netscape Navigator had established itself as the leading browser in the market. At this point in time, Microsoft decided to infuse more resources into the development of its own browser called Internet Explorer 3. Soon after the launch of Netscape Navigator 3, Microsft released a new version of its web browser called Internet Explorer 3 which had an implementation of JavaScript. To avoid any licensing issues with Netscape Communications, Microsft renamed its implementation of Javascript as JScript. This adoption and aggressive implementation of JScript by Microsoft was going to prove as a major leap forward for JavaScript. So by now, there were two different versions of JavaScript floating around with no standardization in place. This was the beginning of the browser incompatibility that we even see today. Unlike all other programming languages like C, C++, Java, etc, JavaScript had no basic standard whatsoever. Submission To ECMA For Standardization Since JavaScript had no standards and with the industry fears mounting over it, JavaScript 1.1 was submitted to European Computer Manufacturers Association (ECMA) for standardization in 1997. A technical committee was created with the objective of standardizing the syntax and semantics of a general-purpose, cross-platform, and a vendor-neutral scripting language. The committee consisted of developers from Sun Microsystems, Netscape Communication, Microsft, and various other companies. They came up with ECMA-262 which is a standard defining a new scripting language and a general-purpose scripting language called the ECMAScript to which all the browsers had to adhere to create their own implementations of JavaScript. Various browsers have tried with varying levels of success to adopt to ECMAScript as a standard for the implementation of JavaScript. There have been success stories as well as failures in the entire journey of adoption of this standard. So Is JavaScript and ECMAScript The Same Thing? The answer is No! Though you would often hear developers use them synonymously, they are very different from each other. ECMAScript is a scripting language based on the standards defined in ECMA-262 and is not tied to web browsers. It is based on ECMAScript that a more robust scripting language like JavaScript has been built. Implementations Of JavaScript JavaScript implementation of ECMAScript has three district parts which collectively are referred to as JavaScript. - This is based on ECMAScript. So the ECMAScript defines the syntax, data types, statements, operators, etc. The Core - This is what the browser renders as a hierarchy of nodes to be displayed to the client or the user. Document Object Model (DOM) - This is what allows the developers to manipulate the browser window. Using the BOM the developers can access the browser features and manipulate them outside the context of the DOM or the page that is being rendered. Browser Object Model (BOM) What’s Next? It is important to understand the three distinct implementations of JavaScript and their history to understand how it has impacted the language over a period of time. In this series of articles, we would next talk about ES6 which was the sixth edition of ECMA-262 released in the year 2015 and how it proved to be the most important enhancement to JavaScript ever made. You can watch this space for the next article or you can and I would notify you as soon as the next article is published. Don’t forget to leave a like or share the article if it was helpful! subscribe to my Newsletter Previously published at https://cloudaffle.com/history-of-javascript/

Winner of Noonies 2020 for the award Hacker Noon Contributor of the Year - DEVELOPER.

2020 - Hacker Noon Contributor of the Year in DEVELOPER

Download My JavaScript CheatSheet

How Javascript Was Created and Why The History Behind It Is Important

The model should conform to these assumptions to produce a best Linear Regression fit to the data. — All the images (plots) are generated and modified by Author. Introduction At first, Linear Regression is a method of modelling the best between the variables and variables. The simplest form of Linear Regression can be defined by the following equation with one independent and one dependent variable: linear relationship independent dependent is the independent variable, x is the dependent variable, y is the coefficient of x, i.e. slope, β 1 is the intercept (constant) which tells the distance of the line from the origin on y-axis. β 0 is a approach to modelling the relationship between a scalar response (or ) and one or more (or ). Linear regression linear dependent variable explanatory variables independent variables — Wikipedia Linear Regression Types — The simplest form of regression which involves one independent variable and one dependent variable, which is explained as above, where we fit a line to the model. Simple Linear Regression — The complex form of regression which involves multiple independent variables and one dependent variable, which can be explained by the following equation: Multiple Linear Regression to are the independent variable, x1 xn is the dependent variable, y to are the coefficients of respective x features, and β 1 β n is the intercept (constant) which tells the distance of the line from the origin on y-axis. β 0 Assumptions in Linear Regression — It is assumed and understood that the relation between the independent variables and dependent variables is linear, i.e. the coefficients must be linear, what we find out using the model building and prediction. 1. Linear Relationship The predictor variables are seen as fixed values and can be any complex function like polynomial, trigonometric, etc. But the coefficients will be strictly linear with the predictor variable. This assumption is used for implementing the , which uses linear regression to fit the response variable as an arbitrary polynomial function of a predictor variable which also makes the linear relationship with the coefficients. Polynomial regression — It is assumed that the residual terms (that is, the “noise” or random disturbance in the relationship between the features and the target) must have the constant variance, i.e. the error term is same across different values of independent features, regardless of the values of the predictor variables. 2. Homoscedasticity (Constant Variance) There should be no clear pattern in the distribution and if there is a specific pattern, the data is heteroscedastic. The leftmost graph shows no definite pattern among the error terms i.e the distribution is varied constantly, whereas the middle graph shows a pattern where the error decreases and then increases with the estimated values violating the constant variance rule and the rightmost graph also reveals a specific pattern where the error terms decrease with the predicted values representing heteroscedasticity. Two or more normal distributions are homoscedastic if they share a common covariance (or correlation) matrix. — It is assumed that the error terms are normally distributed, i.e. the mean of error terms is zero and the sum of error terms is also equal to zero. A less widely known fact is that, as the sample size goes high, the normality assumption for the residuals is not needed anymore. 3. Multivariate Normality The above q-q plot shows that the errors or residuals are normally distributed. The error term can be seen as the composite of some minor residuals or errors. As the number of these minor residuals increases, the distribution of the error term tends to approach the normal distribution. This tendency is called the Central Limit Theorem where the t-test and F- test are only applicable if the error term is normally distributed. — Multicollinearity is defined as the degree of inter-correlations among the independent variables used in the model. It is assumed that the independent feature variables are not at all or very less correlated among each other, which makes them independent. So in practical implementation, the correlation between two independent features must not be greater than 30% as it weakens the statistical power of the model built. For identification of highly correlated features, pair plots (scatter plot) and heatmaps (correlation matrix) can be used. 4. No Multicollinearity Highly correlated features should not be used in the model to maintain the strong relationship between the model and all its features present as the features tend to change in unison. Hence, with the change in one feature, the change in correlated feature does not make the latter constant as the model requires it while predicting the outcome using the weighted coefficients and the expected interpretation of regression coefficient does not conform. — It is assumed that there should be no auto-correlation among the features in the data. It mainly occurs when there is a dependency between residual errors, i.e. the residual error should not be correlated positively or negatively, and it should have a good spread all over. This usually occurs in time series models where the next instant is dependent on the previous instant. The presence of correlation in the residual terms also reduces the model’s predictability. 5. No Auto-correlation Autocorrelation can be tested with the help of the Durbin-Watson test. The Durbin-Watson test statistics is defined as: The Durbin-Watson test statistics will always have a value between 0 and 4. An exact value of 2.0 states that there is no autocorrelation detected in the sample. Values between 0 and 2 indicate positive autocorrelation and values between 2 and 4 indicates negative autocorrelation. — Extrapolation is an estimation that can exist beyond the original observation range. It is assumed that the trained model will be able to predict the values for the dependent variable on independent feature values only for the data that lies within the range of the training data. Therefore, the model cannot guarantee the predicted values that are beyond the range of trained independent feature values. 6. No Extrapolation Conclusion: We have explained the most important assumptions which must be focussed before implementing a Linear Regression Model to a given set of data. These assumptions are just a formal measure to ensure that the predictability of the built linear regression model is good enough to give us the best possible results for a given data set. These assumptions if not satisfied will not stop a Linear regression model to be built but will provide good confidence to the predictability of the model. Thanks for reading. You can find my other . Machine Learning related posts here I hope this post has been useful. I appreciate feedback and constructive criticism. If you want to talk about this article or other related topics, you can drop me a text here or at . LinkedIn Previously published at https://towardsdatascience.com/assumptions-in-linear-regression-528bb7b0495d

Scalar

More Writings!

Little-known Linear Regression Assumptions

The Terminal hasn't changed much since the 1980s. Every other aspect of your workflow is radically different. Especially in the last decade, we’ve seen companies transform industries with the advent of collaborative software: Figma (collaborative design), Github (collaborate code), G Suite (collaborative Word and Excel) and more. We expect real-time collaboration to dominate future markets as the pandemic runs its course and forces people and businesses online en masse. An entire frontier is now wide open to ambitious engineers. What, then, is next? We believe Terminal.app is ripe for innovation. What would this look like in multiplayer mode, built for teams? Surely there's an easier way to manage SSHing to AWS/GCP. How about chat, CMD+R across the team, and other plum integrations (Stack Overflow, Ruby docs, GPT-3)? At the very least, reimagining the engineer’s command center is an interesting experiment. And if successful, your work has the potential to fundamentally change the work of every developer on the planet. We’re excited to announce our second Pioneer Challenge: Build a New Terminal. Get started: https://pioneer.app/challenge How It Works This challenge has three distinct parts. Here's an overview: A 48-hour hackathon anyone can join, starting August 7th. Phase 1: The Hackathon. 5 winners are selected from the hackathon and are incorporated for free into companies, if they'd like. No equity is exchanged. (All participants receive free credits.) Repl.it The 5 companies spend a month building out their prototype. Weekly progress videos are shared on Frontier. Phase 2: The Prototype Month. After a month, the winning team is selected both by popular vote and our team. The winner is awarded Pioneer Gold: $20,000 in exchange for 5% of the company. Phase 3: The Championship. To get started, register with your name and past projects . here Scouting If you refer the winning team to us, we'll grant you 0.5% equity in the company. You'll need them to register with a unique link you can generate for yourself . here There might be extreme circumstances where we grant you the equity (depending on the country you're in, for example). We'll attempt to exhaust all reasonable legal options to make good on this. can't Apply now: https://pioneer.app/challenge FAQ How do I know what to build? Spend time talking to other developers, asking them about their problems, and build a product around it. Get creative. Here's a video to get your wheels spinning: If I win the Hackathon, must I compete in the Challenge? Yes! Where will the Hackathon take place? All submissions will be hosted through . You are not expected to post on Frontier until Friday, August 7th. Frontier Can we start hacking before the Hackathon begins? No. All participants will start hacking on Friday, 12:00pm PT. Finalists will be required to share their code with the organizers to confirm they did not start coding before the event began. What are the judging criteria for the Hackathon? Judging criteria & prizes for the Hackathon and Challenge winners are outlined . here Who’s judging the Hackathon & Challenge? The Pioneer team will select 5 Hackathon winners. The top 3 projects in the Challenge will be chosen by popular vote. Pioneer will choose the winner from those remaining 3. Can I incorporate and not participate in the Challenge? In order to incorporate your company, you must commit to joining the 4-week Challenge. Do I need co-founders? No! You can get started alone. Our introductory audio-only call might be a fun way to meet others in a low-key fashion, if you'd like. Our Slack Group is another chance to connect. How do you keep track of competitors' progress during the Challenge? On Tuesdays and Fridays, the five competing teams will provide progress updates on Frontier. These updates are in addition to an internal WhatsApp group dedicated to more casual, frequent conversation. I'm already in the Pioneer Tournament, how does this interact with winning the normal tournament? Dual victory conditions is a bit of a non-goal. We recommend that you either (a) continue in the normal tournament, working on anything you'd like or (b) completely shift to trying this challenge. Doing both simultaneously would be hard. What if I have more questions? Email us at team@pioneer.app. Previously published at https://pioneer.app/blog/pioneer-challenge-a-new-terminal/

Slack

Pioneer Challenge: Overview

Photo of a brand new car without a plate number. After reading this article you will see the reason why your project is not recognized or noticed by visitors on the platforms. Imagine having a brand new car without a plate/registration number! What do you think will happen to the car? If the car is involved in negative situations for examples; If the car is stolen by hoodlums. If the car has an accident. Let us judge the two scenarios; in the first case, would the car be recovered from the hoodlums by security operatives? The answer is NO! In the second case, would it be easy to get the identity of the car owner for any form of support? The answer is still NO! This is because the car is not registered with the government so the details of the car and the owner are unknown. As you can see not having a registration plate number on a car is very disastrous the disadvantages outweighed the advantages. The bottom line in both cases is the lack of recognition, no security agent would be able to identify the car and the owner's details. The above-narrated situation is an analogy of a programming project without or with a poorly written , that project would not be recognized or noticed by anyone on whatever platform in which you placed your project, for example; GitHub. READme file WHAT IS A READme FILE? A READme file is a text file (markdown) that introduces and explains a project, it is like a landing page for most open source projects. It contains information needed to understand what a particular project is all about. A good READme file must be able to communicate to visitors on how to install, how to use, and how other developers can collaborate with you. It is important to note that the READme file tells others to use or contribute to your project so every programming project must have it. READme file is also essential to retrieve a project on Git Hub as well as in search engines, for example; google. TIME TO CREATE A READme FILE. It is best to create a READme file along with creating your on GitHub for it to be the first file to be created on your project. repo STEP BY STEP WAYS OF CREATING A READme FILE. Navigate to your GitHub platform. Click plus icon on the top right-hand side and select the 'New Repository'. In the repository name section, type your repository name. Click the checkbox with ' Initialize this repository with a README ' Click ' Create repository ' button. Then your repository is created with a READme file. Screenshot of how to create a READme file on GitHub If you forgot to create your READme file with your Repo creation, you can create it on your code editor terminal; Change directory (cd) to your project folder on the terminal. Type ' touch ' keyword with READme on the terminal. READme file is created. Screenshot of how to create READme file on the terminal Whichever way you chose in creating your READme file it must be properly written and completed before showing your project to other people or make it public. INGREDIENTS OF A GOOD READme FILE. A READme file title should concisely explain what you have in the project and must be related to the content. Title: This is about two or three sentences explaining what the project is all about and who it is meant for. Introduction/Summary: This is necessary to allow visitor(s) to have the first feel of how your project looks like before opening your repo files. Screenshot: This can be any prerequisite knowledge, tools, or technologies anyone who wants to use the project might need before they begin. For example; Ruby language, you can instruct them to install ruby. Prerequisites: This should be step by step process on how to install it if the installation is necessary. Guide on Installation: Once the installation has been completed this step should also explain how to use the project with examples and references explaining the command options and flags. How to use the project: A good READme file must contain live-demo and video(optional) links for visitors to check and watch. Live demo: This should explain the step(s) to be taken or followed in order to contribute to the project. Contribution: This should include those that contributed to the success of the project, it is a way to make an open-source feels like a team effort and acknowledge everyone who has contributed to it. Do not forget to include their contact information if given permission to disclose that. If it is a solo project put only your contact information there. The author section: This is where you acknowledge anyone, organization, or group of people that you have used their work for example; designs, images, or code to achieve success in your own project. Acknowledgment: This is where you include the license information if applicable in order to obey the copyright law. License information: If you also care about making your READme file looks great you can add logos, badges, shields, emojis, and so on. The screenshot of a typical READme file Inconclusion, for a project to be visible, easily recognized on flatform(s), and also for the purpose of search engine optimization such a project must have a well-written READme file. Henceforth, do not forget to include the READme file in your programming project.

Why My Project Lacks Recognition On The Platforms

Did you know that recursion can be optimized using a concept which works similar to the way how we jump on a  trampoline. Let me elaborate, in recursion a function calls itself without finishing its own execution. As execution cannot be finished until the next function call returns (or finished execution) and this goes on for further function calls. This creates a stack of unfinished function calls and due to memory constraints there can only be a certain limit of function calls allowed in a stack. But situations with a large number of recursion calls in javascript does not lead to stack overflow, javascript prevents from reaching that limit by throwing a range error. Consider this example, { { (test > ) test; increment(++test);
  } { .log( , test);
  }
} ( ) function increment test try if 1000000 return return catch console "test===>" If you run the above code, it will go into the catch block pretty soon. On a macbook air 2k19, it maxed out around 10K. That means there was a stack overflow. Well in the above example the function calls itself in the return statement, this adds on to the stack one more function call to and this call will again add on to the stack one more function call to and this keeps going on until it reaches the base condition, when becomes greater than , but that never happens. increment increment increment initialValue 1000000 But here if you notice that recursion call is a tail call. A tail call is when a recursion call only happens at the end of the function in a return statement. Why do we need to concern ourselves with the tail call and how does it matter? The trampoline technique that we are about to learn only works when we have recursion defined as a tail call. This will become more clear once we look at the trampoline, let's hop on to it. 'Tramploine of recursion' is given by Kyle Simpson. A trampoline is nothing but a wrapper around our recursive function. In recursion, we call a function from within a function. In a trampolined recursion, we call a function that in turn returns a function ( not a function call & therefore completing the current function execution) and this returned function again calls our recursive function. That is a bit confusing, I know but read it again I am sure you will get it. Here's how the trampoline wrapper looks like, { { result = f(...args); ( result === ) {
      result = result();
    } result;
  };
} ( ) function trampoline f return ( ) function enclose ...args let while typeof "function" return The above trampoline function, takes a function as an argument and returns a function called , you can call it anything you want. enclose In order to use the above wrapper, we need to modify our recursive function a little bit. { { (test > ) test; increment(++test);
  } { .log( , test);
  }
} ( ) function increment test try if 1000000000 return return => () catch console "test===>" Did you notice the difference, we modified our tail call by returning an anonymous function instead. So that its not a function call anymore, but it returns a function that calls our recursive function which is in this case increment Now, if we pass in the above function to our trampoline wrapper, our code will look like this. trampolinedIncrement = trampoline( { { (test > ) test; increment(++test);
  } { .log( , test);
  }
}); .log(trampolinedIncrement( )); const ( ) function increment test try if 1000000000 return return => () catch console "test===>" console 1 When you run the above code, there will be no range error, and you will get as output. The above code took around on a Macbook Air. 1000000001 20 sec But what just happened? Our code just got on a trampoline, LOL! Here is the whole code, { { result = f(...args); ( result === ) {
      result = result();
    } result;
  };
} trampolinedIncrement = trampoline( { { (test > ) test; increment(++test);
  } { .log( , test);
  }
}); .log(trampolinedIncrement( )); ( ) function trampoline f return ( ) function enclose ...args let while typeof "function" return const ( ) function increment test try if 1000000000 return return => () catch console "test===>" console 1 When we trampolined the function, we got the definition of function in with being the function. So now, when we call the function with initial arguments, it calls and the output of that execution is an anonymous function which is stored in the . Then we keep on calling function until it returns a function and at the end will return the result. increment enclose trampolinedIncrement f increment trampolinedIncrement f result result typeof If you are still confused just ask in the comments, I will clarify. You can reach out to me on twitter on @heypran Just to wrap it up, in a normal recursion the stack size keeps on increasing because the current function call has not finished its execution and yet it calls another function. This function again calls another function and this keeps on repeating. This leads to stack overflow. In a trampolined scenario, we are not calling the recursive function directly, instead we are finishing the current function call by returning a function, that in turn calls our recursive function. This makes our stack size always increase only by one and when it finishes it again reduces by one. This jumping between the functions is referred as . I find it pretty cool! trampoline Cheers! Have a Happy Day! @heypran

Twitter

2022 - HackerNoon Contributor of the Year - Cryptocurrency

2022 - HackerNoon Contributor of the Year - Dapps

2022 - HackerNoon Contributor of the Year - Typescript

Follow me on Twitter

Nominated for 2022 - HackerNoon Contributor of the Year - Typescript

Nominated for 2022 - HackerNoon Contributor of the Year - Dapps

Nominated for 2022 - HackerNoon Contributor of the Year - Cryptocurrency

On Recursion and Trampolining

One of the largest misconceptions about succeeding as a software engineer is that people think if they simply work hard, they will become successful and will receive the recognition they deserve. They couldn’t be further from the truth. Technical competence and productivity are incredibly important to succeeding as a software engineer, but most engineers drastically underestimate the importance of how your coworkers perceive you, let alone know how to systematically improve your image over time. Plenty of resources exist out there for the former, but no one talks about the latter which arguably contributes more to a raise or promotion. Many engineers, including myself when I was starting out, want to exclude themselves from this “game” of having to constantly self-advocate. It feels disingenuous and desperate, but there’s nothing disingenuous and desperate about self-advocating if you approach it with structure, objectivity, and consistency. Self-advocating is genuine and effective if you approach it with structure, objectivity, and consistency. The following is the advice that no one really tells you and the advice I wish I had when I was starting my career. I hope it helps you avoid having to learn them from mistakes and missed opportunities. Manage Expectations Well A mentor of mine once told me that the secret to being a successful CEO is to accurately set and meet expectations on a consistent basis. The same principle applies to software engineers: the most successful ones do this really well when it comes to scoping work. The most successful software engineers are able to manage expectations the best when it comes to scoping work. Scope work and multiply the time estimate by 2 or 3 depending on how many unknowns there are. It takes some time to feel comfortable doing this, and it can feel like you’re deceiving your stakeholders, but you are actually doing them favor. If they push you on being more aggressive, stand your ground. Product Managers use your estimates to communicate a delivery date to even more important stakeholders. They depend on you to make themselves look good. Committing to an aggressive deadline and missing it not only makes look bad, but it also makes your Product Manager resent you for making look bad in the face of executives. you them Additionally, stakeholders can have a completely different perception of what is considered an aggressive deadline. I once estimated a project to take a week to do, told my stakeholders it would take two weeks, shipped it in 8 days, and was complimented for my ability to commit to ship features only. Basically, underpromise and overdeliver. Most of the time stakeholders won’t argue with your estimates, if they need something sooner, open up a conversation around what be delivered in the time frame they want. . Product Managers would rather deliver a feature ahead of schedule than late. It will make them look good, and it will make you look good. can DO NOT under any circumstance take away your estimated padding just to appease them Don’t Work Too Hard It may be incredibly counter-intuitive, but the most successful engineers are the ones that only do a moderate amount of work, not the most, not the least. There’s a really good reason for this. When you finish work ahead of schedule, use that time to chill out. This gives you the ability to be able to handle emergencies, help people with last-minute requests, or to simply destress which ultimately makes you more likable and approachable to work with. Coworkers take note of when you go out of your way to help them. You can’t do that when you are pressed for time trying to ship your own work. Don’t work too hard. Not working hard gives you the ability to handle emergencies, help people with last minute requests, or destress to maintain a good mood. Engineers that constantly work around the clock are chronically stressed out, typically hard to communicate with, and do not leave any room for flexibility. They are perceived as not being able to manage their time wisely or efficiently. Those that work the least are perceived as slackers or incompetent: not able to complete tasks in a reasonable amount of time. The effects compound. With every person you help and every last minute request you fulfill, people will take note that you are getting your work done ahead of schedule and somehow find the time to help others in need. You may not be more (in the raw sense of the word) than your peers, but others will you as more productive, more capable, more helpful, and most importantly more likable. productive perceive Note: many would argue that you are actually overall more productive for your team if you are helping teammates and being more collaborative. Managers definitely take note of this. Track Your Work and Accomplishments After previously writing an article about how to get into tech and become a senior engineer in 4 years, a lot of people reached out to me asking about the importance of tracking work and how useful it really is. Short answer: . You cannot improve what you cannot track. it is essential You cannot improve what you cannot track. The first aspect of tracking work involves knowing how efficient you are with your time. How much time do you waste context switching? In meetings? Emailing? Slacking (both in the semantic use of the term and the messaging app)? Did you scope the last project with too little time? Too much time? You should be able to answer all of these questions in a general sense and record them daily. The second aspect of tracking work involves building your track record of accomplishments over time. It’s difficult enough to remember what you did yesterday or the week before, imagine what you’ll miss when you need to show your manager all the work you completed for the month, quarter, or year for a performance review? Leaving out important accomplishments would mean you aren’t maximizing your chances for a raise or promotion. Not many tools exist to help with tracking your time and keep a record of your accomplishments. I ended up doing this primarily through a Google Doc, but things got quite messy and unmanageable as time went on. Spending a few minutes a day actually felt like a lot of effort, but it was totally worth it. I am actually focused on building a tool specifically for this use case, and my company . We are automating a lot of the work it takes to visualize what you’ve done, chart your progress, and build your track record of accomplishments. Please check us out and back us if you think this could help you. recently launched our waitlist Advocate for Yourself Weekly For way too long, I relied on others to advocate for me, and it rarely happened. If it did, it was never at the extent I felt was deserved. Do not rely on others to build your reputation. You have to do it yourself, and it’s totally fine to do so. Do not rely on others to build your reputation. You have to do it yourself, and it’s totally fine to do so. Advocating for yourself is tough. If you are more extroverted, bragging about your work can come off as brash. For those who are more introverted, it takes a lot of effort to speak up and brag about yourself at all. Luckily, there’s a way to inform your manager of all the important things you accomplish in a way that is direct, neutral, and consistent. Even if you have weekly meetings with your manager, you should send her a weekly email consisting of what was expected of you that week, what you shipped, any difficulties you had along the way, and potential insights you had that might be helpful. Use a tone that is neutral and to the point. Be concise and objective in the items you report. Language that fluffs or exaggerates the extent of your work can come off as disingenuous. If your email is too long, your manager won’t read it. The weekly update email to your manager provides consistency and visibility. This is especially important in a post-COVID world where a large majority of software engineers are working remotely, and managers no longer gain visibility into their teams by simply being present in the same physical location. Their sense of knowing that you are consistently productive and transparent every week will go miles when it’s time for a promotion or a raise. Tying It All Together It takes almost just as much effort to set yourself up for success in the work environment as it does to get your actual work done, but it’s necessary, and the most successful engineers at top companies do this incredibly well. Don’t exclude yourself from this aspect of your career. Don’t treat it as additional stuff you need to do outside of your normal work hours. Spend 80% of your work hours working, and the other 20% advocating. No manager or peer has ever given me this advice and frankly, aren’t incentivized to do so. Most people when entering the workforce are tossed into a job without any sort of guidance on how to do the more unspoken, but totally legitimate, tactics of becoming successful at work. I had to figure this stuff out myself, but I hope that this article gives you some insight and actionable advice into this very ambiguous part of building your career. Questions? Comments? Hit me up on Twitter @wrgoto or check out our project called Rize .

HackerNoon Tech Stories Archive September 5th, 2020

15 Must-read Machine Learning Articles for Data Scientists

@limarc
Limarc Ambalina

The Sectors Most at Risk from Remote Work and How to Secure Them

@isaac-kohen-teramind
Isaac Kohen

Glossary of Security Terms: HTTPS

@mozilla
Mozilla Contributors

Glossary of Security Terms: Key

@mozilla
Mozilla Contributors

XinFin Hybrid Blockchain vs EOS.IO: The Battle of XDPoS vs DPoS

@XinFin_Official
XinFin_Hybrid_Blockchain

How Artificial intelligence is Transforming the Gaming Industry

@rafiqulislam
Rafiqul Islam

The Impact of AI Transformers on the Customer Experience

@federico
Federico Cesconi

Building Real-Time Vehicle Detection System

@sabina
Sabina

Introduction To Distributed Tracing Pattern

@elan-srinivasan
Elankumaran Srinivasan

How To Monitor Your API Server

@johnjardin
John Jardin

How Javascript Was Created and Why The History Behind It Is Important

@manik
Manik

Little-known Linear Regression Assumptions

@imsparsh
Sparsh Gupta

Pioneer Challenge: Overview

@jackson-pioneer
Jackson

Why My Project Lacks Recognition On The Platforms

@adesoji-adewumi
Adesoji Adewumi

On Recursion and Trampolining

@heypran
Pran B.

Revealing The Secrets of Top Software Engineers

@wrgoto
Will Goto

HackerNoon Tech Stories Archive September 5th, 2020

@limarcLimarc Ambalina

@isaac-kohen-teramindIsaac Kohen

@mozillaMozilla Contributors

@mozillaMozilla Contributors

@XinFin_OfficialXinFin_Hybrid_Blockchain

@rafiqulislamRafiqul Islam

@federicoFederico Cesconi

@sabinaSabina

@elan-srinivasanElankumaran Srinivasan

@johnjardinJohn Jardin

@manikManik

@imsparshSparsh Gupta

@jackson-pioneerJackson

@adesoji-adewumiAdesoji Adewumi

@heypranPran B.

@wrgotoWill Goto

@limarc
Limarc Ambalina

@isaac-kohen-teramind
Isaac Kohen

@mozilla
Mozilla Contributors

@mozilla
Mozilla Contributors

@XinFin_Official
XinFin_Hybrid_Blockchain

@rafiqulislam
Rafiqul Islam

@federico
Federico Cesconi

@sabina
Sabina

@elan-srinivasan
Elankumaran Srinivasan

@johnjardin
John Jardin

@manik
Manik

@imsparsh
Sparsh Gupta

@jackson-pioneer
Jackson

@adesoji-adewumi
Adesoji Adewumi

@heypran
Pran B.

@wrgoto
Will Goto