kWh Analytics is the market leader in solar risk management

kWh Analytics

kwhanalytics

Image credit: on Shubham Dhage Unsplash It's no secret that large language models (LLMs) like ChatGPT have transformed how we work today. The trading landscape is no different. Crypto With the introduction of these powerful and new technologies, Crypto traders have thought of various ways they can use these technologies to revolutionize the way traders approach market analysis, generate trading signals, automate trading processes, and even democratize decision-making. In this article, I will mention some ways LLMs can be used to change the way we trade and potentially reshape the trading landscape: Sentiment analysis based trading Strategy LLM models, which are deep learning algorithms, can perform various tasks. One of these tasks is natural language processing (NLP) tasks. This means they can process large volumes of textual data from news articles, social media, and other sources to provide a comprehensive market analysis that can be used for sentiment analysis-based trading. This can save time for traders as they no longer need to sift through large volumes of news sources. So, how does this work? A sentiment analysis-based trading strategy allows traders to gauge the market sentiment of some cryptocurrencies and identify market-moving events. Traders can use the result from this sentiment for market trends analysis, allowing traders to swiftly react to news and decide when to buy and sell. Zanyang talked a bit more about this in his , where he used algorithmic trading platforms like QuantConnect. Use of NLP-Powered sentiment analysis in trading strategy paper Customer support and communication Besides Traders, LLMs can benefit everyone in the crypto space, including cryptocurrency founders, project owners, community leaders, and trading platforms. For community and trading platforms, LLMs can be used for knowledge-based databases, Q&A bots, and customer support. This way, they can seamlessly help users navigate the platform, improve customer experience, answer project-related questions, resolve issues, and assist in technical support. The best thing about this system is that, unlike traditional conversational AI, LLM-powered bots can understand human language and respond better 24/7. Some AI Chatbots you can explore for this are and . AwesomeQA UltimateGPT Generate trading signals A trading signal is a trigger telling a trader when to buy or sell orders. Triggers are generated by simply analyzing indicators, patterns, or data points. These common technical indicators include MACD, RSI, KD, and Bollinger Bands. This talks a lot more about these indicators. article by BYDFi With LLMs, you can generate trading signals in various ways: Building trading algorithms that depend on pre-established rules. Identify trends through sentiment. Interprets news events that can cause market movements. Recognize patterns and interpret news events. With these signals, traders can use the insights to capitalize on anticipated price reactions and make more informed decision-making. The best part is that this signal generation can be automated, thus reducing any latency in trade execution. This exciting discussion on the . Numerai forum discusses using LLMs to create trading signals Build, Build, and Build with LLMs You’ve probably used LLMs in code generation or to assist you in generating codes; If not, I have got an [Analytics Vidhya’s article on How to Build LLMs for Code?](https://www.analyticsvidhya.com/blog/2023/09/llms-for-code/#:~:text=A%20Large%20Language%20Model%20(LLM,understand%20and%20generate%20computer%20code.) for you. Based on your prompt, LLM can be used to write code faster and much more efficiently. You can also use it to generate a whole project. Bringing this idea to the crypto space, you can use LLM models like , , and to generate codes, write smart contracts, and test. These models can also read, analyze and modify your codes or even improve your codebase. WizardCoder StarCoder Codeium All this can be done using simple prompts and knowing what you want. Democratizing decision-making and knowledge One of my favorite things about LLM models is that they make information accessible. You can get information about just about anything by writing a simple prompt. Imagine a model trained on real-world financial data that can understand financial concepts, formulas, and market data. This means you can ask this model about anything in this domain, and it will provide you with information and help you make better decisions. Such a custom LLM would benefit not just market leaders but novice traders and beginners in the crypto space. One of my favorite Crypto knowledge bases is , an AI-powered search engine. MinMax Wrap-up You can use LLMs in various ways, from generating codes to providing 24/7 customer support. While many of these models are still under development, they are becoming increasingly powerful and more accurate as the day passes. As this happens, you will likely use them for even more tasks in the crypto industry in the future.

How LLMs like ChatGPT Can Change the Way We Trade

Source [Link](https://www.analyticsinsight.net/when-humans-fall-in-love-with-robots-technology-gets-complicated/) Dating in 2023 feels strange. After a 4-year relationship, I've been single for a year. Things have changed since 2018. and raised the bar for commitment. Nowadays, a pretty girl has the whole world potentially peeking into her life, liking her photos, and sliding into her DMs. Instagram Tinder Now, the bar for settling down has soared to new heights. The pressure to stand out and be more captivating feels like it's turned 11. I've given Bumble and OKCupid a shot, but it's disheartening. My matches are few; looks overshadow words there. It's tough out there. But I am not alone. OKCupid did a .). They looked at how men and women rate each other's physical attractiveness. study For men, many women were average, with balanced ratings for more and less attractive. Like a typical distribution, nothing odd. Weird, isn’t it? There are two ways to see it: only less attractive guys are using the app, or women assess men's looks more critically. It's likely the second one. And if that's true, it means women have high standards for looks. Well, 20% of women are seen as really good-looking by guys, but only 2% of guys are seen that way by women. So, most women feel like they're settling for less attractive guys. The competition gets fierce. The top guys have it easy while the rest women fight over them. Dating for women is like shopping. Dating for men is like trying to find life in outer space. -Wheat Waffles And then there's the rest of the guys like me, like 80% of them. Women don't see them the same way, so they're left out. So what option do we have now? Facing these difficulties in dating app dynamics, men like me are increasingly drawn to AI companion apps. However, it's important to understand that AI models cannot truly experience emotions as humans do. Yet, can AI convince us they feel emotions? Certainly, perception matters. AI models can learn to understand and mimic emotions in two ways. Firstly, supervised learning involves training with labeled emotional text and code. For instance, text from books, movies, and social media is tagged with emotions like happiness or sadness. The AI learns patterns and predicts emotions in the new text. Then there's reinforcement learning. The AI gets rewarded for correctly predicting emotions. Through trial and error, it figures out patterns and improves its understanding. It can even learn from audio, video, and physiological data like heart rate. While AI models can simulate emotions through learning patterns, the concept extends further in creating custom AI companions. Now there are tutorials on Github that can help you create your AI girlfriend. You can customize their personalities and backstories, making them potential romantic partners. With customizable personalities, men are likelier to find an AI companion that aligns with their preferences and desires. These AI girlfriends can provide companionship, engage in conversations, and offer emotional support. In the near future, we could see humanoid robots equipped with AI intelligence and resembling human skin. These robots will look like us and provide a lifelike and close experience. As this idea progresses, the thought of marrying such an AI moves beyond mere speculation. If AI can bring happiness and help with loneliness, maybe giving it a shot isn't such a bad idea after all, right?

Beyond Swipes and Likes: The Intriguing Rise of AI-Driven Companionship

No matter your age, background or job, you almost certainly use data daily. You might access a database to get the information necessary to run a report for your boss. Perhaps data from a mapping app helped you find your way to a new restaurant last weekend. Data also defines many interactions with professionals, from doctors to bank employees. However, a related principle called data governance underpins all those engagements. It encompasses: Who can access the data? What are the valid access reasons? How long will an organization hold the data? What does a company do with the data? Will an organization share the data with others? What are a company’s obligations for securing the data? Regardless of how your company uses data and for what reasons, it’s essential to put data governance into practice. How can you do that? 1. Identify Your Data Usage Norms Begin by clarifying your company’s relationship with data. For example, what kind of data does the organization collect and from whom? Does the content come from internal sources, external ones, or both? Is the current data volume manageable or starting to get out of control? How much do you see that changing over the next few years? The are two other things to consider. Is your data cleansed and in a format you could immediately use, or does it require preparation first? quality and sensitivity of the data Do you hold data related to someone’s finances, health, or other personal matter? If so, the information likely requires particular processing measures to remain compliant. The sooner you know how your business uses data, the easier it’ll be to establish data governance rules. However, assessing whether growth, a new product, or another change could soon alter how your company works with data is essential. If so, account for that now rather than when it might be too late. 2. Establish Access Control Measures Another vital part of data governance is ensuring people can seamlessly access the information and data-based programs necessary for their work. However, far too few companies take that approach. A 2021 Varonis study of manufacturers showed every employee to access at least 1,000 sensitive files. four in 10 businesses allowed Another astounding takeaway was that employees could access an average of six million files from their first days at work. Consider options like role-based access, taking into account that some people will rightfully access files from multiple places, especially during company-mandated travel. A related data governance strategy is to when using business data. Doing that keeps the information safer, especially when the content is intellectual property or confidential. specify an employee’s obligations 3. Create a Data Governance Team Data governance is a collective effort, so everyone is responsible for making it work. However, you’re most likely to succeed by appointing specific people to lead your business’s aims. A manager could oversee individuals tasked with particular data governance milestones. That supervisor must also clearly and its associated responsibilities. communicate a person’s data governance role Fostering an open-communication strategy is an excellent way to keep everyone motivated and on the same page. Team members should also realize data governance goals may change over time. That’s especially likely if the company begins using data differently or collecting another type. Data governance leaders should encourage people to be open to change while making it clear they’re available to hear feedback about team members’ concerns. 4. Connect Your Data Governance Practices to Company Needs Some business leaders make the mistake of thinking they should use data in specific ways, primarily because they’ve seen peers doing it. However, a better approach is to tie all data governance strategies to overarching company goals. A surprising find from a 2023 Gartner study was only believe their teams effectively provide value to their organizations. 44% of data and analytics leaders However, another statistic indicated 44% of the best-performing leaders in that group devoted time to personal development. Those who oversee data governance must determine the most practical and responsible ways to use data throughout an organization. Personal development might factor into that, such as if a data governance leader about new technologies. needs to learn more Data Governance Matters Creating and upholding a data governance plan that works for your business could take months or years — and that’s okay. The vital thing to do now is to set a firm intention of getting to that point. This is a time when consumers freely provide their data to companies for various reasons. However, they also expect the organizations to handle that information responsibly. These data governance suggestions can help you do that. Then, your business will gradually earn and maintain a reputation for handling information responsibly, which could promote growth and customer respect. Sources: https://dataconomy.com/2023/03/how-to-create-a-data-governance-strategy/?utm_content=cmp-true https://www.varonis.com/blog/manufacturing-data-risk-report-reveals-files-open-to-all-employees https://www.humanresourcesonline.net/safeguarding-a-company-s-proprietary-information-amidst-employee-exits https://www.codescan.io/blog/salesforce-data-governance-vs.-data-management/ https://www.gartner.com/en/newsroom/press-releases/03-21-2023-gartner-survey-reveals-less-than-half-of-data-and-analytics-teams-effectively-provide-value-to-the-organization https://www.analyticsinsight.net/top-10-data-governance-tools-to-know-in-the-year-2023/

4 Ways Businesses Can Practice Effective Data Governance

In this article, we will explore how to create, build and deploy every component behind this Bikes recommendation website: 🔗 demo 🎯 Goal High-level The idea behind this project is to test the opportunity to build a recommendation system using , (ML) models, and only resources. public data unsupervised machine learning free To achieve this we will: Leverage to gather the data Reddit as ML framework Spacy transformers to run the ML model, to host the back-end, to host the front-end Google Colab Heroku GitHub pages Implementation is a mountain bike ( ) recommendation system. In other words, you could choose a bike brand or model that you like and then the system will suggest 3 bikes considered interesting and related to your chosen bike. Garrascobike MTB when people talk about some on the same subreddit thread those bikes . So we could extract the names and/or brands from one thread’s comments and intersect that information with other Reddit threads with similar bikes. The idea behind the recommendation system is: Bikes should be related in some way Bike’s 📢 Foreword The goal of this guide is to chase all the aspects involved in the creation of a Web App that serves a simple recommendation system, . trying to keep the complexity level lower as possible So the technical level of this experiment won’t be too deep and we don’t follow industrial-level best practices, nevertheless, this is the guide I would like to have one year ago before starting a simple project: create a WebApp with an ML model at is core. 🗺️ Roadmap What we will do could be summarized in the following steps: Download text comments from Reddit 🐍 Extract interesting entities from the comments 🐍🤖 Create a simple recommendation system model 🐍🤖 Deploy the model on a back-end 🐍 Create a front-end that expose the model predictions 🌐 🐍 = blocks that use Python 🤖 = blocks with Machine Learning topics involved 🌐 = blocks that use HTML, CSS & Javascript 🚵‍♀️ Garrascobike The project is structured in five major sections: scraping, entities extraction, ML train, back-end, front-end (and they coincide with the five chapters of this post). On the above image, the first four sections (front-end is excluded) are reported with their dependencies and where necessary the platform where the code should be executed. Let’s dive deep into each component. 1. Download text comments from Reddit 🐍 Intro First of all: we need the data and Reddit is an amazing social network where people talk about topic. Moreover, expose some API that python packages like could use to scrape the data. any Reddit praw Prerequisites We need to choose some that talk about the topic on which we would create the recommendation system subreddits In we want to build a Mountain Bike recommendation system, so we need subreddits that talk about those bikes. Garrascobike The subreddits selected for this PoC project are: Whichbike, MTB, Mountainbiking, Bicycling How to 👨‍💻Code to use hosted on Github: subreddit-comments-dl Follow this guide to download Reddit comments: https://towardsdatascience.com/how-download-subreddit-comments-f79557c99170 More information is also on my blog: https://www.pistocop.dev/posts/subreddit_downloader/ Outcome Two CSV files with all the information about the subreddit comments. We will create two CSV files, if you are not familiar with Reddit could be useful to read this little glossary comments.csv Contain all the comments with related information, e.g. the comment’s text, the creation date, the and thread where the comment was written subreddit submissions.csv Contain all the information about the : a submission is a post that appears in each subreddit submissions 2. Extract interesting entities from the comments 🐍🤖 Intro Entities are words or sections of a text that belong to some pre-defined “family” e.g. from a text like “ ” we would like to extract something like "2019” is a and "Enduro” is a I have a 2019 Enduro DATE PRODUCT The entities extraction is related to an ML task called Named Entity Recognition (NER), for more examples see the official Spacy webpage So this block get the Reddit comments and will extract the Bikes names contained within them ( ) extraction.parquet That information will be uploaded to an instance Elasticsearch Prerequisites The comments extracted from the desired subreddit How many comments? More comments are directly proportional to better performances For this project, we have downloaded comments from 800.000 103.159 submissions How to 👨‍💻Code to use hosted on Github: garrascobike-core Extract the entities Steps Clone the repository https://github.com/pistocop/garrascobike-core Copy the scraped data under /data/01_subreddit_extractions/ Run the extraction with the script /garrascobike/01_entities_extraction.py Under the hood the script will use system with a ML model Spacy NER spacy-transformers 💡Tip: use this to run the above tasks Colab Notebook Because the process will use a machine-learning model it’s suggested to execute the code on a machine with a GPU. If you aren’t rich like me there is a big chance that you haven’t a powerful GPU for ML tasks, so use the above notebook and leverage the Google GPU’s for free 💘 Load on Elasticsearch What and why is a documental NoSQL database that could store data even if we don’t know the data structure Elasticsearch Why we should use Elasticsearch (ES)? It wasn’t mandatory to use this technology if we remain in the zone, and a python script could be enough to do the same job required for this project. POC However, use ES pledge us with the opportunity to be ready if we go out of the POC and/or we want to reuse the data in different ways for other projects. ES was used during the first data scouting phase, when wasn’t clear which data the ML model has extracted and if it’s possible to shape that information to our requirements Using ES on this scouting phase was very helpful to data discovery and express queries like “ ”, “ ” or “ ” how many threads exist with at least 1 product entity get all the entities typer list all the PRODUCT entities Run the Elasticsearch service locally Install Docker Following to run a mini-cluster locally, the cluster is suggested this guide 01_single-node Load the data Run the script script providing the parquet file and the ES endpoints /garrascobike/02_es_uploader.py # folder garrascobike-core
$ python garrascobike/02_es_uploader.py --es_index my_index \
                                        --es_host http://localhost \
										--es_port 9200 \
                                        --input_file ./data/02_entities_extractions/extraction.parquet 3. Create a simple recommendation system model 🐍🤖 Intro The parsed comments are now associated with text snippets named from the entities extraction process product After a look at those snippets, we could see that the entities here are the Bikes names and brands that users on subreddits talk about product 🦂 Of course there is also noise labelled as e.g. “canyon/giant” is a snippet that contains two brands and should not be included and/or split into two entities “canyon” and “giant”. Moreover, “12speed” isn’t a brand but the number of gears of the bike. product, For this toy project, we don’t care about the noise, so the predictions may result in suggesting bikes that don’t exist. In a real-world scenario, we should apply some revision to those products, e.g. search each brand on an images search engine and see if a bike picture is returned. Nevertheless, some brands cleaning and filtering is actually performed - code link The recommendation system will be built around the intersection of bikes brands and the comments threads to which the comments belong We will create a semifinished product: (new code version store a file named this file contain the entities found for each subreddit correlations.npz presences.csv), product thread In the last step, will train ad store an AI model based on the KNN algorithm able to make the bikes recommendations process 04_recommendation_trainer.py More about this process could be found in those good articles: https://towardsdatascience.com/how-did-we-build-book-recommender-systems-in-an-hour-part-2-k-nearest-neighbors-and-matrix-c04b3c2ef55c https://www.analyticsvidhya.com/blog/2020/11/create-your-own-movie-movie-recommendation-system/ Finally, we will upload by hand (no automatic script provided) the files create by 04_recommendation_trainer.py Use is suggested and supported by the next chapter section, it provides 10 GB of storage for free backblaze.com Prerequisites An ES instance running with the entities extracted products An account on backblaze.com How to 👨‍💻Code to use hosted on Github: https://github.com/pistocop/garrascobike-core Run the script, parameters: 03_correlation_extraction.py the Elasticsearch instance address es_host: the Elasticsearch instance port es_port: the Elasticsearch indexes names. They could be more than one because it’s possible to join more entities extraction. es_index_list: However, for this guide, we could use only one parameter: my_index. $ python garrascobike/03_correlation_extraction.py --es_host localhost \
                                                   --es_port 9200 \
                                                   --es_index_list my_index Run the script, parameters: 04_recommendation_trainer.py : the path to the file created by presence_data_path 03_correlation_extraction.py the model to use to build the recommendation system, only is currently available ml_model: knn : where to save the recommendation system model files, those files should be then uploaded on output_path backblaze $ python garrascobike/04_recommendation_trainer.py --presence_data_path ./data/03_correlation_data/presence_dataset/20210331124802/presences.csv \
                                                   --output_path ./data/04_recommendation_models/knn/ Outcome The recommendation system model trained to predict bikes suggestion from one bike name provided All the recommendation model files should be uploaded on Backblaze 4. Deploy the model on a back-end 🐍 Intro To make the recommendation we need a machine that runs the prediction model We will wrap the model inside a back-end that: Is in Python and use framework FastAPI Run on free Dyno as the hardware provider heroku Run for 30m and then go to sleep mode If it’s in the sleep node, need ~1m to wake up and be ready to use The back-end will expose those API: that take a bike name and return 3 bikes suggested /recommender/{bike_name} that return the list of supported bike names /brands/available that returns a timestamp and will be used to check if the back-end is up and in running state /health Prerequisites We need the list of bikes that the recommendation system model can manage: brands.csv The recommendation model that the back-end will use to make the predictions How to 👨‍💻Code to use hosted on Github: https://github.com/pistocop/garrascobike-be Set the correct credentials and path according to your backblaze account: To set the Backblaze bucket name and path modify the file at: /garrascobike_be/ml_model/hosted-model-info.json To set the Backblaze connection credentials modify and rename to the file: .env /garrascobike_be/.env_example Create an account on and publish the back-end: follow this guide for more detailed information: heroku 🔗 https://towardsdatascience.com/how-to-deploy-your-fastapi-app-on-heroku-for-free-8d4271a4ab9 Outcome A back-end up and running on Heroku, you could get the API URL from the “open app” button inside the Heroku webpage The back-end openAPI specification could be found under and will look like this: https://<heroku-app-url>/docs Note: you may need to wait ~1m before seeing this OpenAPI specification 5. Create a front-end that expose the model predictions 🌐 Intro We need a website that the user could use to choose the desired bike and receive the suggestions The website must use the API exposed by the back-end However, for this reason, the FE need to show some kind of “wait disclaimer” while Heroku wake up the App The FE should also provide succinctly the list of bikes that the user could choose from We will build an autocomplete text field using the amazing framework autoComplete.js We will create a static website with HTML, CSS & JS The website is then hosted for free on GitHub Pages Prerequisites The Garrascobike back-end up and running online The URL of the back-end program to run the website locally - live-server npm How to 👨‍💻Code to use hosted on Github: https://github.com/pistocop/garrascobike-fe Steps to set up a new front-end Fork the project Replace with your back-end URL this line Commit the changes Follow to enable the GitHub pages service on your repository (note: the repo must be public!) this guide Follow the link generated by the GitHub interface Enjoy your new front-end! Outcome Demo of the website: https://pistocop.github.io/garrascobike-fe/ The website you will see should resemble like this: 🎌 Other languages All the project is built and work for English Subreddits Other languages could be supported, the only section that requires a change is the spacy model in the entities extraction task https://github.com/pistocop/garrascobike-core Spacy transformers models of different languages could be found here: language support 💭 Final Thoughts We have just seen how to build a recommendation system from the scraping process to a WebApp. All the data are scraped from Reddit and processed with unsupervised ML models (Spacy), using the Google Colab platform. This has saved us a lot of work and supplied a straightforward path that could be automatized also. Finally, a back-end free hosted on Heruko and a front-end free hosted on GitHub pages complete the project thanks to a web app page. From here, a lot of work could be done to refine each of the components, e.g. if we want to move from a POC project like this to production and more serious system we should: Filter and remove all the entities extracted that aren’t bikes Scrape and process more data Build and test different parameters or families for the recommendation system As final, I want to say that it was nice to build this project, I haven’t experience with the front-end world and the simple website is made after following free course. theodinproject.com 🍺 And now, after completing every component and the is up and running, I must admit that I feel satisfied, hope you’ve enjoyed the journey! website **📧 Found an error or have a question? let’s [connect](https://www.pistocop.dev/)** This article was first published here: https://www.pistocop.dev/posts/garrascobike/

How a recommendation system Web App was Built

Cryptocurrencies are blooming into a fresh asset class and creating a new era for the global finance industry. These digital assets are appealing to more and more people who would like to enter the market and gain from the potential of decentralized currency. The first generation of crypto services focused on things like exchanges. Market analysis tools were scarce, as most people were highly focused on Bitcoin, and little else. Thousands of exchanges were created so investors can buy, sell, and trade cryptocurrencies. Some of these currencies allow people to make secure and untraceable transactions, and most have gone up in value substantially. People know Bitcoin, Litecoin, Ethereum, and XRP, but there are thousands more that present opportunities for any kind of investor. Some of the smaller tokens can create outsized gains, which is appealing to investors. However, new entrants to the crypto markets need to learn about crypto, so they can make smart decisions about where to deploy their capital. Analytics are key, and there are a few good ways to use big data to any crypto investor's advantage. Simply put – the advent of AI in the crypto markets will change the way investments are made. Just like the big data systems that are used in the equity markets, any investor who has access to AI-driven crypto analytics will have an advantage. Cryptocurrency Investment and Data Analytics With thousands of blockchains out there, market data in the crypto space isn't as simple to analyze as it would be on a centralized exchange. But just like any other financial market, data analytics are extremely important and allow investors to make informed decisions. Analytics technology is most valuable where investors can use data to make real-time decisions at any time frame. Some investors look to buy a token for days or weeks, while other need to take on longer-term positions. Using analytics tools is a good idea for investors, and as the quants on Wall St. have demonstrated, these systems can help investors avoid huge losses. Many data analytics platforms were designed to offer investors solutions for understanding how the crypto market works, and one of the most interesting new ones is Defy Trends. The Miami-based platform is developing its first product, which takes advantage of Artificial Intelligence (AI) to create data analytics that works for any investor. Unlike many of the trading bot platforms, this one allows investors to access far more information, and dive deeper into the markets. The Rise of Data Analytics Blockchain-based data analytics overcomes the disparate nature of decentralized platforms and allows for direct comparisons between thousands of blockchain platforms. In addition, some recent advancements allow investors to understand the energy requirements of a blockchain, as many use tremendous amounts of power. As blockchain and crypto adoption has been growing among industries, blockchain data analytics has also become a necessary part of the market. Developers, investors, institutional investors, and governments are looking for ways to better understand the blockchain space. Data can guide investment strategies, manage risk, conduct cybersecurity investigations, and a lot more. The demand for blockchain and crypto data analytics is predicted even greater in the future when venture capitalists are pouring money into companies providing these services, and larger institutional investors need hard data on these systems. A Popular Industry There is no doubt that early-stage investors see the potential of data analytics in the blockchain space. In June of this year, the San Francisco-based startup TRM Labs closed a US$14 million Series A funding round. Later, Chainalysis raised a US$100 million Series E funding round and nailed a US$4.2 billion valuation. Singapore-based blockchain data analytics platform Nansen secured a $12 million Series A fundraising round led by Andreessen Horowitz. In The Startup Showdown, Panoramic Venture Capital’s high-energy competition, which took place early this month, saw judges choose two winners, with one of them operating in the blockchain analytics space. As such, each startup received a $120K investment from Panoramic instead of only one being awarded. DeFy Trends was one of the companies that won funding in the competition. Why AI? AI-based algorithms can analyze historical data to predict future trends and help investors make constructive decisions. These new projects can offer a range of tools such as an Index, which scores coins and informs traders which are under- and over-valued as a way to see the indicators that are most important to their reference. Data analytics tools like the suite that Defy Trends created can help to educate people about crypto markets via bringing huge amounts of data together – and making it understandable to a single person. Analytics technology can contribute to creating a safer cryptocurrency environment through its data. The combination of data analysis and constant monitoring will make it easier to identify opportunities and great crypto traders. Investors and traders can use analytics platforms to look at the blockchain network that operates behind the team that created the cryptocurrency to make smarter choices. Simply put, AI-based data analytics allow investors to understand markets in a way that would not be possible without the technology. The advantages they provide are impossible to match with any other technology, and will likely grow from here. Sources https://www.analyticsinsight.net/how-cryptocurrency-benefits-from-big-data-analytics/ https://refreshmiami.com/7-things-to-know-in-miamitech-the-lowdown-on-startup-showdown-michael-saymans-book-talk-thermabands-backer-much-more/ https://www.europeanbusinessreview.com/what-are-the-4-important-benefits-of-cryptocurrency-to-big-data-analytics-in-2021/ https://www.smartdatacollective.com/using-analytics-to-decide-which-cryptocurrency-to-invest-in/ https://www.jigsawacademy.com/what-is-blockchain-analytics/ https://forkast.news/blockchain-data-analytics-nansen/ Disclaimer: The opinions in this article belong to the author alone and should not be considered investment advice.

Investors Clamor for Digestible Data Analytics in the Fledgling Crypto Industry

FloC brings the model to the data instead of sending the data to the centralized model Overview Federated Learning is a relatively new and evolving machine learning technique that decentralizes the training of data from one central machine/ data center to multiple devices, including mobile phones. Federated Learning of Cohorts, or FloC for short, is a form of web tracking enabled through Federated Learning in which individuals are grouped into “cohorts” based on similar browsing behavior. Machine learning is a branch of Artificial Intelligence and computer science that leverages data and algorithms to make computers mimic human learning and decision making. Federated Learning takes advantage of edge computing principles, bringing computation and data storage closer to where it is needed. This principle allows for reduced response times, bandwidth conservation, and personalization, amongst other benefits. This article seeks to provide an easy-to-digest overview of Federated Learning of Cohorts for business professionals and others interested in the new technology. What is Federated Learning As with any machine learning task, data is necessary to train and retrain algorithmic models to improve on them, identify patterns, make decisions, and bring enhanced features to a system. But, as we know, data is everywhere and sometimes siloed in different locations, making it hard to access, especially within a business organization that may produce a multitude of data across various sources. Regarding data, other considerations are privacy issues, regulatory requirements, and lack of resources to bring segmented data together from a single source. The typical process of a machine learning project seeks to establish data pipelines to a central location to then be cleaned, pre-processed, and trained. Federated Learning aims to improve this process by bringing training models closer to where data is located. In other words, Federated learning brings training models to the source of data-generating entities. As Li et al. put it, Federated Learning: Explores training statistical models directly on remote devices. For example, mobile phones and other devices download the training model locally, contribute to the machine learning algorithm with local data, then generate a summary of changes made to the machine learning model as a minor “update.” This “update” is sent to the cloud, where other “updates” are amassed from various sources, ultimately resulting in an improved shared and distributed machine learning algorithm. It is important to note a secure and encrypted aspect of data sharing between the Federated Learning enabled device and the Cloud service performing the aggregation and updating the new consensus of the machine learning algorithm. The encrypted and private “updates” mentioned above only occur when it doesn’t negatively impact the user experience on target devices. Federated Learning is essential for numerous reasons; namely, it opens practitioners up to a new paradigm shift in how machine learning can be implemented. These new techniques pave the way for decentralized AI and other available machine learning models, enabling new tools to tackle scaled machine learning problem sets. More specifically, FloC enhances the machine learning process through “faster deployment and testing of smarter models, lower latency, and less power consumption, all while ensuring privacy (Bhattacharya, 2019)”. Federated Learning also enables the target device to immediately use and take advantage of the evolving machine learning algorithm that the device itself is helping to improve. For example: The improved model on your phone can also be used immediately, powering experiences personalized by the way you use your phone (McMahan & Ramage, 2017). Federated Learning of Cohorts Powered by Federated Learning Algorithms It is important to remember that Federated Learning of Cohorts is powered through Federated Learning (Machine Learning) models and aims to improve web advertising and phase out third-party cookies (metadata used to track users across different websites). FloC is a new protocol developed by Google as a part of its “Privacy Sandbox” initiative, a long-term feat aiming to eliminate 3rd party cookies, potentially uprooting a long-standing digital advertising ecosystem. More specifically, Google’s “privacy sandbox” seeks to transform digital advertising through: A secure environment for personalization that also protects user privacy (Schuh, 2019) Online privacy is the core topic of the criticism surrounding the design of FloC protocols and Federated Learning techniques. Currently, 3rd-party cookies offer little protection against privacy because developers and ad agencies can extrapolate critical identifiers using “fingerprinting” techniques. Schuh notes, With fingerprinting, developers have found ways to use tiny bits of information that vary between users, such as what device they have or what fonts they have installed to generate a unique identifier which can then be used to match a user across websites (Schuh, 2019). Although this model paves the way for exciting engineering and user experience use cases, it does come with its criticisms. Who should care about Federated Learning of Cohorts “The ability to share behavioral labels across the web in a manner that enables ‘learning’ of unique user behavior in an automated and algorithmically governed fashion has the potential to create a future where every interaction online can be leveraged to manipulate you.” – RichardMM Privacy Harming and Criticism of Federated Learning of Cohorts The Brave organization (Brave Browser) and the Electronic Frontier Foundation (EFF) are prominent advocates against Google’s FloC and “Privacy Sandbox” initiatives. Brave states that “FLoC shares information about your browsing behavior with sites and advertisers that otherwise wouldn’t have access to that information (Snyder & Eich, 2021).” Although your browsing information is anonymized through a grouped cohort of users with similar interests and browsing habits, FloC protocols do not prevent sites within your intra-browsing environment from accessing unique data about you. More simply put, your browsing data is shared with website B after visiting website A, which allows website C to learn about your behavior and unique interests that would not have been otherwise shared. Here is a brief excerpt from the Brave foundation: “For example, you may have an existing account with Walgreens, possibly to fill prescriptions. Walgreens necessarily knows who you are. FLoC tells Walgreens things that Walgreens has no business knowing about you (not a pseudonymous you or a cohort including you, but your Walgreens login identifies you) based on your browsing behavior. Chrome telling Walgreens (and Twitter, and GitHub, and Facebook, and any other site you have an account with) is unquestionably harming your privacy by telling sites information about you that they otherwise wouldn’t have, the information you didn’t decide to share with those sites, and information that is likely unrelated to why you chose to visit those sites where you do login (Snyder & Eich, 2021).” The Electronic Frontier Foundation weighs in on this topic by highlighting how FloC protocols will “avoid the privacy risks of third-party cookies, but it will create new ones in the process (Cyphers, 2021).” The ability to share behavioral labels across the web in a manner that enables “learning” of unique user behavior in an automated and algorithmically governed fashion has the potential to create a future where every interaction online can be leveraged to manipulate you. Federated Learning of Cohorts is a tool leveraged in a new digital frontier where emerging technologies can better user experience and offer great value. Although Federated Learning can bring about new capacities to our devices by enabling a new paradigm of machine learning and AI techniques, it also paves the way for designing tools that can be used maliciously. In this instance, the design of FloC protocols may not initially be architected with malignant intent, but do present futures where these technologies can produce unintended consequences. The EFF presents the following point of view on the potential of the future use of FloC protocols: In one, users get to decide what information to share with each site they choose to interact with. No one needs to worry that their past browsing will be held against them—or leveraged to manipulate them—when they next open a tab. In another, each user’s behavior follows them from site to site as a label, inscrutable at a glance but rich with meaning to those in the know. Their recent history, distilled into a few bits, is “democratized” and shared with dozens of nameless actors that take part in the service of each web page. Users begin every interaction with a confession: here’s what I’ve been up to this week, please treat me accordingly - (Cyphers, 2021). In either instance, it is imperative for technologists, designers, and large corporations to have diverse teams that enable many considerations as emerging technologies are being built. Federated Learning is an exciting technology and can provide good in this world, but there may be unintended consequences of its use (as highlighted through Federated Learning of Cohort techniques) as with any other technology. Bibliography Bhattacharya, D. S. (2019, February 2). The New Dawn of AI: Federated Learning. Medium. . https://towardsdatascience.com/the-new-dawn-of-ai-federated-learning-8ccd9ed7fc3a Cyphers, B. (2021, April 9). Google’s FLoC Is a Terrible Idea. Electronic Frontier Foundation. . https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea Marboer. (2021, April 6). Federated Learning: Introduction to Federated Learning. Analytics Vidhya. . https://www.analyticsvidhya.com/blog/2021/04/federated-learning-for-beginners/ McMahan, B., & Ramage, D. (2017, April 6). Federated Learning: Collaborative Machine Learning without Centralized Training Data. Google AI Blog. . https://ai.googleblog.com/2017/04/federated-learning-collaborative.html Schuh, J. (2019, August 22). Building a more private web. Google. . https://www.blog.google/products/chrome/building-a-more-private-web/ Snyder, P., & Eich, B. (2021, April 20). Why Brave Disables FLoC. Brave Browser. . https://brave.com/why-brave-disables-floc/ Useful Links https://hackernoon.com/a-beginners-guide-to-federated-learning-b29e29ba65cf https://ai.googleblog.com/2017/04/federated-learning-collaborative.html https://hackernoon.com/federated-learning-a-decentralized-form-of-machine-learning-nr4635rg https://federated.withgoogle.com/ https://towardsdatascience.com/the-new-dawn-of-ai-federated-learning-8ccd9ed7fc3a https://www.ibm.com/blogs/research/2020/08/ibm-federated-learning-machine-learning-where-the-data-is/ https://towardsdatascience.com/the-new-dawn-of-ai-federated-learning-8ccd9ed7fc3a https://www.techradar.com/news/theres-more-to-googles-floc-than-meets-the-eye https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea https://techcrunch.com/2021/03/24/google-isnt-testing-flocs-in-europe-yet/ https://techcrunch.com/2019/08/22/google-proposes-new-privacy-and-anti-fingerprinting-controls-for-the-web/ Also published on . https://richardmm.com/federated-learning-of-cohorts-floc-a-quick-brief/

A Quick Introduction to Federated Learning of Cohorts [FloC]

In 2012, Harvard Business Review called data scientists the sexiest job of the 21st century. However, correctly answering data science interview questions to get a job as a data scientist is very tricky. During the interview, the interviewer can ask questions from different data science topics such as statistics, programming, data analysis, data pre-processing and modeling. Your skills will be put to test, and you need to prepare yourself if you want to pursue a career in data science. In this article, I have compiled a list of common data science interview questions with guides on how you can answer them and a list of resources to learn more about the specific topic presented in each interview question. 1.What is Logistic Regression? State an Example When You Have Used Logistic Regression Recently Logistic regression is a popular algorithm used to solve classification problems. In this question, you need to explain what logistic regression is, how it works, and an example of a data science problem you solved by using logistic regression. Here are resources to help you get started crafting your response: https://hackernoon.com/introduction-to-machine-learning-algorithms-logistic-regression-cbdd82d81a36 https://www.youtube.com/watch?v=yIYKR4sgzI8 2.Why do we Need Evaluation Metrics? What is a  Confusion Matrix? Machine learning models must be evaluated to check their performances. In this question, you need to explain how you can use the confusion matrix to evaluate the model performance. You can further mention other metrics to evaluate regression and classification models. Here are resources to help you get started crafting your response: https://machinelearningmastery.com/confusion-matrix-machine-learning/ https://www.youtube.com/watch?v=prWyZhcktn4 3.How is Data Science Different from Traditional Application Programming? A good way to answer this question is by using examples of how the program is created in both cases. Traditional Programming Approach Data Science Approach Here are resources to help you get started crafting your response: https://mathsgee.com/qna/24080/data-science-different-traditional-application-programming 4.Explain the Difference Between Supervised and Unsupervised Learning. Supervised and unsupervised learning are among the types of machine learning techniques. The best way to answer this question is by explaining their differences in terms of the kind of datasets you can use in each technique and examples of algorithms. Here are resources to help you get started crafting your response: https://www.guru99.com/supervised-vs-unsupervised-learning.html https://www.youtube.com/watch?v=cfj6yaYE86U 5.What is a Decision Tree? A decision tree is another supervised learning algorithm that can be used to solve regression or classification problems. It is recommended to explain how the decision tree algorithm learns from the data and the advantages and disadvantages of using a decision tree algorithm. Here are resources to help you get started crafting your response: https://hackernoon.com/what-is-a-decision-tree-in-machine-learning-15ce51dc445d https://www.youtube.com/watch?v=RmajweUFKvM 6.What is Cross-Validation? The purpose of the question is to determine if you know some techniques used to assess the effectiveness of the machine learning model. For example, when you want to avoid overfitting. When answering this question, it is recommended to explain some methods of cross-validation you have applied in any data science projects. Here are resources to help you get started crafting your response: https://www.analyticsvidhya.com/blog/2018/05/improve-model-performance-cross-validation-in-python-r/ https://www.youtube.com/watch?v=fSytzGwwBVw 7.What is a Normal Distribution? This term is commonly used when solving a data science problem. In this question, you can explain the meaning of normal distribution, its properties, and why it is important to check if your data is normally distributed. Here are resources to help you get started crafting your response: https://statisticsbyjim.com/basics/normal-distribution/ https://www.youtube.com/watch?v=rzFX5NWojp0 8.What is a Random Forest Algorithm? Random forest is one of the popular machine learning algorithms. When answering this question, it is recommended to explain how the algorithm learns from the data and when it is recommended to use the random forest algorithm over other machine learning algorithms. Here are resources to help you get started crafting your response: https://www.analyticsvidhya.com/blog/2020/12/lets-open-the-black-box-of-random-forests/ https://www.youtube.com/watch?v=eM4uJ6XGnSM 9.Explain Univariate, Bivariate, and Multivariate Analyses These three types of analyses are used to summarize variables in the dataset and help you get some insights. You can also share their differences and when you can apply them with examples. Here are resources to help you get started crafting your response: https://www.geeksforgeeks.org/univariate-bivariate-and-multivariate-data-and-its-analysis/ https://www.youtube.com/watch?v=JG8GRlMjp3c 10.How can we Handle Missing Data? Some datasets may have missing data or values and can cause a problem when training machine learning models. It is important to mention some techniques that can be used to handle missing data. You can also share your experience of how you handle missing data in your last data science project. Here are resources to help you get started crafting your response: https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/ https://www.youtube.com/watch?v=P_iMSYQnqac 11.what is the Benefit of Dimensionality Reduction? Dimensionality reduction is the technique to reduce the number of features or variables in the dataset. There are different advantages or benefits of dimensionality reduction you can explain when answering this question. It is recommended to explain why and when you need to apply this technique. Here are resources to help you get started crafting your response: https://hackernoon.com/dimensionality-reduction-using-pca-a-comprehensive-hands-on-primer-ph8436lj https://www.youtube.com/watch?v=1eYJKD0TQ8U 12.How can we deal with Outliers? An outlier is a data point that deviates significantly from the rest. In this question, you can explain how one can identify outliers and different techniques used to deal with outliers. Here are resources to help you get started crafting your response: https://www.kdnuggets.com/2017/01/3-methods-deal-outliers.html https://www.youtube.com/watch?v=DDpym2j_ILY 13.What is Ensemble Learning? In machine learning, ensemble learning is a process of using multiple algorithms to obtain better predictive performance than could be obtained from any algorithms alone. When answering this question, you can also share your experience the last time you implemented ensemble methods in a data science project. Here are resources to help you get started crafting your response: https://www.analyticsvidhya.com/blog/2015/08/introduction-ensemble-learning/ https://www.youtube.com/watch?v=WtWxOhhZWX0 14.Explain how Machine Learning is Different from Deep Learning? The best way to explain the difference between machine learning and deep learning is the way they solve problems. You can go further by explaining some of the problems that can be solved by either machine learning or deep learning techniques. Here are resources to help you get started crafting your response: https://www.simplilearn.com/machine-learning-vs-deep-learning-major-differences-you-need-to-know-article https://www.simplilearn.com/tutorials/artificial-intelligence-tutorial/ai-vs-machine-learning-vs-deep-learning 15.What are the Differences Between Overfitting and Underfitting? The best way to explain the difference is not just with definition but through examples. You can also share your personal experience when faced with overfitting or underfitting problems in a data science project. Here are resources to help you get started crafting your response: https://www.geeksforgeeks.org/underfitting-and-overfitting-in-machine-learning/ https://www.simplilearn.com/tutorials/machine-learning-tutorial/overfitting-and-underfitting 16.What is Regularisation? Why is it Useful? When answering this question, you can also go further by explaining the two common regularization techniques L1 norm and L2 norm. Here are resources to help you get started crafting your response: https://dzone.com/articles/what-when-amp-why-of-regularization-in-machine-learning https://www.youtube.com/watch?v=rPBFvvw2OM4 17.What is Selection Bias? It is not enough to define Selection Bias. If possible you can explain different types of bias, their effects, and how to avoid them. Here are resources to help you get started crafting your response: https://imotions.com/blog/selection-bias/ https://academy4sc.org/video/selection-bias-dont-forget-about-me/ 18.Can you Explain the Difference Between a Validation Set and a Test Set? In this question, after explaining their differences, you can explain the advantage of having a validation set and a test set in a data science project. Here are resources to help you get started crafting your response: https://machinelearningmastery.com/difference-test-validation-datasets/ https://www.youtube.com/watch?v=pGlQLMPI46g 19.What is the Difference Between Regression and Classification ML Techniques? We all know that regression and classification are supervised learning and the only difference is their output. When you answer this question, you can mention a few algorithms that can be used to solve regression problems or classification problems. Also, try to share how their models are evaluated. Here are resources to help you get started crafting your response: https://www.javatpoint.com/regression-vs-classification-in-machine-learning https://www.youtube.com/watch?v=TJveOYsK6MY 20.What are Artificial Neural Networks? In this question don't just define Artificial Neural Networks but also explain their advantages and where you can use them. Here are resources to help you get started crafting your response: https://hackernoon.com/overview-of-artificial-neural-networks-and-its-applications-2525c1addff7 https://searchenterpriseai.techtarget.com/definition/neural-network 21.What Tools and Devices do you Plan to use in Your Role as a Data Scientist? This question is straightforward but it is recommended to mention tools you have used before or you are planning to use in the future project. You can also share your experience of how the tools help you implement the data science project successfully. Keep in mind, you will use different tools for different projects. For example, some tools can be used for an NLP project and others for a Time-series project. Here are resources to help you get started crafting your response: https://sdacademy.dev/13-tools-every-data-scientist-needs-to-know/ https://searchbusinessanalytics.techtarget.com/feature/15-data-science-tools-to-consider-using 22.What is Natural Language Processing? State some Real-Life Examples of NLP. You have to define Natural language processing in a simple way and how it can be used to solve business problems. Then share some real-life examples. If possible you can also share some of the NLP projects you have done or collaborate with others. Here are resources to help you get started crafting your response: https://www.ibm.com/cloud/learn/natural-language-processing https://intellipaat.com/blog/applications-of-nlp/ 23.What is Normalisation? Difference between Normalisation and Standardization? Normalization and standardization are techniques used to pre-process the data before applying machine learning algorithms. The purpose of the question is to explain the differences between these two techniques and at what condition of the dataset, you should apply one over another. Here are resources to help you get started crafting your response: https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/ https://www.youtube.com/watch?v=mnKm3YP56PY Final Thoughts on Data Science Interview Questions Reviewing these common data science interview questions will actually boost your confidence during the interview. Don't expect the interviewer to ask you all questions mentioned in this article. However, most of the interview questions will come from the same topics. For example, instead of asking " ". The interviewer can ask you to “E ”. Explain the difference between supervised and unsupervised learning xplain some supervised learning algorithms and how they learn from the data If you are interested in learning and reading more data science interview questions, I recommend you take your time to read the following resources I have compiled for you. https://www.springboard.com/blog/data-science/data-science-interview-questions/ https://huyenchip.com/ml-interviews-book/ https://hackernoon.com/160-data-science-interview-questions-415s3y2a It is recommended to practice your coding skills because some questions during the interview require you to code the solution. I hope these data science interview questions will help you in preparing for your interview and I wish you the best of luck in your data science career. If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in python and data science? in the footer below Subscribe to our newsletter

23 Common Data Science Interview Questions for Beginners

Data Science is no doubt the "sexiest" career path of the 21st century, made up of people with strong intellectual curiosity and technical expertise to dig out valuable insights from humongous volumes of data. This helps firms add value by improving their productivity, unlocking insights for better decision making and profit gains, just to mention a few. The knowledge of Data Science is desirable and useful across various industries. The journey of a Data Scientist is full of twists and turns that will mold you. However, it is not these twists and turns that molds, rather how you handle the ones thrown at you. Many of these challenges can be prevented or minimized by having a foreknowledge of the right tool kits before kickstarting the journey or maneuvering your way in the journey of being a successful data scientist. This article provides you with this key information needed so you can spend your time efficiently and navigate a data science career path smartly. Hence, a guide to help you find your way through the Data Science maze. Top ✔️ 10 Websites for Data Science Analytics Vidhya Kaggle Coursera Udacity Datacamp EdX Udemy KDNuggets R-bloggers Khan Academy Top ✔️ 10 Skills for Data Science Probability & Statistics Linear Algebra Python R SQL Tableau/Power BI AWS/Azure Spark Excel DevOps Top ✔️ 10 Algorithms for Data Science Linear Regression Logistics Regression K-means Clustering PCA Support Vector Machine Decision Tree Random Forest Gradient Boosting Machines Naïve Bayes Classifier Artificial Neural Networks Top ✔️ 10 Data Science Roles Data Scientist Decision Maker Analyst ETL Engineer Machine Learning Engineer Data Engineer Analytics Manager Tableau Developer Researcher BI Analyst Top ✔️ 10 Data Science Experts to follow on LinkedIn Bernard Marr DJ Patil Francesca Lazzeri, PhD Carla Gentry Dennis R. Mortensen Andrew Ng Gregory Piatetsky-Shapiro Tom Davenport Randy Lao ️ NABIH IBRAHIM BAWAZIR Top ✔️ 10 Python Libraries for Data Science Pandas Numpy Scikit-Learn Keras PyTorch LightGBM Matplotlib SciPy Theano TensorFlow Top ✔️ 10 Industries for Data Science Technology Finance Retail Telecom Healthcare & Pharma Manufacturing Automotive Cybersecurity Energy Utilities Top ✔️ 10 Data Science-related Hashtags to follow on LinkedIn and to also use for engaging posts #innovation #technology #bigdata #businessintelligence #analytics #datamining #data #artificialintelligence #machinelearning #datascience Top ✔️ 10 Data Science groups to join on LinkedIn Big Data and Analytics Advanced Analytics and Data Science Big Data, Analytics, Business Intelligence & Visualization Experts Community Data Science, Big Data, Machine Learning, Artificial Intelligence Professionals | DataScience.US Data Mining, Statistics, Big Data, Data Visualization, and Data Science Research Methods and Data Science Big Data, Analytics, IoT (Internet of Things) & Blockchain Big Data|Artificial Intelligence|Machine Learning|Predictive Analytics|Data Mining|Data Science IBM Big Data and Analytics Advanced Analytics, Predictive Modeling & Statistical Analyses Professionals Group Top ✔️ 10 Free Dataset sources for Data Science project Kaggle UCI Machine Learning Repository Google Custom Dataset Search gov Reddit Quandl VisualData GitHub world Google Cloud Public Datasets Good luck on your journey to becoming a top-notch Data Science expert that you desire to become. Nothing is impossible, believe it! References https://towardsdatascience.com/top-10-great-sites-with-free-data-sets-581ac8f6334 https://www.dataquest.io/blog/free-datasets-for-projects/ https://www.analyticsinsight.net/top-10-big-data-analytics-linkedin-groups/ https://www.linkedinsights.com/the-hot-100-linkedin-hashtags/ https://www.geeksforgeeks.org/11-industries-that-benefits-the-most-from-data-science/ https://datafloq.com/read/top-industries-for-data-science-professionals/6698 https://www.linkedin.com/pulse/top-12-data-science-experts-best-career-articles-linkedin-tanmoy-ray/ https://www.kdnuggets.com/2016/05/10-big-data-data-science-leaders-linkedin.html https://towardsdatascience.com/the-top-10-ml-algorithms-for-data-science-in-5-minutes-4ffbed9c8672 https://medium.com/@exastax/top-20-data-science-blogs-and-websites-for-data-scientists-d88b7d99740 Also published on: https://www.kdnuggets.com/2020/08/top-10-lists-data-science.html This article was written by Mojeed Abisiga. is a Data Scientist & Machine Learning Engineer with vast experience in successfully applying Machine Learning-based solutions to real world problems and leveraging his proficiency in tools and techniques for finding patterns and digging out insights from large volume of data to help firms drive growth, make valuable decisions, and gain competitive advantage on their data journey. Bio: Mojeed Abisiga He is currently a Data Scientist & RPA Specialist in the Data & Analytics unit of KPMG Nigeria where he has built several enterprise-level Intelligent Automations, Business Intelligence, and Machine Learning Models that cuts across different domains and industries like Telecoms, Banking, Human Resources, and FMCG.

ML Essentials: Top 10 Lists Every Data Scientist Should Know

Sentiment Analysis using Deep Learning techniques with India Elections 2019 — A Case study

Idea / inspiration I’m creating a project that can spot trees, using satellite data. I got this idea from the company . Which uses satellite imagery and AI to track forests and vegetation around the world. They are many uses for this technology. You can track deforestation in an area. By tracking the amount of missing trees over a certain timescale. A company called uses satellite imagery and AI. To prevent fires and power outages by spotting trees growing too close to power lines. 20tree.ai Spacept Using satellite imagery saves time and money. As inspecting forestry using a helicopter or drone can cost a lot of money. And will cost a lot of time, to set up the helicopter and drone to fly over the forestry. Dataset When I was first trying to get the dataset together for the machine learning model.  I settled on using . I got this idea from this . Which used satellite imagery to spot landslides. In the blog post he highlighted the issue of getting satellite data. Where if you wanted to get images from the . You will have to do them manually. And you cannot add coordinates of an area you want to capture. google earth engine medium blog post ESA Copernicus SciHub website So, you get continent sized images. Which I can agree from my personal use of the service as well. When I tried to get an image that covered a small area in my city. Instead I got large images covering all of France and Britain. With Google earth engine you can filter by date, area, and cloud cover. Which is why Google earth engine was recommended.  I used the code from the blog to create a dataset. I added customisations to the code for my dataset. The blog post described the process below: How our final (semi) automated pipeline ended up working was as follows: 1. Enter coordinates intocropping_coordinates.py to calculate the coordinates of a 10 km square. 2.  Copy the output and replace the variables at the top of the JavaScript code in the Google Earth Engine console (the JavaScript code we used). Don’t forget to change the date range to one that you’d like. 3.  Download the images with less than 100 for the mean cloud density given by clicking the link printed in the console output. When I got the images, they looked like this: Image by author In the beginning I did not know why this was the case. But later learned that satellite images were separated into different bands. Bands are certain wavelengths that the satellite captures. Doing this allows the user to view the image with features that are not visible with human eye or can be highlighted with a chosen wavelength. For example, a common use of satellite imagery is tracking farmland. By using the infrared spectrum, it is more easier spot vegetation. Figure 1 This false-colour image of Florida combines shortwave infrared, near infrared, and green light. (NASA by Matt Radcliff with data from the USGS ) image Landsat 5 Earth Explorer. Also found here: https://earthobservatory.nasa.gov/features/FalseColor gives a nice list of wavelengths and their uses: Wikipedia The wavelengths are approximate; exact values depend on the particular satellite's instruments: Blue, 450–515..520 nm, is used for atmosphere and deep water imaging, and can reach depths up to 150 feet (50 m) in clear water. Green, 515..520–590..600 nm, is used for imaging vegetation and deep water structures, up to 90 feet (30 m) in clear water. Red,600..630–680..690 nm, is used for imaging man-made objects, in water up to 30 feet (9 m) deep, soil, and vegetation. Near infrared (NIR), 750–900 nm, is used primarily for imaging vegetation. Mid-infrared (MIR), 1550–1750 nm, is used for imaging vegetation, soil moisture content, and some forest fires . Far-infrared (FIR), 2080–2350 nm, is used for imaging soil, moisture, geological features, silicates, clays, and fires. Thermal infrared , 10400-12500 nm, uses emitted instead of reflected radiation to image geological structures, thermal differences in water currents, fires, and for night studies. Radar and related technologies are useful for mapping terrain and for detecting various objects. Also, : combinations True-color uses only red, green, and blue channels, mapped to their respective colors. As a plain color photograph, it is good for analyzing man-made objects, and is easy to understand for beginner analysts. Green-red-infrared, where the blue channel is replaced with near infrared, is used for vegetation, which is highly reflective in near IR; it then shows as blue. This combination is often used to detect vegetation and camouflage. Blue-NIR-MIR, where the blue channel uses visible blue, green uses NIR (so vegetation stays green), and MIR is shown as red. Such images allow the water depth, vegetation coverage, soil moisture content, and the presence of fires to be seen, all in a single image. I wanted a simple RGB image. Which is simple visible light. As its best to start simple. To develop that, we need an image that combines all 3 bands into one. So, using some of the google earth engine example code and my personal changes. I was able to get some images. The red square covering the map is the area that google earth engine will capture for the image. Image by author The image below: Image by author As we can the image quality is horrible to be frank. I started to notice problems when setting up the red square. You see in the image of the general area of outside the red square. You can see that image lacks clarity. Which means if you zoom in even further you cannot tell what’s in the image because the quality is pretty bad. The reason for the problem was the resolution of images. The satellites sentinel-2 and Landsat only have a resolution 30 meters. This explains spatial resolution perfectly: website Spatial resolution refers to the size of one pixel on the ground. A pixel is that smallest ‘dot’ that makes up an optical satellite image and basically determines how detailed a picture is. Landsat data, for example, has a 30m resolution, meaning each pixel stands for a 30m x 30m area on the ground. It’s considered a medium-resolution image, which can cover an entire city area alone, but the level of detail isn’t fine enough to distinguish individual objects like houses or cars. Custom Dataset All open source data goes down to 30 meters. For finer data one must pay. So I had to find a solution that allows me get higher quality images. Without forking small fortune. After a lot thinking (and worrying). I found the solution hidden in plain sight. When using navigation systems on your phone. Your phone can give you a satellite image of your area. You can see which buildings are which. And differentiate between objects. So I decided to use (normal) google earth and capture images straight from the website. To make sure the screen captures where all the same size used a program called PicPick. Here’s one image I captured: Image by author Labelling data Now we have a small sample size. We need a solution for the machine learning model to know it ‘s looking at a tree. So used a program called Which I used to manually label the trees in the images. LabelImg from GitHub. Image by author I did not label every single tree in every image. As that will take forever. But wanted to have some labelling so the model can pick one or two trees in the test image. Same reasoning why the dataset is very small. The reasoning is so if I need to make changes to the dataset. It won’t take a long time. Also, just to create a model as simple prototype before scaling up the work. Model This project is an object detection problem. As I’m dealing with image data. I will be using some type of Convolutional Neural Network. One example I was thinking of using for the model is . But quickly learned that as the model was already trained the model won’t be able to read satellite image for trees. The blog post mentioned you do transfer learning or fine-tuning. But said t that it was outside the scope of the article. So, I had to find a new model to use. here R-CNN After a while of googling around. I found an interesting model that I can use called R-CNN. More precisely faster R CNN. R CNN which stands for Region-Based Convolutional Neural Network. Where the model selectively goes through many 2000 proposal regions of the image. After the proposal regions are selected, they then shaped for the input of the Convolutional Neural Network. After Support vector machines classify the regions. Then a bounding box regression is used to work out where the box goes. Figure 2 https://www.analyticsvidhya.com/blog/2018/10/a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1/ But they are a few problems relating to using a traditional R CNN. Mainly that processing an image takes a lot of time. As the model is extracting 2000 regions to check. Prediction with the model takes around 40 seconds. Fast R-CNN A better version of R CNN was created by the same person. Called Fast R CNN. Main difference is instead of extracting thousands of regions from the image. Fast R-CNN inputs the whole image. Then the CNN produces a convolutional feature map from that the chosen region proposals. The region proposals are then shaped for the pooling layer. After that it gets reshaped again to be 4entered into a full connected layer. For each region of interest feature vector, it is moved to a SoftMax layer to predict the class. The use of bounding box regression to find the boxes. One problem is that as Fast R-CNN uses selective search is slowing down testing. But Fast R-CNN is way faster than the traditional R-CNN. As Fast R-CNN takes around 2 seconds to predict. Faster R-CNN Now we have faster R-CNN. The examples above uses selective search to go through proposal regions. Faster R-CNN uses a separate network of predict the region proposals. Using the convolutional feature map. Then the proposals are then reshaped for a pooling layer. For classification of the region. Also bounding box regressor used to place the images. Figure 3 https://www.analyticsvidhya.com/blog/2018/10/a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1/ Coding up the of model I followed tutorial from . The tutorial used a dataset of blood cells. I replaced the dataset with my custom dataset of google earth photos. To get the data ready for the model I had to convert the bounding boxes from XML to CSV. here To do that I used pandas and python’s ElementTree module. And borrowed code from and the export file in the tutorial. machine learning mastery linked list_dataframe = []
folder = file os.listdir(folder):
    path = file = path + file
    print(file) filename = re.sub( , , file)
    filename = filename.replace( , )
    filename = filename.replace( , )
    tree_type = tree = ElementTree.parse(file) root = tree.getroot() boxes = list() box root.findall( ):
        xmin = int(box.find( ).text)
        ymin = int(box.find( ).text)
        xmax = int(box.find( ).text)
        ymax = int(box.find( ).text) row_list = [filename, tree_type, xmin, xmax, ymin, ymax]
        print(row_list) list_dataframe.append(row_list)


print(list_dataframe)

df = pd.DataFrame(list_dataframe, columns=[ , , , , , ])

print(df.head())

df.to_csv( , index= ) 'Custom_dataset\\Labels' for in 'Custom_dataset\Labels\\' # filename = os.path.splitext(file)[0]+'.png' 'Custom_dataset\\\Labels' "" "\\" "" ".xml" ".png" 'Tree' # get the root of the document # extract each bounding box for in './/bndbox' 'xmin' 'ymin' 'xmax' 'ymax' # coors = [xmin, xmax, ymin, ymax] # boxes.append(coors) # df = df.reset_index(drop=True) 'filename' 'tree_type' 'xmin' 'xmax' 'ymin' 'ymax' 'testing.csv' False df = pd.DataFrame( , columns=[ , , , , , ])
list_dataframe = []
folder = None 'filename' 'tree_type' 'xmin' 'ymax' 'ymix' 'ymax' 'Custom_dataset\\Labels' This section sets up the variables to be used for the loop. An empty dataframe is set up with the name of the filenames, class and the bounding boxes. The path of folder containing the images are saved in the folder variable. file os.listdir(folder):
  path = file = path + file
  print(file) for in 'Custom_dataset\Labels\\' The start of the for loop starts by listing the files in the folder. Then the file path then printed by combining the folder path and name of the file. This is done as a normal for loop without os.listdir did not work on my laptop. filename = re.sub( , , file)
filename = filename.replace( , )
filename = filename.replace( , ) 'Custom_dataset\\\Labels' "" "\\" "" ".xml" ".png" When printing the filename it had too many backslashes. So to get rid of it str.replace functions were used. Also, as the folder it using the label names not the images. The filenames ended in XML. So, replaced to PNG filetype. tree_type = tree = ElementTree.parse(file) root = tree.getroot() boxes = list() 'Tree' # get the root of the document # extract eachbounding box The section parses through the xml file. Crates the name of the class. In tree_type. tree = ElementTree.parse(file) parses the file. Then root = tree.getroot() gets the root of the file. And saves bonding box in a list. box root.findall( ):
     xmin = int(box.find( ).text)
     ymin = int(box.find( ).text)
     xmax = int(box.find( ).text)
    ymax = int(box.find( ).text)

   row_list = [filename, tree_type, xmin, xmax, ymin, ymax]
   print(row_list)
   List_dataframe.append(row_list) for in './/bndbox' 'xmin' 'ymin' 'xmax' 'ymax' The for loop goes through each of the boxes and extracts the coordinates. Of the four corners of the box. Then a list is created which saves the filename, class, and coordinates. Then appended to the dataframe from earlier. print(list_dataframe)
	
df = pd.DataFrame(list_dataframe, columns=[ , , , , , ])
	
print(df.head())
	
df.to_csv( , index= ) 'filename' 'tree_type' 'xmin' 'xmax' 'ymin' 'ymax' 'testing.csv' False After that list_dataframe containing all the data of the files. Is then added into another dataframe with the column names. And df.to_csv turns the dataframe into a CSV file. Implementing the model Now the tutorial asks for use to convert the CSV file into txt file. As the model uses a different format. filepath, x1,y1,x2,y2,class_name Where: filepath is the path of the training image x1 is the xmin coordinate for bounding box y1 is the ymin coordinate for bounding box x2 is the xmax coordinate for bounding box y2 is the ymax coordinate for bounding box class_name is the name of the class in that bounding box pandas pd

data = pd.DataFrame()
data[ ] = train[ ] i range(data.shape[ ]):
    data[ ][i] = + data[ ][i] i range(data.shape[ ]):
    data[ ][i] = data[ ][i] + + str(train[ ][i]) + + str(
        train[ ][i]) + + str(train[ ][i]) + + str(train[ ][i]) + + train[ ][i]

data.to_csv( , header= , index= , sep= ) import as 'format' 'image_names' # as the images are in train_images folder, add train_images before the image name for in 0 'format' 'train_images/' 'format' # add xmin, ymin, xmax, ymax and class as per the format required for in 0 'format' 'format' ',' 'xmin' ',' 'ymin' ',' 'xmax' ',' 'ymax' ',' 'cell_type' 'annotate.txt' None None ' ' Now the tutorial asks us clone this When I ran the model I got a few errors. So, I went inside the repository and edited the code. Most errors were AttributeError: module 'keras.backend' has no attribute 'image_dim_ordering'. I found this helpful solving the problem. This is because some of the code used is deprecated. And you need to use the new keras API which has different names for the functions. GitHub repository. GitHub issue answer So, after a few hours of googling other issues the machine learning model started to run. I used a google cloud deep learning VM to train my model. Image by author Due to the amount of time. A decided to reduce the amount epochs to 5. As each epoch took 4 hours. And the default number was 1000. Which will take couple of days to train and quite a bit of money. Now I needed to test model. I did this on my laptop. As Google’s VM did not show images To do this I ran the statement: python test_frcnn.py -p test_images Image by author The output shows it found some trees: Images by author The model was able to find some areas with trees. Mainly thick forestry. But others, not at all: Images by author It looks like the model can’t point individual trees. But it can point to big areas forestry. Improvements Like I mentioned above the model can’t spot individual trees. But areas of forestry. But even then with the areas of forestry. It can only spot some sections. A few small boxes cover a large area of forestry. The dataset was kept brief. So, I can get a workable prototype up and running. Labelling can be improved by boxing every single tree in an image. And having custom segmentation for areas forestry. Instead of boxes to cover an area of forestry. The dataset is very small (20 images) and trained for only 5 epochs. This can be improved by simply adding more images to the dataset. And spending more time training the model. A serious question to ask is, how I would label a large dataset. Labelling 20 images was already long enough. By manually labelling the trees. Took around hour or so. Can’t imagine how to label thousand images by hand. Also, with custom segmentation for large areas forestry I’m not sure how to make custom shapes. To cover the shape of the forestry. As the program I was using only allows for boxes. Not custom shapes like polygons. Published from my website

How To Identify Trees with Deep Learning

Scrapy is a web crawling framework which divide the whole process of crawling to small processes so that the crawling process is well organize! Scrapy Crawl Data ( ) -> Rotate proxy or ip ( ) -> spider.py middlewares.py Clean Data ( )-> Store Data( ) items.py pipeline.py With all the settings ( ). setting.py The biggest feature is that it is built on Twisted, an asynchronous networking library, so Scrapy is implemented using a non-blocking (aka asynchronous) code for concurrency, which makes the spider performance is very great. — Michael Yin Selenium Selenium is a free automated testing suite for web applications across different browsers and platforms. Although it was created for automated testing on web app, it is really easy to apply to scrape websites! You just need to -> Download chrome, firefox or other drivers -> Use their API to scrape websites. Issues I faced using Selenium: Speed is quite slow. Need quite a lot memory if you want to build a multi threading crawler to speed the process up. Issues I faced using Scrapy: It is harder to debug. Harder to connect to if you are implementing Scrapy-Splash. Tor Lesser reference to refer when you want to use Scrapy-Splash. Sharing about my experiences: At first, I learned Selenium as it is much easier to learn and debug as I need to render JavaScript websites. When I first use selenium, it satisfies all my needs, crawling all the web-pages in required time frame. Then speed it up by using multi threading and everything goes really smooth. Yeah Really Smooth! But one day, one particular website block me by implementing I was really stuck but I was required to figure out a way to solve this problem. So, after I tried all the ways to solve the captcha, I think why not I use another framework to try and see whether it can bypass the captcha. Completely Automated Public Turing test to tell Computers and Humans Apart (Captcha). Bang my head and hope something magical come to my mind :( At last I found Scrapy framework and not only solve captcha problems but a start for me to learn a really powerful crawling framework! The learning curve for Scrapy is much steeper than Selenium but it definitely worth it base on the below: five points Write your crawler code in a much shorter Python script compare to selenium. Crawl a lot faster than selenium. If you are using scrapy-splash, there is a great terminal Splash render on localhost:8050 so that u can try your Lua script. Organize your crawler code in a really structured way so that you can attain maximum satisfaction :) Scrapy can scale well if you project need to crawl a lot of websites. Scrapy-Splash is definitely worth trying out to render heavy loaded Javascript websites but compare to Splash, Scrapy-Splash have much lesser resources compare to Scrapy. Here are some resources I find useful to learn Scrapy-Splash. https://splash.readthedocs.io/en/stable/faq.html (chinese website) https://www.cnblogs.com/shaosks/p/6950358.html (chinese website) https://juejin.im/post/5afe47b3f265da0b767db40e Here are some really useful resources to learn Scrapy. https://doc.scrapy.org/en/latest/intro/tutorial.html https://www.analyticsvidhya.com/blog/2017/07/web-scraping-in-python-using-scrapy/ https://blog.michaelyin.info/scrapy-tutorial-5-how-create-simple-scrapy-spider/ https://python.gotrained.com/scrapy-tutorial-web-scraping-craigslist/ Here are some really useful resources to learn Selenium. https://selenium-python.readthedocs.io/ https://www.softwaretestingmaterial.com/selenium-tutorial/ https://www.tutorialspoint.com/selenium All resources are based on Python. Happy Learning! If you are interested to know more about tutorials for Scrapy-Splash, Scrapy or Selenium, feel free to comment below! Feel free to too:) reach out to me

Scrapy or Selenium?

Is Object Detection a Done Deal?

Neural networks and Genetic algorithms are our naive approach to imitate nature. They work well for a class of problems but they do have various hurdles such as overfitting, local minima, vanishing gradient and much more. There is another set of algorithms that do not get much recognition(in my opinion) compared to others and they are boosting algorithms. What is Boosting? Boosting is a method of converting a set of weak learners into strong learners. Suppose we have a binary classification task. A weak learner has an error rate that is slightly lesser than 0.5 in classifying the object, i.e the weak learner is slightly better than deciding from a coin toss. A strong learner has an error rate closer to 0. To convert a weak learner into strong learner, we take a family of weak learners, combine them and vote. This turns this family of weak learners into strong learners. The idea here is that the family of weak learners should have a minimum correlation between them. Here, let A, B and C be different classifiers. Their area A represents where the classifier A misclassifies(goes wrong) and area B represents where the classifier B misclassifies and area C represents where the classifier C misclassifies. Since there is no correlation between the errors of each classifier, combining them and using a technique of democratic voting to classify each object, this family of classifiers will never go wrong. I guess this would’ve provided a basic understanding of boosting. Moving on to types of boosting . algorithm Types of boosting algorithms: I would like to explain different boosting algorithms without any of the math involved in it cause I feel it would complicate things and defeat the purpose of this article, which is simplicity(hopefully). The different types of boosting algorithms are: AdaBoost Gradient Boosting XGBoost These three algorithms have gained huge popularity, especially XGBoost, which has been responsible for winning many data science competitions. AdaBoost(Adaptive Boosting): The Adaptive Boosting technique was formulated by Yoav Freund and Robert Schapire, who won the Gödel Prize for their work. AdaBoost works on improving the areas where the base learner fails. The base learner is a algorithm which is a weak learner and upon which the boosting method is applied to turn it into a strong learner. Any machine learning algorithm that accept weights on training data can be used as a base learner. In the example taken below, are used as the base learner. machine learning Decision stumps We take the training data and randomly sample points from this data and apply decision stump algorithm to classify the points. After classifying the sampled points we fit the decision tree stump to the complete training data. This process iteratively happens until the complete training data fits without any error or until a specified maximum number of estimators. After sampling from training data and applying the decision stump, the model fits as showcased below. Decision Stump 1 We can observe that three of the positive samples are misclassified as negative. Therefore, we exaggerate the weights of these misclassified samples so that that they have a better chance of being selected when sampled again. Decision Stump 2 When data is sampled next time, the decision stump 2 is combined with decision stump 1 to fit the training data. Therefore we have a miniature ensemble here trying to fit the data perfectly. This miniature ensemble of two decision stumps misclassifies three negative samples as positive. Therefore, we exaggerate the weights of these misclassified samples so that that they have a better chance of being selected when sampled again. Decision Stump 3 The previously misclassified samples are chosen and decision stump 3 is applied to fit the training data. We can find that two positive samples are classified as negative and one negative sample is classified as positive. Then the ensemble of three decision stumps(1, 2 and 3) are used to fit the complete training data. When this ensemble of three decision stumps are used the model fits the training data perfectly. Ensemble of 3 Decision Stumps The drawback of AdaBoost is that it is easily defeated by noisy data, the efficiency of the algorithm is highly affected by outliers as the algorithm tries to fit every point perfectly. You might be wondering since the algorithm tries to fit every point, doesn’t it overfit? No, it does not. The answer has been found out through experimental results, there has been speculations but no concrete reasoning available. Code: # AdaBoost Algorithmfrom sklearn.ensemble import AdaBoostClassifier clf = AdaBoostClassifier()# n_estimators = 50 (default value)# base_estimator = DecisionTreeClassifier (default value)clf.fit(x_train,y_train)clf.predict(x_test) Continuing to explain Gradient Boosting and XGBoost will further increase the length of this already pretty long article. Therefore I have decided to write them as another article. Please follow the link below. Link: https://medium.com/@grohith327/gradient-boosting-and-xgboost-90862daa6c77 References: https://www.analyticsvidhya.com/blog/2015/11/quick-introduction-boosting-algorithms-machine-learning/

Stories, News & Mentions about KWHANALYTICS

Kwhanalytics’s Stories on HackerNoon