Google trends (GT) is an under-utilized superweapon and harvests a massive amount of search data. But, it hasn't been possible to use GT for real time machine learning tasks, such as predicting stock price or crypto currency movements, until now....In this blog, we'll explain the problem with GT for machine learning, the fix to GT data and the edge we've built in crypto trading models at edgebase.io.We are currently looking for experienced crypto traders as beta testers for our product - please reach out to hello@edgebase.io! Edgebase.io is a no-code platform for building your own AI trading signals (initially cryptos only). What's the problem? Let's assume you want to test whether Google trends keywords, such as "bitcoin", are leading indicators of the price movement for BTCUSD on the Kraken Exchange. You'll need an method to grab historical data for building ML models and the equivalent real time data to evaluate those models and make predictions. Easy. Not quite. GT gives different results across different time windows Bitcoin for past 30 days on 26th April 2021 Bitcoin for past 90 days on 26th April 2021 Above we can see that the "interest over time" of "bitcoin" for the past 30 days and the past 90 days . Looking back 30 days, the value on 23rd April 2021 is 100, whereas looking back 90 days the value on 23rd April is 76. Google trends considers the relative search volume within a time window. In the past 90 days, the day with the highest search counts was February 9th 2021 and thus the 23rd April gets reduced from 100 to 71. This is a problem for machine learning because historical GT data cannot align with new real time data. In fact, the problem is even worse! GT also uses a sample of the true search count. Even if we query within the same date range, the results can suffer from huge variances due to small sample sizes. A group of German researchers made headlines in 2020, when they published a paper showing huge discrepancies for "kurzarbeit", the German word for placing workers on partial furlough. This was particularly concerning because German politicans were actively using "kurzarbeit" GT data for setting policy during the coronavirus pandemic. What's the fix? Our fix at edgebase.io is to use our proprietary correction algorithm for GT data. Starting from a 'seed date', we iteratively track changes in the GT reindexing and correct any reindexing over time. This means that different GT sample windows can be stitched together and provide a consistent view of the relative search count over time.We also correct for low sample sizes in GT data, by taking multiple samples from the same time window and creating an average sample using the magic of the central limit theorem. The sample error reduces by a factor of sqrt(n), where n is the number of GT queries taken for the same window. Google doesn't provide daily data beyond 90 days, and if you try to stitch together rolling 30 day windows of "bitcoin" GT data you get: Above there are many peaks in search volume, some do correspond to bitcoin movements, but this is very noisy and unreliable data. However, if you use the edgebase correction algorithm you get: Visually, you can see that our correction has significantly less spikes at 100, closely follows the Bitcoin price and scales over time. But can this be used to predict bitcoin price movements... What's the edge? At edgebase.io we quickly train ML models with new datasets, such as our new corrected GT bitcoin data at scale.Let's start with a simple bitcoin price model. We can drag and drop datasets and create features off these dataset using the Edgebase model builder. In the example below , we take the 11 daily log difference of our new "Bitcoin" GT data and use it to predict the daily bitcoin Kraken closing price: Now we have constructed our simple model, let's hit "Train Model": Pretty impressive! The F1 score is 52, which means we predicted the bitcoin price movement about 52% of the time. Our profit is about 40% over 3 months and our Sharpe ratio is 1.24, which approximately means we earned 1.24 units of reward for every unit of trading risk taken.We can improve the model by feeding the logarithmic 3 day (weighted) difference of bitcoin prices into our model as well: Even better ! The F1 score is still 52, but with a slightly improved Sharpe ratio of 2.12 and higher returns at 75%: This is only one dataset we provide to predict cryptocurrencies at edgebase.io . We also have volume data, futures data and other proprietary datasets for bitcoin, Ethereum, dogecoin and other altcoins.Thank you for reading and if you are an experienced crypto trader please reach out to hello@edgebase.io as we are looking for beta testing users! Google trends (GT) is an under-utilized superweapon and harvests a massive amount of search data. But, it hasn't been possible to use GT for real time machine learning tasks, such as predicting stock price or crypto currency movements , until now.... In this blog, we'll explain the problem with GT for machine learning, the fix to GT data and the edge we've built in crypto trading models at edgebase.io . We are currently looking for experienced crypto traders as beta testers for our product - please reach out to hello@edgebase.io ! Edgebase.io is a no-code platform for building your own AI trading signals (initially cryptos only). predicting stock price or crypto currency movements the problem the fix the edge edgebase.io experienced crypto traders hello@edgebase.io What's the problem? What's the problem? Let's assume you want to test whether Google trends keywords, such as "bitcoin", are leading indicators of the price movement for BTCUSD on the Kraken Exchange . You'll need an method to grab historical data for building ML models and the equivalent real time data to evaluate those models and make predictions. Easy. Not quite. Kraken Exchange historical data real time data GT gives different results across different time windows Bitcoin for past 30 days on 26th April 2021 Bitcoin for past 30 days on 26th April 2021 Bitcoin for past 30 days on 26th April 2021 Bitcoin for past 90 days on 26th April 2021 Bitcoin for past 90 days on 26th April 2021 Bitcoin for past 90 days on 26th April 2021 Above we can see that the "interest over time" of "bitcoin" for the past 30 days and the past 90 days . Looking back 30 days, the value on 23rd April 2021 is 100 , whereas looking back 90 days the value on 23rd April is 76 . Above Above "bitcoin" 100 76 Google trends considers the relative search volume within a time window. In the past 90 days, the day with the highest search counts was February 9th 2021 and thus the 23rd April gets reduced from 100 to 71. This is a problem for machine learning because historical GT data cannot align with new real time data. relative This is a problem for machine learning In fact, the problem is even worse! GT also uses a sample of the true search count. Even if we query within the same date range, the results can suffer from huge variances due to small sample sizes . A group of German researchers made headlines in 2020, when they published a paper showing huge discrepancies for "kurzarbeit", the German word for placing workers on partial furlough. This was particularly concerning because German politicans were actively using "kurzarbeit" GT data for setting policy during the coronavirus pandemic. sample sample sample the results can suffer from huge variances due to small sample sizes paper What's the fix? What's the fix? Our fix at edgebase.io is to use our proprietary correction algorithm for GT data. Starting from a 'seed date', we iteratively track changes in the GT reindexing and correct any reindexing over time. This means that different GT sample windows can be stitched together and provide a consistent view of the relative search count over time. We also correct for low sample sizes in GT data, by taking multiple samples from the same time window and creating an average sample using the magic of the central limit theorem . The sample error reduces by a factor of sqrt(n) , where n is the number of GT queries taken for the same window. edgebase.io magic of the central limit theorem sqrt(n) n Google doesn't provide daily data beyond 90 days, and if you try to stitch together rolling 30 day windows of "bitcoin" GT data you get: Above there are many peaks in search volume, some do correspond to bitcoin movements, but this is very noisy and unreliable data. However, if you use the edgebase correction algorithm you get: Above Above edgebase correction algorithm Visually, you can see that our correction has significantly less spikes at 100, closely follows the Bitcoin price and scales over time. But can this be used to predict bitcoin price movements... What's the edge? What's the edge? At edgebase.io we quickly train ML models with new datasets, such as our new corrected GT bitcoin data at scale . Let's start with a simple bitcoin price model. We can drag and drop datasets and create features off these dataset using the Edgebase model builder. In the example below , we take the 11 daily log difference of our new "Bitcoin" GT data and use it to predict the daily bitcoin Kraken closing price: edgebase.io at scale below Kraken Now we have constructed our simple model, let's hit "Train Model": Pretty impressive! The F1 score is 52, which means we predicted the bitcoin price movement about 52% of the time. Our profit is about 40% over 3 months and our Sharpe ratio is 1.24, which approximately means we earned 1.24 units of reward for every unit of trading risk taken. We can improve the model by feeding the logarithmic 3 day (weighted) difference of bitcoin prices into our model as well: F1 score Even better ! The F1 score is still 52, but with a slightly improved Sharpe ratio of 2.12 and higher returns at 75%: F1 score This is only one dataset we provide to predict cryptocurrencies at edgebase.io . We also have volume data, futures data and other proprietary datasets for bitcoin, Ethereum, dogecoin and other altcoins. Thank you for reading and if you are an experienced crypto trader please reach out to hello@edgebase.io as we are looking for beta testing users! edgebase.io experienced crypto trader hello@edgebase.io

Ethereum

Trade Crypto with Machine Learning Based On Google Trends

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps