Regrello

regrello

Stock Market Prediction - the Holy Grail of Time Series Data

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Method

HDR or SDR? A Study of Scaled and Compressed Videos: Objective Assessment

Comparison of Machine Learning Methods: Conclusions and Future Work, and References

On-Device AI Models and Core ML Tools: Insights From WWDC 2024

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrea Roncoli, Department of Computer, Science (University of Pisa);
(2) Aleksandra Ciprijanovi´c´, Computational Science and AI Directorate (Fermi National Accelerator Laboratory) and Department of Astronomy and Astrophysics (University of Chicago);
(3) Maggie Voetberg, Computational Science and AI Directorate, (Fermi National Accelerator Laboratory);
(4) Francisco Villaescusa-Navarro, Center for Computational Astrophysics (Flatiron Institute);
(5) Brian Nord, Computational Science and AI Directorate, Fermi National Accelerator Laboratory, Department of Astronomy and Astrophysics (University of Chicago) and Kavli Institute for Cosmological Physics (University of Chicago). Table of Links Abstract and Intro Data and Methods Results Conclusions Acknowledgments and Disclosure of Funding, and References Additional Plots 3 Results Latent space organization Isomaps are two-dimensional projections of the multi-dimensional latent space [35]. Figure 1 shows the difference in the latent space structure without (top row) and with (bottom row) DA. Ellipses in the top right isomap highlight how the two distributions are encoded in different regions of the latent space. Without the MMD loss, the model encodes samples with very different values of Ωm close to each other, if they originate from different simulations (circles and triangles of different colors are overlapping). This scenario leads to the fragility of the regressor, which cannot learn to output different values for the same latent space encodings. On the contrary, the DA-GNN (bottom right plot) correctly encodes the samples in a domain-invariant way. Visually, circle and triangle distributions are overlapping, which indicates domain mixing. Furthermore, the direction in the color gradient shows that the DA-GNN encodes information such that the regressor can now more correctly predict cosmological parameters based on the encodings of both simulations. [5] In [41], authors get slightly better results for the same domain, and slightly worse for the cross-domain tests. We impute these differences to choices such as batch sizes and optimization techniques we took due to computational and time constraints. [6] In Appendix A, the IllustrisTNG counterpart of this plot is presented. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrea Roncoli, Department of Computer, Science (University of Pisa); (2) Aleksandra Ciprijanovi´c´, Computational Science and AI Directorate (Fermi National Accelerator Laboratory) and Department of Astronomy and Astrophysics (University of Chicago); (3) Maggie Voetberg, Computational Science and AI Directorate, (Fermi National Accelerator Laboratory); (4) Francisco Villaescusa-Navarro, Center for Computational Astrophysics (Flatiron Institute); (5) Brian Nord, Computational Science and AI Directorate, Fermi National Accelerator Laboratory, Department of Astronomy and Astrophysics (University of Chicago) and Kavli Institute for Cosmological Physics (University of Chicago). This paper is available on arxiv under CC 4.0 license. available on arxiv Authors: Authors: (1) Andrea Roncoli, Department of Computer, Science (University of Pisa); (2) Aleksandra Ciprijanovi´c´, Computational Science and AI Directorate (Fermi National Accelerator Laboratory) and Department of Astronomy and Astrophysics (University of Chicago); (3) Maggie Voetberg, Computational Science and AI Directorate, (Fermi National Accelerator Laboratory); (4) Francisco Villaescusa-Navarro, Center for Computational Astrophysics (Flatiron Institute); (5) Brian Nord, Computational Science and AI Directorate, Fermi National Accelerator Laboratory, Department of Astronomy and Astrophysics (University of Chicago) and Kavli Institute for Cosmological Physics (University of Chicago). Table of Links Abstract and Intro Abstract and Intro Data and Methods Data and Methods Results Results Conclusions Conclusions Acknowledgments and Disclosure of Funding, and References Acknowledgments and Disclosure of Funding, and References Additional Plots Additional Plots 3 Results Latent space organization Isomaps are two-dimensional projections of the multi-dimensional latent space [35]. Figure 1 shows the difference in the latent space structure without (top row) and with (bottom row) DA. Ellipses in the top right isomap highlight how the two distributions are encoded in different regions of the latent space. Without the MMD loss, the model encodes samples with very different values of Ωm close to each other, if they originate from different simulations (circles and triangles of different colors are overlapping). This scenario leads to the fragility of the regressor, which cannot learn to output different values for the same latent space encodings. On the contrary, the DA-GNN (bottom right plot) correctly encodes the samples in a domain-invariant way. Visually, circle and triangle distributions are overlapping, which indicates domain mixing. Furthermore, the direction in the color gradient shows that the DA-GNN encodes information such that the regressor can now more correctly predict cosmological parameters based on the encodings of both simulations. [5] In [41], authors get slightly better results for the same domain, and slightly worse for the cross-domain tests. We impute these differences to choices such as batch sizes and optimization techniques we took due to computational and time constraints. [6] In Appendix A, the IllustrisTNG counterpart of this plot is presented.

Adaptive Graph Neural Networks for Cosmological Data Generalization: Results

Starting Simple: The Strategic Advantage of Baseline Models in Machine Learning

Are we on the brink of debunking Moravec’s Paradox? In the 1980s, AI and robotics researcher Hans Moravec highlighted a counterintuitive aspect of AI: tasks requiring high-level reasoning — like chess or Go — are easier for to master than basic sensory and motor skills — such as walking or identifying your mom’s face — which humans find instinctive. Adding complexity, these “simpler” skills actually demand much more computational power. This insight sheds light on the complexity of replicating human-like perception and dexterity, outcomes of millions of years of evolution, as opposed to logical reasoning, a more recent development. In today’s AI and ML landscape, this paradox underscores the challenges in creating robots and AI systems capable of seamlessly navigating and interacting with the physical world. AI However, last week, Bernt Bornich, CEO and founder of 1x, a humanoid robotics company, , “ .” I suspect that this has something to do with the advancements in foundation models. Originally known for their ability to perform a wide range of tasks based on a single type of data (like text for language models), these models become “multimodal” when integrating and interpreting information across different sensory inputs, closely mirroring human-like understanding. wrote New progress update on the droids dropping in 4 weeks, looks like Moravec’s paradox might be debunked, and we just didn’t have the data Could the embodiment of AI, with all its sensory inputs, plus reasoning-imitation algorithms like LLMs, be the pool of data that disproves Moravec’s paradox? Another intriguing development caught my attention. Huang Jensen, Nvidia’s CEO, to a question from Wired about what current development could change everything. Jensen replied, “ ” responded There are a couple of things. One doesn’t really have a name, but it’s part of the work we’re doing in foundational robotics. If you can generate text and images, can you also generate motion? The answer is probably yes. And if you can generate motion, you can understand the intent and generate a generalized version of articulation. Therefore, humanoid robotics should be right around the corner. Something to observe in the coming weeks! In related news from the robotics universe, Figure AI, a humanoid robotics startup, made headlines by raising approximately in funding. What’s more impressive is the list of backers: Amazon, NVIDIA, Microsoft, OpenAI, Intel, LG, and Samsung. This indicates a strong belief in the potential of humanoid robotics to disrupt various sectors. $675 million Yet, there are skeptical voices. Rodney Brooks, who coined Nouvelle AI*, last week: “ ” posted Tele-op robots presented as autonomous, like the Tesla Optimus humanoid folding a shirt, and 1X humanoid robots, are misrepresentations of what robots are actually doing, which can also be called LIES. Note that the Stanford robot cooking and cleaning videos are also tele-operated. If 2023 was the year of LLMs, are we ready to evolve to an embodied AI and make 2024 the year of robots? 🎁 Bonus: The freshest research papers from the week of Feb 19 — Feb 25 Enhancing Large Language Models (LLMs) : Expands the processing capability of LLMs to handle over 2 million tokens, pushing the boundaries of context window sizes for more comprehensive understanding and generation tasks. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Read the paper. : Demonstrates the versatility of LLMs in performing numerical regression tasks, suggesting their potential as universal tools for predictive modeling across a variety of domains. . OmniPred: Language Models as Universal Regressors Read the paper : Investigates efficient strategies for distilling large models into smaller, more manageable ones, particularly for reasoning tasks, emphasizing the importance of decomposition over problem-solving. . Divide-or-Conquer? Which Part Should You Distill Your LLM? Read the paper : Proposes a method to enhance the efficiency of self-attention mechanisms in LLMs, crucial for improving performance and reducing resource consumption. . ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition Read the paper : Introduces a framework for personalizing LLM interactions using user embeddings, enhancing the model’s responsiveness to individual user preferences and histories. USER-LLM: Efficient LLM Contextualization with User Embeddings Multimodal and Multi-Agent Systems : Explores integrating video and language for advanced AI understanding and interaction, leveraging a novel RingAttention mechanism for efficient multimodal learning. . World Model on Million-Length Video and Language with RingAttention Read the paper : Develops a multi-agent platform that enhances cooperation and flexibility among agents, addressing the complexity of multi-agent systems and their practical applications. . AgentScope: A Flexible yet Robust Multi-Agent Platform Read the paper : Focuses on the design and analysis of small-scale multimodal models, proving that with strategic optimizations, smaller models can achieve or surpass the performance of larger counterparts. . TinyLLaVA: A Framework of Small-scale Large Multimodal Models Read the paper : Introduces a versatile multimodal language model that processes speech, text, images, and music, demonstrating the power of discrete representations in unifying various data modalities within a single framework. This approach simplifies the integration of new modalities without needing to modify the underlying architecture or training methodologies. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling read the paper : Presents a novel dataset that enhances multimodal understanding by incorporating touch with vision and language, aiming to advance touch-vision-language alignment and understanding through a tactile encoder and text generation model. A Touch, Vision, and Language Dataset for Multimodal Alignment read the paper Advancements in Specific Domains : Introduces a novel object detection model that leverages programmable gradient information for enhanced accuracy and efficiency in learning. . YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information Read the paper : Demonstrates the application of Transformers in complex planning tasks, offering a method that surpasses traditional search algorithms in efficiency and effectiveness. . : Better Planning with Transformers via Search Dynamics Bootstrapping* Beyond A Read the paper Developer Tools and APIs : Presents a vast dataset designed for training LLMs to interact with APIs, addressing the challenge of creating effective models for API usage and integration. . API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs Read the paper : Develops an open-source system for code generation, execution, and refinement, facilitated by a dataset of multi-turn interactions, aiming to bridge the gap between code generation models and practical coding tasks. . OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement Read the paper Security and Adversarial Research : Explores the susceptibility of LLMs to a wide range of adversarial attacks, highlighting the need for comprehensive security measures to protect against unintended behaviors and data extraction. . Coercing LLMs to do and reveal (almost) anything Read the paper Model Efficiency and Quantization : Discusses a novel framework for quantizing LLM weight matrices to 1-bit to drastically reduce storage and computational demands while maintaining performance, enabling efficient deployment of LLMs on resource-constrained devices. OneBit: Towards Extremely Low-bit Large Language Models read the paper Instruction Tuning and Data Quality : Introduces REALIGN, a method for refining instruction data quality for LLMs to better align with human values, emphasizing the importance of instruction data quality in model alignment and suggesting areas for further exploration in LLM science. Reformatted Alignment read the paper : Proposes a novel method for instruction tuning that generates synthetic instruction data across all disciplines, showcasing a scalable and customizable approach to instruction tuning without relying on specific training data. Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models read the paper : Proposes a pre-instruction-tuning method to enhance LLMs’ knowledge updating capabilities, demonstrating significant improvements in factual knowledge absorption and cross-domain generalization. Instruction-tuned Language Models are Better Knowledge Learners read the paper *Also published . * here

Is 2024 The Year of Advanced Robotics?

Options market pricing depends on implied volatility and is priced based on how the volatility forecast would look in the future. This story focuses primarily on predicting the volatility with time series models in the S&P500 index based on historical data. The Options market is an attractive source of investment for people across the globe and has been substantially growing at a double-digit pace every year from 2003 through 2008. , despite a brutal bear market in stocks, total U.S. options volume reached 10.32 billion contracts, up 4.6% vs. 2021. In 2022 Various option pricing models are being used in predicting the price of the options. One commonly used model is the Black Scholes model. Implied volatility (IV) is the expectation of future volatility in the options market. Higher implied volatility results in high option premiums and is also a forward-looking measure. IV differs across the options with different expiration dates. The assumption about volatility does not hold in the real world as the volatility keeps on changing. Predicting volatility using historical data seems to be a good alternative as the financial market repeats itself over some time. ARIMA and GARCH time series forecasting models have been developed which predict the IV based on the historical data and have a better accuracy. Black Scholes Model The Black-Scholes model is the primary model used in predicting the price of the options. It is given by the equation, C0=N(d1 )-  X/e^(rf*T)  N(d2) Where N(d*) is the cumulative normal distribution function for (d*). C is the theoretical call premium, S is the Current stock price, t is the time until the option expiration, K is the option striking price, and rf is the risk-free interest rate. The assumptions in the Black Scholes model include the direction of the stock’s price following a random walk, constant volatility, normal distribution of returns, and risky assets not paying any dividends during the options life. Implied volatility is based on the pricing of the at-the-money call and put options. Forecasting the IV would help to determine the option premium accurately and can also be used by the investors to generate profits based on the straddle and strangle strategy. ARIMA modeling VIX levels were modelled in Figure 1 and we could see that the VIX level remained considerably stable from 2000 to 2007. However, in 2008 we could that the VIX index was very volatile because of the global financial crisis and there was a huge spike in 2020 because of Covid and turbulent times in financial markets. The historical data from 2000 were plotted as residuals as shown below. To model the residuals, ACF and PACF plots were plotted to determine the number of lags and the model to be used. The above figures depict the autocorrelation and partial autocorrelation plot for the log of the VIX levels and its 1st difference levels. From the ACF plot we could see that 1st difference level with MA (2) and from PACF AR (2) seems to be a good fit. On the above information, initially, an ARIMA (2,1,2) was plotted and checked for the p-values. However, the p values were insignificant and hence an ARIMA (1,1,1) was incorporated to predict the residuals. Below are the specifications for the ARIMA (1,1,1) model. We can see that AR and MA coefficients are significant and the model has a low AIC and BIC which suggests ARIMA (1,1,1) better fits the VIX data. ARIMA+GARCH modeling To also consider the volatility shocks just like the events that happened in 2008 and 2020, we tried combining GARCH with the ARIMA model to see if we could get an improved forecast in the VIX index. On modeling GARCH(1,1) we got the regression results The coefficients of the GARCH model were significant at 99% level. From the residual plot below, it could be inferred that the model picked up the overall shocks in the volatility as it is far less fluctuating but in 2020 the market was so volatile that the model could not pick up the variation. Volatility Forecasting After fitting the ARIMA+GARCH model with exogenous regressors, this paper analyzes on the prediction efficiency of the model. We have split the input datasets into training and test samples. The training dataset contains data from 04/01/2000 to 12/31/2020 and the test dataset contains data from 01/04/2021 to 11/30/2023. After training the model with the fixed training regression window, the below result was obtained. The above result is the VIX observed and the predicted first differenced levels. We could see that the model picked up the variations in 2021 and till the middle of 2022. As inflation was starting to pick up in mid to late 2022, the predicted level remained higher and the was quite a difference in those time frames. Conclusion VIX forecasting involves a combination of qualitative and quantitative metrics, incorporating factors from various domains. Traders and investors should be aware of the inherent uncertainties in predicting volatility and approach forecasting with a well-informed and adaptive mindset. Forecasting VIX helps us to gauge the market sentiments to some extent even if not accurately. The main drawback of the Black-Scholes model is that the volatility is assumed to be constant. If we use the historical forecasting method, it can be accurately priced to avoid mispricing. Based on the results we obtained VIX prediction is highly likely with the historical IVs using ARIMA and GARCH model with improved accuracy and efficiency. References https://www.investors.com/research/options/options-trading-today-after-50-years-of-growth/ https://finance.yahoo.com/ The Bond & Money Markets, Strategy, Trading, Analysis, A volume in Securities Institution Professional Reference Series, Book,2001 Engle, Robert. 2001. "GARCH 101: The Use of ARCH/GARCH Models in Applied Econometrics." Journal of Economic Perspectives, 15 (4): 157-168.DOI: 10.1257/jep.15.4.157

Volatility Modeling and Forecasting in Financial Markets

Recently, due to the various changes in macroeconomic factors, many businesses have faced difficulty coming up with forecasts that are reliable. According to , about 90% of the forecasting models do not go into production. This has been attributed to a complex pipeline that’s unscalable and a generally non-exploratory nature. Venturebeat Additionally, we also notice that a single model is generally not an answer to a range of forecasting needs as forecasting for an organization involves a lot of moving parts. In this article, we will cover general methods for putting time series forecasting models into practice while creating an ensemble approach that backtests a variety of models and chooses the best approach. Table of Contents Business Problem Approach Pipeline Overview Benchmarking Automation Conclusion Business Problem Predict the revenue a product would make over the financial year Revenue impact on the forecast, assuming a change in assumptions (like Take Rate, CPI) A scale to understand Contribution to Growth Anomaly Detection For a typical organization, forecasting is a process that goes hand in hand with budgeting, planning, and allocation. We typically look at a few metrics that we want to forecast and we run a time series model on historical data. During this process, we decompose the forecast into its corresponding trends, seasonality, and residual use. This result is then used to tune and come up with a forecast. Based on the nature and understanding of the business problem in hand we can also build assumptions in the form of regressors that can help instruct the forecast. These assumptions are based on features like spend actuals & intended spend, take rate, conversion efficiency, economic data like publicly available CPI forecasts, custom seasonality flags, etc. Approach Data preprocessing & preparation Forecast Decomposition into Trend, Seasonality, Holidays and Error Improve Accuracy by Cross Validation Top Down Approach vs Bottom Up Approach Running Backtests We now go ahead and come up with the overall approach we want to follow. Some of the questions we would ask during this step are how much data we have, how clean is the data, and how to parse the data to train/test datasets. Additionally, during this stage we also come up with the various models we want to try out and a metric we would consider for benchmarking these models against each other in the form of backtests. Sometimes we could also want to combine the results of multiple models together in an ensemble approach. A Top-down vs bottom-up approach is the methodology of determining whether we want to forecast at the higher level and decompose the forecast or if we want to forecast at a product level and aggregate it to come up with the overall forecast. Pipeline Overview Data Prep using BigQuery Set up VM Instance to deploy Pipeline Machine Learning Libraries like ScikitLearn, Prophet, Arima etc. Control pipeline using Bash Script Deploy Persistent Machines on Cloud Storage For a Typical forecasting project, we start the project by querying the data and creating a dataset that we can use as a data input. In certain cases, we would also go ahead and further create features that make the most sense to use in our training (for example converting categorical variables to Numeric) and then run a model after defining our target variable. Models like Facebook Prophet, Arima, MVAR, etc can all be used during this step. We then look at the results by charting them as well as aggregating the data to multiple granularities (like converting daily forecasts to weekly & monthly). Look at the results. Benchmarking Cross Validation Errors() MAPE = Mean Absolute Percentage Error Comparison (Compare to other metrics) After running a model, we then go ahead and cross-validate the data, using a metric like to understand the model performance. We optimize the forecast using this metric and run backtests and then perform hyperparameter tuning to derive the best parameters that fit the model. MAPE Automation Forecasting at Scale Predict the same data at hourly, daily, or weekly aggregations Run for Multiple Merchants in Parallel using the same Models Test Models in Production Once the core models and forecasting approaches have been selected, the subsequent step involves devising a scalable approach to run the pipeline. This can be achieved through the use of tools such as Airflow to schedule the job or simple chron jobs to create a schedule. During this process, it is important to understand the various files that need to be executed and the models that need to be run. An effective pipeline can be created by following a sequence of steps, starting with Data Preparation, followed by data cleaning and model preparation before executing the model run. Subsequently, data validation and scoring can be carried out before rerunning the model and back-testing the results. The results can then be written into a data table, which can be used to feed a dashboard or outputted as an excel sheet to test assumptions and gain a better understanding of the forecast. Conclusion In conclusion, forecasting is an integral part of business decision-making. It provides organizations with insights into market trends, potential risks, and growth opportunities. It is crucial to ensure that any forecasting automation implemented adds value to the organization by considering factors such as forecast accuracy, data reliability, and potential impact on the business. By doing so, we can provide effective and efficient forecasting automation that supports the organization’s goals and objectives, enabling them to make informed decisions that drive success. Some of the questions we ask ourselves by automating this forecast would be: Who Asked for the forecast? Man Hours Saved Business Impact Operational Impact Lead Image generated with stable diffusion.

Ensemble Time Series Forecasting at Scale

Welcome to other chapters of Let’s Understand Chrome V8 When we learn V8, the SharedFunction and JSFunction are two very common and important terminologies that may confuse beginners. So what are they? What is the difference between them? In general, the main role of the ShardFunction is the place that lays the bytecodes, there is other stuff like the trampoline which I’ll talk about in the future. “Shared”, you can roughly think of the SharedFunction as a DLL library that can be called in anywhere. The compiler compiles a JavaScript function to a SharedFunction, and the SharedFunction can be used anywhere, like a DLL. This is the reason its name is Shared-function. Roughly, a JSFunction equals a SharedFunction + a context. When V8 is going to call a SharedFunction, V8 needs to bind a context to the ShareFunction, the bind is a term of V8 which means it binds a SharedFuncton and a context together, namely a JSFunction. A JSFunction is an executable instance that can be executed by V8 interpreter ignition. 1. SharedFunction The following code is SharedFucntion class. 1.  // SharedFunctionInfo describes the JSFunction information that can be
2.  // shared by multiple instances of the function.
3.  class SharedFunctionInfo : public HeapObject {
4.  public:
5.  V8_EXPORT_PRIVATE Code GetCode() const;
6.  V8_EXPORT_PRIVATE static void SetScript(
7.      Handle<SharedFunctionInfo> shared, Handle<Object> script_object,
8.       int function_literal_id, bool reset_preparsed_scope_data = true);
9.   V8_EXPORT_PRIVATE int EndPosition() const;
10.    V8_EXPORT_PRIVATE int StartPosition() const;
11.    V8_EXPORT_PRIVATE void SetPosition(int start_position, int end_position);
12.    inline bool IsApiFunction() const;
13.    inline bool is_class_constructor() const;
14.    inline FunctionTemplateInfo get_api_func_data();
15.    inline void set_api_func_data(FunctionTemplateInfo data);
16.    inline bool HasBytecodeArray() const;
17.    inline BytecodeArray GetBytecodeArray() const;
18.    inline void set_bytecode_array(BytecodeArray bytecode);
19.    inline Code InterpreterTrampoline() const;
20.    inline bool HasInterpreterData() const;
21.    inline InterpreterData interpreter_data() const;
22.    inline void set_interpreter_data(InterpreterData interpreter_data);
23.    // builtin_id corresponds to the auto-generated Builtins::Name id.
24.    inline bool HasBuiltinId() const;
25.    inline int builtin_id() const;
26.    inline void set_builtin_id(int builtin_id);
27.    inline bool HasUncompiledData() const;
28.    inline UncompiledData uncompiled_data() const;
29.    inline void set_uncompiled_data(UncompiledData data);
30.    inline bool HasUncompiledDataWithPreparseData() const;
31.    inline UncompiledDataWithPreparseData uncompiled_data_with_preparse_data()
32.        const;
33.    inline void set_uncompiled_data_with_preparse_data(
34.        UncompiledDataWithPreparseData data);
35.    inline bool HasUncompiledDataWithoutPreparseData() const;
36.    inline LanguageMode language_mode() const;
37.    inline void set_language_mode(LanguageMode language_mode);
38.    DECL_PRIMITIVE_ACCESSORS(syntax_kind, FunctionSyntaxKind)
39.    inline bool is_wrapped() const;
40.    DECL_BOOLEAN_ACCESSORS(has_duplicate_parameters)
41.    DECL_BOOLEAN_ACCESSORS(native)
42.    DECL_BOOLEAN_ACCESSORS(is_asm_wasm_broken)
43.    DECL_BOOLEAN_ACCESSORS(name_should_print_as_anonymous)
44.    DECL_BOOLEAN_ACCESSORS(is_oneshot_iife)
45.    DECL_BOOLEAN_ACCESSORS(are_properties_final)
46.    DECL_BOOLEAN_ACCESSORS(is_safe_to_skip_arguments_adaptor)
47.    DECL_BOOLEAN_ACCESSORS(has_reported_binary_coverage)
48.    DECL_BOOLEAN_ACCESSORS(private_name_lookup_skips_outer_class)
49.    inline FunctionKind kind() const;
50.    DECL_INT_ACCESSORS(function_map_index)
51.    inline void clear_padding();
52.    inline void UpdateFunctionMapIndex();
53.    inline bool optimization_disabled() const;
54.    inline BailoutReason disable_optimization_reason() const;
55.    void DisableOptimization(BailoutReason reason);
56.    DECL_BOOLEAN_ACCESSORS(requires_instance_members_initializer)
57.    bool HasSourceCode() const;
58.    static Handle<Object> GetSourceCode(Handle<SharedFunctionInfo> shared);
59.    static Handle<Object> GetSourceCodeHarmony(Handle<SharedFunctionInfo> shared);
60.    inline bool IsSubjectToDebugging();
61.    inline bool IsUserJavaScript();
62.    inline bool CanDiscardCompiled() const;
63.    void UpdateExpectedNofPropertiesFromEstimate(FunctionLiteral* literal);
64.    void UpdateAndFinalizeExpectedNofPropertiesFromEstimate(
65.        FunctionLiteral* literal);
66.    DECL_CAST(SharedFunctionInfo)
67.    // Constants.
68.    static const uint16_t kDontAdaptArgumentsSentinel = static_cast<uint16_t>(-1);
69.    static const int kMaximumFunctionTokenOffset = kMaxUInt16 - 1;
70.    static const uint16_t kFunctionTokenOutOfRange = static_cast<uint16_t>(-1);
71.    STATIC_ASSERT(kMaximumFunctionTokenOffset + 1 == kFunctionTokenOutOfRange);
72.    DEFINE_FIELD_OFFSET_CONSTANTS(HeapObject::kHeaderSize,
73.                                  TORQUE_GENERATED_SHARED_FUNCTION_INFO_FIELDS)
74.//............omit..............
75.    }; Lines 1–2 comment that a SharedFunction can be shared by multiple instances. Here, I give you some important members of a SharedFunction. Line 5, the GetCode() returns code, I would shed some more light on that the code is a Builtin code but not your JavaScript source code, the Builtin code is used to construct the function prologue for interpreter ignition. Before calling a function, V8 needs to construct a call stack and get the first instruction address, for which all of this is responsible by the Builtin code; Line 6, SetScript() sets JavaScript source code into the SharedFunction; Lines 9–11, they set offset that describes the corresponding position between bytecode and JavaScript source code; Lines 12–15, they describe if the SharedFunction is an API function; Lines 16–18, they get and set bytecode array for a SharedFunction; Line 19, it is the InterpreterTrampoline address that will be talked about in the future; Lines 27–35, they are the optimization reason for TurboFan. The following list is the optimization reason. #define BAILOUT_MESSAGES_LIST(V)                                            \
  V(kNoReason, "no reason")                                                 \
                                                                            \
  V(kBailedOutDueToDependencyChange, "Bailed out due to dependency change") \
  V(kCodeGenerationFailed, "Code generation failed")                        \
  V(kCyclicObjectStateDetectedInEscapeAnalysis,                             \
    "Cyclic object state detected by escape analysis")                      \
  V(kFunctionBeingDebugged, "Function is being debugged")                   \
  V(kGraphBuildingFailed, "Optimized graph construction failed")            \
  V(kFunctionTooBig, "Function is too big to be optimized")                 \
  V(kLiveEdit, "LiveEdit")                                                  \
  V(kNativeFunctionLiteral, "Native function literal")                      \
  V(kNotEnoughVirtualRegistersRegalloc,                                     \
    "Not enough virtual registers (regalloc)")                              \
  V(kOptimizationDisabled, "Optimization disabled")                         \
  V(kNeverOptimize, "Optimization is always disabled") Lines 36–37, set the mode as strict or sloppy; Lines 40–48, the macro DECL_BOOLEAN_ACCESSORS’ definition is below: #define DECL_PRIMITIVE_ACCESSORS(name, type) \
  inline type name() const;                  \
  inline void set_##name(type value);

#define DECL_BOOLEAN_ACCESSORS(name) DECL_PRIMITIVE_ACCESSORS(name, bool) Line 72, it defines the SharedFunction’s memory layout, the following is the source code and Figure 1 shows the memory layout: #define DEFINE_FIELD_OFFSET_CONSTANTS(StartOffset, LIST_MACRO) \
  enum {                                                       \
    LIST_MACRO##_StartOffset = StartOffset - 1,                \
    LIST_MACRO(DEFINE_ONE_FIELD_OFFSET)                        \
  };
//==================omit===========================
#define TORQUE_GENERATED_SHARED_FUNCTION_INFO_FIELDS(V) \
V(kStartOfWeakFieldsOffset, 0) \
V(kFunctionDataOffset, kTaggedSize) \
V(kEndOfWeakFieldsOffset, 0) \
V(kStartOfStrongFieldsOffset, 0) \
V(kNameOrScopeInfoOffset, kTaggedSize) \
V(kOuterScopeInfoOrFeedbackMetadataOffset, kTaggedSize) \
V(kScriptOrDebugInfoOffset, kTaggedSize) \
V(kEndOfStrongFieldsOffset, 0) \
V(kLengthOffset, kUInt16Size) \
V(kFormalParameterCountOffset, kUInt16Size) \
V(kExpectedNofPropertiesOffset, kUInt16Size) \
V(kFunctionTokenOffsetOffset, kUInt16Size) \
V(kFlagsOffset, kInt32Size) \
V(kFunctionLiteralIdOffset, kInt32Size) \
V(kUniqueIdOffset, kInt32Size) \
V(kSize, 0) \
//==============omit====================
#define FLAGS_BIT_FIELDS(V, _)//see line 70 above 2. JSFunction The function NewFunctionFromSharedFunctionInfo reads a SharedFunction and generates the corresponding JSFunction. The following is the source code: 1.  Handle<JSFunction> Factory::NewFunctionFromSharedFunctionInfo(
2.      Handle<SharedFunctionInfo> info, Handle<Context> context,
3.      AllocationType allocation) {
4.    Handle<Map> initial_map(
5.        Map::cast(context->native_context().get(info->function_map_index())),
6.        isolate());
7.    return NewFunctionFromSharedFunctionInfo(initial_map, info, context,
8.                                             allocation);
9.  }
10.  //==============分隔线============
11.  Handle<JSFunction> Factory::NewFunction(Handle<Map> map,
12.                                          Handle<SharedFunctionInfo> info,
13.                                          Handle<Context> context,
14.                                          AllocationType allocation) {
15.    Handle<JSFunction> function(JSFunction::cast(New(map, allocation)),
16.                                isolate());
17.    function->initialize_properties(isolate());
18.    function->initialize_elements();
19.    function->set_shared(*info);
20.    function->set_code(info->GetCode());
21.    function->set_context(*context);
22.    function->set_raw_feedback_cell(*many_closures_cell());
23. //........omit...............
24.  } In the above code, line 7 calls the NewFunction(). Here, I give you some details about the NewFunction. Lines 17–18, they describe some properties and elements that a SharedFunction doesn’t have. Line 21, it is the bound context that I mentioned above. Lines 19–20, they set the corresponding SharedFunction and Builtins code into the JSFunction. Okay, a SharedFunction can be shared by multiple instances and a JSFunction is an executable instance. Okay, that wraps it up for this share. I’ll see you guys next time, take care! Please reach out to me if you have any issues. : qq9123013 : WeChat Email v8blink@outlook.com Also published . here

Understanding Chrome V8 - Chapter 24: How does V8 Describe Your JavaScript Function

A decision Tree is a supervised machine learning algorithm that works on the basis of recursively answering some questions (if-else conditions). The algorithm is used both for regression and classification. However mostly for classification problems. The questions in boxes are called the internal nodes where the answers to the questions split it into branches. The nodes which do not split anymore are called leaf, which represent the decision/output of the model. This tree of course is much bigger and more complex for bigger datasets when compared to our simple example above. The tree grows and forms according to the data we provide to it (training the model). However, this simple diagram also shows how simple actually the algorithm works. You can already imagine that to be able to split the data properly, you need to ask the right questions starting from the top node. This means which features and what conditions to use are crucial to building a good performing decision tree. Well, how is this possible? Firstly, the root node feature is selected based on the results from the Attribute Selection Measure(ASM). Afterward, ASM is applied to all nodes emerging recursively until no more split is possible (when we reach the leaf). Attribute Selective Measure (ASM) Attribute Subset Selection Measure is a method used for data reduction. The data reduction is necessary to make better analysis and prediction of the target variable. This is how the decision tree chooses the nodes to make the best splits for the data. The two main ASM techniques: Gini index Information Gain (ID3) Gini Index Gini index or Gini impurity is a measure of impurity (degree of probability of the feature being classified incorrectly) used for creating the decision tree. A feature with low Gini index value should is preferred for the decision of nodes while creating the decision tree. Gini index is used to create only binary splits in the tree. Information Gain (ID3) Information Gain tells how informative a feature is by measuring the changes in the entropy after splitting the data on that feature. The decision tree algorithm always tries to maximize the Information Gain, in which the node with the highest information gain is chosen as the first node (first split). Therefore the tree is first split by the feature with the highest entropy, decreasing the entropy all the way down the leafs. The entropy formula is: We can use the DecisionTreeClassifier model from the scikit learn library (DecisionTreeClassifier documentation): Let’s use the cancer dataset from scikit-learn library and apply the Decision Tree model: # Import train_test_split function and the dataset
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier

cancer = load_breast_cancer()
y = cancer.target
X = cancer.data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf = DecisionTreeClassifier(criterion='entropy', random_state=0)
clf.fit(X_train, y_train)

print("Training accuracy:{:.2f}".format(clf.score(X_train,y_train)))
print("Test accuracy: {:.2f}".format(clf.score(X_test,y_test))) Our training accuracy being higher than our test accuracy shows us that our model is overfitting to the data. Let’s plot our decision tree and examine it’s complexity. from sklearn import tree
import matplotlib.pyplot as plt

fig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (16,8), dpi=100)
tree.plot_tree(clf, feature_names = cancer.feature_names, class_names=cancer.target_names, filled = True, fontsize = 5); Changing the asm criterion from gine to entropy: clf = DecisionTreeClassifier(criterion='entropy', random_state=0)
clf.fit(X_train, y_train)

print("Training accuracy:{:.2f}".format(clf.score(X_train,y_train)))
print("Test accuracy: {:.2f}".format(clf.score(X_test,y_test))) Our test accuracy improved while using the entropy attribute selection measure as the splitting criterion. We can have a look at the tree again to see if there are any changes to the splits done. fig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (16,8), dpi=100)
tree.plot_tree(clf, feature_names = cancer.feature_names, class_names=cancer.target_names, filled = True, fontsize = 5); As we can see, our tree is pretty complex which is resulting in overfitting the data. We can reduce the complexity of the tree by providing a maximum depth (max_depth) and prevent overfitting. clf = DecisionTreeClassifier(max_depth=3)
clf.fit(X_train, y_train)

print("Training accuracy:{:.2f}".format(clf.score(X_train,y_train)))
print("Test accuracy: {:.2f}".format(clf.score(X_test,y_test))) By reducing the maximum depth of our decision tree to three, we were able to decrease overfitting and increase our test accuracy slightly. If we examine our tree diagram now, we will be seeing a much simpler tree… By setting the max_depth equal to 3, we reduced the complexity of the Decisions tree by pruning it. This pruned model is less complex and a little easier to understand in comparison to the previous model where the tree kept splitting until all leaves are pure (gini impurtiy = 0). Decision Tree Regression Let’s have a look at how the decision tree works for a regression problem. For this we can again generate a random dataset: import numpy as np
np.random.seed(5)

X = np.sort(5 * np.random.rand(40, 1), axis=0)
y = np.sin(X).ravel()
# Add noise to targets
y[::5] += 1 * (0.5 - np.random.rand(8)) The scikit-learn libary provides us with a decision tree model for regression called DecisionTreeRegressor: from sklearn.tree import DecisionTreeRegressor

dt_reg = DecisionTreeRegressor(criterion="squared_error", random_state=0)
dt_reg.fit(X, y)

fig, axes = plt.subplots(nrows = 1, ncols = 1, figsize=(16,8), dpi=100)
tree.plot_tree(dt_reg, feature_names='X', filled=True, fontsize=5); #generate a random Test data
T = np.linspace(0, 5, 100)[:, np.newaxis]

#creating two regression trees with different depths
dt_reg_1 = DecisionTreeRegressor(max_depth = 10, random_state=0)
dt_reg_2 = DecisionTreeRegressor(max_depth = 3, random_state=0)

#training the models
dt_reg_1.fit(X, y)
dt_reg_2.fit(X, y)

#making predictions for the random test data we generated above
y_pred_1 = dt_reg_1.predict(T)
y_pred_2 = dt_reg_2.predict(T)

#comparison plot to see the effect of tree depth
plt.figure()
plt.scatter(X, y, s=40, c="orange", label="actual")
plt.plot(T, y_pred_1, color="b", label="max_depth=10", linewidth=2)
plt.plot(T, y_pred_2, color="g", label="max_depth=3", linewidth=2)
plt.xlabel("X")
plt.ylabel("y")
plt.title("Decision Tree Regression")
plt.legend()
plt.show() Looking at the figure, we can see how the decision tree regressor model with max depth set to ten, overfits the data capturing all the noise in the data. Moreover, the tree with max depth set to three is much better in generalizing and creating a better fit to the data without capturing all the noise. Advantages Easy to understand, interpret and visualize Usually, no feature scaling or normalization and feature selection are needed Functions well with multiple data types (categorical, numerical, binary) in the dataset (easier data preprocessing) It is also suitable for multi-output problems Disadvantages Decision Trees tend to overfit and do not generalize very well Mostly need an ensemble of trees of other models for better generalization performance Decision trees tend to form biased trees if there is data imbalance. Datasets with dominating classes should be balanced. If you like this tutorial, feel free to follow me for more free machine-learning tutorials and courses! Also Published Here

What are Decision Trees in Machine Learning?

Scikit-learn is the most popular open-source and free python machine learning library for and Machine learning practitioners. The scikit-learn library contains a lot of efficient tools for machine learning and statistical modeling including classification, regression, clustering, and dimensionality reduction. Data scientists In this article, I’m happy to share with you the top 5 new features presented in the new version of scikit-learn (1.0). TABLE OF CONTENTS Install Scikit-learn v1.0 New Flexible Plotting API Feature Names Support Pearson’s R Correlation Coefficient OneHot Encoder Improvements Histogram-based Gradient Boosting Models are now stable Install Scikit-learn v1.0 Firstly, make sure you install the latest version (with pip): pip install --upgrade scikit-learn If you are using conda, use the following command: conda install -c conda-forge scikit-learn Note: Version 1.0.0 of scikit-learn requires python 3.7+, NumPy 1.14.6+ and scipy 1.1.0+. Optional minimal dependency is matplotlib 2.2.2+ Now, let’s look at the new features! 1. New Flexible Plotting API Scikit-learn 1.0 has introduced new flexible plotting API such as metrics.PrecisionRecallDisplay, metrics.DetCurveDisplay, and inspection.PartialDependenceDisplay. This Plotting API comes with two class methods: (a) from_estimator() This class method allows you to fit a model and plot the results at the same time. Let's look at an example by using PrecisionRecallDisplay to visualize Precision and Recall. matplotlib.pyplot plt sklearn.datasets make_classification sklearn.metrics PrecisionRecallDisplay sklearn.model_selection train_test_split sklearn.ensemble RandomForestClassifier
 
X, y = make_classification(random_state= )
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size= )
                                                    
classifier= RandomForestClassifier(random_state= )
classifier.fit(X_train, y_train)
 
disp_confusion = PrecisionRecallDisplay.from_estimator(classifier,
                                                       X_test,
                                                       y_test) 
                                    
                                                    
plt.show() import as from import from import from import from import 42 0.2 42 (b) from_predicitons() In this class method, you can just pass prediction results and get your plots. Let's look at an example by using ConfusionMatrixDisplay to visualize the confusion matrix. matplotlib.pyplot plt sklearn.datasets make_classification sklearn.metrics confusion_matrix, ConfusionMatrixDisplay sklearn.model_selection train_test_split sklearn.ensemble RandomForestClassifier
 
X, y = make_classification(random_state= )
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size= )
                                                    
classifier= RandomForestClassifier(random_state= )
classifier.fit(X_train, y_train)
 
predictions = classifier.predict(X_test)
    
disp_confusion = ConfusionMatrixDisplay.from_predictions(predictions,
                                                       y_test,
                               display_labels=classifier.classes_) 
                                    
                                                    
plt.show() import as from import from import from import from import 42 0.2 42 2. Feature Names Support (Pandas Dataframe) In the new version of scikit-learn, you can track the names of the columns of your pandas dataframe when working with transformers or estimators. When you pass a dataframe to an estimator and call the fit method, the estimator will store the features name in the attribute. feature_names_in_ sklearn.preprocessing StandardScaler pandas pd
 
X = pd.DataFrame([[ , , ], [ , , ]], columns=[ , , ])
scalar = StandardScaler().fit(X)
 
print(scalar.feature_names_in_) from import import as 1 2 3 4 5 6 "age" "days" "duration" array(['age', 'days', 'duration'], dtype=object) feature names support is only enabled when the column names in the dataframe are all strings. Note: 3. Pearson’s R Correlation Coefficient This is a new feature in that can measure the linear relationship between each feature and the target for the regression tasks. It is also known as the pearson’s r. feature selection The cross-correlation between each regressor and the target is computed as ((X[:, i] - mean(X[:, i])) * (y - mean_y)) / (std(X[:, i]) * std(y)). Where X is the features of the dataset and y is the target variable. Note: The following example shows how you can compute the Pearson’s r for each feature and the target. sklearn.datasets fetch_california_housing sklearn.feature_selection r_regression 
 
X, y = fetch_california_housing(return_X_y= )
 
print(X.shape)
 
p = r_regression(X,y) 
 
print(p) from import from import True (20640, 8) [ 0.68807521  0.10562341  0.15194829 -0.04670051 -0.02464968 -0.02373741  -0.14416028 -0.04596662] 4. OneHot Encoder Improvements The OneHot Encoder in scikit-learn 1.0 can accept values it has not seen before. You just need to set a parameter called handle_unknown to 'ignore' (handle_unknown='ignore') when instantiating the transformer. When you transform data with an unknown category, the encoded columns for this feature will be all zero values. In the following example, we pass an unknown category when we transform the data given. sklearn.preprocessing OneHotEncoder
 
enc = OneHotEncoder(handle_unknown= )
 
X = [[ ], [ ], [ ]]
 
enc.fit(X)
 
transformed = enc.transform([[ ], [ ],[ ]]).toarray()
 
print(transformed) from import 'ignore' 'secondary' 'primary' 'primary' 'degree' 'primary' 'secondary' [[0. 0.] [1. 0.] [0. 1.]] In the inverse transform, an unknown category will be labeled as None. Note: 5. Histogram-based Gradient Boosting Models are now Stable The two supervised learning algorithms introduced in the previous version of scikit-learn 0.24 (HistGradientBoostingRegressor and HistGradientBoostingClassifier) are no longer experimental and you can simply import and use them as: sklearn.ensemble HistGradientBoostingClassifier, HistGradientBoostingRegressor from import There are more new features in scikit-learn 1.0.0 that I did not mention in this article. You can find the highlights of other features released in scikit-learn 1.0.0 . here Congratulations, you have made it to the end of this article! I hope you have learned something new that will help you on your next machine learning project. If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter @Davis_McDavid. And you can read more articles like this . here

Scikit Learn 1.0: New Features in Python Machine Learning Library

Machine learning models represent the learning output from a machine in such a way that, it can be used in the future to predict or understand similar kinds of data by which the model had been trained. In the process of building a Machine Learning model, there is a trade-off between bias and variance. We all know this. But what exactly does it mean when we say a model has a high bias or high variance? Can we visualize what is happening? I have explored these questions using python and some basic visualization libraries. We will be using NumPy for data generation and calculations. Matplotlib and Seaborn are useful for visualizing the generated data points and predicted data points. We will be using Scikit-learn for model building and Pandas for handling the data. # Importing Libraries
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set()
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression What are bias and variance in simple terms? In most real-world problems, we don't really know what is the true relation between independent variables(features) and the dependent variable(target variable). In statistical learning, we try to estimate this true function using various models. Since it is not possible to estimate the true function accurately, this will induce error in the predictions. This error can be broken down into 3 categories: . bias, variance, and irreducible error Bias is the error that is caused by the model when we try to approximate a complex real-world problem using a simple model. Linear regression often has a high bias since we assume a linear relation which is a simple one. Because of this reason, we will use Linear regression as one of our models to visualize. Variance is the amount by which the model would change if we use different sample data for model training. A decision tree is a model that has high variance but low bias. Decision trees make almost no assumption about the true function. But if we change the training data, the model will vary too much. The decision tree is the second model that we use for our prediction. Irreducible error is caused because of some unknown reason. In terms of features, we can say that there can be some other factors that affect the target variable which is not a part of our available features. Generating data We will use NumPy to create our feature X which lies between -4 and 4. We are generating our target variable Y as a sine function of X. But we know that there can be an irreducible error which we won't be able to learn from our available features. In order to simulate this irreducible error, we randomly generate very small values and add them to Y. n = 2000
np.random.seed(63)
X = np.linspace(-4,4,n)                   # generating data with noise
Y = np.sin(X) + 0.4*np.random.randn(n)    # for sine function We can visualize the generated data using a scatter plot.\ plt.figure(figsize=(12,6))
plt.scatter(X,Y)
plt.xlabel('X')
plt.ylabel('Y')
plt.show() Note that, we are using only one feature variable so that it will be easier to visualize. But in real-world problems, we will have many features and they cannot be visualized. Our objective here is to see what is happening in terms of bias and variance. Creating subsets from the generated data Now we will store this data as a Pandas data frame and make 5 subsets of this data by randomly sampling 100 indices of our data without replacement. We are performing random sampling without replacement to make sure that each of these subsets has different data. One important thing that we should realize is, out of the 2000 data points that we generated, we are only using 100 data points for each subset. We are doing this to make sure that, even though the data in each subset is different, each of these subsets should be a good representation of our population of 2000 data points. df = pd.DataFrame({'X':X,'Y':Y}) # Dataframe that contains our population

# creating subsets
np.random.seed(63)
X1 = df.iloc[np.random.choice(range(2000),100,replace=False),:]
X2 = df.iloc[np.random.choice(range(2000),100,replace=False),:]
X3 = df.iloc[np.random.choice(range(2000),100,replace=False),:]
X4 = df.iloc[np.random.choice(range(2000),100,replace=False),:]
X5 = df.iloc[np.random.choice(range(2000),100,replace=False),:] Building the models We can build 5 different linear regression models using the 5 subsets that we have created. Similarly, we can also build 5 different decision tree models. Our objective is to build a five linear regression model and visualize its prediction in the interval of -4 to 4(interval of our population). To make this process easier, we can create a python function that will take the training data as input and return the prediction for 2000 evenly spaced numbers between -4 to 4. # For Linear regressi/on
def LR_gen(data):
    lr1 = LinearRegression()
    lr1.fit((data['X'].values).reshape(-1,1),data['Y'])
    prediction = lr1.predict(np.linspace(-4,4,2000).reshape(-1,1))
    return prediction

# For Decision tree regressor
def DT_gen(data):
    dt1 = DecisionTreeRegressor()
    dt1.fit((data['X'].values).reshape(-1,1),data['Y'])
    prediction = dt1.predict(np.linspace(-4,4,2000).reshape(-1,1))
    return prediction Visualizing the predictions Now we can simply train a model using a subset of data and get the predictions using the function that we defined. We can visualize these 5 predictions along with the ground truth in one plot. Linear Regression models plt.figure(figsize=(12,6))
for i,x in enumerate([X1,X2,X3,X4,X5]):
    plt.plot(np.linspace(-4,4,2000),LR_gen(x),label=f'Prediction_{i+1}')
plt.scatter(df['X'],df['Y'],label='Ground Truth',color='b',alpha=0.1)
plt.legend()
plt.show() What does the plot say? From the plot, we can make two observations: All the linear regression models differ very much from the ground truth/true values. All the five linear regression models are close to each other. The first observation tells us about the high bias of linear regression. That is, in reality, our true function is a non-linear one. Since we can't possibly know this, we assumed that the true function is linear and build linear regression models. This resulted in models that predict very poorly. If we take the average of all 5 predictions from the models and compare it with the true function, it will have a high difference. The second observation tells us about the low variance of linear regression models. All 5 models are close to each other. This implies that, when we change the training data, the model will get affected very little. mean_predictions = []      # To store predictions from 5 models
for i in [X1,X2,X3,X4,X5]:
    lr1 = LinearRegression()
    lr1.fit((i['X'].values).reshape(-1,1),i['Y'])
    prediction = lr1.predict((df['X'].values).reshape(-1,1))
    mean_predictions.append(prediction)
mean_prediction = np.mean(mean_predictions,axis=0) # average of all 5 models

# Plotting
fig, (ax1,ax2) = plt.subplots(2,1,figsize=(12,8),sharey=True)
# Plotting the average prediction of all the model and the true function
ax1.scatter(df['X'],mean_prediction,label='Mean prediction',color='green')
ax1.scatter(df['X'],df['Y'],label='Ground Truth',color='b',alpha=0.1)
ax1.title.set_text('Average prediction v/s Ground truth')
ax1.legend()
for i,x in enumerate([X1,X2,X3,X4,X5]):
    ax2.plot(np.linspace(-4,4,2000),LR_gen(x),label=f'Prediction_{i+1}')
ax2.scatter(df['X'],mean_prediction,label='Mean prediction',color='green')
ax2.title.set_text('Average prediction v/s models with different data')
ax2.legend()
plt.show() Decision tree models plt.figure(figsize=(12,6))
for i,x in enumerate([X1,X2,X3,X4,X5]):
    plt.plot(np.linspace(-4,4,2000),DT_gen(x),label=f'Prediction_{i+1}')
plt.scatter(df['X'],df['Y'],label='Ground Truth',color='b',alpha=0.1)
plt.legend()
plt.show() What does the plot say? From the plot, we can make two observations: All the decision tree models are closer to the ground truth(true values). All the five decision tree models vary drastically from each other. The first observation tells us about the low bias of the decision tree. That is, in reality, our true function is a non-linear one. Since we didn't make any assumption about the form of the function and built decision tree models, it was able to learn the underlying form up to an extent. This resulted in models that predict well. If we take the average of all 5 predictions from the models and compare it with the true function, it will be very close to the true function. The second observation tells us about the high variance of the decision tree model. All 5 models vary very much from each other. This implies that, if we change the training data the model will get affected too much. mean_predictions = []
for i in [X1,X2,X3,X4,X5]:
    dt1 = DecisionTreeRegressor()
    dt1.fit((i['X'].values).reshape(-1,1),i['Y'])
    prediction = dt1.predict((df['X'].values).reshape(-1,1))
    mean_predictions.append(prediction)
mean_prediction = np.mean(mean_predictions,axis=0)

# Plotting
fig, (ax1,ax2) = plt.subplots(2,1,figsize=(12,8))
ax1.scatter(df['X'],mean_prediction,label='Mean prediction',color='green')
ax1.scatter(df['X'],df['Y'],label='Ground Truth',color='b',alpha=0.1)
ax1.title.set_text('Average prediction v/s Ground truth')
ax1.legend()
for i,x in enumerate([X1,X2,X3,X4,X5]):
    ax2.plot(np.linspace(-4,4,2000),DT_gen(x),label=f'Prediction_{i+1}')
ax2.scatter(df['X'],mean_prediction,label='Mean prediction',color='green')
ax2.title.set_text('Average prediction v/s models with different data')
ax2.legend()
plt.show() Takeaways We are able to understand bias and variance visually. But in practice, we will have many features rather than just one. This makes it impossible to visualize and tune our model in order to attain reduced bias as well as variance. We can use various statistical methods during the model training phase to achieve minimum bias and variance. References: [1] Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An introduction to statistical learning: with applications in R (2013), Springer [2] Hunter, J. D, Matplotlib: A 2D graphics environment(2007), Computing in Science & Engineering, 9(3), 90–95. [3] Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., … Oliphant, T. E, Array programming with NumPy(2020), Nature, 585, 357–362. https://doi.org/10.1038/s41586-020-2649-2 [4] McKinney, W., & others, Data structures for statistical computing in Python(2010), In Proceedings of the 9th Python in Science Conference (Vol. 445, pp. 51–56). [5] Pedregosa, F., Varoquaux, Ga"el, Gramfort, A., Michel, V., Thirion, B., Grisel, O., … others, Scikit-learn: Machine learning in Python(2011), Journal of Machine Learning Research, 12(Oct), 2825–2830.

How to Visualize Bias and Variance

AI Image Synthesis: What The Future Holds

PyTorch Geometric Temporal is a deep learning library for neural spatiotemporal signal processing. This library is an open-source project. It consists of various dynamic and temporal geometric deep learning, embedding, and Spatiotemporal regression methods from a variety of published research papers. In addition, it comes with an easy-to-use dataset loader, an iterator for dynamic and temporal graphs, and gpu-support. It also comes with a number of benchmark datasets with temporal and dynamic graphs (you can also create your own datasets). In the following, we will overview a case study where l can be used to solve a real-world relevant machine learning problem. PyTorch Geometric Tempora Learning from a Discrete Temporal Signal We are using the Hungarian Chickenpox Cases dataset in this case study. We will train a regressor to predict the weekly cases reported by the counties using a recurrent graph convolutional network. First, we will load the dataset and create an appropriate spatio-temporal split. torch_geometric_temporal.data.dataset ChickenpoxDatasetLoader torch_geometric_temporal.data.splitter discrete_train_test_split

loader = ChickenpoxDatasetLoader()

dataset = loader.get_dataset()

train_dataset, test_dataset = discrete_train_test_split(dataset, train_ratio= ) from import from import 0.2 Next, we will define the architecture used for solving the supervised task. The constructor defines a DCRNN layer and a feedforward layer. It is important to note that the final non-linearity is not integrated into the recurrent graph convolutional operation. This design principle is used consistently and it was taken from PyTorch Geometric. recurrent graph neural network Because of this, we defined a ReLU non-linearity between the recurrent and linear layers manually. The final linear layer is not followed by a non-linearity as we solve a regression problem. In the next steps, we will define the architecture used for solving the supervised task. recurrent graph neural network torch torch.nn.functional F torch_geometric_temporal.nn.recurrent DCRNN super(RecurrentGCN, self).__init__()
        self.recurrent = DCRNN(node_features, , )
        self.linear = torch.nn.Linear( , ) h = self.recurrent(x, edge_index, edge_weight)
        h = F.relu(h)
        h = self.linear(h) h import import as from import : class RecurrentGCN (torch.nn.Module) : def __init__ (self, node_features) 32 1 32 1 : def forward (self, x, edge_index, edge_weight) return Let us define a model (we have 4 node features) and train the model on the training split (first 20% of the temporal snapshots) for 200 epochs. We backpropagate when the loss from every snapshot is accumulated. We will use the with a learning rate of . The function is used for measuring the runtime need for each training epoch. Adam optimizer 0.01 tqdm tqdm tqdm

model = RecurrentGCN(node_features = )

optimizer = torch.optim.Adam(model.parameters(), lr= )

model.train() epoch tqdm(range( )):
    cost = time, snapshot enumerate(train_dataset):
        y_hat = model(snapshot.x, snapshot.edge_index, snapshot.edge_attr)
        cost = cost + torch.mean((y_hat-snapshot.y)** )
    cost = cost / (time+ )
    cost.backward()
    optimizer.step()
    optimizer.zero_grad() from import 4 0.01 for in 200 0 for in 2 1 Using the holdout we will evaluate the performance of the trained recurrent graph convolutional network and calculate the mean squared error across . all of the spatial units and time periods model.eval()
cost = time, snapshot enumerate(test_dataset):
    y_hat = model(snapshot.x, snapshot.edge_index, snapshot.edge_attr)
    cost = cost + torch.mean((y_hat-snapshot.y)** )
cost = cost / (time+ )
cost = cost.item()
print( .format(cost)) Accuracy: 0 for in 2 1 "MSE: {:.4f}" >>> 0.6866 Previously published at https://pytorch-geometric-temporal.readthedocs.io/en/latest/notes/introduction.html

Machine-Learning Neural Spatiotemporal Signal Processing with PyTorch Geometric Temporal

Time-series or sequential data can be defined as any data that has time dependency. , huh, Well, a bit everywhere, from credit card transactions, my everyday routine and whereabouts to medical records, such as ECG and EEG’s. Cool but where can I find sequential data? Generate synthetic sequential data with TimeGAN Although sequential data is pretty common to be found and highly useful, there are many reasons that lead to not leverage it — from to the scarcity of its existence. In one of my previous posts, I’ve covered the ability of that preserves the utility and fidelity of a real datasets, nevertheless to generate tabular data is far more simple than generating datasets that should preserve privacy regulations Generative Adversarial Netoworks (GANs) to learn and generate new synthetic data temporal dynamics. To model successfully time-series data means that a model must, not only capture the datasets features distributions within each time-point but also, it should be able to capture the complex dynamics of those features across time. We must not forget also that each time sequence as a variable length associated. But being a challenging task, does not mean it is impossible! and have proposed, in 2019, — — that I’ll be covering with a practical example throughout this blog post. Jinsung Yoon Daniel Jarret a novel GAN architecture to model sequential data TimeGAN Time-series Generative Adversarial Networks TGAN or Time-series Generative Adversarial Networks, was proposed in 2019, as a GAN based framework that is able to generate realistic time-series data in a variety of different domains, meaning, sequential data with different observed behaviors. Different from other GAN architectures (eg. ) where we have implemented an unsupervised adversarial loss on both real and synthetic data, TimeGAN architecture introduces the concept of the model is encouraged to capture time conditional distribution within the data by using the original data as a supervision. WGAN supervised loss — Also, we can observe the introduction of an that is responsible to reduce the adversarial learning space dimensionality. embedding network block diagram and training scheme as depicted in the paper TimeGAN As mentioned above, TimeGAN is a framework to synthesize sequential data compose by 4 networks, that play distinct roles in the process of modelling the data: the expected and , but also, by a and models. generator discriminator recovery embedder Implementation with TensorFlow 2 What’s new about TimeGAN? Different from other the proposed framework is able to generate it’s training to handle a mixed-data setting, where both static and sequential data are able to be generated at the same time. GAN architectures for sequential data, (attributes) (features) Less sensitive to hyper parameters changes A more stable training process, when compared to other architectures. framework instantiated with RNNs For the purpose of this example, I’ve decided to keep it simple with a very similar architecture for all the 4 elements: A 3 layers network. But it’s possible to have this architectures change to more or less layers, and also to chose between and . TimeGAN GRU GRU LSTMs net_type== : i range(n_layers):
            model.add(GRU(units=hidden_units,
                      return_sequences= ,
                      name= )) : i range(n_layers):
            model.add(LSTM(units=hidden_units,
                      return_sequences= ,
                      name= ))

    model.add(Dense(units=output_units,
                    activation= ,
                    name= )) model : def make_net (model, n_layers, hidden_units, output_units, net_type= ) 'GRU' if 'GRU' for in True f'GRU_ ' {i + } 1 else for in True f'LSTM_ ' {i + } 1 'sigmoid' 'OUT' return Which results in the definition of each of the networks elements as per the code snippet below. self.hidden_dim=hidden_dim model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim) model : class Supervisor (Model) : def __init__ (self, hidden_dim) : def build (self, input_shape) 'Supervisor' 2 return self.hidden_dim = hidden_dim
        self.net_type = net_type model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim,
                         net_type=self.net_type) model self.hidden_dim = hidden_dim
        self.net_type=net_type model = Sequential(name= )
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units= ,
                         net_type=self.net_type) model self.hidden_dim=hidden_dim
        self.n_seq=n_seq recovery = Sequential(name= )
        recovery.add(Input(shape=input_shape, name= ))
        recovery = make_net(recovery,
                            n_layers= ,
                            hidden_units=self.hidden_dim,
                            output_units=self.n_seq) recovery self.hidden_dim=hidden_dim embedder = Sequential(name= )
        embedder.add(Input(shape=input_shape, name= ))
        embedder = make_net(embedder,
                            n_layers= ,
                            hidden_units=self.hidden_dim,
                            output_units=self.hidden_dim) embedder self.hidden_dim=hidden_dim model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim) model : class Generator (Model) : def __init__ (self, hidden_dim, net_type= ) 'GRU' : def build (self, input_shape) 'Generator' 3 return : class Discriminator (Model) : def __init__ (self, hidden_dim, net_type= ) 'GRU' : def build (self, input_shape) 'Discriminator' 3 1 return : class Recovery (Model) : def __init__ (self, hidden_dim, n_seq) return : def build (self, input_shape) 'Recovery' 'EmbeddedData' 3 return : class Embedder (Model) : def __init__ (self, hidden_dim) return : def build (self, input_shape) 'Embedder' 'Data' 3 return : class Supervisor (Model) : def __init__ (self, hidden_dim) : def build (self, input_shape) 'Supervisor' 2 return In what concerns the losses, the TimeGAN is composed by three: The , which refers to the auto-encoder (embedder & recovery), that in a nutshell compares how well was the reconstruction of the encoded data when compared to the original one. reconstruction loss The that, in a nutshell, is responsible to capture how well the generator approximates the next time step in the latent space. The , this one it’s already familiar to us, a it reflects the relation between the generator and discriminator networks (min-max game) supervised loss unsupervised loss Given the architecture choice and the defined losses we have three training phases: Training the autoencoder on the provided sequential data for optimal reconstruction; Training the supervisor using the real sequence data to capture the temporal behavior of the historical information, and finally; The combined training of four components while minimizing all the three loss functions mentioned previously. The full code detailing the training phases can be found at . ydata-synthetic The original implementation of TimeGAN can be found using TensorFlow 1 here . Synthetic stock data The data used to evaluate the synthetic data generated by the framework, refers to . The data has : Open, High, Low, Close, Adj Close and Volume. TimeGAN Google stock data 6 time dependent variables Prior to synthesize the data we must, first, ensure some preprocessing: Scale the series to a range between [0,1]. For convenience, I’ve decided to leverage scikit-learn’s MinMaxScaler; Create rolling windows — following the original paper recommendations, I’ve create rolling windows with overlapping sequences of 24 data points. ori_data = data[:: ] scaler = MinMaxScaler().fit(ori_data)
    ori_data = scaler.transform(ori_data) temp_data = [] i range( , len(ori_data) - seq_len):
        _x = ori_data[i:i + seq_len]
        temp_data.append(_x) idx = np.random.permutation(len(temp_data))
    data = [] i range(len(temp_data)):
        data.append(temp_data[idx[i]]) data # Method implemented here: https://github.com/jsyoon0823/TimeGAN/blob/master/data_loading.py # Originally used in TimeGAN research : def real_data_loading (data: np.array, seq_len) """Load and preprocess real-world datasets.
    Args:
      - data_name: Numpy array with the values from a a Dataset
      - seq_len: sequence length
    Returns:
      - data: preprocessed data.
    """ # Flip the data to make chronological data -1 # Normalize the data # Preprocess the dataset # Cut data by sequence length for in 0 # Mix the datasets (to make it similar to i.i.d) for in return Following the recommendations from the original paper, I’ve decided to train the synthesizer for 10000 iterations nevertheless, bare in mind, that these values must the optimized for each data set in order to return optimal results. You can find in the full flow including the data download, processing and synthesizing. this notebook Measuring synthetic data fidelity and utility Now that we were able to synthesize our data, it’s time to check whether the new data is able to reproduce properly the behavior observed in the original stock data data. Visual comparison One of my favorites ways to compare real with synthetic data is through visualization. Of course that in terms of automation is not the ideal to validate the quality of the new synthetic data, but it gives us already a pretty good idea. To ensure a 2D visualization of the results, it was applied both a and a with 2 components. TSNE PCA The results are pretty promising, as we see an almost perfect overlap between the synthetic and the real data points. Synthetic data utility — Train synthetic test real There are a plenty of different metrics that can be used to measure the utility of the data such as the . SRA and the TSTR In this example we’ve decided to go for the to validate the utility of our synthetic data. To be used as a regressor model, I’ve decided for a simple architecture with a single layer GRU with 12 units. TSTR method opt = Adam(name= )
    loss = MeanAbsoluteError(name= )
    model = Sequential()
    model.add(GRU(units=units,
                  name= ))
    model.add(Dense(units= ,
                    activation= ,
                    name= ))
    model.compile(optimizer=opt, loss=loss) model : def RNN_regression (units) 'AdamOpt' 'MAE' f'RNN_1' 6 'sigmoid' 'OUT' return Prior to train the model, we had to prepare the inputs: 75% of the data was used as train data (for both synthetic and real datasets) 25% of the real data was used as the test set. For both the model trained on synthetic and the one trained on the real data, I’ve used as the validation set loss as the early stopping criteria. Below are the summarized results obtained for the test set. Conclusion The results obtained with this experiment are very promising and exciting in what concerns the generation of synthetic sequential data. Nevertheless, there are a few caveats that are needed to be pointed: the data under study can be considered to be quite simple, as it was relatively small, with a daily time frequency, there were no surprises in terms of missing values and the dimensionality (columns wise) was relatively low. Not to mention, the very intensive training time. But without a doubt, Generative Adversarial Networks are impressive frameworks, that can be leveraged for much more than synthetic data generation. For those of you, that are looking to keep exploring synthetic data generation have a look into our . We’ll be updating it with new generative algorithms as well as new data, and we invite you to collaborate! GitHub repository Fabiana Clemente is CDO at YData . Improved and synthetic data for AI. YData provides the first dataset experimentation platform for Data Scientists. Also published here

A GAN approach To Synthetic Time-Series Data

Multi-output Machine Learning — MixedRandomForest Introduction Multi-output learning subsumes many learning problems in multiple disciplines and deals with complex decision-making in many real-world applications. It has a multivariate nature and the multiple outputs may have complex interactions. Many strucurted inference problems have been architected by multi-output machine learning. The output values have diverse data types, depending on the type of ML problem. For example, 0/1 based output values can refer to a multi-label classification problem. Such examples also range from : Binary Nominal output values to a multi-dimensional classification problem Ordinal output values to label ranking problem Real-valued outputs to a multi-target regression problem. Special Use Cases of Multi-output Learning : Multi-class classification can be categorized as a traditional single-output learning paradigm when the output class is represented by the integer encoding. It can also be extended to a multi-output learning scenario if each output class is represented by the one-hot vector. Multi-class Classification : In this type of classification, though the vector representation is the same as fine-grained classification outputs to the multi-class classification outputs, their internal structures of the vectors are different. Labels under the same parent tend to have a closer relationship than the ones under different parents in the label hierarchy. Fine-grained Classification : Multi-task learning aims at learning multiple related tasks simultaneously, where each task outputs one single label, and learning multiple tasks is similar to learning multiple outputs. It leverages the relatedness between tasks to improve the performance of learning models. The major difference between multi-task learning and multi-output learning is that different tasks might be trained on different training sets or features in multi-task learning, while output variables usually share the same training data or features in multi-output learning. Multi-task Learning In , each instance in the dataset has two or more output values (nominal or real-valued)— i.e., the output value is a vector rather than a scalar. They are solved by any of the following methods: Multi-output pattern recognition problems By transforming the multi-label (or multi-output) into multiple single-output problems. By adopting a so that it directly handles multi-output data. pattern recognition algorithm However, there are certain pros and cons to the above-mentioned approaches: The first approach of training an inductive classifier or regression model can be a — particularly so when training data sets are very large. When multiple models need to be trained using the same input data — but with different output data — the training time is unusually too high making it unsuitable for large datasets. Consequently, this also impacts the processing requirements. time-consuming task The second adaption approach enables to create a model that simultaneously predicts a set of two or more classification labels, regression values, or even outputs from only a single training iteration. If the prediction tasks are related (i.e., there is a correlation or covariance between output values), training a coherent multi-output model can potentially bring benefits in the form of increased predictive performance compared to training . joint classification-regression multiple disjoint models Here in this blog, we discuss a model, that supports : Mixed/Multi-target RandomForest "multi-output problems with multiple classification outputs, multiple regression outputs, as well as arbitrary joint classification-regression outputs”. Further, the algorithm provides support for mixed-task multi-task learning, i.e., it is possible to train the model on any number of classification tasks and regression tasks, simultaneously. The Random Forest predictor lets each individual ensemble member vote for the most probable output according to its learned decision rule. The ensemble members’ votes are tallied and aggregated, as a combined classifier — with and — to yield a mode for classification mean for regression common ensemble output. Multi-Output vs Multi-Label Classification In multi-output classification, the goal is to learn a classification rule whose output is a set, or vector, of labels i.e. y1 belong to Y1, y2 belonging to Y2, yn belonging to Yn and v, the vector is composed of y1, y2, y3, …yn. Multi-label classification involves classifying instances into several labels that share semantics, for example, the problem of classifying songs according to their genre — it is possible to classify a dance as either ballet or traditional, but also possible to classify a dance as both (i.e. ”ballet traditional”). Here, pop and rock share semantics: they both relate to the songs’ genre, and are thus two different values of the same label. There is also no a priori knowledge regarding the size of the output — it is very possible that a song cannot be classified as any previously known genre, or that it is best classified as several different genres. The problem of multi-output classification is effectively the opposite of multi-label classification — the output values do not share semantics, but the number of outputs is known a priori. In a classification problem where the goal is to simultaneously predict temperature (low, medium, or high) and pressure (low, medium, or high) inside a pressure cooker — in this case, the model is expected to output exactly two values, one value for the temperature label and another for the pressure label. The machine learning task of solving a multi-output problem thus involves building a predictive model that simultaneously outputs a set of (two or more) labels that measure different concepts — essentially two or more separate (although related) classification problems are solved concurrently within the same model. A multi-output classification is multitask-classification — which illustrates the fact that a multi-output classification problem is effectively equivalent to multiple simultaneous (multi-tasked) single-label classification problems. Multi-label problems can be transformed into multi-output problems, the opposite is not necessarily true. Joint Classification-Regression Problems In the multi-output problem containing both classification tasks and regression tasks, solving unrelated problems need not be more difficult than training a set of classifiers and regressors on the individual tasks. If the tasks are related, the algorithm adaption method can be used to provide the best results in terms of predictive performance. joint classification-regression can be solved with a tree induction algorithm that simultaneously solves one classification task and one regression task. Much like MT-DT and MRT, the joint classification-regression tree (JCRT) solves multiple simultaneous prediction tasks by modifying the node-split function in the inductive step and marking terminal nodes with appropriate values for each task. Due to the nature of joint classification-regression problems, the modified split function is required to consider the error of both the classification part and the regression part simultaneously. Joint Classification-Regression Trees The split function uses an entropy function consisting of three parts: Shannon entropy is computed for the classification part. A weighted differential entropy is calculated for the regression part Shannon entropies and differential entropies exist in different ranges, hence a normalization step is applied to combine the two entropies. Joint classification-regression forests, when evaluated on spatially structured data in the form of CT scans perform two tasks are: Classify pixels according to objects (organs) Estimate the distance from each pixel to the object’s boundary. Research results demonstrate that joint forests are not only able to provide accurate estimates of object boundaries, but also improve the accuracy of the classification of objects. Reference Implementation Classification-Regression with MixedRandomForest The below example has referred Github source https://github.com/donlnz/morfist morfist MixedRandomForest, cross_validation sklearn.ensemble RandomForestRegressor, RandomForestClassifier sklearn.model_selection cross_val_score sklearn.datasets dst numpy np n_trees = x_reg, y_reg = dst.load_linnerud(return_X_y= )
x_cls, y_cls = dst.load_digits(return_X_y= )
x_mix_1, y_mix_1 = x_reg, np.vstack([y_reg, y_reg < y_reg.mean()]).T
x_mix_2, y_mix_2 = x_cls, np.vstack([y_cls, y_cls]).T from import from import from import import as import as # Config 20 True True The available default set of hyper-parameters for MixedRandomForest are: n_estimators= ,
max_features= ,
min_samples_leaf= ,
choose_split= ,
class_targets= 10 'sqrt' 5 'mean' None The below code snippet demonstrates comparative results between Scikit-learn’s RandomForest’s Classifier and Multi-output MixedRandomForest. cls_rf = MixedRandomForest(
   n_estimators=n_trees,
   min_samples_leaf= ,
   class_targets=[ ]
)

cls_skrf = RandomForestClassifier(n_estimators=n_trees)

cls_scores = cross_validation(
   cls_rf,
   x_cls,
   y_cls,
   class_targets=[ ],
   folds= )

scores = cross_val_score(
   cls_skrf,
   x_cls,
   y_cls
)

print( )
print( .format(cls_scores.mean()))
print( .format(scores.mean())) 1 0 0 10 'Classification with Single output: ' '\t morfist (accuracy): {}' '\t scikit-learn (accuracy): {}' Results morfist (accuracy): scikit-learn (accuracy): 0.9632721202003339 0.928187335049821 The below code snippet demonstrates comparative results between Scikit-learn’s RandomForestRegressor and Multi-output MixedRandomForest. reg_rf = MixedRandomForest(
   n_estimators=n_trees,
   min_samples_leaf= )

reg_skrf = RandomForestRegressor(n_estimators=n_trees)

reg_scores = cross_validation(
   reg_rf,
   x_reg,
   y_reg,
   folds= )
scores = cross_val_score(
   reg_skrf,
   x_reg,
   y_reg,
   scoring= )
print( )
print( .format(reg_scores.mean()))
print( .format(np.sqrt(-scores.mean()))) 5 10 'neg_mean_squared_error' 'Multivariate Regression multiple outputs: ' '\t morfist (rmse): {}' '\t scikit-learn (rmse): {}' Results morfist (rmse): 11.758534341303097
scikit-learn (rmse): 17.79445305492162 The below code snippet demonstrates multiple output labels on a classification dataset. mix_rf = MixedRandomForest(
   n_estimators=n_trees,
   min_samples_leaf= ,
   class_targets=[ ]
)
mix_scores = cross_validation(
   mix_rf,
   x_mix_2,
   y_mix_2,
   folds= ,
   class_targets=[ ]
)
print( )
print( .format(mix_scores[ ]))
print( .format(mix_scores[ ])) 1 0 10 0 'Mixed output on Classification Dataset: ' '\t Task 1 (original) (accuracy): {}' 0 '\t Task 2 (additional) (rmse): {}' 1 Results Mixed output on Classification Dataset:
task (original) (accuracy): task (additional) (rmse): 1 0.9627156371730662 2 1.117109567700808 Other Python Libraries : A python-based multi-output stream/batch learning framework, can be used within Jupyter Notebooks and can be used with scikit-learn. The following figure depicts the multi-output capabilities of different java and python libraries. scikit-multiflow Multi/Single output learning frameworks, Source Performance Evaluation of Multi-Output Learning Label identification and evaluation are one of the primary steps to quantify the quality of labels and label representations. It also plays a key role in the performance of multi-output tasks. Learning models with different multi-outputs can be used to determine and then improve upon the label quality in terms of different tasks. Labels can be evaluated from three different perspectives. To add valid checks in order to determine whether the annotation has good quality (Step A).To determine and infer on the possible labels, so as to conclude how well the chosen label representation can actually represent the labels (Step B).To determine the coverage, so as to evaluate how the provided label set well covers the dataset (Label Set). After the evaluation, there occurs a human intervention when the human expert explores and addresses the underlying issues, and provides feedback to improve different aspects of labels accordingly. Real-World Applications of Multi-output Learning The sample figure below explains the real application of different labels applied to an image in the social network. These labels correspond to varying output structures fitted for multi-output learning. Reference Independent vector is the vector with independent dimensions, where each dimension represents a particular label that does not necessarily depend on other labels. This includes tags, attributes, bag-of-words, bad-of-visual-words, hash codes and etc. of a given data. Independent Vector: Provides the information of probability distribution for each dimension, like a tag with the largest weight. Distribution: It shows the tags ordered from the most important to the least. Examples of its application are text categorization ranking, question answering, and visual object recognition. Ranking: : Text can be in the form of keywords, sentences, paragraphs or even documents. Applications for text outputs can be document summarization and paragraph generation. Text : Sequence (used in speech recognition, language translation) is usually a sequence of elements selected from a label set or word set. Each element prediction is dependent on past predicted outputs and present input. An output sequence often corresponds with an input sequence. Sequence : The Tree is represented as a hierarchical labeled structure to display the outputs. The outputs have the hierarchical internal structure where each output belong to a label as well as its ancestors in the tree, useful in syntactic parsing. Tree : One of the output objects are images consisting of multiple pixel values. A single pixel is predicted depending on the input and the pixels around it to consider an overall region prediction. Image output applications include super-resolution construction, text-to-image synthesis, which generates images from natural language descriptions, and face generation. Image The bounding box is often used to find the exact locations of the objects that appeared in an image and it is commonly used in object recognition and object detection Bounding Box: : A partitioned social network with edges represents the friendship of the users, the goal is to predict whether two currently unlinked users will be friends in the future. Link : A graph made up of a set of nodes and edges and it is used to model the relations between objects, where each object is represented by a node. The connected objects are linked by an edge. Graph : Contour and polygons are similar to the bounding box which can be used to localize objects in an image. In information retrieval, the output can be a list of data objects that are similar to the given query. In image segmentation, the output is usually segmentation masks for different objects, used for detecting common saliency on the multiple images. Others Conclusion The key motivation for solving multi-output pattern recognition problems using algorithm adaption is that a single model trained on a set of related tasks will show an improvement in predictive performance as compared to a set of individual models, each trained on a single task. It is seen that MixedRandomForest gives when applied on Scikit-learn’s dataset — better accuracy for Classification and better RMSE for Regression linnerud dataset (multivariate regression), and digits dataset (classification). But MixedRandomForest takes more time to run, than scikit-learn. There’s a slight increase in accuracy on the multi-output classification dataset. You can read  more on the deep learning mechanisms of Keras’s multi-output at and classification / https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses at also provides a method to evaluate the multi-output regression model. multi-label classification https://www.pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/ skflow-tensorflow References https://github.com/donlnz/morfist https://pdfs.semanticscholar.org/4219/f87ed41c558d43cf78f63976cf87bcd7ebb0.pdf https://arxiv.org/pdf/1901.00248.pdf http://www.jmlr.org/papers/volume19/18-251/18-251.pdf https://www.researchgate.net/publication/330102137_A_Survey_on_Multi-output_Learning

Stories, News & Mentions about REGRELLO

Regrello’s Stories on HackerNoon