ZenML GmbH, a startup working to simplify artificial intelligence development, today announced that it has raised $3.7 million in funding.

ZenML raises $3.7M for its open-source MLOps platform

The annual analytics contest, powered by Amazon Web Services (AWS), will focus on applying advanced data from Next Gen Stats to analyze tackling.

NFL selects tackling as theme for sixth annual Big Data Bowl competition

Cellarity Announces Partnership with Chan Zuckerberg Initiative to Advance New Machine Learning Algorithms for Single-Cell Analysis

The Kaggle AI Report for 2023 is a testament to the remarkable progress in the field of artificial intelligence. This annual report celebrates the growing global community of AI and machine ...

Kaggle AI Report 2023: A Deep Dive into AI Advancements

Responsible AI Institute (RAI Institute), the leading non-profit building AI Assessments and a Certification program dedicated to converting responsible AI principles into practice, today announced the appointment of Var Shankar as its new Executive Director.

Responsible AI Institute Announces Leadership Transition, Welcoming Var Shankar as Executive Director

A completed Kaggle submission. My leaderboard ranking has fallen precipitously since its peak in the Top 30. This has been a rewarding and exciting first foray into to the world of Kaggle ...

Data Mining for Gold on Kaggle

Seattle's Gable.ai launched out of stealth today, with $7 million in seed funding, to build the 'GitHub for data.'

With ‘GitHub for data,’ Gable.ai wants to connect software engineers and ML developers

Raises $7M in Seed funding, led by<br />Zetta Venture Partners, Crane Venture Capital and Essence Venture Capital; investors say Gable.a

Gable.Ai Unveils Data Collaboration Platform Set To Transform Analytics And Ai Workflows

Siddharth Singh has worked as a data scientist at Karmalife.AI since 2021. The Kaggle expert from Bangalore has been participating in hackathons for the past three years. MachineHack has been Singh’s go-to platform for competing with some of the greatest minds in data science.

Climate Crisis Meets Data Genius: Top 3 Winners Announced from Genpact-Google Hackathon

YuviPep is one of the leading companies worldwide, specialising in Innovation programmes. Its Innovation Base Course (IBC) is specifically designed for students in grades 1 to 12.

Bengaluru School Girl's Innovative Can Be A Blessing For Cancer Patients

Kaggle

kaggle

Leveraging SwarmUI & Stable Diffusion 3 on Cloud Platforms: A Guide to Kaggle (No-Cost), Massed Comp

If you are in the data science field or just interested in getting into it, this interview series is for you. In this episode, I interview Chris Deotte, one of the very few . This is a very prestigious title that most people in the data science world have dreamed of having or at least, recognize as “being successful”. Kaggle quadruple grandmasters This specific episode is for you if you’ve ever wondered how to build a strong data science resume, how to become a Kaggle grandmaster (even a quadruple one), what it’s like at NVIDIA, how to prepare for data science interviews, or how to tackle current data science challenges… about his current role at NVIDIA, where Chris is literally getting paid thanks to his Kaggle prowess! Learn more from Chris Deotte in this video interview P.s. this is the second episode in collaboration with NVIDIA GTC, in which they are giving away an RTX 4080 to a lucky listener! More details on how to participate in the video. https://youtu.be/NjGnnG3evmE?embedable=true You can also listen to this episode on Spotify or Apple Podcast where I will host all future podcast episodes… https://podcasters.spotify.com/pod/show/louis-bouchard/episodes/What-is-Data-Science-like-at-NVIDIA--With-Meriem-Bendris-and-RTX-4080-giveaway-e1vmmm4?embedable=true https://podcasts.apple.com/us/podcast/whats-ai-by-louis-bouchard/id1675099708?i=1000603382690&embedable=true

An Interview with Quadruple Kaggle Grandmaster Chris Deotte from NVIDIA

Hi, my name is and in this blog post, I share some tricks and tips to compete in Kaggle competitions and some code snippets which help in achieving results in limited resources. Here is . Prashant Kikani my Kaggle profile For Deep Learning competitions 1. TTA(test time augmentation) in inference similar to be done in training. TTA is making predictions on the same data-sample multiple times but each time, the sample will be augmented. So, the overall sample is the same, but we do some data augmentation similar to be done while training. Doing TTA is a common way to make our model predictions more robust & reduce the variance in our predictions. Therefore, ultimately improving the score. 2. Change image size at inference time While training a computer vision model, we often resize our images to 512x512 or 256x256 size. We do that to fit our data and model in the GPU memory. In most cases, we can't train the model with original high-resolution images. But, at inference time, we can keep the images in the original shape or in high resolution. Because we don't do/can't do back-propagation at inference time. Doing this helps because we are giving our model more relevant data in the form of those extra pixels. 3. Ensemble of multiple diverse models Ensemble is a technique in which we combine multiple diverse models (which are mostly trained on the same data) by using all the models at inference time. For example, averaging predictions of all the models on a test data-sample. The goal of ensemble is to reduce the biad and/or variance in our predictions. Here's a to learn more about ensemble with Python code. Also here is a about the ensemble. great notebook great blog In Kaggle, people share their codes with certain performances on the public leaderboard. What we can do is, we can train our own model and we can ensemble our model with that best public model. 4. Gradient Accumulation or effective batch size. Generally, GPU RAM becomes a hurdle train bigger models in robust manner. Kaggle GPU provided 16 GB of GPU memory. But in some cases, we can't fit higher in that RAM. Higher is good to train robust models. So, we can do gradient accumulation to make our effectively higher. Here's a sample code in PyTorch from this . batch_size batch_size batch_size gist model.zero_grad() i, (inputs, labels) enumerate(training_set):
    predictions = model(inputs) loss = loss_function(predictions, labels) loss = loss / accumulation_steps loss.backward() (i+ ) % accumulation_steps == : optimizer.step() model.zero_grad() (i+ ) % evaluation_steps == : evaluate_model() # Reset gradients tensors for in # Forward pass # Compute loss function # Normalize our loss (if averaged) # Backward pass if 1 0 # Wait for several backward steps # Now we can do an optimizer step # Reset gradients tensors if 1 0 # Evaluate the model when we... # ...have no gradients accumulated We do not every batch, but for every . Which will also change the back-propagation period. And that will effectively change our higher. optimizer.step() accumulation_steps batch_size 5. Post-processing on predictions Once we get our predictions on the test data, we can do some processing on it based on the metric of the competition & nature of data. For example, If is the metric measure performance, then average to ensemble models will perform better than a simple average. You can find a sample Python code for this . AUC rank here If is the metric of the competition, then simple average of labels gives the best naive baseline. Also, multiplying all the metrics with 0.99 or 1.01 helps in metric. LogLoss LogLoss Sometimes, rather than a simple average of predictions of models in the ensemble, geometric mean can be better. is a sample Python for this. Here Most post-processing techniques depend on the nature of the competition data. Training data may have some signals to tweak the model predictions to improve the score. 6. Feature Engineering and Data Augmentation This is very obvious and famous thing to do in Kaggle. Using all the given data in a competition, can we make more relevant data to make our model more robust & better? In tabular competitions, we combine multiple columns from our data to make more relevant columns in our data. For example, and columns of a house are given, we can create its by multiplying those two columns. Lots of things can be done in feature engineering - we just need to use our brain in how we can create more relevant data out of existing data. height width total_area Data augmentation is mostly done in image and text data. In image data, we can apply all sorts of transformations to make more data with the same label as the original image like: Rotate the image by any degree between 0-360. Crop the irrelevant part. Changing the opacity of the image. Flip the image horizontally/vertically. In text-data, we can augment using back-translation. Like, given an English sentence, we translate that sentence to let say German & translate it back from German to English. Now, that new English sentence may not be exactly the same as the original sentence but the meaning will be more or less the same. Feature engineering is a skill that requires creativity & logical thinking. And that's what differentiates a good Kagglers from a novice! If you have enjoyed this blog, you may find interesting! some of my other blogs Happy Kaggling. Cheers!

Top Tips For Competing in a Kaggle Competition

In this article, I will discuss some great tips and tricks to improve the performance of your structured data binary classification model. These tricks are obtained from solutions of some of Kaggle’s top tabular data competitions. Without much lag, let’s begin. These are the five competitions that I have gone through to create this article: Home credit default risk Santander Customer Transaction Prediction VSB Power Line Fault Detection Microsoft Malware Prediction IEEE-CIS Fraud Detection Dealing with larger datasets One issue you might face in any machine learning competition is the size of your data set. If the size of your data is large, that is 3GB + for kaggle kernels and more basic laptops you could find it difficult to load and process with limited resources. Here is the link to some of the articles and kernels that I have found useful in such situations. Faster data loading with pandas. Data compression techniques to . reduce the size of data by 70% Optimize the memory by r educing the size of some attributes. Use open-source libraries such as , it performs parallel computing and saves up memory space. Dask to read and manipulate the data Use . cudf Convert data to format. parquet Converting data to format. feather Reducing memory usage for . optimizing RAM Data exploration Data exploration always helps to better understand the data and gain insights from it. Before starting to develop machine learning models, top competitors always read/do a lot of exploratory data analysis for the data. This helps in feature engineering and cleaning of the data. EDA for microsoft malware detection. Time Series EDA for malware detection. Complete . EDA for home credit loan prediction Complete EDA for Santader prediction. EDA for VSB Power Line Fault Detection. Data preparation After data exploration, the first thing to do is to use those insights to prepare the data. To tackle issues like class imbalance, encoding categorical data, etc. Let’s see the methods used to do it. Methods to t . ackle class imbalance Data augmentation by . Synthetic Minority Oversampling Technique Fast inplace . shuffle for augmentation Finding synthetic samples in the dataset. used in signal processing competitions. Signal denoising Finding . patterns of missing data Methods to handle . missing data An overview of various encoding techniques for categorical data. Building model to predict missing values. Random to create new synthetic training set. shuffling of data Feature engineering Next, you can check the most popular feature and feature engineering techniques used in these top kaggle competitions. The feature engineering part varies from problem to problem depending on the domain. Target for better encoding. encoding cross validation Entity embedding to . handle categories Encoding c yclic features for deep learning. Manual . feature engineering methods Automated feature engineering techniques . using featuretools Top hard crafted features used in . microsoft malware detection Denoising NN for . feature extraction Feature engineering using RAPIDS framework. Things to remember while processing f eatures using LGBM. Lag features and moving averages. for dimensionality reduction. Principal component analysis LDA for . dimensionality reduction Best hand crafted LGBM features for . microsoft malware detection Generating frequency features. Dropping variables with different train and test distribution. for home credit competition. Aggregate time series features used in home credit default risk. Time Series features Scale,Standardize and  n . ormalize with sklearn Handcrafted features for Home default risk competition. Handcrafted features used in Santander Transaction Prediction. Feature selection After generating many features from your data, you need to decide which all features to use in your model to get the maximum performance out of your model. This step also includes identifying the impact each feature is having on your model. Let’s see some of the most popular feature selection methods. Six ways to do f . eatures selection using sklearn . Permutation feature importance . Adversarial feature validation Feature selection using null importances. Tree explainer using SHAP. DeepNN explainer using . SHAP Modeling After handcrafting and selecting your features, you should choose the right Machine learning algorithm to make your prediction. These are the collection of some of the most used ML models in structured data classification challenges. Random forest classifier . XGBoost : Gradient boosted decision trees. for distributed and faster training. LightGBM to handle categorical data. CatBoost classifier. Naive bayes model. Gaussian naive bayes used in 3rd place solution of Santander Customer Transaction Prediction LGBM + CNN  model n Neural Network. Knowledge distillation i method. Follow the regularized leader Comparison between (goss, gbdt and dart). LGB boosting methods experiment. NN + focal loss Keras NN with timeseries splitter. 5th place for Santander Transaction prediction. NN architecture with code Hyperparameter tuning LGBM methods. hyperparameter tuning Automated methods. model tuning Parameter tuning with . hyperopt for hyperparameter tuning. Bayesian optimization . Gpyopt Hyperparameter Optimisation Evaluation Choosing a suitable validation strategy is very important to avoid huge shake-ups or poor performance of the model in the private test set. The traditional 80:20 split wouldn’t work for many cases. Cross-validation works in most cases over the traditional single train-validation split to estimate the model performance. There are different variations of KFold cross-validation such as group k-fold that should be chosen accordingly. . K-fold cross-validation . Stratified KFold cross-validation Group KFold to check if train and test distributions are similar or not. Adversarial validation validation. Time Series split Extensive time series splitter. Note: There are various metrics that you can use to evaluate the performance of your tabular models. A bunch of useful classification metrics are listed and explained here. Other training tricks for LGBM. GPU acceleration . Use the GPU efficiently . Free keras memory to save runtime and memory. Save and load models Ensemble If you’re in the competing environment one won’t get to the top of the leaderboard without ensembling. Selecting the appropriate ensembling/stacking method is very important to get the maximum performance out of your models. Let’s see some of the popular ensembling techniques used in kaggle competitions: . Weighted average ensemble Stacked generalization ensemble. . Out of folds predictions . Blending with linear regression Use to determine blending weights. optuna . Power average ensemble Power 3.5 blending strategy. . Blending diverse models Different stacking approaches. AUC weight optimization. for low correlation predictions. Geometric mean Weighted . rank average Final thoughts In this article, you saw many popular and effective ways to improve the performance of your tabular data binary classification model. Hopefully, you will find them useful in your projects. This article was originally written by Shahul ES and posted on the Neptune blog . You can find more in-depth articles for machine learning practitioners there.

How To Run Text Categorization: All Tips and Tricks from 5 Kaggle Competitions

In early February I saw that a “March Madness” competition was being hosted on . I thought this might be something I could do since I am somewhat interested in sports, but when I looked at the competition I was pleasantly surprised . . . there was an analytics portion. had three portions this year: A Competition predicting the Men’s Tournament, the Women’s Tournament, and a separate analytics competition. I had been planning on working on my EDA and visualization skills anyway, so I thought this would be the perfect training ground. Kaggle The Google Cloud & March Madness Competition The competition originally began on February 13 and would end on April 6th. The challenge is “to tell a data story about college basketball through a combination of both narrative text and data exploration.” The rules for the notebook were relatively broad, so the topics you could explore were endless as long as they related to March Madness. I began exploring the data and saw that the opportunities were basically endless. There was data about both the Men’s and Women’s Tournament and Regular Season going back over 20 years. Fearful of drowning in a sea of data, I began brainstorming topics that would allow me to be creative and produce good visualizations, as well as have a chance at finishing this notebook on time. After exploring some data on points per game (PPG) and the track records of different teams, an idea popped into my head. It was just a passing thought at first, but then, after a few hours, I could not stop thinking about it. I had begun to deeply wonder about a connection between the men’s and women’s data. The more I thought, the more the idea took shape—Schools that have two good teams (men and women) could be more likely to win games in the NCAA Tournament. It was a crazy thought, I know. I didn’t even have any evidence or experience to back up the idea. But on some level, it just clicked with me. So, from there I began my exploration into what I have dubbed the “Two-Team Factor.” I needed to first establish what exactly this factor did, and how I could explain it in my notebook. Basically, the Two-Team Factor shows us that when programs/schools (like, say Duke) have both their Men’s and Women’s teams competing in both NCAA Tournaments in the same season, their teams are more likely to win games and advance in the tournament than programs that only have one team in the tournament. I know that is a mouthful so I tried to condense to just saying Two-Team programs and Single Team programs. Two-Team programs are special, they are an anomaly; they are schools that are lucky enough to be able to watch both their Men’s and Women’s teams play in March Madness in the same year. As this idea began to take shape and my fantasies about maybe even winning my first competition grew, the world suddenly changed. COVID-19 (Coronavirus) had spread to the U.S., and in early March the NCAA canceled the remaining college basketball season, including March Madness. This was a shock, and soon the prediction competitions for both the Men’s and Women’s Tournaments were canceled too. As I worried about whether my hard work had been wasted, Kaggle announced that the analytics competition for Google Cloud & March Madness would continue and the deadline would be extended to April 30. This was great news, which also allowed many contestants from the prediction competitions to move into the analytics competition. Surprisingly, this influx of competitors only motivated me to work harder on my idea. Now that I had extra time, I could really flesh it out and make some great visualizations. This competition forced me to learn more about pandas and visualizations. For days I dove into , , and . Each library offered something different that I could use. is the classic, the one I, and everyone else, starts out using. It is really great for being able to describe how each and every detail of your visualization will look, but this also means it can be a very complex and long block of code. is a visualization library that is built on top of . It allowed me to use shorter code to make my plots. However, was really the library that I had never used before, and it became a large part of my arsenal. is a web-based library that allowed me to build interactive plots. This means I could build plots like this: Matplotlib Plotly seaborn Matplotlib Seaborn Matplotlib Plotly Plotly that allow users to hover over the columns and see the different values. It also allows me to make bar charts like this that can compare data from two different variables. As I progressed, I learned that a notebook can get out of hand very quickly. I had done this entire competition so far in one notebook. It consisted of many of my experiments and many failed attempts. As the notebook got longer and longer it became more difficult to keep up with the variables I created and the different datasets. This alerted me to the fact that I need to make a plan and outline earlier in the competition to keep up with everything, maybe even multiple notebooks like I do in normal competitions. So with this realization, I started to prepare another notebook that would be my final draft. My original notebook had served its purpose. I was able to explore the data and reach the conclusion that the Two-Team Factor had merit. I discovered that in each round the “Two-Team” programs, as a percentage of the teams in the round, grew from under 30% in the first round to over 60% in the final rounds. This held true for both the men’s and women’s tournaments, and showed that these few teams must be winning most of their matchups. I was able to then provide visualizations that showed the winning percentages of each type of program (Two-Team & Single Team) in each round of the tournament. As I had expected, the Two-Team programs had a higher winning percentage (60% or higher) in every round except one. With this information I created the basis of my argument and decided to continue exploring this factor in relation to other common factors associated with team performance. During this entire experimentation period, the submission date was growing nearer. It caused a certain amount of stress, but just enough to motivate me. Fortunately, I had a lot of extra time due to the unfortunate event of the Coronavirus. This allowed me to really put in the extra hours in the last couple weeks. I eventually came to a point where there was no more time for exploration, and it was now time to create a beautiful notebook. My personal requirements for a beautiful notebook were: (1) Clear and concise writing, (2) Great visualizations, and (3) understandable, concise code. The writing was not very difficult because I had a perfect idea of what I wanted to convey. It just took time and many revisions. The visualizations, however, were something I needed to spend a lot of time on. I discovered that I could change the style of each chart with a single line of code. This was really a game-changer for me because I think the chart style adds a lot to a visualization. Some charts need the style, and others need a simple “fivethirtyeight” “seaborn-whitegrid.” Finding the perfect fit took some tinkering, but I think it improves the value of a visualization by a lot. Finally, I came to the code. I had decided I would use the hide feature on most of it in the final notebook for readability. But I still wanted my code to by clear, efficient, and understandable. The only problem was I had written so much unstructured code in my original notebook it was hard to piece it all together in the final one. My actual code needed good variable names and good documentation that explained what the code was doing so it is reproducible and understandable. This is something that is easy to forget when using a . Jupyter Notebook I’ve heard for months now that Jupyter notebooks can be dangerous to developers, but data scientists love them. I love them! They are easy to use and great for data exploration, but as soon as you start running cells out of order things can get out of hand fast. To make more readable and useful code I realized I needed to actually use my (very weak) python programming skills. I needed to define objects clearly, write notes, and use classes and functions. These can be very useful for shortening code and creating repeatable processes. It also allows me to understand and run the code in order. This was a big hurdle, but I now believe developer skills are much more valuable to a data scientist than I originally thought. Even though it is a , we need reproducible code that has structure. This is something I want to focus on in the near future, and develop these programming skills further. jupyter notebook The deadline was now only days away and I scrambled to make sure my writing was clear and concise, and all of my visualizations had labels and made sense. On March 30, 2020, my first live Kaggle competition ended and I turned in the notebook, “The Two-Team Factor: Gender in March Madness.” There were many great notebooks turned in to this competition. On May 18, 2020, the competition creators chose five winners in the . I, not surprisingly, was not chosen. The winners, whose notebooks can be viewed , had very impressive and creative notebooks. For my first competition, I was very proud of my submission. It was a semi-clear, creative notebook with what I thought were good visualizations. March Madness Analytics Competition here I could not have asked for a better first live competition. The other competitors were great and I was able to really dive deep into some of the skills I wanted to work on. I think this competition has really improved my dataset manipulation and visualization skills. This is something that I hope will be extremely helpful in creating EDAs in future competitions, as well as creating great notebooks to share.

Impressions From Attending Live Kaggle Competition 

In the Birthday AMA Episode, Sanyam Bhutani had shared a small series of "exciting updates" coming to CTDS.Show: This post is an announcement, a (slightly advance) celebration of 1 year of running the podcast and a discussion of these announcements: 1 Year Anniversary In the efforts of sharing stories of many ML Heroes' journey with the community, here are a few numbers about what we've achieved thanks to amazing community: CTDS has consistently released 2 interviews/week, on Thursday and Sunday sharp at 9 AM PT (Throughout 2020, all non-monetised and ad-free) In the slightly under an year of being live, we've published over 75 interviews, crossed a little over 75k streams and have reached 110+ Countries! We are approaching 1 year anniversary of the podcast and to celebrate with the community, we'll be running a 3 week Kaggle Contest As you might know a large number of interviews are with Kagglers, so what better way to celebrate than running a Kaggle Contest and workshops: CTDS Kaggle Contest About: The competition will be aimed at articulating insights from the Interviews with ML Heroes. Check the judging criteria below for complete details. We will also be hosting sessions and sharing resources aimed at: Introduction to Kaggle Data Exploration: EDA Session NLP Session There will also be a few sessions on getting started on Kaggle and EDA with active support throughout the run on the forums. Keep an eye out on this for all details discussions thread You can also sign up here to be notified once the contest goes live Data That's cool! So, what is the data? The dataset consists of detailed Stats, Transcripts of CTDS.Show. Any External data usage is allowed as long as it’s shared with other participants, even the usage of video and audio files of the podcast is also allowed, participants are free to download these from YouTube/Audio hosting for usage with this competition. The goal is to use these and come up with interesting insights or stories based on the 100 interviews with ML Heroes. (Participants are allowed to use the earlier 25-blog interviews as well to discover insights) Okay, when are you going to run it? Launch: On the anniversary recording date of CTDS, 21st June, 2020 Session/Workshop Dates: (Weekends/TBD) End: 14th July, 2020. Will there be prizes? Or is this free labour? We're glad you asked! Prizes: Participants will be awarded a certificate by , and if they submit at least 1 Kernel that is original work, the notebook should ideally talk about a particular theme/story instead of simple EDA and shouldn’t follow a voting ring pattern. All MLT DSNet Made With ML Winners: A female and a male winner will be invited for the 1 year anniversary interview episode on the podcast to share their insights. (Can you pinpoint to the date using the dataset? 😉) Books: (The only 4x Kaggle Grandmaster in the world) has very kindly agreed to gift 7 Physical copies of his book to all of the winners (2 + 5 categorical). We'd highly recommend checking out his for many videos along the lines of the book title. Abhishek Thakur Approaching (Almost) any ML Problem YouTube Channel Manning publications has agreed 4 (2 each) of their books and winners can decide between either of them: - - Build a Career in Data Science Data Science Bookcamp They have also provided 40% discount coupons valid across all of their materials! Please use: coupon once the contest goes live to utilise it. mtpctds20 Disclaimer: None of the links above are affiliate links, both Abhishek Thakur and Manning publications were kind enough to support our contest and is just a reflection of their support for the Data science community. Note, we're reaching out to more sponsors and will try to add more prizes during launch. Judging Criteria: The scoring will be based on points out of 100, based on 5 criteria highlighted below. Every participant will be evaluated out of 5 points by the 4 judges allowing the maximum score achieved to be 100. (5 points * 5 criteria * 4 judges) Winners will be chosen based on the highest score achieved overall along with special recognition to participants with highest score achieved in individual criterion. Final LB will be published on the 1 year anniversary of the Podcast. : How well is the notebook written in terms of code quality, text description and grammar? Presentation : Is there a natural flow of the story that connects various points? Storytelling : Are the visualisations appealing, understandable and aligned with description? Visualisations : Are the insights relevant, useful and actionable? Insights : How novel and creative are the ideas and approaches? Innovation Judge Panel: So, who will judge the submissions? 🤓 The best of the best: We are really grateful to the stellar panel of 4 Kernel Heroes for their time: 💡: All of them have been kind enough to do interviews on the podcast, maybe start by using their advice as a story-telling tip 🤔 Andrey Lukyanenko | Kaggle: , , @artgor Linkedin Twitter Martin Henze | Kaggle: , , @headsortails Linkedin Twitter Parul Pandey | Kaggle: , , @parulpandey Linkedin Twitter Shivam Bansal | Kaggle: , , @shivamb Linkedin Twitter Communities/Participation rules: Participants from all communities or anywhere in the world are invited. We’re also grateful for , , , and for their support. WiMLDS Hyd MLT DSNet MadeWithML CTDS Blogs! Wait, Wait! That's not it, we also have a few announcements that we're VERY excited about: Until now, all of the interviews were released and shared just in Video and Audio format, which is how interviews and podcasts are meant to be. However, if you remember the original version of the ML Hero interviews started out as a blog. So in the effort of making these interviews available in all formats to be consumed: All of the interviews' blog versions (an adapted version of the conversations) will be posted on our hackernoon page. Don't go looking, you're looking at it! 😁 We're really grateful to the HackerNoon team for helping share these interviews. CTDS.News In just after 1 year of running the podcast, the time has arrived where CTDS asked itself: Should we start a new podcast!? After having a lot of strong 🍵 and a lot of brainstorming and very kind advices from many amazing people, we announce: CTDS.News: A short news podcast, this will be the little/baby version of CTDS and will be aimed at being short, ideally, Bi-Monthly and sharing all of the following amazing things happening in Data Science: Papers News Open Source Projects Kaggle related Projects Meetups/Events If you're still reading, yes, all of these will go out in a podcast format. But, wait! There's more! Fortunately you won't just be hearing Sanyam Bhutani's voice, unfortunately he will still remain the host. 🤔 Co-Hosts: Many amazing people have said yes to the idea of being rotating co-hosts: Aakash Nain Elvis Saravia Goku Mohandas Nirant Kaliswal So, what exactly is going on here? I thought you said news? 📰 Yes! News, and More! All of the amazing people above contribute to amazing paper summaries, one has an amazing website: (Think, Producthunt for ML + More 👌) MadeWithML Elvis is the founder of the amazing , runs an amazing newsletter. dair.ai community Nirant as well is an NLP Expert! Aakash is a CV + DL Expert and author of many that he has agreed to help port over to podcasts too! amazing paper summaries So depending on if it's a Computer Vision heavy week or a NLP intense week, you might hear from Aakash or Elvis or Nirant as our co-hosts. In complete honesty, many of the summaries would just be picked from the trending projects on MadeWithML so instead, Goku will be kind enough to appear on many of these. Goku shared our enthusiasm for this podcast and already was very helpful in planning for the podcast, even for our 1st episode. The same lazy engineer approach of ours can be extended to all of the co-hosts: For NLP, we'd be picking stories from dair.ai newsletters, Elvis was very kind to agree to the idea as well. 🎙Checkout the 27 second teaser by init27 (Sanyam Bhutani), now available across the majority of Podcast Platforms Website: CTDS.News One Last thing: Do you really think you're Steve Jobs? 📱 Woah! Your words, not ours! 😁 There will also be mini-series on the show consisting of paper summaries, concept explanations and more! If you have any suggestions, tweet at us or send a pigeon! 🍵

CTDS: 1 Year anniversary | Kaggle Contest | CTDS.News Launch

In this article (originally posted by on the ), I will discuss some great tips and tricks to improve the performance of your text classification model. These tricks are obtained from solutions of some of Kaggle’s top NLP competitions. Shahul ES Neptune blog Namely, I’ve gone through: — $65,000 Jigsaw Unintended Bias in Toxicity Classification — $35,000 Toxic Comment Classification Challenge — $25,000 Quora Insincere Questions Classification — $25,000 Google QUEST Q&A Labeling — $50,000 TensorFlow 2.0 Question Answering and found a ton of great ideas. Without much lag, let’s begin. Dealing with larger datasets One issue you might face in any machine learning competition is the size of your data set. If the size of your data is large, that is 3GB + for Kaggle kernels and more basic laptops you could find it difficult to load and process with limited resources. Here is the link to some of the articles and kernels that I have found useful in such situations. Optimize the memory by reducing the size of some attributes Use open-source libraries such as , it performs parallel computing and saves up memory space Dask to read and manipulate the data Use cudf Convert data to format parquet Convert data to format feather Small datasets and external data But, what can one do if the dataset is small? Let’s see some techniques to tackle this situation. One way to increase the performance of any machine learning model is to use some external data frame that contains some variables that influence the predicate variable. Let’s see some of the external datasets. Use of data for Question Answering tasks squad Other for QA tasks datasets Wikitext long term dependency language modeling dataset Stackexchange data Prepare a dictionary of commonly misspelled words and corrected words. Use of for cleaning helper datasets is the process of adding confidently predicted test data to your training dataUse different data Pseudo labeling sampling methods Text augmentation by Exchanging words with synonyms Text augmentation by noising in RNN Text augmentation by translation to other languages and back Data Exploration and Gaining insights Data exploration always helps to better understand the data and gain insights from it. Before starting to develop machine learning models, top competitors always read/do a lot of exploratory data analysis for the data. This helps in feature engineering and cleaning of the data. Twitter data exploration methods Simple EDA for tweets for Quora data EDA in R for Quora data EDA Complete with stack exchange data EDA My previous article on EDA for natural language processing Data Cleaning Data cleaning is one of the important and integral parts of any NLP problem. Text data always needs some preprocessing and cleaning before we can represent it in a suitable form. Use this to clean social media data notebook for BERT Data cleaning Use to correct misspellings textblob for pre-trained embeddings Cleaning for multilingual tasks Language detection and translation Preprocessing for Glove and part 1 part 2 to get more from pre-trained word embeddings Increasing word coverage Text Representations Before we feed our text data to the Neural network or ML model, the text input needs to be represented in a suitable format. These representations determine the performance of the model to a large extent. Pretrained vectors Glove Pretrained vectors fasttext Pretrained vectors word2vec My previous article on these 3 embeddings Combining . This can help in better representation of text and decreasing OOV words pre-trained vectors embeddings Paragram Universal Sentence Encoder Use USE to generate sentence-level features 3 methods to combine embeddings Contextual embeddings models Bidirectional Encoder Representations from Transformers BERT GPT a Robustly Optimized BERT Roberta a Lite BERT for Self-supervised Learning of Language Representations Albert a lighter version of BERT Distilbert XLNET Modeling Model architecture Choosing the right architecture is important to develop a proper machine learning model, sequence to sequence models like LSTMs, GRUs perform well in NLP problems and is always worth trying. Stacking 2 layers of LSTM/GRU networks is a common approach. Stacking Bidirectional CuDNNLSTM Stacking LSTM networks LSTM and 5 fold Attention Bidirectional LSTM with 1D convolutions Unfreeze and tune embeddings BiLSTM with Global maxpooling Attention weighted average GRU+ Capsule network I nceptionCNN with flip Plain vanilla network with BERT CuDNNGRU network TextCNN with pooling layers BERT embeddings with LSTM Multi-sample dropouts Siamese transformer network Global Average pooling of hidden layers BERT Different Bert based models Distilling BERT — BERT performance using Logistic Regression Different learning rates among the layers of BERT Finetuning Bert for text classification Loss functions Choosing a proper loss function for your NN model really enhances the performance of your model by allowing it to optimize well on the surface. You can try different loss functions or even write a custom loss function that matches your problem. Some of the popular loss functions are for binary classification Binary cross-entropy for multi-class classification Categorical cross-entropy used for unbalanced datasets Focal loss for multilabel classification Weighted focal loss for multiclass classification Weighted kappa to get sigmoid cross-entropy BCE with logit loss Custom used in bias classification competition mimic loss Jigsaw unintended used in bias classification competition MTL custom loss jigsaw unintended Optimizers Stochastic gradient descent RMSprop allows the learning rate to adapt based on parameters Adagrad for fast and easy convergence Adam to enable warmup state to Adam algorithm Adam with warmup for Bert based models Bert Adam for stabilizing training and accelerating convergence Rectified Adam Callback methods Callbacks are always useful to monitor the performance of your model while training and trigger some necessary actions that can enhance the performance of your model. for monitoring and saving weights Model checkpoint to change the learning rate based on model performance to help converge easily Learning rate scheduler Simple custom callbacks using lambda callbacks Custom Checkpointing Building your for various use cases custom callbacks to reduce the learning rate when a metric has stopped improving Reduce on plateau to stop training when the model stops improving Early Stopping to get a variety of model checkpoints in one training Snapshot ensembling Fast geometric ensembling Stochastic Weight Averaging (SWA) Dynamic learning rate decay Evaluation and cross-validation Choosing a suitable validation strategy is very important to avoid huge shake-ups or poor performance of the model in the private test set. The traditional 80:20 split wouldn’t work for many cases. Cross-validation works in most cases over the traditional single train-validation split to estimate the model performance. There are different variations of KFold cross-validation such as group k-fold that should be chosen accordingly. K-fold cross-validation Stratified KFold cross-validation Group KFold to check if train and test distributions are similar or not Adversarial validation CV analysis of different strategies Runtime tricks You can perform some tricks to decrease the runtime and also improve model performance at the runtime. to save runtime and improve performance Sequence bucketing when the input sentence is larger than 512 tokens Get sentences from its head and tail Use the GPU efficiently Free keras memory to save runtime and memory Save and load models Don’t Save Embedding in RNN Solutions Load without key vectors word2vec vectors Model ensembling If you’re in the competing environment one won’t get to the top of the leaderboard without ensembling. Selecting the appropriate ensembling/stacking method is very important to get the maximum performance out of your models. Let’s see some of the popular ensembling techniques used in Kaggle competitions: Weighted average ensemble Stacked generalization ensemble Out of folds predictions Blending with linear regression Use to determine blending weights optuna Power average ensemble Power 3.5 blending strategy Final thoughts In this article, you saw many popular and effective ways to improve the performance of your NLP classification model. Hopefully, you will find them useful in your projects. This article was originally posted on the Neptune blog. If you liked it, you may like it there :) You can also find me tweeting @Neptune_a i or posting on LinkedIn about ML and Data Science stuff.

Text Classification Models: All Tips And Tricks From 5 Kaggle Competitions

Imagine if you could get all the tips and tricks you need to hammer a Kaggle competition. I have gone over 39 Kaggle competitions including – $1,000,000 Data Science Bowl 2017 – $100,000 Intel & MobileODT Cervical Cancer Screening – $100,000 2018 Data Science Bowl – $60,000 Airbus Ship Detection Challenge – $60,000 Planet: Understanding the Amazon from Space – $50,000 APTOS 2019 Blindness Detection – $37,000 Human Protein Atlas Image Classification – $30,000 SIIM-ACR Pneumothorax Segmentation – $25,000 Inclusive Images Challenge – and extracted that knowledge for you. Dig in. Contents External Data Preprocessing Data Augmentations Modeling Hardware Setups Loss Functions Training Tips Evaluation and Cross-validation Ensembling Methods Post Processing External Data Use of the data because it contains detailed annotations from radiologists LUng Node Analysis Grand Challenge Use of the data because it had radiologist descriptions of each tumor that they found LIDC-IDRI Use , Flickr CC Wikipedia Commons datasets Use Human Protein Atlas Dataset Use dataset IDRiD Data Exploration and Gaining insights with the 0.5 threshold Clustering of 3d segmentation Identify if there is a substantial difference in train/test label distributions Preprocessing Perform blob Detection using the . Used the implementation available in package. Difference of Gaussian (DoG) method skimage Use of in order to reduce the time of training patch-based inputs for training Use for loading data instead of because it has a faster reader cudf Pandas Ensure that all the images have the same orientation Apply contrast limited adaptive histogram equalization Use for all general image preprocessing OpenCV Employ and adding manual annotations automatic active learning in order to apply the same model to scans of different thicknesses Resize all images to the same resolution into normalized 3D numpy arrays Convert scan images Apply single using Dark Channel Prior Image Haze Removal Convert all data to Hounsfield units Find duplicate images using pair-wise correlation on RGBY Make labels more balanced by developing a sampler Apply p to test data in order seudo labeling to improve score (CLAHE) with kernel size 32×32 Scale down images/masks to 320×480 Histogram equalization Convert DCM to PNG Calculate the when there are duplicate images md5 hash for each image Data Augmentations Use package for augmentations albumentations Apply random rotation by 90 degrees Use h orizontal, vertical or both flips Attempt : Elastic Transform, PerspectiveTransform, Piecewise Affine transforms, pincushion distortion heavy geometric transformations Apply random HSV Use of for generalization to prevent loss of useful image information loss-less augmentation Apply channel shuffling Do based on class data augmentation frequency Apply gaussian noise Use for data augmentation lossless permutations of 3D images by a random angle from 0 to 45 degrees Rotate by a random factor from 0.8 to 1.2 Scale changing Brightness Randomly change hue, saturation and value Apply augmentations D4 Contrast limited adaptive Use the histogram equalization Auto augmentation strategy Augment Modeling Architectures Use of a based architecture. Adopted the concepts and applied them to 3D input tensors U-net Employing automatic active learning and adding manual annotations The for training features with different receptive fields inception-ResNet v2 architecture with adversarial training Siamese networks , , x 5 with Dense (FC) layer as the final layer ResNet50 Xception Inception ResNet v2 Use of a which returns a fixed-length output no matter the input size global max-pooling layer Use of stacked dilated convolutions VoxelNet Replace plus sign in with concat and conv1x1 LinkNet skip connections Generalized mean pooling Keras to train the model from scratch using 224x224x3 NASNetLarge Use of the to slide over the imagesImagenet-pre-trained as the feature extractor 3D convnet ResNet152 Replace the final fully-connected layers of ResNet by 3 fully connected layers with dropout Use in the decoder ConvTranspose Applying the Implementing the network with adjusted receptive fields and a 64 unit bottleneck layer on the end of the network VGG baseline architecture C3D Use of type architectures with pre-trained weights to improve convergence and performance of binary segmentation on 8-bit RGB input images UNet since it’s fast and memory efficient LinkNet MASKRCNN BN-Inception Fast Point R-CNN and Seresnext UNet Deeplabv3 Faster RCNN SENet154 ResNet152 NASNet-A-Large EfficientNetB4 ResNet101 GAPNet PNASNet-5-Large Densenet121 AC-GAN XceptionNet (96), XceptionNet (299), Inception v3 (139), InceptionResNet v2 (299), DenseNet121 (224) from AlbuNet (resnet34) ternausnets from SpaceNet Resnet50 selim_sef SpaceNet 4 SCSE from Unet (seresnext50) selim_sef SpaceNet 4 A custom Unet and Linknet architecture FPNetResNet50 (5 folds) FPNetResNet101 (5 folds) FPNetResNet101 (7 folds with different seeds) PANetDilatedResNet34 (4 folds) PANetResNet50 (4 folds) EMANetResNet101 (2 folds) RetinaNet Deformable R-FCN Deformable Relation Networks Hardware Setups Use of the AWS GPU instance p2.xlarge with a NVIDIA K80 GPU Pascal Titan-X GPU Use of 8 TITAN X GPUs 6 GPUs: 2 1080Ti + 4 1080 Server with 8×NVIDIA Tesla P40, 256 GB RAM and 28 CPU cores Intel Core i7 5930k, 2×1080, 64 GB of RAM, 2x512GB SSD, 3TB HDD GCP 1x P100, 8x CPU, 15 GB RAM, SSD or 2x P100, 16x CPU, 30 GB RAM NVIDIA Tesla P100 GPU with 16GB of RAM Intel Core i7 5930k, 2×1080, 64 GB of RAM, 2x512GB SSD, 3TB HDD 980Ti GPU, 2600k CPU, and 14GB RAM Loss Functions because it works well with imbalanced data Dice Coefficient whose aim is to reduce the distance between the predicted segmentation and the ground truth Weighted boundary loss that creates a criterion that optimizes a multi-label one-versus-all loss based on max-entropy, between input  and target MultiLabelSoftMarginLoss Balanced cross entropy (BCE) that involves weighing the positive and negative examples by a certain coefficient with logit loss that performs direct optimization of the mean intersection-over-union loss in neural networks based on the convex Lovasz extension of sub-modular losses Lovasz obtained by summing the Focal and Lovasz losses FocalLoss + Lovasz that incorporates margin in order to maximise face class separability Arc margin loss that computes the npairs loss between y_true and y_pred. Npairs loss A combination of functions BCE and Dice loss – a pairwise ranking that is is smooth everywhere and thus is easier to optimize LSEP that simultaneously learns a center for deep features of each class and penalizes the distances between the deep features and their corresponding class centers Center loss that augments standard loss functions such as Softmax that trains a network to embed features of the same class at the same time maximizing the embedding distance of different classes Ring Loss Hard triplet loss that involves subtracting the BCE and DICE losses then adding 1 1 + BCE – Dice that is the binary cross-entropy minus the log of the dice loss Binary cross-entropy –  log(dice) of BCE, dice and focal Combinations that  loss performs direct optimization of the mean intersection-over-union loss Lovasz Loss -Dice loss is  obtained by calculating smooth dice coefficient function BCE + DICE that is an improvement to the standard cross-entropy criterion Focal loss with Gamma 2 – this is basically a summation of the three loss functions BCE + DICE + Focal that incorporates the area and size information and integrates the information in a dense deep learning model Active Contour Loss 1024 * BCE(results, masks) + BCE(cls, cls_target) – Kappa is a loss function for multi-class classification of ordinal data in deep learning. In this case we sum it and the focal loss Focal + kappa — Additive Angular Margin Loss for Deep Face Recognition ArcFaceLoss s – Soft Dice uses predicted probabilities oft Dice trained on positives only which is a custom loss used by the Kaggler 2.7 * BCE(pred_mask, gt_mask) + 0.9 * DICE(pred_mask, gt_mask) + 0.1 * BCE(pred_empty, gt_empty) that creates a criterion that uses a squared term if the absolute element-wise error falls below 1 and an L1 term otherwise nn.SmoothL1Loss() Use of the in scenarios where it seems to work better than . Mean Squared Error objective function binary-cross entropy objective function Training tips Try different learning rates Try different batch sizes Use SDG with momentum with manual rate scheduling Too much will reduce the accuracy augmentation Train on image on full images crops and predict Use of Keras’s to the learning rate ReduceLROnPlateau() Train then apply soft and hard augmentation to some epochs without augmentation until plateau last one and use 1000 images from Freeze all layers except the Stage1 for tuning Make labels more balanced by Use of developing a sampler class aware sampling Use dropout and augmentation while tuning the last layer to improve score Pseudo Labeling Use Adam reducing LR on plateau with patience 2–4 Use Cyclic LR with SGD Reduce the by a factor of two if validation loss does not improve for two consecutive epochs learning rate Repeat the of 10 batches worst batch out Train with default UNET so that each edge pixel is covered twice Overlap tiles Hyperparameter tuning: learning rate on training, non-maximum suppression and score threshold on inference with low confidence score Remove low bounding box Train different then build an ensemble convolutional neural networks Stop is decreasing training when the F1 score with gradual reducingTrain ANNs in 5 folds and 30 repeats Differential learning rate a stacking way using Track of your experiments using . Neptune Evaluation and cross-validation Split on by classes non-uniform stratified Avoid by applying while the last layer overfitting cross-validation tuning 10-fold CV ensemble for classification Combination ensembles for detection of 5 10-fold CV Sklearn’s 5 stratified K fold function K Fold Cross-Validation Adversarial Validation & Weighting Ensembling methods Use simple for ensemble majority voting on the the z-location and the XGBoost max malignancy at 3 zoom levels, amount of strange tissue for models classes. This was done for raw data features only. LightGBM with too many for CatBoost a second-layer model Training with 7 features for the gradient boosting classifier Use to speed up model training. In this technique, models are first trained on simple samples then progressively moving to hard ones. ‘curriculum learning’ Ensemble with ResNet50, InceptionV3, and InceptionResNetV2 for object detection Ensemble method An ensemble of , , and architectures n with a classification network — architecture Mask RCNN YOLOv3 Faster RCNN DenseNet-121 Post Processing Apply — presenting an image to a model several times with different random transformations and average the predictions you get test time augmentation Equalize test prediction instead of only using predicted classes probabilities Apply to the geometric mean predictions is covered at least thrice because UNET tends to have bad predictions around edge areas. Overlap tiles during inferencing so that each edge pixel and bounding box shrinkage Non-maximum suppression to detach objects in instance segmentation problems. Watershed post processing Final Thoughts Hopefully, this article gave you some background into image segmentation tips and tricks and given you some tools and frameworks that you can use to start competing. We’ve covered tips on: architectures training tricks, losses, pre-processing, post processing ensemblingtools and frameworks. If you want to go deeper down the rabbit hole, simply follow the links and see how the best image segmentation models are built. Happy segmenting! This article was originally posted by Derrick Mwiti on the Neptune blog . If you liked it, you may like it there :) You can also find me tweeting @Neptune_ai or posting on Linkedin about ML and Data Science stuff.

Image Segmentation: Tips and Tricks from 39 Kaggle Competitions

In this post, we will see how to use the platform and get a submission that achieves a respectable 83% Accuracy on the test set. This accuracy may seem low but the current agenda for the platform is to also finally have API to use these models with as low latency as possible. So currently, the models used strike the perfect balance between accuracy and latency. Based on the awesome . Huggingface Transformers Library So let’s start . . . Aasaan.ai Website First, we will visit the website. Aasaan.ai Click on try now to start the demo. It is completely free! Import the dataset First, we will download the dataset from the Kaggle Challenge website. We are tackling the Disaster Tweets Challenge — https://www.kaggle.com/c/nlp-getting-started/data Upload the train.csv and wait for a few seconds. Select the text and the labels Aasaan.ai shows you a preview of the dataset showing you the columns. The text column is column 3 and the label column is column 4, so we will select these for training the model. Train the model Now we need to just train the model. Yes. It is that easy! Now we wait. It takes roughly 2 minutes to train the model. Have a coffee. Create the submission File Move to the CSV tab. Upload the test.csv previously downloaded, and wait for a few seconds and the submission.csv will be downloaded. Upload to Kaggle and view the results. I got to top 24% of all participants! Interested? We have successfully built, trained and deployed a data classifier without - Writing a single line of code Configuring any GPU Looking at daunting documentations Reading and understanding of machine learning papers. Visit us at to try for yourself and be an early beta tester and join our waiting list! aasaanai Aasaan.ai is a No Code platform for building, training and deploying Deep Learning models without worrying about GPUs, Infrastructure, Transformers or anything.

How I Got to Top 24% on a Kaggle Text Classification Challenge Without Writing a Single Line of Code

There are many great resources for learning data science and machine learning out there, but the one thing that might be missing is a live accounting of a non-technical individual learning these skills. I use the term “outsider” in the title because I don’t feel like I have the typical background that most people do on Kaggle. I am not a machine learning expert, mathematician, or expert computer programmer. I have experience in finance and law, not computer science or statistics. My goal in documenting my Kaggle progress is to: Hold myself accountable Share my experiences This is by no means a perfect explanation of how to enter and win a Kaggle competition. This is my journey, and hopefully I will get immensely better at both Kaggle competitions and writing. WHERE TO BEGIN? One of the toughest things about this was just committing to doing it. Learning these skills seems like an overwhelming task when new information is coming out everyday. So I began looking for good resources which would give me a framework of where to start. I have a little experience teaching myself Python over the last few years, and I decided working with real data in a competition would be the best way to learn. Approaching this, I completed most of the learning courses on Kaggle, some MOOCs, and I dabbled in a couple competitions (not live). I needed a repeatable process I could use when approaching any Kaggle competition. The article by Alex Kruegger was a very good resource for establishing a framework to approach Kaggle competitions. This is a Towards Data Science article written by a Kaggle Competitions Master on how he reached this feat in a short amount of time. He did a wonderful job of breaking down what goals can be accomplished on Kaggle and how to accomplish them. How to “farm” Kaggle in the right way The main takeaways for me were the iterative process he used in competitions. It laid out an outline for what steps in a competition produce the best model and output. Then the article recommended going back through the public kernels after your submission and writing down any successful approaches so they can be added to our repertoire for future competitions. One important point I want to make is the reason for implementing other Kaggler’s best solutions. We do not want to simply copy the best kernel in the competition and submit it to try to move up the leader board. We want to read the best Kaggle solutions and try to understand them so we can implement those solutions in future competitions. The point is not to medal in competitions, but to learn how to use these tools. After reading the article I decided I would try to implement my framework while also learning from the MOOC “How to Win a Data Science Competition” on Coursera. This way I could learn new skills while also completing a real competition. While working through this MOOC and competition I housed my Kaggle notebooks on Github. There are a few reasons for doing this. First, I want to learn how to use git and github (A few great articles are and ). The second reason is I think this is a good way to keep track of my progress and look back on old notebooks at any time. Here here ENTERING THE COMPETITION Predict Future Sales I begin my Kaggle journey by entering the playground contest “Predict Future Sales” because it is the competition that goes with the “How to Win a Data Science Competition” MOOC on Coursera. I did not register for the course, I simply audited it. Understand The first thing we need to do is get an understanding of the competition. To do this we read the information provided by the competition creators. This should give us an idea of the data and goals of the competition. I recommend competitions with smaller sets of data for beginners, as larger sets will be more difficult to control and possibly require more compute power than you have available. Next, we look at the rules of the competition, which outline how we are to compete. This will include the timeline, whether teams are allowed, and anything else the creators want to specify. Finally, we look at the evaluation of the competition. This tells us what our final prediction value should look like and how our predictions are scored. Be careful at this point. Our output file is required to list the items sold each month. The training dataset, however, only has the number of items sold per day. So the first thing we need to do is change the items sold per day to items sold per month. Competition Workflow EDA Modeling and Initial Submission Feature Engineering Modeling Ensemble EDA The first step in my Kaggle competition is the Exploratory Data Analysis (EDA). This is exactly what is sounds like, an analysis that explores the data so we can learn more about the datasets we’re using. Most of my time was spent on learning about the dataset and fixing small mistakes I had made when preprocessing, cleaning, and engineering features. This process is focused on the small details. If you can get those right, you can save yourself significant time. The best way to do this is to explore the data after any changes. From reading other notebooks of more experienced kagglers, I have developed a simple system for understanding the data. I initially find the shape of the data. This data was in form (similar to an excel notebook), this means it has clearly identifiable rows and columns. The code ( is what I am using as the name of the training data dataset; you can choose to name your dataset anything you want) gives us an output describing the data. csv train_data.shape train_data The code shows the shape of the train and test dataset. The rows are on the left and columns on the right. So train would have 2,935,849 rows and 6 columns. Next, I call the command to get an idea of what the values in the dataset look like. head() We can view which features (columns) are numerical and which are categorical (words like the shop names), and get a feel for what kind of data we are dealing with. There are many other functions such as , , and , which I use to understand the data. More functions can be found in the documentation for the library. This is the python library used with data structures and tables. describe() info() unique() pandas After exploring the data a little, I started to figure out the time series format. The data was given to us in days and we needed to group it into months for the submission. We do this with the sum function and then fill in the missing months with zero, as there were zero sales of the items in those months. A. Visualizations Visualization is a way to understand and explore the data. For those starting out, I would advise looking at the the course on Kaggle. Reading other notebooks is also an excellent way to learn many of the visualization techniques. I usually do a few basic things like make histograms, scatter plots, etc. Data Visualization These are two types of plots we make in Kaggle. They both depict the number of items sold in each month. Graphs and plots like these allow us to see overarching trends in the data. B. Univariate and Bivariate Data Analysis There are two ways to explore the data. The first is the univariate analysis, which uses visualizations and statistics to explore a single feature in the dataset. This is a useful initial evaluation. I explored each feature individually and made a short summary in my notebook. This allowed me to have a basic understanding of each feature and how it might impact the target feature. The Bivariate data analysis can be more in depth and revealing. A bivariate analysis analyzes two or more features at once. We are able to see their correlations and which features impact others. C. Preprocessing and Cleaning Preprocessing and cleaning the data can be one of the most time consuming tasks in the competition, but they are an important part of the EDA.  We usually start with finding out how many missing values there are in the dataset. Most datasets have missing values and they will usually be filled with . We use the below function to determine how many missing values are in each column. NaN Once we have found the missing values we can decide what to do with them. We can drop the column if there are too many, or we can fill them with zero or another number such as the median value of that feature. Preprocessing also consists of finding any outliers in the data, such as negative numbers or extremely large numbers. We drop these values because they can have an outsized effect on the model. MODELING and INITIAL SUBMISSION Even after completing most of my EDA I still struggled to understand how the data was structured and how it could be used to predict the next month’s sales. Time series data is exactly what it sounds like, data which is recorded at different times. In a classic classification challenge we are given data to train on, such as passengers on the titanic and whether they lived or died. Then we are given completely different passengers for the test data to determine if they lived or died. In a time series competition the same item is given in the training and testing dataset, but at different periods in time. In this competition, the training data holds items and which store they were sold in. We are then given the same items and stores in the test data, but it is at a later date and we must predict how many are sold at this later time. To try to understand this data I created a pivot table to train on before going in depth in feature engineering. I got the idea from this by . notebook NowYSM The first two columns list the shop and item id, and then the next thirty-three are the consecutive months from January 2013 to October 2015. In each row the number of items sold for that month is listed. I was now able to visualize what the time series data looked like, and I decided to run it through a model. I decided to use an model to determine the feature importance. Training the pivot table on this model showed me which months were most predictive (it was the most recent), because those were my features. Doing this helped me to understand the features and how the models worked. XGBoost XGBoost Aside After you commit your code in Kaggle it will show a message that looks like this: The columns show the difference between this version and the last version you committed. The far right column (in red) is the number of lines of code you deleted or changed. The column to the left of this one (in green) shows a positive number. This is the number of lines of code added. For example, in my latest version, , I deleted (or changed) two lines of code, and added nine lines of code. Diff Version 11 FEATURE ENGINEERING After my initial model, we return to the feature engineering step to add new predictive features. The data is back in its original format and not in the pivot table. We can add extra features (columns) to the dataset. This process is called feature engineering, and it is when we use existing features in the dataset to create new ones. After looking at our feature importance and model performance on the pivot table, we were able to determine different months of sales had different predictive value. I decided that making a few moving average features should be a good start. I made a previous month feature, which created a column holding the value of the number of sales in the previous month. I then created a 6-month and 12-month moving average of sales for each item. Next, I did some simple feature engineering by creating columns with the maximum number of each item sold, and the maximum price the item sold at. Finally, we move to a powerful form of feature engineering called mean encoding or target encoding. This was a form of feature engineering the MOOC focused on because it is the simplest way to raise the accuracy of your model. Mean encoding takes a feature, such as item id, and finds the mean (average) target value for that specific item. In simple terms, if item #10 was sold twice this month and zero times next month then the mean encoded feature for this item would be 1 [(2+0) / 2]. This means the average number of items sold per month is one. If we make a feature with these values, we can produce a more accurate prediction. Not So Fast! We cannot forget about our validation strategy When we build a model we need to validate it. Validation involves splitting the training dataset into two parts. One set will be the data we use to train our model and the other part will be used to test the accuracy of our model. The validation set acts as our test set, and allows us to test our models before running them on our test values. The key to our validation strategy is when we split the training dataset. We need to split it before certain forms of feature engineering to make sure it gives us an accurate prediction of how our model will perform. This means we perform our split before we mean encode or there will be data leakage. train/validation Data leakage is when values from the validation set leak into our training data and cause our model to perform better than it would in the real world. A simple example of data leakage would be mean encoding all our data including the validation dataset. MODELING After we finish feature engineering we are ready to begin building our models. The modeling portion of data science is probably the most talked about portion. However, in reality this is where you will spend the least amount of your time. And even the time spent here will be mostly spent waiting for a model to train. Most of the work will be done in exploring, cleaning, and processing the data, and feature engineering. Modeling is still, however, an important part of the Kaggle competition. If you want an overview of the different models I suggest taking some of the Kaggle Learn courses such as . The MOOC I am taking also has some excellent sections on modeling and how to tune hyperparameters. These skills are important for competitions, but in the beginning a much higher score will be achieved by simply focusing on good data processing habits and creating good features. Intro to Machine Learning The main models I focused on and learned about in the beginning were Decision Trees, Random Forests, Linear/Logistic Regression, K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Naïve Bayes Classifier, and Gradient Boosting models such as XGBoost and CatBoost. We have our training data split into train and validation, so for each set we need to prepare our data for the model we are going to use. To prepare it I like to separate the train data into two sections: the input (X) and the output (y). These two parts of the data are used to train the model. We take the train data and make the input by dropping the target column. Then we create the output dataset containing only the target value column ( ). Now, when we train the model it knows the input will be all of the test data we are given, and the output will be the target value we are trying to predict. We will do the exact same thing for the validation set we have created. It will usually look something like this: item_cnt_month Once we have done this, we train the model on the training set data (X & y). Then we try to find the target value of the validation set using the model we recently trained. So we input only the X portion of the validation data into the model. We then take the output the model gives us and compare it to the true target value for the validation data (y). ENSEMBLE There is another way to get an even higher score in a competition. It is called stacking models or the ensemble method. In this method we create another model which makes predictions based on the outputs of the first models we used. The thinking behind this is each model we used above performs differently. The Random Forest and Linear Regression models create outputs in different ways and therefore they are each more accurate on different portions of the data. So even though random forest or may have the best score, we can get an even better score by incorporating all of the models. XGBoost For this competition I used a simple Linear Regression model for the ensemble. The idea and process for doing this came from a by . In this notebook he creates a simple ensemble model using linear regression on all of the predictions made by his other models. notebook DimitreOliveira The picture above is the one used to describe the model in notebook. I did not create this image. DimitreOliviera’s This seemed like the best way to get introduced to model ensembles. It is relatively simple, and did not have any meaningful impact on my score. Its not something I am too worried about as a beginner, but it is a very powerful tool that I hope master in the future. DATA SIZE PROBLEMS During the commit process I ran into a few problems in some of my notebooks. The main problem was trouble committing the code. It appeared the data was too large for the Kaggle environment. This may not be a problem if you have your own private environment you run your models in, but for a beginner like me who is doing most of his work on Kaggle, this was a problem. A found a couple solutions to the problem which involved downsizing the data. Hopefully someone will correct me if I am wrong, but downsizing is when you change the data type of each cell (or data point). Large data types like , which is the default for Kaggle, take up much more space when loading a dataset into a notebook. If you have a value in each cell that is not extremely large, you should be able to use a smaller data type such as , , or (If you want to learn more about data size and bytes, there are some good resources on Coursera). int64 int32 int16 float32 I found a couple of scripts online that solved this problem and downsized the data when it was loaded into the notebook ( and ). This saved me a lot of time running the notebooks and allowed Kaggle to process all of the data. Even if you have not encountered this problem yet, it is definitely important to study. As data continues to grow we will have larger and larger datasets that will require skill in manipulating data types and size. This downsizing of data should not cause any data loss. here here CONCLUSION I’m not promising that doing anything in this notebook will help you win a Kaggle competition. My submission scores are about average (50%), which isn’t great, but also isn’t horrible for my first real attempt. What I am hoping to do is to share my process in an understandable way. Hopefully this will inspire others to dive in and feed their curiosity.

Stories, News & Mentions about KAGGLE

Kaggle’s Stories on HackerNoon

Kaggle's News & Mentions