How to use Keras sparse_categorical_crossentropy

In this quick tutorial, I am going to show you two simple examples to use the loss function and the metric when compiling your Keras model. sparse_categorical_crossentropy sparse_categorical_accuracy Example one — MNIST classification As one of the multi-class, single-label classification datasets, the task is to classify grayscale images of handwritten digits (28 pixels by 28 pixels), into their ten categories (0 to 9). Let’s build a Keras CNN model to handle it with the last layer applied with “softmax” activation which outputs an array of ten probability scores(summing to 1). Each score will be the probability that the current digit image belongs to one of our 10 digit classes. For such a model with output shape of (None, 10), the conventional way is to have the target outputs converted to the one-hot encoded array to match with the output shape, however, with the help of the loss function, we can skip that step and keep the integers as targets. sparse_categorical_crossentropy All you need is replacing with when compiling the model like this. categorical_crossentropy sparse_categorical_crossentropy After that, you can train the model with integer targets, i.e. a one-dimensional array like array([5, 0, 4, 1, 9 ...], dtype=uint8) Note this won’t affect the model output shape, it still outputs ten probability scores for each input sample. Example two — character level sequence to sequence prediction We’ll train a model on the combined works of William Shakespeare, then use it to compose a play in the similar style. Every character in the text blob is first converted to an integer by calling Python’s built-in function which returns an integer representing of a character as its ASCII value. For example, returns the integer . As a result, we have a list of integers to represent the whole text. ord() ord('a') 97 Given a moving window of sequence length 100, the model learns to predict the sequence one time-step in the future. In other words, given characters of timesteps T0~T99 in the sequence, the model predicts characters of timesteps T1~T100. Let’s build a simple sequence to sequence model in Keras. We can further visualize the structure of the model to understand its input and output shape respectively. The training model Even though the model has 3-dimensional output, when compiled with the loss function , we can feed the training targets as sequences of integers. Similarly to the previous example, without the help of , one need first to convert the output integers to one-hot encoded form to fit the model. sparse_categorical_crossentropy sparse_categorical_crossentropy The training model is, non-stateful seq_len =100 batch_size = 128 Model input shape: (batch_size, seq_len) Model output shape: (batch_size, seq_len, MAX_TOKENS) Once the model is trained, we can make it “stateful” and predict five characters at a time. By making it stateful, the LSTMs’ last state for each sample in a batch will be used as the initial state for the sample in the following batch, or put it simply, those five characters predicted at a time and following predicted batches are characters in one sequence. The prediction model loads the trained model weights and predicts five chars at a time, it is, stateful seq_len =1, one character/batch batch_size = 5 Model input shape: (batch_size, seq_len) Model output shape: (batch_size, seq_len, MAX_TOKENS) Need to call before prediction to reset LSTMs' initial states. reset_states() For more implementation detail of the model, please refer to my . GitHub repository Conclusion and further reading This tutorial explores two examples using to keep integer as chars' / multi-class classification labels without transforming to one-hot labels. So, the output of the model will be in softmax one-hot like shape while the labels are integers. sparse_categorical_crossentropy To learn the actual implementation of and , you can find it on TensorFlow repository. Don’t forget to download the source code for this tutorial on my GitHub. keras.backend.sparse_categorical_crossentropy sparse_categorical_accuracy _Examples to use Keras sparse_categorical_crossentropy - Tony607/keras_sparse_categorical_crossentropy_github.com Tony607/keras_sparse_categorical_crossentropy Share on Twitter Share on Facebook Originally published at www.dlology.com .