Deeper Still: Convolutional Neural Networks

Two weeks ago, we began our machine study in earnest by constructing a full neural network. But this network was still quite simple by deep learning standards. In this article, we’re going to tackle a much more difficult problem: image recognition. Of course, we’ll still be using a well known data set with well-known results, so this is only the tip of the iceberg. We’ll be using the . This set classifies images of handwritten digits as the numbers 0–9. This problem is so well-known that the folks at Tensor Flow refer to it as the “Hello World” of machine learning. MNIST data set We’ll start this problem by using a very similar approach to what we used with the Iris data set. We’ll make a fully-connected neural network with two layers, and then use the “Adam” optimizer. This will give us some decent results by our beginner standards. But MNIST is a well known problem with a very large data set. So we’re going to hold ourselves to a higher standard of accuracy this time. This will force us to use some more advanced techniques. But to start with, let’s examine what we need to change to adapt our Iris model to work for the MNIST problem. As with the last couple weeks, the code for all this is on if you want to follow along. Github Re-use and Recycle! Generally, we can re-use most of the code we had with Iris, which is good news! We still have to make a few adjustments here and there though. First, we’ll use some different constants. We’ll use in place of , and instead of . We’ll also bump up the size of our hidden layer and the number of samples we’ll draw on each iteration: mnistFeatures irisFeatures mnistLabels irisLabels mnistFeatures :: Int64mnistFeatures = 784 mnistLabels :: Int64mnistLabels = 10 numHiddenUnits :: Int64numHiddenUnits = 1024 sampleSize :: IntsampleSize = 1000 We’ll also change our model to use as the result type instead . Word8 Int64 data Model = Model { train :: TensorData Float -> TensorData Word8 -- Used to be Int64 -> Session () , errorRate :: TensorData Float -> TensorData Word8 -- Used to be Int64 -> SummaryTensor -> Session (Float, ByteString) } Now we have to change how we get our input data. Our data isn’t in CSV format this time. We’ll use helper functions from the Tensor Flow library to extract the images and labels: import TensorFlow.Examples.MNIST.Parse (readMNISTSamples, readMNISTLabels)…runDigits :: FilePath -> FilePath -> FilePath -> FilePath -> IO ()runDigits trainImageFile trainLabelFile testImageFile testLabelFile = withEventWriter eventsDir $ \eventWriter -> runSession $ do -- trainingImages, testImages :: [Vector Word8] trainingImages <- liftIO $ readMNISTSamples trainImageFile testImages <- liftIO $ readMNISTSamples testImageFile -- traininglabels, testLabels :: [Word8] trainingLabels <- liftIO $ readMNISTLabels trainLabelFile testLabels <- liftIO $ readMNISTLabels testLabelFile -- trainingRecords, testRecords :: Vector (Vector Word8, Word8) let trainingRecords = fromList $ zip trainingImages trainingLabels let testRecords = fromList $ zip testImages testLabels ... Our “input” type consists of vectors of elements. These represent the intensity of various pixels. Our “output” type is , referring to the actual labels (0-9). We read the images and labels from separate files. Then we zip them together to pass to our processing functions. We’ll have to make a few changes to these processing functions for this data set. First, we have to generalize the type of our randomization function: Word8 Word8 -- Used to be IrisRecord SpecificchooseRandomRecords :: Vector a -> IO (Vector a) Next we have to write a new encoding function that will put our data into the format. This looks like our old version, except dealing with the new tuple type instead of the . TensorData IrisRecord convertDigitRecordsToTensorData :: Vector (Vector Word8, Word8) -> (TensorData Float, TensorData Word8)convertDigitRecordsToTensorData records = (input, output) where numRecords = Data.Vector.length records input = encodeTensorData [fromIntegral numRecords, mnistFeatures] (fromList $ concatMap recordToInputs records) output = encodeTensorData [fromIntegral numRecords] (snd records) recordToInputs :: (Vector Word8, Word8) -> [Float] recordToInputs rec = fromIntegral (toList . fst) rec And then we just have to substitute our new functions and parameters in, and we’ll be able to run our digit trainer! Current training error 89.8Current training error 19.300001Current training error 13.300001Current training error 11.199999Current training error 8.700001Current training error 6.5999985Current training error 6.999999Current training error 5.199999Current training error 4.400003Current training error 5.000001Current training error 2.3000002 test error 6.830001 So our accuracy is 93.2%. This seems like an alright number. But imagine being a Post office and having 6.8% of your mail sorted into the wrong Zip Code! (This was the original use case of this data set). So let’s see if we can do better. Convolution and Max Pooling Now we could train our model longer. This will tend to improve our error rate. But we can also help ourselves by making our model more complex. The fundamental flaw with what we’ve got so far is that it doesn’t account for the 2D nature of the images. This means we’re losing a ton of useful information. So the first thing we’ll do is treat our images as being 28x28 tensors instead of 1x784. This way, our model can pick out specific areas that are significant for the identification of the digit. One thing we want to account for is that our image might not be in the center of the frame. To account for this, we’re going to apply convolution. When using convolution, we break the image into many different overlapping tiles. In our case, we’ll make our strides size “1” in every direction, and we’ll use a patch size of 5x5. So this means we’ll center a 5x5 tile around each different pixel in our image, and then come up with a score for it. That score tells us if this part of the image contains any important information. We can represent this score as a vector with many features. So with 2D convolution, we’ll be dealing with 4-dimensional tensors. The first dimension is the sample size. The second two dimensions are the shape of the image. The final dimension is the number of features of the “score” for each part of the image. So each original image starts our with a single feature for the “score” of each pixel. This score is the actual intensity of that pixel! Then each layer of convolution will act as a mini neural network per pixel, making as many features as we want. Max pooling is a form of down-sampling. After our first convolution step, we’ll have scores on the 28x28 image. We’ll use 2x2 max-pooling, meaning we divide each image into 2x2 squares. Then we’ll make a new layer that is 14x14, using only the “best” score from each 2x2 box. This makes our model more efficient while keeping the most important information. Simple demonstration of max-pooling Implementing Convolutional Layers We’ll do two rounds of convolution and max pooling. So we’ll make a function that creates a layer that performs these two steps. This will look a lot like our other neural network layer. We’ll take parameters for the size of the input and output channels of the layer, as well as the tensor itself. So our first step will be to create the weights and bias tensors using these parameters: patchSize :: Int64patchSize = 5 buildConvPoolLayer :: Int64 -> Int64 -> Tensor v Float -> Text -> Build (Variable Float, Variable Float, Tensor Build Float)buildConvPoolLayer inputChannels outputChannels input layerName = withNameScope layerName $ do weights >= initializedVariable bias >= initializedVariable ... where weightsShape :: [Int64] weightsShape = [patchSize, patchSize, inputChannels, outputChannels] Now we’ll want to call our convolution and max pooling functions. These are still a little rough around the edges (the Haskell library is still quite young). The C versions of these functions have many optional, named attributes. For the moment there don’t seem to be any functions that use normal Haskell values for these arguments. Instead, we’ll be using values, assign bytestring names to values. OpAttr where ... convStridesAttr = opAttr "strides" .~ ([1,1,1,1] :: [Int64]) poolStridesAttr = opAttr "strides" .~ ([1,2,2,1] :: [Int64]) poolKSizeAttr = opAttr "ksize" .~ ([1,2,2,1] :: [Int64]) paddingAttr = opAttr "padding" .~ ("SAME" :: ByteString) dataFormatAttr = opAttr "data_format" .~ ("NHWC" :: ByteString) convAttrs = convStridesAttr . paddingAttr . dataFormatAttr poolAttrs = poolKSizeAttr . poolStridesAttr . paddingAttr . dataFormatAttr The argument for convolution refers to how much we shift the window each time. The argument for pooling refers to how big the windows will be that we perform the pooling over. In this case, it's 2x2. Now that we have our attributes, we can call the library functions and . This gives our resulting vector. We also throw in a call to between these steps. strides strides conv2D’ maxPool’ relu buildConvPoolLayer :: Int64 -> Int64 -> Tensor v Float -> Text -> Build (Variable Float, Variable Float, Tensor Build Float)buildConvPoolLayer inputChannels outputChannels input layerName = withNameScope layerName $ do weights >= initializedVariable bias >= initializedVariable let conv = conv2D' convAttrs input (readValue weights) `add` readValue bias let results = maxPool' poolAttrs (relu conv) return (weights, bias, results) where ... Modifying our Model Now we’ll make a few updates to our model and we’ll be in good shape. First, we need to reshape our input data to be 4-dimensional. Then, we’ll apply the two convolution/pooling layers: imageDimen :: Int32imageDimen = 28 createModel :: Build ModelcreateModel = do let batchSize = -1 -- Allows variable sized batches let conv1OutputChannels = 32 let conv2OutputChannels = 64 let denseInputSize = 7 * 7 * 64 :: Int32 -- 3136 let numHiddenUnits = 1024 inputs <- placeholder [batchSize, mnistFeatures] outputs <- placeholder [batchSize] let inputImage = reshape inputs (vector [batchSize, imageDimen, imageDimen, 1]) (convWeights1, convBiases1, convResults1) <- buildConvPoolLayer 1 conv1OutputChannels inputImage "convLayer1" (convWeights2, convBiases2, convResults2) <- buildConvPoolLayer conv1OutputChannels conv2OutputChannels convResults1 "convLayer2" Once we’re done with that, we’ll apply two fully-connected (dense) layers as we did before. Note we’ll reshape our result from four dimensions back down to two: let denseInput = reshape convResults2 (vector [batchSize, denseInputSize])(denseWeights1, denseBiases1, denseResults1) <- buildNNLayer (fromIntegral denseInputSize) numHiddenUnits denseInput "denseLayer1" let rectifiedDenseResults = relu denseResults1(denseWeights2, denseBiases2, denseResults2) <- buildNNLayer numHiddenUnits mnistLabels rectifiedDenseResults "denseLayer2" And after that we can treat the rest of the model the same. We’ll update the parameter names and add the new weights and biases to the that the model can change. params As a review, let’s look at the dimensions of each of the intermediate tensors here. Then we can see the restrictions on the dimensions of the different operations. Each convolution step takes two four-dimensional tensors. The final dimension of argument 1 must match the third dimension of argument 2. Then the result will swap in the final dimension of argument 2. Meanwhile, pooling with a 2x2 stride size will take this resulting 4-dimensional tensor and halve each of the inner dimensions. input: n x 784inputImage: n x 28 x 28 x 1convWeights1: 5 x 5 x 1 x 32convBias1: 32conv (first layer): n x 28 x 28 x 32convResults1: n x 14 x 14 x 32convWeights2: 5 x 5 x 32 x 64conv (second layer): n x 14 x 14 x 64convResults2: n x 7 x 7 x 64denseInput: n x 3136denseWeights1: 3136 x 1024denseBias1: 1024denseResults1: n x 1024denseWeights2: 1024 x 10denseBias2: 10denseResults2: n x 10 So for each input, we’ll have a probability for all 10 of the possible inputs. We pick the greatest of these as the chosen label. Results We’ll run our model again, only this time we’ll use a smaller sample size (100 per training iteration). This allows us to train for more iterations (20000). This takes quite a while to train, but we get these results (printed every 1000 iterations). Current training error 91.0Current training error 6.0Current training error 2.9999971Current training error 2.9999971Current training error 0.0Current training error 0.0Current training error 0.99999905Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0Current training error 0.0 test error 1.1799991 Not too bad! Once it got going, we saw very few training errors, though still ended up with a model that was a tad overfit. The Tensor Flow MNIST suggests using a dropout factor. This helps diminish the effects of overfitting. But this option isn’t available to us yet in Haskell. Still, we got close to 99% accuracy, which is a success for us! expert tutorial And here’s what our final graph looks like. Notice the extra layers we added for convolution: Conclusion So that’s it for convolutional neural networks! Our goal was to adapt our previous neural network model to recognize digits. Not only did we pick a harder problem, but we also wanted higher accuracy. We achieved this by using more advanced machine learning techniques. Convolution allowed us to use the full 2x2 nature of the image. It also checked for the digit no matter where it was in the image. Max pooling enabled us to make our algorithm more efficient while keeping the most important information. If you’re itching to see what else Haskell can do with Tensor Flow, check out our . It’ll walk you through some of the trickier parts of getting the library working on your local machine. It will also go through the most important information you need to know about the types in this library. Tensor Flow Guide If you’re new to Haskell, here are a couple more resources you can dig into before you try your hand at Tensor Flow. First, there’s our . This will point you toward some helpful resources for learning the language. Next, you can check out our so you can learn how to organize a project! If you want to use Tensor Flow in Haskell, Stack is a must! Getting Started Checklist Stack mini-course