tensorflow, machine learning, gpu, setup-guide Intro …For devs wanting to run some cool models or experiments with TensorFlow (on GPU for more intense training). This probably isn’t for the professional data scientists or anyone creating actual models — I imagine their setups are a bit more verbose. This blog post will cover my manual implementation of setting up/building TensorFlow with support. I’ve spent decent time reading posts and going through walkthroughs… and learned a ton from them… so I pieced together this installation guide to which I’ve been routinely using since (should have a CloudFormation script soon). There are obviously simple installs via pip/conda, but we want to manually build out TF and get it running on GPU… still this installation guide is for simple/default configurations and settings. GPU The Core (we will cover these as we go along): AWS EC2 p2.xlarge Ubuntu 16.04 CUDA 8.0 (along with CUDDN) TensorFlow 1.0 Python 2.x and libs (libs not directly needed here, but I base my installations around Python usage) Launch your AWS EC2 Instance Choose OS type: Ubuntu Server 16.04 Choose Instance Type: GPU Compute > p2.xlarge Why p2? They were made specifically for what we want to do, which is to run intense computations on the GPU. p2.xlarge offers us 12gb GPU. We will be able to do (batched) computationally-intensive just fine on this machine learning p2 instances, while running, are — So I suggest you only keep the instance state running while you are performing tasks such as training. WARNING: $0.90/hour Configuration: Basic configurations are fine, you will need to use a VPC (default VPC and subnet settings are fine as well) Storage: when adding storage, account for the amount of data you’ll most likely be training on. For example, the COCO 2014 dataset (images) is around 15GB, and the trained neural net most commonly used for COCO is .5GB. Also, we will be downloading 100s of MBs of individual software and libraries. I’ve opted for in my work, and have been using around 30–40GB of that. Note: 50GB storage Continue on, and setup with your preferences or the defaults. There is no other special settings that are needed to proceed, so launch and wait for the instance to be started. Launch the instance, wait for it to be , and we should be good to move on. running Installation Guide SSH into your newly launched instance. Ubuntu updates $ sudo apt-get update$ sudo apt-get upgrade If you get the response: Note: “new version of boot/grub/menu.lst …” — I keep the local version in my testing environments. For production use, due diligence is required. Dependencies useful tools we will need: $ sudo apt-get -y git vim curl build-essential cmake pkg-config zip this is also where I install the python libraries I will use (not required for tensorflow install): $ sudo apt-get -y install python-pip python-dev python-numpy python-scipy ipython python-matplotlib python-sklearn python-wheel other system libraries i’ve found to be required when building and using tensorflow: $ sudo apt-get -y install -y libpng12-dev ibjasper-dev libfreetype6 libjpeg-dev libtiff5-dev libgtk2.0-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev swig now create a directory for us to work out of: $ mkdir install && cd install Building TensorFlow We are going to be building TensorFlow from source. There is a simple pip installation, but we will get — in some use cases — by building better performance https://www.tensorflow.org/performance/performance_guide#build_and_install_from_source Bazel is TenorFlow’s build tool. Bazel needs to use our ip and because we are in a VPC, we need to change the hosts file. To , run in the terminal… you’ll get an output, so look for and note that ip — (also available in the AWS Console) get your private ip-address ifconfig inet adds modify /etc/hosts: $ sudo vim /etc/hosts// [vim opens the file …]## change this line:127.0.0.1 localhost## to (if your private ip is 155.10.100.10)127.0.0.1 ip-155–10–100–10 2. install Java (needed for Bazel): $ sudo add-apt-repository ppa:webupd8team/java// [press enter]$ sudo apt-get update$ sudo apt-get install -y oracle-java8-installer// [accept terms] 3. install Bazel (0.5.1): $ wget https://github.com/bazelbuild/bazel/releases/download/0.5.1/bazel-0.5.1-installer-linux-x86_64.sh $ chmod +x bazel-0.5.1-installer-linux-x86_64.sh$ ./bazel-0.5.1-installer-linux-x86_64.sh Bazel will now be installed. Let’s add it in .bashr modify ~/.bashrc (add the 2 lines to the end of the file): $ vim ~/.bashrc// [vim opens the file] # Add to end of file:# BAZELsource /home/ubuntu/.bazel/bin/bazel-complete.bashexport PATH=”$PATH:$HOME/bin” let’s load .bashrc into our shell now: $ source ~/.bashrc We now have Bazel. Ok, before we get TensorFlow and build, we need to install our dependencies for GPU-support. These include CUDA and CUDNN. For more info on these: https://en.wikipedia.org/wiki/CUDA install CUDA: $ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_8.0.44-1_amd64.deb $ sudo dpkg -i cuda-repo-ubuntu1404_8.0.44–1_amd64.deb$ sudo apt-get -y update$ sudo apt-get -y upgrade$ sudo apt-get install -y cuda // [will take a couple of minutes] $ sudo sh -c “sudo echo ‘/usr/local/cuda/lib64’ > /etc/ld.so.conf.d/cuda.conf”$ sudo ldconfig To test that we have this installed correctly, run a simple command: $ nvidia-smi ## note, using:## watch -n 0.5 nvidia-smi## is a useful way to launch a “live” monitor of GPU stats, use this when you train your models For CUDNN, you will need a Nvidia developer account. You can get this here: Note: https://developer.nvidia.com/rdp/cudnn-download After you have setup your account, go ahead and download the tarball to your own machine. 8.0-linux-x64-v5.1 We will transfer this file (and all others coming from our machine) to our EC2 instance using scp. Before you scp the cudnn tarball, edit the command below with your ec2 and correct paths Note: run command from your own machine (replace ip): scp $ scp -i \~/keys/learner-key.pem \~/Downloads/cudnn-8.0-linux-x64-v5.1.tgz \ubuntu@ec2–34–203–215–27.compute-1.amazonaws.com:~/install/ ## the syntax:## scp -i <path_to_ec2_private_key> <path_to_file_to_transfer> <receiving_machine_and_path_to_dest> After the tarball transfers, unzip and place: $ tar -zxf cudnn-8.0-linux-x64-v5.1.tgz$ sudo cp -P cuda/lib64/* /usr/local/cuda/lib64/ TENSOR TIME Yea.. cool. Let’s go and get TensorFlow. Using v1 here. $ git clone $ cd tensorflow$ git checkout v1.0.0 https://github.com/tensorflow/tensorflow $ ./configure // [launches configuration] This will launch the configuration script. You will need to configure one setting as non-default: CUDA usage. So [press enter] to get all default settings, until you see the question asking if you want to support CUDA. we do. The following CUDA settings can be default as well. Note: YES Ok, TensorFlow is configured and ready to build. The build process is timely. Usually lasting around 1-hour on p2.xlarge. So to combat this, and not leave your process bound to the open shell, we will use is a tool that will allow us to launch a new window in the shell, start a process, and then from that window… and eventually allow us to come back/re-attach to that window/process. See more here: screen. Screen detach https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/ (screen also comes in handy when training your models and running those respective processes) launch a new screen window: $ screen We should still be in the install/tensorflow directory build TensorFlow with Bazel: $ bazel build -c opt — config=cuda //tensorflow/tools/pip_package:build_pip_package // [bazel builds tensorflow here — long process] This is a long process, expect this to take an hour. Thanks to screen, we can detach from this window (close the ssh connection if you’d like) and take a break Note: To detach from the screen window where Bazel is building: ctrl-a + d … time period of tensorflow building … After this break, reattach to the screen window that is building/built TensorFlow: $ screen -r install the new build using pip: $ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg $ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.0.0-cp27-cp27mu-linux_x86_64.whl We want to now make sure TensorFlow can find cuda before running. We need to add it into our environment. modify ~/.bashrc $ vim ~/.bashrc// [vim opens the file] # Add to end of file:# CUDAexport LD_LIBRARY_PATH=”$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64" export CUDA_HOME=/usr/local/cuda cd out to the home directory (make sure you are out of tensflow directory) and TENSORFLOW IS INSTALLED! let’s test this in interactive Python: $ python// [python interpreter opens] >>>>>> import tensorflow as tf I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locallyI tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally … more like output like this (good sign that we see CUDA) ## TEST SCRIPT: >>> with tf.Session() as sess: …… sess.run(tf.global_variables_initializer()) …… print sess.run(output) …… [press enter] // [script runs — you will see an initial output] and to confirm we are using GPU, look for this output: Now go do some cool s***. follow me
