Content Overview Overview


Analysis Workflow



Is your tf.data pipeline producing data fast enough?




Are you prefetching data?



Are you reaching high CPU utilization?


Bottleneck Analysis


Additional resources Overview Overview Analysis Workflow Analysis Workflow Is your tf.data pipeline producing data fast enough? Is your tf.data pipeline producing data fast enough? Is your tf.data pipeline producing data fast enough? Are you prefetching data? Are you prefetching data? Are you prefetching data? Are you reaching high CPU utilization? Are you reaching high CPU utilization? Bottleneck Analysis Bottleneck Analysis Additional resources Additional resources Overview This guide assumes familiarity with the TensorFlow Profiler and tf.data. It aims to provide step by step instructions with examples to help users diagnose and fix input pipeline performance issues. Profiler tf.data To begin, collect a profile of your TensorFlow job. Instructions on how to do so are available for CPUs/GPUs and Cloud TPUs. CPUs/GPUs Cloud TPUs The analysis workflow detailed below focuses on the trace viewer tool in the Profiler. This tool displays a timeline that shows the duration of ops executed by your TensorFlow program and allows you to identify which ops take the longest to execute. For more information on the trace viewer, check out this section of the TF Profiler guide. In general, tf.data events will appear on the host CPU timeline. this section tf.data Analysis Workflow Please follow the workflow below. If you have feedback to help us improve it, please create a github issue with the label “comp:data”. Please follow the workflow below. If you have feedback to help us improve it, please create a github issue with the label “comp:data”. create a github issue 1. Is your tf.data pipeline producing data fast enough? tf.data Begin by ascertaining whether the input pipeline is the bottleneck for your TensorFlow program. To do so, look for IteratorGetNext::DoCompute ops in the trace viewer. In general, you expect to see these at the start of a step. These slices represent the time it takes for your input pipeline to yield a batch of elements when it is requested. If you’re using keras or iterating over your dataset in a tf.function, these should be found in tf_data_iterator_get_next threads. IteratorGetNext::DoCompute tf.function tf_data_iterator_get_next Note that if you’re using a distribution strategy, you may see IteratorGetNextAsOptional::DoCompute events instead of IteratorGetNext::DoCompute(as of TF 2.3). distribution strategy IteratorGetNextAsOptional::DoCompute IteratorGetNext::DoCompute If the calls return quickly (<= 50 us), this means that your data is available when it is requested. The input pipeline is not your bottleneck; see the Profiler guide for more generic performance analysis tips. If the calls return quickly (<= 50 us), Profiler guide If the calls return slowly, tf.data is unable to keep up with the consumer’s requests. Continue to the next section. If the calls return slowly, tf.data 2. Are you prefetching data? The best practice for input pipeline performance is to insert a tf.data.Dataset.prefetch transformation at the end of your tf.data pipeline. This transformation overlaps the input pipeline’s preprocessing computation with the next step of model computation and is required for optimal input pipeline performance when training your model. If you’re prefetching data, you should see a Iterator::Prefetch slice on the same thread as the IteratorGetNext::DoCompute op. tf.data.Dataset.prefetch tf.data Iterator::Prefetch IteratorGetNext::DoCompute If you don’t have a prefetch at the end of your pipeline, you should add one. For more information about tf.data performance recommendations, see the tf.data performance guide. If you don’t have a prefetch tf.data tf.data performance guide If you’re already prefetching data, and the input pipeline is still your bottleneck, continue to the next section to further analyze performance. If you’re already prefetching data 3. Are you reaching high CPU utilization? tf.data achieves high throughput by trying to make the best possible use of available resources. In general, even when running your model on an accelerator like a GPU or TPU, the tf.data pipelines are run on the CPU. You can check your utilization with tools like sar and htop, or in the cloud monitoring console if you’re running on GCP. tf.data tf.data sar htop cloud monitoring console If your utilization is low, this suggests that your input pipeline may not be taking full advantage of the host CPU. You should consult the tf.data performance guide for best practices. If you have applied the best practices and utilization and throughput remain low, continue to Bottleneck analysis below. If your utilization is low, tf.data performance guide Bottleneck analysis If your utilization is approaching the resource limit, in order to improve performance further, you need to either improve the efficiency of your input pipeline (for example, avoiding unnecessary computation) or offload computation. If your utilization is approaching the resource limit You can improve the efficiency of your input pipeline by avoiding unnecessary computation in tf.data. One way of doing this is inserting a tf.data.Dataset.cache transformation after computation-intensive work if your data fits into memory; this reduces computation at the cost of increased memory usage. Additionally, disabling intra-op parallelism in tf.data has the potential to increase efficiency by > 10%, and can be done by setting the following option on your input pipeline: tf.data tf.data.Dataset.cache tf.data dataset = ...
options = tf.data.Options()
options.experimental_threading.max_intra_op_parallelism = 1
dataset = dataset.with_options(options) dataset = ...
options = tf.data.Options()
options.experimental_threading.max_intra_op_parallelism = 1
dataset = dataset.with_options(options) 4. Bottleneck Analysis The following section walks through how to read tf.data events in the trace viewer to understand where the bottleneck is and possible mitigation strategies. tf.data Understanding tf.data events in the Profiler tf.data Each tf.data event in the Profiler has the name Iterator::<Dataset>, where <Dataset> is the name of the dataset source or transformation. Each event also has the long name Iterator::<Dataset_1>::...::<Dataset_n>, which you can see by clicking on the tf.data event. In the long name, <Dataset_n> matches <Dataset> from the (short) name, and the other datasets in the long name represent downstream transformations. tf.data Iterator::<Dataset> <Dataset> Iterator::<Dataset_1>::...::<Dataset_n> tf.data <Dataset_n> <Dataset> For example, the above screenshot was generated from the following code: dataset = tf.data.Dataset.range(10)
dataset = dataset.map(lambda x: x)
dataset = dataset.repeat(2)
dataset = dataset.batch(5) dataset = tf.data.Dataset.range(10)
dataset = dataset.map(lambda x: x)
dataset = dataset.repeat(2)
dataset = dataset.batch(5) Here, the Iterator::Map event has the long name Iterator::BatchV2::FiniteRepeat::Map. Note that the datasets name may differ slightly from the python API (for example, FiniteRepeat instead of Repeat), but should be intuitive enough to parse. Iterator::Map Iterator::BatchV2::FiniteRepeat::Map Synchronous and asynchronous transformations For synchronous tf.data transformations (such as Batch and Map), you will see events from upstream transformations on the same thread. In the above example, since all the transformations used are synchronous, all the events appear on the same thread. tf.data Batch Map For asynchronous transformations (such as Prefetch, ParallelMap, ParallelInterleave and MapAndBatch) events from upstream transformations will be on a different thread. In such cases, the “long name” can help you identify which transformation in a pipeline an event corresponds to. Prefetch ParallelMap ParallelInterleave MapAndBatch For example, the above screenshot was generated from the following code: dataset = tf.data.Dataset.range(10)
dataset = dataset.map(lambda x: x)
dataset = dataset.repeat(2)
dataset = dataset.batch(5)
dataset = dataset.prefetch(1) dataset = tf.data.Dataset.range(10)
dataset = dataset.map(lambda x: x)
dataset = dataset.repeat(2)
dataset = dataset.batch(5)
dataset = dataset.prefetch(1) Here, the Iterator::Prefetch events are on the tf_data_iterator_get_next threads. Since Prefetch is asynchronous, its input events (BatchV2) will be on a different thread, and can be located by searching for the long name Iterator::Prefetch::BatchV2. In this case, they are on the tf_data_iterator_resource thread. From its long name, you can deduce that BatchV2 is upstream of Prefetch. Furthermore, the parent_id of the BatchV2 event will match the ID of the Prefetch event. Iterator::Prefetch tf_data_iterator_get_next Prefetch BatchV2 Iterator::Prefetch::BatchV2 tf_data_iterator_resource BatchV2 Prefetch parent_id BatchV2 Prefetch Identifying the bottleneck In general, to identify the bottleneck in your input pipeline, walk the input pipeline from the outermost transformation all the way to the source. Starting from the final transformation in your pipeline, recurse into upstream transformations until you find a slow transformation or reach a source dataset, such as TFRecord. In the example above, you would start from Prefetch, then walk upstream to BatchV2, FiniteRepeat, Map, and finally Range. TFRecord Prefetch BatchV2 FiniteRepeat Map Range In general, a slow transformation corresponds to one whose events are long, but whose input events are short. Some examples follow below. Note that the final (outermost) transformation in most host input pipelines is the Iterator::Model event. The Model transformation is introduced automatically by the tf.data runtime and is used for instrumenting and autotuning the input pipeline performance. Iterator::Model tf.data If your job is using a distribution strategy, the trace viewer will contain additional events that correspond to the device input pipeline. The outermost transformation of the device pipeline (nested under IteratorGetNextOp::DoCompute or IteratorGetNextAsOptionalOp::DoCompute) will be an Iterator::Prefetch event with an upstream Iterator::Generator event. You can find the corresponding host pipeline by searching for Iterator::Model events. distribution strategy IteratorGetNextOp::DoCompute IteratorGetNextAsOptionalOp::DoCompute Iterator::Prefetch Iterator::Generator Iterator::Model Example 1 Example 1 The above screenshot is generated from the following input pipeline: dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(parse_record)
dataset = dataset.batch(32)
dataset = dataset.repeat() dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(parse_record)
dataset = dataset.batch(32)
dataset = dataset.repeat() In the screenshot, observe that (1) Iterator::Map events are long, but (2) its input events (Iterator::FlatMap) return quickly. This suggests that the sequential Map transformation is the bottleneck. Iterator::Map Iterator::FlatMap Note that in the screenshot, the InstantiatedCapturedFunction::Run event corresponds to the time it takes to execute the map function. InstantiatedCapturedFunction::Run Example 2 Example 2 The above screenshot is generated from the following input pipeline: dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(parse_record, num_parallel_calls=2)
dataset = dataset.batch(32)
dataset = dataset.repeat() dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(parse_record, num_parallel_calls=2)
dataset = dataset.batch(32)
dataset = dataset.repeat() This example is similar to the above, but uses ParallelMap instead of Map. We notice here that (1) Iterator::ParallelMap events are long, but (2) its input events Iterator::FlatMap (which are on a different thread, since ParallelMap is asynchronous) are short. This suggests that the ParallelMap transformation is the bottleneck. Iterator::ParallelMap Iterator::FlatMap Addressing the bottleneck Source datasets If you’ve identified a dataset source as the bottleneck, such as reading from TFRecord files, you can improve performance by parallelizing data extraction. To do so, ensure that your data is sharded across multiple files and use tf.data.Dataset.interleave with the num_parallel_calls parameter set to tf.data.AUTOTUNE. If determinism is not important to your program, you can further improve performance by setting the deterministic=False flag on tf.data.Dataset.interleave as of TF 2.2. For example, if you’re reading from TFRecords, you can do the following: tf.data.Dataset.interleave num_parallel_calls tf.data.AUTOTUNE deterministic=False tf.data.Dataset.interleave dataset = tf.data.Dataset.from_tensor_slices(filenames)
dataset = dataset.interleave(tf.data.TFRecordDataset,
  num_parallel_calls=tf.data.AUTOTUNE,
  deterministic=False) dataset = tf.data.Dataset.from_tensor_slices(filenames)
dataset = dataset.interleave(tf.data.TFRecordDataset,
  num_parallel_calls=tf.data.AUTOTUNE,
  deterministic=False) Note that sharded files should be reasonably large to amortize the overhead of opening a file. For more details on parallel data extraction, see this section of the tf.data performance guide. this section tf.data Transformation datasets If you’ve identified an intermediate tf.data transformation as the bottleneck, you can address it by parallelizing the transformation or caching the computation if your data fits into memory and it is appropriate. Some transformations such as Map have parallel counterparts; the tf.data performance guide demonstrates how to parallelize these. Other transformations, such as Filter, Unbatch, and Batch are inherently sequential; you can parallelize them by introducing “outer parallelism”. For example, supposing your input pipeline initially looks like the following, with Batch as the bottleneck: tf.data caching the computation Map tf.data Filter Unbatch Batch Batch filenames = tf.data.Dataset.list_files(file_path, shuffle=is_training)
dataset = filenames_to_dataset(filenames)
dataset = dataset.batch(batch_size) filenames = tf.data.Dataset.list_files(file_path, shuffle=is_training)
dataset = filenames_to_dataset(filenames)
dataset = dataset.batch(batch_size) You can introduce “outer parallelism” by running multiple copies of the input pipeline over sharded inputs and combining the results: filenames = tf.data.Dataset.list_files(file_path, shuffle=is_training)

def make_dataset(shard_index):
  filenames = filenames.shard(NUM_SHARDS, shard_index)
  dataset = filenames_to_dataset(filenames)
  Return dataset.batch(batch_size)

indices = tf.data.Dataset.range(NUM_SHARDS)
dataset = indices.interleave(make_dataset,
                             num_parallel_calls=tf.data.AUTOTUNE)
dataset = dataset.prefetch(tf.data.AUTOTUNE) filenames = tf.data.Dataset.list_files(file_path, shuffle=is_training)

def make_dataset(shard_index):
  filenames = filenames.shard(NUM_SHARDS, shard_index)
  dataset = filenames_to_dataset(filenames)
  Return dataset.batch(batch_size)

indices = tf.data.Dataset.range(NUM_SHARDS)
dataset = indices.interleave(make_dataset,
                             num_parallel_calls=tf.data.AUTOTUNE)
dataset = dataset.prefetch(tf.data.AUTOTUNE) Additional resources tf.data performance guide on how to write performance tf.data input pipelines
Inside TensorFlow video: tf.data best practices
Profiler guide
Profiler tutorial with colab tf.data performance guide on how to write performance tf.data input pipelines tf.data performance guide tf.data Inside TensorFlow video: tf.data best practices Inside TensorFlow video: tf.data Profiler guide Profiler guide Profiler tutorial with colab Profiler tutorial with colab Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. TensorFlow TensorFlow

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Apache

How to Save, Load, and Deploy Models Using TensorFlow SavedModel

The Shape of Data: Broadcasting, Indexing, and Encoding with RaggedTensors

Too Long; Didn't Read

Make resilience your competitive advantage

How to Speed Up Your TensorFlow tf.data Pipeline

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Detailed Overview of TensorFlow Core APIs

10 Must-Try Open Source Tools for Machine Learning

59 Stories To Learn About Tensorflow

9 Things You Should Know About TensorFlow

A Quick Introduction to Artificial Intelligence, Machine Learning, Deep Learning and TensorFlow

A Detailed Overview of TensorFlow Core APIs

10 Must-Try Open Source Tools for Machine Learning

59 Stories To Learn About Tensorflow

9 Things You Should Know About TensorFlow

A Quick Introduction to Artificial Intelligence, Machine Learning, Deep Learning and TensorFlow

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps