TensorFlowで複数のGPUを使用する方法(コード変更は必要ありません)

コンテンツ概要 セットアップ 概要 ロゴ装置の配置 マニュアル装置の配置 GPUメモリの成長を制限する 1 つの GPU を複数の GPU システムで使用する 複数のGPUを使用 TensorFlowコード、および モデルは単一のGPU上で透明に実行され、コード変更は必要ありません。 tf.keras 注: tf.config.list_physical_devices('GPU') を使用して TensorFlow が GPU を使用していることを確認します。 利用 TensorFlow が GPU を使用していることを確認します。 Note: tf.config.list_physical_devices('GPU') 複数の GPU で、1 台または複数のマシンで実行する最も簡単な方法は、ディストリビューション戦略を使用することです。 このガイドは、これらのアプローチを試したユーザー向けであり、TensorFlow が GPU を使用する方法を精密に制御する必要があります。 ガイド TensorFlow GPU のパフォーマンスを最適化 セットアップ 最新の TensorFlow gpu リリースがインストールされていることを確認してください。 import tensorflow as tf print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Num GPUs Available: 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 概要 TensorFlow は、CPU と GPU を含むさまざまな種類のデバイスで計算を実行することをサポートしています。 "/device:CPU:0": あなたのマシンのCPU。 "/GPU:0": TensorFlow に表示されるマシンの最初の GPU の短いノート。 "/job:localhost/replica:0/task:0/device:GPU:1": TensorFlow に表示されるあなたのマシンの 2 番目の GPU の完全な資格名称。 TensorFlow オペレーションに CPU と GPU の両方の実装がある場合、デフォルトでは、オペレーションが割り当てられるときに GPU デバイスが優先されます。 CPU と GPU の両方のカーネルがあり、デバイス付きのシステムで そして , the デバイスが実行するように選択 別のデバイスで実行することを明示的に要求しない限り。 tf.matmul CPU:0 GPU:0 GPU:0 tf.matmul TensorFlow オペレーションが GPU を実装していない場合、オペレーションは CPU デバイスに戻ります。 CPU カーネルのみを有し、デバイス付きのシステムで そして , the デバイスが実行するように選択 たとえ、走ることを求められても、 デバイス tf.cast CPU:0 GPU:0 CPU:0 tf.cast GPU:0 ロゴ装置の配置 どのデバイスにあなたの操作やテンサーが割り当てられているかを調べるには、 デバイスの配置ログを有効にすると、Tensor の割り当てや操作が印刷されます。 tf.debugging.set_log_device_placement(True) tf.debugging.set_log_device_placement(True) # Create some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) 上記のコードは、表記を印刷します。 OPが処刑された。 . MatMul GPU:0 マニュアル装置の配置 自動的に選択されているデバイスではなく、選択したデバイスで特定の操作を実行したい場合は、 デバイスコンテキストを作成し、そのコンテキスト内のすべての操作が同じ指定されたデバイス上で実行されます。 with tf.device tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # Run on the GPU c = tf.matmul(a, b) print(c) Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) あなたは今見るだろう そして に割り当てられている。 装置が明示的に指定されていないため、 オペレーションと利用可能なデバイスに基づいて TensorFlow ランタイムが選択されます( この例では)および必要に応じて、デバイス間で自動的にテンサーをコピーします。 a b CPU:0 MatMul GPU:0 GPUメモリの成長を制限する デフォルトでは、TensorFlow はすべての GPU のほぼすべての GPU メモリをマッピングします。 )は、プロセスに表示されます。これは、メモリの断片化を減らすことによって、デバイスの比較的貴重なGPUメモリリリソースをより効率的に使用するために行われます。 方法です。 CUDA_VISIBLE_DEVICES tf.config.set_visible_devices gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e) Visible devices cannot be modified after being initialized 場合によっては、プロセスが利用可能なメモリのサブセットのみを割り当てるか、またはプロセスに必要なメモリの使用量を増やすだけであることが望ましい場合があります。 最初の選択肢は、呼び出しによってメモリの成長を有効にすることです。 実行時間の割り当てに必要なだけのGPUメモリを割り当てようとします:それは非常に少ないメモリを割り当て始め、プログラムが実行され、より多くのGPUメモリが必要になるにつれて、GPUメモリ領域はTensorFlowプロセスのために拡張されます。 tf.config.experimental.set_memory_growth gpus = tf.config.list_physical_devices('GPU') if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e) Physical devices cannot be modified after being initialized このオプションを有効にするもう一つの方法は、環境変数を設定することです。 2位 この構成はプラットフォーム特有です。 TF_FORCE_GPU_ALLOW_GROWTH true 2つ目の方法は、仮想 GPU デバイスを構成することです。 GPU に割り当てる総メモリにハード リミットを設定します。 tf.config.set_logical_device_configuration gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized これは、TensorFlow プロセスに利用可能な GPU メモリの量を本当に制約したい場合に有用です. This is common practice for local development when the GPU is shared with other applications such as a workstation GUI. 1 つの GPU を複数の GPU システムで使用する システムに 1 つ以上の GPU がある場合は、最低の ID を持つ GPU がデフォルトで選択されます。 tf.debugging.set_log_device_placement(True) try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) except RuntimeError as e: print(e) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 指定したデバイスが存在しない場合は、 : : . RuntimeError .../device:GPU:2 unknown device TensorFlow が既存およびサポートされているデバイスを自動的に選択して、指定されたデバイスが存在しない場合に操作を実行するようにしたい場合は、 . tf.config.set_soft_device_placement(True) tf.config.set_soft_device_placement(True) tf.debugging.set_log_device_placement(True) # Creates some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) 複数のGPUを使用 複数の GPU 用に開発すると、モデルが追加のリソースでスケールできるようになります。単一の GPU 用にシステムで開発する場合は、仮想デバイスで複数の GPU をシミュレートできます。 gpus = tf.config.list_physical_devices('GPU') if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized ランタイムに複数の論理 GPU が利用できるようになったら、複数の GPU を使用できます。 あるいは手動で配置する。 tf.distribute.Strategy 同 tf.distribute.戦略 tf.distribute.戦略 複数のGPUを使用するための最善の実践は、 以下は簡単な例です。 tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') strategy = tf.distribute.MirroredStrategy(gpus) with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2)) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3') Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0 y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0 key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 sk:0/device:CPU:0 StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0 dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 ocalhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 このプログラムは、それぞれのGPU上でモデルのコピーを実行し、入力データをそれらの間で分割します。 「」 データパラレル 配布戦略に関する詳細については、ガイドをご覧ください。 . ここ マニュアル配置 デバイス間で計算を複製することによってキャップの下で動作します. You can manually implement replication by constructing your model on each GPU. For example: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0 tf.Tensor( [[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32) 最初はTensorFlowのウェブサイトに掲載され、この記事はここに新しいタイトルの下に表示され、CC BY 4.0でライセンスを受けています。 最初はTensorFlowのウェブサイトに掲載され、この記事はここに新しいタイトルの下に表示され、CC BY 4.0でライセンスを受けています。 TensorFlowについて TensorFlowについて